Abstract
Peer review is the widely accepted mechanism to determine the quality of scientific work. Even though peer-reviewing has been an integral part of academia since the 1600s, it frequently receives criticism for the lack of transparency and consistency. Even for humans, predicting the peer review outcome is a challenging task as there are many dimensions and human factors involved. However, Artificial Intelligence (AI) techniques can assist the editor/chair anticipate the final decision based on the reviews from the human reviewers. Peer review texts reflect the reviewers' opinions/ sentiments on various aspects (e.g., novelty, substance, soundness, etc.) of the paper concerning the research in the paper, which may be valuable to predict a manuscript's acceptance or rejection. The exact types and number of aspects could vary from one to the other venue (i.e., the conferences or journals). Peer review texts, however, which often contain rich sentiment information about the reviewers and, therefore, their overall opinion of the paper's research, can be useful in predicting a manuscript's acceptance or rejection. Here in this work, we study how we could take advantage of aspects and their corresponding sentiment to build a generic controllable system to assist the editor/chair in determining the outcome based on the reviews of a paper to make better editorial decisions. Our proposed deep neural architecture considers three information channels, including reviews, review aspect category, and its sentiment, to predict the final decision. Experimental results show that our model can achieve up to 76.67% accuracy on the ASAP-Review dataset (Aspect-enhanced Peer Review) consisting of ICLR and NIPS reviews considering the sentiment of the reviews. Empirical results also show an improvement of around 3.3 points while aspect information is added to the sentiment information1.
Original language | English |
---|---|
Title of host publication | JCDL 2022 - Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2022 |
Publisher | Institute of Electrical and Electronics Engineers Inc. |
ISBN (Electronic) | 9781450393454 |
DOIs | |
State | Published - Jun 20 2022 |
Externally published | Yes |
Event | 22nd ACM/IEEE Joint Conference on Digital Libraries, JCDL 2022 - Virtual, Online, Germany Duration: Jun 20 2022 → Jun 24 2022 |
Publication series
Name | Proceedings of the ACM/IEEE Joint Conference on Digital Libraries |
---|---|
ISSN (Print) | 1552-5996 |
Conference
Conference | 22nd ACM/IEEE Joint Conference on Digital Libraries, JCDL 2022 |
---|---|
Country/Territory | Germany |
City | Virtual, Online |
Period | 06/20/22 → 06/24/22 |
Funding
Sandeep Kumar acknowledges the Prime Minister Research Fellowship (PMRF) program of the Govt of India for its support. Asif Ekbal acknowledges the Young Faculty Research Fellowship (YFRF), supported by Visvesvaraya PhD scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia). Tirthankar Ghosal acknowledges Cactus Communications, India.
Keywords
- Aspect-based Sentiment Analysis
- Decision Prediction
- Deep Neural Network
- Peer Reviews