Abstract
Selecting a potential reviewer to review a manuscript, submitted at a conference is a crucial task for the quality of a peer-review process that ultimately determines the success and impact of any conference. The approach adopted to find the potential reviewer needs to be consistent with its decision of allocation. In this work, we propose a framework for evaluating the reliability of different NLP approaches that are implemented for the match-making process. We bring various algorithmic approaches from different paradigms and an existing system Erie, implemented in IEEE INFOCOM conference, on a common platform to study their consistency of predicting the set of the potential reviewers, for a given manuscript. The consistency analysis has been performed over an actual multi-track conference organized in 2019. We conclude that Contextual Neural Topic Modeling (CNTM) with a balanced combinatorial optimization technique showed better consistency, among all the approaches we choose to study.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Due to the data privacy and confidentiality conditions, the original conference’s name is not revealed.
References
Orcid. https://orcid.org/. Accessed 14 July 2021
Semantic scholar \(|\) AI-powered research tool. https://www.semanticscholar.org/. Accessed 14 July 2021
You are crossref - crossref. https://www.crossref.org/. Accessed 14 July 2021
An automated conflict of interest based greedy approach for conference paper assignment system. J. Informetr. 14(2), 101022 (2020). https://doi.org/10.1016/j.joi.2020.101022. https://www.sciencedirect.com/science/article/pii/S1751157719301373
Bianchi, F., Terragni, S., Hovy, D.: Pre-training is a hot topic: contextualized document embeddings improve topic coherence. arXiv preprint arXiv:2004.03974 (2020)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3(null), 993–1022 (2003)
Campos, R., Mangaravite, V., Pasquali, A., Jorge, A., Nunes, C., Jatowt, A.: YAKE! Keyword extraction from single documents using multiple local features. Inf. Sci. 509, 257–289 (2020). https://doi.org/10.1016/j.ins.2019.09.013
Cer, D., et al.: Universal sentence encoder (2018)
Charlin, L., Zemel, R.: The Toronto paper matching system: an automated paper-reviewer assignment system (2013)
Di Mauro, N., Basile, T., Ferilli, S.: GRAPE: an expert review assignment component for scientific conference management systems, pp. 789–798, June 2005. https://doi.org/10.1007/11504894_109
Flach, P.A., et al.: Novel tools to streamline the conference review process: experiences from SIGKDD’09. SIGKDD Explor. Newsl. 11(2), 63–67 (2010). https://doi.org/10.1145/1809400.1809413
Goldsmith, J., Sloan, R.: The AI conference paper assignment problem. In: AAAI Workshop - Technical Report, January 2007
Grootendorst, M.: BERTopic: leveraging BERT and c-TF-IDF to create easily interpretable topics (2020). https://doi.org/10.5281/zenodo.4430182
Hartvigsen, D., Wei, J., Czuchlewski, R.: The conference paper-reviewer assignment problem*. Decis. Sci. 30, 865–876 (2007). https://doi.org/10.1111/j.1540-5915.1999.tb00910.x
Jin, J., Geng, Q., Zhao, Q., Zhang, L.: Integrating the trend of research interest for reviewer assignment. In: Proceedings of the 26th International Conference on World Wide Web Companion. WWW 2017 Companion, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, pp. 1233–1241 (2017). https://doi.org/10.1145/3041021.3053053
Kalmukov, Y.: Architecture of a conference management system providing advanced paper assignment features. Int. J. Comput. Appl. 34, 51–59 (2011). https://doi.org/10.5120/4083-5888
Kalmukov, Y.: Describing papers and reviewers’ competences by taxonomy of keywords. Comput. Sci. Inf. Syst. 9, 763–789 (2012). https://doi.org/10.2298/CSIS110906012K
Kou, N.M., Leong Hou, U., Mamoulis, N., Gong, Z.: Weighted coverage based reviewer assignment. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, New York, NY, USA, pp. 2031–2046. Association for Computing Machinery (2015). https://doi.org/10.1145/2723372.2723727
Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logist. Q. 2(1–2), 83–97 (1955)
Li, B., Hou, Y.T.: The new automated IEEE INFOCOM review assignment system. IEEE Netw. 30(5), 18–24 (2016). https://doi.org/10.1109/MNET.2016.7579022
Li, X., Watanabe, T.: Automatic paper-to-reviewer assignment, based on the matching degree of the reviewers. Procedia Comput. Sci. 22, 633–642 (2013). https://doi.org/10.1016/j.procs.2013.09.144. https://www.sciencedirect.com/science/article/pii/S187705091300937X. 17th International Conference in Knowledge Based and Intelligent Information and Engineering Systems - KES 2013
Liu, X., Suel, T., Memon, N.: A robust model for paper reviewer assignment. In: Proceedings of the 8th ACM Conference on Recommender Systems, RecSys 2014, New York, NY, USA, pp. 25–32. Association for Computing Machinery (2014). https://doi.org/10.1145/2645710.2645749
Medakene, A.N., Bouanane, K., Eddoud, M.A.: A new approach for computing the matching degree in the paper-to-reviewer assignment problem. In: 2019 International Conference on Theoretical and Applicative Aspects of Computer Science (ICTAACS), vol. 1, pp. 1–8 (2019). https://doi.org/10.1109/ICTAACS48474.2019.8988127
Merelo-Guervós, J.J., Castillo-Valdivieso, P., et al.: Conference paper assignment using a combined greedy/evolutionary algorithm. In: Yao, X. (ed.) PPSN 2004. LNCS, vol. 3242, pp. 602–611. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30217-9_61
Mihalcea, R., Tarau, P.: TextRank: bringing order into text. In: EMNLP (2004)
Nguyen, J.: Knowledge aggregation in people recommender systems: matching skills to tasks (2019)
Peng, H., Hu, H., Wang, K., Wang, X.: Time-aware and topic-based reviewer assignment. In: Bao, Z., Trajcevski, G., Chang, L., Hua, W. (eds.) DASFAA 2017. LNCS, vol. 10179, pp. 145–157. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-55705-2_11
Price, S., Flach, P.A.: Computational support for academic peer review: a perspective from artificial intelligence. Commun. ACM 60, 70–79 (2017)
Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using siamese BERT-networks (2019)
Rose, S., Engel, D., Cramer, N., Cowley, W.: Automatic keyword extraction from individual documents. Text Min. Appl. Theory 1, 1–20 (2010). https://doi.org/10.1002/9780470689646.ch1
Zhang, D., Zhao, S., Duan, Z., Chen, J., Zhang, Y., Tang, J.: A multi-label classification method using a hierarchical and transparent representation for paper-reviewer recommendation. ACM Trans. Inf. Syst. 38(1), February 2020. https://doi.org/10.1145/3361719
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kotak, N., Roy, A.K., Dasgupta, S., Ghosal, T. (2021). A Consistency Analysis of Different NLP Approaches for Reviewer-Manuscript Matchmaking. In: Ke, HR., Lee, C.S., Sugiyama, K. (eds) Towards Open and Trustworthy Digital Societies. ICADL 2021. Lecture Notes in Computer Science(), vol 13133. Springer, Cham. https://doi.org/10.1007/978-3-030-91669-5_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-91669-5_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-91668-8
Online ISBN: 978-3-030-91669-5
eBook Packages: Computer ScienceComputer Science (R0)