short-paper

A Case Study of Multi-class Classification with Diversified Precision Recall Requirements for Query Disambiguation

Authors:

Yingrui Yang,

Christopher Miller,

Peng Jiang, and

Azadeh MoghtaderiAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

Pages 1633 - 1636

https://doi.org/10.1145/3397271.3401315

Published: 25 July 2020 Publication History

Get Access

Abstract

We introduce a new metric for measuring the performance of multi-class classifiers. This metric is a generalization of the f1 score that is defined on binary classifiers, and offers significant improvement over other generalizations such as micro- and macro-averaging. In particular, one can select coefficients that weight the per-class precision and recall, as well as the overall class importance, with a robust mathematical interpretation. When certain parameters are selected our metric yields macro-averaged statistic as a special case. We demonstrate the efficacy of this metric on an application in genealogical search.

References

[1]

Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. http://tensorflow.org/ Software available from tensorflow.org.

Google Scholar

[2]

Vincent Van Asch. 2013. Macro-and micro-averaged evaluation measures [ [ BASIC DRAFT ] ].

Google Scholar

[3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. (2018). https://arxiv.org/abs/1810.04805

Google Scholar

[4]

Peng Jiang, Yingrui Yang, Gann Bierner, Fengjie Alex Li, Ruhan Wang, and Azadeh Moghtaderi. 2019. Family History Discovery through Search at Ancestry. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'19). Association for Computing Machinery, New York, NY, USA, 1389--1390. https://doi.org/10.1145/3331184.3331430

Digital Library

Google Scholar

[5]

Juri Opitz and Sebastian Burst. 2019. Macro F1 and Macro F1. arXiv preprint arXiv:1911.03347 (2019).

Google Scholar

[6]

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. (2018). https://arxiv.org/abs/1802.05365

Google Scholar

[7]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language Models are Unsupervised Multitask Learners. (2018). https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

Google Scholar

[8]

Tommaso Teofili. 2018. Deep Learning for Search. Manning Early Access Program.

Google Scholar

[9]

C.J. Van Rijsbergen. 1979. Information Retrieval. Butterworths. 78040725 https://books.google.com/books?id=t-pTAAAAMAAJ

Digital Library

Google Scholar

Cited By

View all

Radojičić DRadojičić NRheinländer T(2024)A comparative study of the neural network models for the stock market data classification—A multicriteria optimization approachExpert Systems with Applications10.1016/j.eswa.2023.122287238(122287)Online publication date: Mar-2024
https://doi.org/10.1016/j.eswa.2023.122287
Zhao XWang JWang JWang JHong RShen TLiu YLiang Y(2023)DTLR-CS: Deep tensor low rank channel cross fusion neural network for reproductive cell segmentationPLOS ONE10.1371/journal.pone.029472718:11(e0294727)Online publication date: 30-Nov-2023
https://doi.org/10.1371/journal.pone.0294727
Suhaimi NOthman ZYaakub M(2022)Comparative Analysis Between Macro and Micro-Accuracy in Imbalance Dataset for Movie Review ClassificationProceedings of Seventh International Congress on Information and Communication Technology10.1007/978-981-19-2394-4_8(83-93)Online publication date: 12-Jul-2022
https://doi.org/10.1007/978-981-19-2394-4_8

Index Terms

A Case Study of Multi-class Classification with Diversified Precision Recall Requirements for Query Disambiguation
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment
    2. Information retrieval query processing
      1. Query intent

Recommendations

Constructing a multi-class classifier using one-against-one approach with different binary classifiers

For the one-against-one approach, all the binary classifiers that form a one-against-one classifier should be sufficiently competent. If some of the classifiers are not competent, the consequences might be invalid classification results. To address the ...
Read More
Multi-class classification via heterogeneous ensemble of one-class classifiers

In this paper, a multi-class classification method based on heterogeneous ensemble of one-class classifiers is proposed. The proposed method consists of two phases: training heterogeneous one-class classifiers for each class using various one-class ...
Read More
Quantum-inspired algorithm for direct multi-class classification
Abstract
Over the last few decades, quantum machine learning has emerged as a groundbreaking discipline. Harnessing the peculiarities of quantum computation for machine learning tasks offers promising advantages. Quantum-inspired machine learning has ...
Highlights
- A quantum-inspired multi-class classifier based on the theory of quantum state discrimination is presented.
- The quantum-inspired classifier achieves higher classification accuracy than many standard classifiers when tested with more ...
Read More

Comments

Information & Contributors

Information

Published In

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
237
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)12

Other Metrics

View Author Metrics

Citations

Cited By

View all

Radojičić DRadojičić NRheinländer T(2024)A comparative study of the neural network models for the stock market data classification—A multicriteria optimization approachExpert Systems with Applications10.1016/j.eswa.2023.122287238(122287)Online publication date: Mar-2024
https://doi.org/10.1016/j.eswa.2023.122287
Zhao XWang JWang JWang JHong RShen TLiu YLiang Y(2023)DTLR-CS: Deep tensor low rank channel cross fusion neural network for reproductive cell segmentationPLOS ONE10.1371/journal.pone.029472718:11(e0294727)Online publication date: 30-Nov-2023
https://doi.org/10.1371/journal.pone.0294727
Suhaimi NOthman ZYaakub M(2022)Comparative Analysis Between Macro and Micro-Accuracy in Imbalance Dataset for Movie Review ClassificationProceedings of Seventh International Congress on Information and Communication Technology10.1007/978-981-19-2394-4_8(83-93)Online publication date: 12-Jul-2022
https://doi.org/10.1007/978-981-19-2394-4_8

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Constructing a multi-class classifier using one-against-one approach with different binary classifiers

Multi-class classification via heterogeneous ensemble of one-class classifiers

Quantum-inspired algorithm for direct multi-class classification