article

Free access

Active Learning with Feedback on Features and Instances

Authors:

Rosie JonesAuthors Info & Claims

The Journal of Machine Learning Research, Volume 7

Pages 1655 - 1686

Published: 01 December 2006 Publication History

Abstract

We extend the traditional active learning framework to include feedback on features in addition to labeling instances, and we execute a careful study of the effects of feature selection and human feedback on features in the setting of text categorization. Our experiments on a variety of categorization tasks indicate that there is significant potential in improving classifier performance by feature re-weighting, beyond that achieved via membership queries alone (traditional active learning) if we have access to an oracle that can point to the important (most predictive) features. Our experiments on human subjects indicate that human feedback on feature relevance can identify a sufficient proportion of the most relevant features (over 50% in our experiments). We find that on average, labeling a feature takes much less time than labeling a document. We devise an algorithm that interleaves labeling features and documents which significantly accelerates standard active learning in our simulation experiments. Feature feedback can complement traditional active learning in applications such as news filtering, e-mail classification, and personalization, where the human teacher can have significant knowledge on the relevance of features.

References

[1]

J. Allan. Topic detection and tracking. Kluwer Academic Publishers, 2002.

Digital Library

[2]

D. Angluin. Computational learning theory: survey and selected bibliography. In Proceedings of the 24th Annual ACM Symposium on the Theory Computation, pages 351-369, 1992.

Digital Library

[3]

P. Anick. Using terminological feedback for web search refinement: a log-based study. In Proceedings of SIGIR '03: The 26th annual international ACM SIGIR conference on Research and development in information retrieval, pages 88-95, 2003.

Digital Library

[4]

Y. Baram, R. El-Yaniv, and K. Luz. Online choice of active learning algorithms. In Proceedings of ICML 03: The 20th International Conference on Machine Learning, pages 19-26, 2003.

[5]

E. B. Baum and K. Lang. Query learning can work poorly when human oracle is used. In International Joint Conference in Neural Netwroks, 1992.

[6]

P. Beineke, T. Hastie, and S. Vaithyanathan. The sentimental factor: Improving review classification via human-provided information. In Proceedings of ACL 04: The 42nd Meeting of the Association for Computational Linguistics, Main Volume, pages 263-270, 2004.

Digital Library

[7]

N. J. Belkin, C. Cool, D. Kelly, S. J. Lin, S. Y. Park, J. Perez-Carballo, and C. Sikora. Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing and Management, 37(3):403-434, 2001.

Digital Library

[8]

J. Brank, M. Grobelnik, N. Milic-Frayling, and D. Mladenic. Feature selection using linear support vector machines. Technical report, Microsoft Research, 2002.

[9]

C. C. Chang and C. J. Lin. Libsvm: a library for support vector machines. Available electronically at http://www.csie.ntu.edu.tw/cjlin/libsvm.

[10]

J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement , 20:27-46, 1960.

[11]

D. A. Cohn, L. Atlas, and R. E. Ladner. Improving generalization with active learning. Machine Learning, 15(2):201-221, 1994.

[12]

W. B. Croft and R. Das. Experiments with query acquisition and use in document retrieval systems. In Proceedings of SIGIR '90: The 13th annual international ACM SIGIR conference on Research and development in information retrieval, pages 349-368, 1990.

Digital Library

[13]

G. DeJong and R. Mooney. Explanation-based generalization: an alternative view. Machine Learning , 1(2):145-176, 1986.

Digital Library

[14]

F. Diaz and J. Allan. When less is more: Relevance feedback falls short and termexpansion succeeds at HARD 2005. In Text REtrieval Conference (TREC 2005) Notebook. Dept. of Commerce, NIST, 2005.

[15]

C. Domeniconi and D. Gunopulos. Incremental support vector machine construction. In Proceedings of ICDM 01:2001 IEEE International Conference on Data Mining, pages 589-592, 2001.

Digital Library

[16]

S. Godbole, A. Harpale, S. Sarawagi, and S. Chakrabarti. Document classification through interactive supervision of document and term labels. In Proceedings of PKDD 04: The 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, pages 185-196, 2004.

Digital Library

[17]

G. F. Hughes. On the mean accuracy of statistical pattern recognizers. IEEE Transactions on Information Theory, 14:55-63, 1968.

Digital Library

[18]

T. Joachims. Text categorization with support vector machines: learning with many relevant features. In ECML 98: The 10th European Conference on Machine Learning, pages 137-142, 1998.

Digital Library

[19]

T. Joachims. Transductive inference for text classification using support vector machines. In ICML '99: Proceedings of the Sixteenth International Conference on Machine Learning, pages 200- 209, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-612-2.

Digital Library

[20]

R. Jones. Learning to extract entities from labeled and unlabeled text. PhD thesis, Carnegie Mellon University, Pittsburgh, USA, 2005.

[21]

G. R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. Biometrics , 33:159-174, 1977.

Digital Library

[22]

K. Lang. Newsweeder: Learning to filter netnews. In Proceedings of ICML 95: The 12th International Conference on Machine Learning, pages 331-339, 1995.

[23]

D. D. Lewis. Naive (Bayes) at forty: The independence assumption in information retrieval. In Proceedings of ECML 98: 10th European Conference on Machine Learning, pages 4-15, 1998.

Digital Library

[24]

D. D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In Proceedings of ICML 94: The 11th International Conference on Machine Learning, pages 148-156, 1994.

[25]

N. Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2(4):285-318, 1988.

[26]

D. J. Lizotte, O. Madani, and R. Greiner. Budgeted learning of naive-bayes classifiers. In Proceedings of UIA 03: The 19th Conference on Uncertainty in AI (UAI), 2003.

Digital Library

[27]

A. K. McCallum. Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. Available electronically at http://www.cs.cmu.edu/~mccallum/bow, 1996.

[28]

T. Mitchell, R. Keller, and S. Kedar-Cabelli. Explanation-based generalization: A unifying view. Machine Learning, 1:47-80, 1986.

Digital Library

[29]

M. J. Pazzani and D. Kibler. The role of prior knowledge in inductive learning. Machine Learning, 9, 54-97., 9, 1992.

Digital Library

[30]

M. Porter. An algorithm for suffix stripping. Automated Library and Information Systems, 14(3): 130-137, 1980.

[31]

H. Raghavan, O. Madani, and R. Jones. Interactive feature selection. In Proceedings of IJCAI 05: The 19th International Joint Conference on Artificial Intelligence, pages 841-846, 2005.

Digital Library

[32]

T. G. Rose, M. Stevenson, and M. Whitehead. The Reuters Corpus Vol. 1 - from yesterday's news to tomorrow's language resources. In Proceedings of International Conference on Language Resources and Evaluation, 2002.

[33]

G. Salton. Automatic information organization and retrieval. McGraw Hill, 1968.

Digital Library

[34]

R. Schapire, M. Rochery, M. Rahim, and N. Gupta. Incorporating prior knowledge into boosting. In Proceedings of ICML 02: The 19th International Conference on Machine Learning, 2002.

Digital Library

[35]

G. Schohn and D. Cohn. Less is more: Active learning with support vector machines. In Proceedings of ICML 00: The 17th International Conference on Machine Learning, pages 839-846, 2000.

Digital Library

[36]

F. Sebastiani. Machine learning in automated text categorization. ACM Computing Surveys, 34(1): 1-47, 2002.

Digital Library

[37]

S. Tong and D. Koller. Support vector machine active learning with applications to text classification. Journal of Machine Learning Research, 2:45-66, 2002. ISSN 1533-7928.

Digital Library

[38]

E. M. Voorhees and L. P. Buckland, editors. Text REtrieval Conference (TREC 2005) Notebook, 2005. Dept of Commerce, NIST.

[39]

X. Wu and R. Srihari. Incorporating prior knowledge with weighted margin support vector machines. In Proceedings of KDD 04: Tenth ACMSIGKDD International Conference on Knowledge Discovery and Data Mining, pages 326-333, 2004.

Digital Library

[40]

X. Zhu. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005.

Cited By

Zhang XWang ZJiang LGao WWang PLiu KLarson K(2024)TFWTProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/284(2570-2578)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/284
Feng KMcdonald D(2023)Addressing UX Practitioners’ Challenges in Designing ML Applications: an Interactive Machine Learning ApproachProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584064(337-352)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584064
Feng KCoppock MMcDonald D(2023)How Do UX Practitioners Communicate AI as a Design Material? Artifacts, Conceptions, and PropositionsProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596101(2263-2280)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596101
Show More Cited By

Index Terms

Active Learning with Feedback on Features and Instances
1. Applied computing
  1. Document management and text processing
2. Computing methodologies
  1. Machine learning

Recommendations

Label distribution learning with label-specific features
IJCAI'19: Proceedings of the 28th International Joint Conference on Artificial Intelligence

Label distribution learning (LDL) is a novel machine learning paradigm to deal with label ambiguity issues by placing more emphasis on how relevant each label is to a particular instance. Many LDL algorithms have been proposed and most of them ...
Active learning by labeling features
EMNLP '09: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1

Methods that learn from prior information about input features such as generalized expectation (GE) have been used to train accurate models with very little effort. In this paper, we propose an active learning approach in which the machine solicits "...
Cost‐effective multi‐instance multilabel active learning
Abstract
Multi‐instance multi‐label (MIML) Active Learning (M2AL) aims to improve the learner while reducing the cost as much as possible by querying informative labels of complex bags composed of diverse instances. Existing M2AL solutions suffer high ...

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research

The Journal of Machine Learning Research Volume 7, Issue

12/1/2006

2725 pages

ISSN:1532-4435

EISSN:1533-7928

Issue’s Table of Contents

Publisher

JMLR.org

Publication History

Published: 01 December 2006

Published in JMLR Volume 7

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

68
Total Citations
View Citations
841
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)12

Reflects downloads up to 10 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang XWang ZJiang LGao WWang PLiu KLarson K(2024)TFWTProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/284(2570-2578)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/284
Feng KMcdonald D(2023)Addressing UX Practitioners’ Challenges in Designing ML Applications: an Interactive Machine Learning ApproachProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584064(337-352)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584064
Feng KCoppock MMcDonald D(2023)How Do UX Practitioners Communicate AI as a Design Material? Artifacts, Conceptions, and PropositionsProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596101(2263-2280)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596101
Wang ZLiu CGombolay M(2022)Heterogeneous graph attention networks for scalable multi-robot scheduling with temporospatial constraintsAutonomous Robots10.1007/s10514-021-09997-246:1(249-268)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1007/s10514-021-09997-2
Ghai BLiao QZhang YBellamy RMueller K(2021)Explainable Active Learning (XAL)Proceedings of the ACM on Human-Computer Interaction10.1145/34329344:CSCW3(1-28)Online publication date: 5-Jan-2021
https://dl.acm.org/doi/10.1145/3432934
van der Stappen AFunk MHammond TVerbert KParra DKnijnenburg BO'Donovan JTeale P(2021)Towards Guidelines for Designing Human-in-the-Loop Machine Training InterfacesProceedings of the 26th International Conference on Intelligent User Interfaces10.1145/3397481.3450668(514-519)Online publication date: 14-Apr-2021
https://dl.acm.org/doi/10.1145/3397481.3450668
Park SWang AKawas BLiao QPiorkowski DDanilevsky MHammond TVerbert KParra DKnijnenburg BO'Donovan JTeale P(2021)Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP ModelsProceedings of the 26th International Conference on Intelligent User Interfaces10.1145/3397481.3450637(585-596)Online publication date: 14-Apr-2021
https://dl.acm.org/doi/10.1145/3397481.3450637
Chen LPaleja RGhuy MGombolay MBelpaeme TYoung JGunes HRiek L(2020)Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network DistillationProceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3319502.3374791(659-668)Online publication date: 9-Mar-2020
https://dl.acm.org/doi/10.1145/3319502.3374791
Gombolay MJensen RStigile JGolen TShah NSon SShah J(2019)Human-machine collaborative optimization via apprenticeship schedulingJournal of Artificial Intelligence Research10.1613/jair.1.1123363:1(1-49)Online publication date: 17-Apr-2019
https://dl.acm.org/doi/10.1613/jair.1.11233
Correia ALecue F(2019)Human-in-the-loop feature selectionProceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v33i01.33012438(2438-2445)Online publication date: 27-Jan-2019
https://dl.acm.org/doi/10.1609/aaai.v33i01.33012438
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents