Scaling up text classification for large file systems

Reviewer: Tania Bedrax-Weiss

Machine learning has been applied in the domain of information retrieval with great success, as witnessed in Web search engines such as Google. In such domains, the number of search results is so large that the objective is to obtain high precision in the top search results. However, there are smaller document repositories where the objective is both high precision and high recall. In the larger of these repositories, machine learning can still take a long time to find the search results, so optimization techniques are necessary. This paper explains one such optimization technique that relies on narrowing the search space to provide significant speedups. More specifically, Forman and Rajaram formulate a two-phase classification, where the first phase selects a small subset of documents that are relevant, by using the index to query the repository, and the second classifies the documents retrieved. The authors describe in detail the different design choices made for each phase and how some of the design choices perform with respect to others. They show that the two-phase approach yields significant savings over the full classification, and conclude that the savings are more pronounced in more difficult cases-large queries. The intuition is that the two-phase classifier performs a balancing act between query time, and fetching the document and analyzing it-extracting features and classifying. Although this paper provides an interesting read as it dives into text classification for information retrieval, there are optimizations that this document does not address. For example, if the corpus does not change often, one could use offline classification for documents that essentially do not change or change very little. Still, Forman and Rajaram discuss possible extensions of their approach that seem promising, and anyone interested in this subject area should feel encouraged to explore. Online Computing Reviews Service