How to Interpret PubMed Queries and Why It Matters

Introduction

This research project was initiated as a part of the continuous work at the National Library of Medicine^™ (NLM) at the National Institutes of Health to improve information access and retrieval from MEDLINE^™, a collection of approximately 17 million bibliographic records as of fall 2007. All records have titles and about half have abstracts. PubMed^™ is a search engine, developed and maintained by the National Center for Biotechnology Information at NLM, which works on MEDLINE^™. Our major effort in this article is to examine queries and search approaches with the objective of providing MEDLINE^™ users with improved retrieval results.

When processing a query, most search engines, by default, interpret a query as a collection of terms and use the logical AND operation on these terms to retrieve records containing the query terms. That search approach is called co-occurrence searching because it retrieves records where the query terms co-occur. The PubMed^™ search engine is based on a co-occurrence search strategy.¹ For example, at the time of writing, PubMed^™ translated the query sudden death syndrome to a search of sudden AND death AND syndrome. Some search engines, including PubMed^™, allow a user to search for the exact query phrase when the query is in quotes. That search approach is called exact phrase searching.

A quoted query is an indication that users have a phrase in mind. With respect to an unquoted query, it is important to identify what users have in mind when they type in a query. Do they intend the query to be a meaningful phrase, or do they simply type terms describing the topic of interest in some random order? Search results may vary depending on how the query is interpreted by the search engine. For example, one could type in the query sudden death syndrome, looking for the records where the words sudden, death, and syndrome all occur. With the same query, one could also be looking for the records containing that exact phrase, rather than records where the query terms simply co-occur. At the time of writing, searching MEDLINE^™ for the exact query phrase sudden death syndrome retrieved 174 records, while searching MEDLINE^™ for a conjunction of query terms (sudden AND death AND syndrome) retrieved 6,237 records. Are the additional 6,063 records retrieved by the co-occurrence search as relevant to the query as the 174 records retrieved by the exact phrase search? What is the optimal retrieval approach?

Analysis of queries in PubMed^™ demonstrates that the majority of queries are multiterm. Many of these multiterm queries include parsing indicators that could help the search engine determine whether the query is a phrase or not. We discuss these in more detail in the Description of Queries in PubMed^™ section. However, a significant fraction of multi-term queries have no such indicators and consist of content words only. We refer to these as queries with no parsing indicators. These queries have no explicit indication of whether the query was intended to be a phrase or a collection of words.

In this article, we focus on multiterm queries with no parsing indicators and address the following two questions:

• What are these queries, phrases, or collections of terms?

• Does it really make any difference whether the query is handled as a phrase or as a collection of words, and, if it does, what is the optimal way to handle such a query?

To address the first question, we studied multiterm queries submitted to PubMed^™ that had no parsing indicators. A significant number of these queries are found in the MEDLINE^™ records as exact phrases, suggesting that they could, indeed, be meaningful phrases. We analyzed whether the user intended the query to be a meaningful phrase by looking at the preference of the word order in the query. For example, if a user typed in a two-word query, word1 word2, and either of the queries word1 word2 or word2 word1 appear in published text, we found that it was predominantly the order word1 word2 that was present in the published text. This suggests that users, in many cases, have a phrase in mind when they create a multiterm query with no parsing indicators. Based on these observations, we conclude that a reasonable fraction of multiterm queries with no parsing indicators are phrases and the users may expect or prefer retrieval of the same form.

To examine the second question of whether it makes a difference how the query is handled, we developed a machine learning based retrieval evaluation method. This machine learning technique is based on the assumption that the best answer to a query, which is a meaningful phrase, is a record containing that phrase. Using our evaluation method, we demonstrate that a machine learner can determine the patterns that describe the class of records containing the search phrase and the class of records containing the search terms, but not the phrase. Based on these patterns, the machine learner can distinguish between the classes. Moreover, we show that the distinction is systematic, depending on the proximity of search terms to each other. Based on term proximity, we can establish the best order of retrieval for the records.

In the next section, we study the queries submitted to PubMed^™ to examine their composition. In the Retrieval Evaluation Framework and Justification section, we explain a general framework for our machine learning based retrieval evaluation method. The Retrieval Evaluation Experiments and Results section uses that method to compare several classes of records. We describe a series of experiments that reveal the differences between classes of records that correspond to various retrieval strategies. These results are consistent with findings in the area of proximity searching, which we review in the Proximity Searching section. In the Discussion section, we discuss our approach, and we draw conclusions in the Conclusions section.

Description of Queries in PubMed^™

The goal of this section is to understand what the person issuing a query had in mind when he or she typed in a query. To do so, we examine the 14 million queries collected from the PubMed^™ search engine during 5 days in June 2006. We refer to this set of 14 million queries as PMQ2006. This data has no user or time specific information. The graph in Figure 1 depicts the distribution of these queries.

In examining these PMQ2006 queries, we found that 15% are single word queries and 85% are multiword queries, which consist of character strings separated by spaces. We observed that some multiword queries contain strings, such as ‘AND,’ ‘[author],’ or ‘+,’ which suggest how to parse that query. We refer to queries that contain such strings as queries with parsing indicators. While we do not attempt to parse these queries, we point out that the indicators they contain could be used to parse a query. These queries represent 45% of all queries and are discussed in detail in the next section. However, 40% of all queries had no such indicators but consisted of multiple content words only. What are such queries, exact phrases, or collections of terms? We look at these queries in detail in the Queries with No Parsing Indicators section in an attempt to understand how to interpret them. We find that many such queries are phrases, rather than collections of words.

Retrieval Evaluation Framework and Justification

We present a retrieval evaluation framework to compare classes of records using machine learning methods. Currently, most retrieval methods are evaluated based on how the retrieval output compares to a gold standard. A gold standard is a set of records judged to be relevant to a query that provides a benchmark against which to measure the quality of search results. This approach is used at the annual Text Retrieval Conference (TREC), run by the National Institute of Standards and Technology (NIST) since 1992 (Voorhees, 2002). Every year, NIST develops a list of queries, called topics, and provides large test collections and uniform scoring procedures. For every TREC topic, the conference participants apply their retrieval methods. Then, human experts judge a select set of records based on retrieval by all participants and produce a gold standard. Retrieved records are taken to be relevant or not relevant to the query, based on their presence or absence in the gold standard. The difficulty with this approach is that a gold standard is created by human experts, which makes the evaluation both expensive and time consuming. Therefore, it would be very desirable if we could avoid manually creating a gold standard.

We suggest using an automated retrieval evaluation method that utilizes machine learning to compare different retrieval approaches without a manually created gold standard. We develop our method based on the following two hypotheses:

• H1. As a class, the records in P systematically differ from the records in C(= S ∪ R).

• H2. The systematic differences between P and C reflect characteristics of these two sets that make records in P more likely than records in C to be useful to a searcher querying with a phrase.

It is important that the reader see why these two hypotheses are highly plausible. This is primarily based on the fact that phrases tend to be much more specific than individual words. A phrase of two or more words tends to define a specific topic of discussion because the meaning of each word is constrained by the context of the other words in the phrase. On the other hand, if the words occur, but not as a phrase, the individual words may occur with varied meanings and different relationships to each other. To illustrate this point, we searched MEDLINE^™ with the query restless legs syndrome as a phrase and as a Boolean conjunction of query terms. A typical record containing that exact search phrase is entitled

Treatment of the restless legs syndrome.

As the title indicates, the above record is very likely to talk about the query subject. On the other hand, an irrelevant record entitled

Three cases of drug-induced akathisia due to antiemetics during cancer palliative care

is retrieved by Boolean conjunction of the query terms because it contains the following text:

This syndrome consists of subjective … and objective components (restless movement, including rocking on one’s feet, walking in position shuffling and tramping the legs, and crossing and uncrossing one’s legs while sitting).

Of course, not every record containing the search phrase is focused on the query subject, as the phrase could be mentioned in passing. Likewise, it is possible for a record to contain the query terms, but not the phrase, and still be talking about the query subject. But these cases tend to be less common. The phrase restless legs syndrome happens to be a MeSH^™ term, and upon examining the sets P and C in MEDLINE^™, we found that 75% of documents in P have the MeSH^™ term “restless legs syndrome” assigned to them, whereas only 47% of the documents in C have this MeSH^™ term assigned.

Not only are the hypotheses H1 and H2 highly plausible, but we are in a position to investigate H1 using machine learning methods. If the patterns of word usage for records in P, Π_P, and records in C, Π_C, are different and the machine learning method is able to identify that difference while training, it will have distinguished between the two classes of records in the test set in a systematic way. If, on the other hand, the patterns of these two classes of records are not significantly different or the machine learner was not able to capture the difference, then we will not have succeeded in distinguishing between the two sets.

In our investigation, we used Naïve Bayes (NB) as the machine learning method as it has been shown to deal successfully with a large variety of text classification problems (Lewis, 1998). We used single words as features and the Binary Independence (Lewis) formulation of NB. Given a training set—a set of records that have already been classified as positive or negative—the learning method computes the weights of the features, based on the probability of a feature appearing in the positive and negative sets. These weights are then used to score the records from a test set according to the features (words) they contain. Based on that score, a method ranks records from most likely to be positive (high score) to most likely to be negative (low score). For convenience, we will refer to such scores as NB scores.

Once the machine learning algorithm is applied and test records are scored, we employ the Receiver Operating Characteristic (ROC) metric (Fawcett, 2006) to evaluate the ability of the classifier to distinguish between the two classes. An ROC curve is a graphical representation of true positives on the y-axis versus false positives on the x-axis as the records in the test set are examined in rank order from highest to lowest NB score. The area under the ROC curve (AUC) is called the ROC score of a classifier, and is the probability that the classifier ranks or scores a randomly chosen positive record higher than a randomly chosen negative record. The value of the ROC score typically ranges from 0.5 to 1.0, where 0.5 represents a classifier that is completely unsuccessful and 1.0 a classifier that is perfectly discriminating. In our application, it is more suggestive to report our scores as rROC = 1-ROC. We do this so that higher scores correspond to sets C, which are more like P. If ROC is high, discrimination is good and C is found to be far from the good set P. In that case, C is relatively bad and gets a low rROC score. If, on the other hand, the ROC is low, then we are having a hard time distinguishing C from the good set P, and this correlates with a good C and a high rROC score.

Our approach is to apply NB machine learning and to test H1. To the extent that we find H1 to be true, we can then use the results of the machine learning to rank documents by NB score that corresponds to their likelihood to be members of P (high score) or C (low score). Hypothesis H2 then applies with the claim that NB scores reflect differences in not only the probability of being a member of P but also the probability of being relevant. We use the term relevant merely to mean that a record is useful to a person issuing a query. Although relevance can have various philosophical meanings (Saracevic, 2007a), (Saracevic, 2007b), it is purely pragmatic in our context.

Retrieval Evaluation Experiments and Results

To generate the data for our experiments, we used queries from PMQ2006 with no parsing indicators, which the order preference test indicated were phrases (as presented in Table 2). To ensure that there is enough data for machine learning, we required that these query phrases retrieve at least 200 records in each of the sets defined in the paragraph below. From these queries, we randomly selected 100 instances each of two-, three-, and four-word queries. We study these 300 queries and refer to them as query phrases.

For each such query phrase, we retrieved records in the following sets. Set P(phrase) contains records that match the query phrase precisely. Set S(sentence) contains records that match the query terms within a single sentence but not as a phrase. Set R(record) contains records that match all query terms but not within a single sentence. The sets P, S, and R are defined so that they do not overlap, and the union of sets P ∪ S ∪ R represents the universe of records that contain all the terms of the query phrase. Each of these sets contain at least 200 records for every query phrase. This definition of the sets is identical to the one in the Queries with No Parsing Indicators section.

We randomly partition the sets P, S, and R into training and test sets

so that the training set is two-thirds of the whole, and test set is the remaining one-third.

We also define a set C(co-occurrence) as a union of sets S and R to represent the records where the query terms co-occur but not as a phrase. Clearly, set C contains at least 400 records. Set C is already partitioned into training and test parts C = C^Train ∪ C^Test, where

Now, we describe a series of experiments designed to determine whether a machine learner can distinguish between the records in the above sets.

Proximity Searching

Having developed proximity searching using purely automated means it is useful to compare with other researchers’ work using manually prepared gold standards.

Proximity searching can retrieve relevant records missed by exact phrase searching while avoiding the large number of records retrieved by co-occurrence searching. It is a flexible search methodology that can be tailored by changing the size of the window, which controls the range of records retrieved by the search.

There is substantial evidence in the literature that records with query terms in close proximity are more relevant than records with terms scattered throughout the record. In some retrieval systems, proximity search capability is explicitly available as a search option. In other systems, proximity is implicitly implemented in the ranking methods.

One of the earliest investigations on proximity searching in text retrieval systems was presented by Keen (1992). He described different systems that utilize the proximity searching capability. In an attempt to standardize the proximity searching terminology, eight variables describing proximity searching were identified and that could determine the retrieval outcome. They include span within which the proximity operates (sentence, paragraph, record, etc), maximal allowable distance between the words, whether stop words are counted, whether the simple occurrence of words is measured or some function of word counts and distance between them, etc. He compared the search results of the Boolean AND strategy with the search results of four proximity strategies: search within the paragraph, within the sentence, within the 10-word span, and five-word span. For the last two search strategies, sentence boundaries were ignored. As the search span narrowed, precision increased from 26% (Boolean AND) to 29% within the paragraph, 33% within the sentence, 36% within the 10-word span, and 43% within the five-word span. Recall percentages dropped with decrease in span but did not go below 59%. These results were based on the averages of seven searches on 6,004 bibliographical records consisting of title, abstract, and two controlled vocabulary fields.

Siadaty, Shu, & Knaus (2007) suggested using sentences as text windows for searching multiterm queries. The authors proposed that the co-occurrence of the query terms in a sentence provides evidence for the existence of a relationship between the words. They developed a system, which ranks MEDLINE^™ records retrieved by PubMed^™. As a part of their ranking system, they compared the group of records where all query terms appeared in at least one sentence in an abstract with the group of records that did not have such a sentence but contained all query terms anywhere within the record text, including abstract title and MeSH^™ terms. Based on examining two queries, they showed that, on average, the precision dropped from about 0.6 for sentence-level co-occurrence to about 0.2 for record-level co-occurrence. With only two queries, this is suggestive but hardly a definitive result.

Butcher, Clarke, & Lushman (2006) integrated term proximity into the existing vector space retrieval method Okapi BM25. Given a query, they computed the relevance score of the record as a sum of the standard TFxIDF score and a proximity measure score, which is proportional to the inverse of the squared distance between the query terms. The authors found that the use of term proximity was more important for larger text collections. For example, precision of the top 10 records with proximity-enhanced BM25 was 0.6 (compared to 0.529 with the regular BM25), and 0.561 of the top 20 records (compared to 0.494 with the regular BM25) for their large collection, GOV2, consisting of 25.2 million records. On their smaller collection, TREC45-CR, consisting of 528,155 records, they did not notice an improvement. Using various size subsets of the GOV2 database, the authors verified that the relative gain achieved by proximity-enhanced BM25, compared to the original BM25, increased as the underlying text collection grew. They did not have enough evidence, however, to support the hypothesis that term proximity is important when the search engine is dealing with longer records.

Beigbeder and Mercier (2005) developed a record-ranking model based on the degree of proximity of query term occurrences in a record. In their model, every query term has an influence region, defined by an influence function of some parameter k, which controls the extent of the term’s influence. The function has a maximum value of 1 in each place where a query term appears in the record. It decreases with a constant slope down to zero at a distance of k words on each side of the term. Given two query terms, the proximity value is positive in a region between the terms where both influence functions are positive. If the terms appear too far from each other, then the proximity value is zero. Setting the parameter k to a low value requires proximity at the phrase level, while high values allow the required proximity to approximate sentence or paragraph levels. The best performance was obtained for parameter values of k ≥ 20. However, for values of k > 3, their model was consistently better than a vector retrieval model.

In their work, Clarke, Cormack, & Burkowski (1995) introduced a shortest substring search model, which searched for the smallest spans in the text record that contained query words. A score was assigned to each of these spans based on the inverse of the span length. The record score was computed as the sum of the scores of the selected spans. This model participated at the TREC-4 ad-hoc retrieval task. For over 65% of the topics, the average precision with this model was above the median average precision for all other models.

Haas and Losee (1994) looked at the composition of text windows. They pointed out that various syntactic and semantic relationships may be found within a limited span of words, and that it is possible to exploit them all by using windows, rather then separately interpreting each relationship and combining the results. They confirmed the previously suggested idea that a window of size 11 words provides the most natural grouping of words. Hence, their research supports the claim that if the query is a phrase, then a useful answer to that query may be found within a limited text window.

Discussion

We found that a significant fraction of queries in PubMed^™ are multiterm queries without parsing instructions. We also found evidence that a large fraction of these queries are frequently used phrases in MEDLINE^™ records. This raised the question of how they should best be interpreted as queries. It also suggests that users may expect answers focused on the specific subject of the phrase. Using our machine learning methodology we have been able to adduce evidence that documents containing the exact query phrase as a class are qualitatively different from documents merely containing the query words and are more likely to be relevant as answers to the query.

The findings in the area of proximity searching show that records with the query terms appearing in close proximity to each other are more relevant to the query than records with query terms appearing further apart. Specifically, they show that the quality of retrieval changes depending on proximity of query terms to each other within the records. All these outcomes of proximity theory are consistent with our results. The advantage of our method is that we have obtained our results using solely automated techniques.

We find that records with the exact phrase, P, are different from records that merely contain the words in the query, R. rROC scores distinguishing these sets are less than 0.18 for all phrase lengths considered. Clearly, these sets are significantly different and the machine learner was able to learn corresponding patterns that were used to recognize the differences in the test set. This provides strong proof for hypothesis H1. Although we do not have a proof of hypothesis H2, there are at least three lines of argument that provide support for it. First, we note that it provides an explanation for the systematic differences seen in our machine learning. If records that contained only the words, but not the phrase, were just as relevant to the subject of the phrase as the records containing the phrase, it is hard to see why or how they could have markedly different distributions of words in them. Second, empirical evidence, such as the example of the phrase restless legs syndrome, shows that when authors use a phrase, indexers are more likely to judge such a document as relevant to the subject of the phrase. (We have confirmed this phenomenon for several examples but have not made a systematic study.) Third, the findings in studies of proximity searching also suggest that a text containing a phrase is more likely to be judged relevant to that phrase as a query than is text that contains the same words at a distance from each other.

Based on our results, we can establish the best retrieval order for the records. Hence, our system can be used as a ranking method in search engines. A simple approach would be to rank the records retrieved based on proximity of query terms to each other. A more sophisticated approach would be to recognize that relevance is not limited to proximity and rank the retrieved records in accordance with the machine learning NB score assigned by our method. This approach could rank some informative records from sets S and R higher than set P records that simply contain the query phrase in passing.

Our approach could also be used as assistance in creating a gold standard. While automated methods cannot completely replace human review, they could be used to boost the process of generating the gold standard. Once a sufficient number of records have been identified as relevant by human experts, one may extract new records that look like these relevant records. This approach is similar to relevance feedback, as described in Ruthven (2003). It provides a direction of search for the possibly relevant records by providing a set of neighboring records to the existing relevant ones.

A more straightforward design of our machine learning-based method might have been to train using the records that contain query terms against the rest of MEDLINE^™, defined as

where M stands for all of MEDLINE^™. Or, one could also train using records containing an exact query phrase, set P, against M′. In fact, we started with such an approach. Although the method could learn the difference between sets P ∪ S ∩ R and M′, the result was not effective in distinguishing the sets P, S, and R from each other. We believe there are two factors that are relevant to this issue. First, such training does not directly train the machine learner to distinguish P, S, and R. Second, NB may perform poorly when trained on unbalanced training sets (Sohn, Kim, Comeau, & Wilbur 2008), and clearly the sets M′ and P are unbalanced, in terms of the number of records they contain. To deal with these problems and improve our results, we performed the training as described in the paper.

Conclusions

A significant fraction of queries in PubMed^™ are frequently used phrases in MEDLINE^™ records. We find evidence of benefit in interpreting such queries as phrases for retrieval purposes. Using our automatic retrieval evaluation framework based on machine learning methods, we demonstrate that records with query terms occurring in close proximity to each other are more like records that contain the exact query phrase as compared to records with query terms occurring farther apart. The extent to which records where query terms only co-occur differ from the records containing the exact phrase is proportional to the distance between the query terms in the sentence. The results generated by our evaluation method are consistent with the results of researchers studying proximity searching based on manually judging the retrieved records. The appeal of our evaluation method is that it does not require a manually constructed gold standard.