Overview of the gene ontology task at BioCreative IV

Task A: retrieving GO evidence text for relevant genes

GO evidence text is critical for human curators to make associated GO annotations. For a given GO annotation, multiple evidence passages may appear in the paper, some being more specific with experimental information while others may be more succinct about the gene function. For this subtask, participants were given as input full-text articles together with relevant gene information. For system output, teams were asked to submit a list of GO evidence sentences for each of the input genes in the paper. Manually curated GO evidence passages were used as the gold standard for evaluating team submissions. Each team was allowed to submit three runs.

Task B: predicting GO terms for relevant genes

This subtask is a step toward the ultimate goal of using computers for assisting human GO curation. As in Task A, participants were given as input full-text articles with relevant gene information. For system output, teams were asked to return a list of relevant GO terms for each of the input genes in a paper. Manually curated GO annotations were used as the gold standard for evaluating team predictions. As in Task A, each team was allowed to submit three runs.

Generally speaking, the first subtask is a text retrieval task while the second can be seen as a multi-class text classification problem where each GO term represents a distinct class label. In the BioNLP research domain, the first subtask is in particular akin to the BioCreative II Interaction Sentence subtask (24), which also served as an immediate step for the ultimate goal of detecting protein–protein interactions. Task A is also similar to the BioCreative I GO subtask 2.1 (22) and automatic GeneRIF identification (18, 28–31). The second subtask is similar to the BioCreative I GO subtask 2.2 (22) and is also closely related to the problem of semantic indexing of biomedical literature, such as automatic indexing of biomedical publications with MeSH terms (32–35).

Corpus annotation

A total of eight professional GO curators from five different MODs—FlyBase (http://flybase.org/); Maize Genetics and Genomics Database (http://www.maizegdb.org/); Rat Genome Database (http://rgd.mcw.edu/); TAIR (http://www.arabidopsis.org/); WormBase (http://www.wormbase.org/)—contributed to the development of the task data. To create the annotated corpus, each curator was asked, in addition to their routine annotation of gene-related GO information, to mark up the associated evidence text in each paper that supports those annotations using a Web-based annotation tool. To provide complete data for text-mining system development (i.e. both positive and negative training data), curators were asked to select evidence text exhaustively throughout the paper (36).

For obtaining high-quality and consistent annotations across curators, detailed annotation guidelines were developed and provided to the curators. In addition, each curator was asked to practice on a test document following the guidelines before they began curating task documents. Because of the significant workload and limited number of curators per group, each paper was only annotated by a single curator.

Evaluation measures

For Task A evaluation, traditional precision (P), recall (R) and F₁ score (F₁) are reported when comparing the submitted gene-specific sentence list against the gold standard. We computed the numbers of true-positive (TP) results and false-positive (FP) results in two ways: the first one (exact match) is a strict measure that requires the returned sentences exactly match the sentence boundary of human markups, while the second (overlap) is a more relaxed measure where a prediction is considered correct (i.e. TP) as long as the submitted sentence overlaps with the gold standard. Although a single character overlap between the text-mined and human-curated sentences would be sufficient for the relaxed measure, the actual overlap is significantly higher as we found in practice (see results below).

For the Task B evaluation, gene-specific GO annotations in the submissions were compared with the gold standard. In addition to the traditional precision, recall and F₁ score, hierarchical Precision (hP), Recall (hR) and F₁ score (hF₁) were also computed where common ancestors in both the computer-predicted and human-annotated GO terms are considered. The second set of measures was proposed to reflect the hierarchical nature of GO: a gene annotated with one GO term is implicitly annotated with all of the term’s parents, up to the root concept (37, 38). Such a measure takes into account that ‘predictions that are close to the oracle label should score better than predictions that are in an unrelated part of the hierarchy’. (37) Specifically, the hierarchical measures are computed as follows:

where ${\widehat{G}}_i$ and ${\widehat{G}}^{\prime }{}_i$ are the sets of ancestors of the computer-predicted and human-annotated GO terms for the ith set of genes, respectively.

The BC4GO corpus

The task participants were provided with three data sets comprising 200 full-text articles in the BioC XML format (39). Our evaluation for the two subtasks was to assess teams’ ability to return relevant sentences and GO terms for each given gene in the 50 test articles. Hence, we show in Table 1 the overall statistics of the BC4GO corpus including the numbers of genes, gene-associated GO terms and evidence text passages. For instance, in the 50 test articles, 194 genes were associated with 644 GO terms and 1681 evidence text passages, respectively. We refer interested readers to (36) for a detailed description of the BC4GO corpus.

Team participation results

Overall, seven teams (three from America, three from Asia and one from Europe) participated in the GO task. In total, they submitted 32 runs: 15 runs from five different teams for Task A, and 17 runs from six teams for Task B.

Post-challenge analysis: classification of FP sentences

To better understand the types of FP sentences returned by the participating text-mining systems, we asked curators to manually review and classify FP predictions using one or more categories described below. For this analysis, each curator was given three test set papers that they previously annotated. In total, seven curators completed this analysis by assigning 2289 classifications to 2074 sentences.

Sentence classifications

• (1) Experiment was performed—These types of sentences relate that an experiment has been performed but do not describe what the actual result was. Such sentences may or may not contain a GO-related concept.

‘To characterize the functions and interrelationships of CSP41a and CSP41b, T-DNA insertion lines for the genes encoding the two proteins were characterized.’

• (2) Previously published result—These sentences refer to experimental findings from papers cited in the test set papers. They often contain a parenthetical reference, or other indication, that the information is from a previously published paper.

‘Molecular studies of the REF-1 family genes hlh-29 and hlh-28 indicate that their gene products are identical, and that loss of hlh-29/hlh-28 activities affects C. elegans embryonic viability, egg-laying, and chemorepulsive behaviors [21].’

• (3) Not GO related—These sentences describe an aspect of biology that is not amenable to GO curation, i.e. it does not describe a biological process, molecular function or cellular component.

‘(A) An anti-Aurora A anti-serum recognizes the NH2-terminal recombinant histidine-tagged protein domain used for immunization (left) and the 47-kD endogenous Aurora A protein kinase in Drosophila embryo extracts (right) by Western blotting.’

• (4) Curator missed—This class of sentences actually represents TP sentences that the curator failed to identify when annotating the test set papers.

‘We found that knockdown of Shank3 specifically impaired mGluR5 signaling at synapses.’

• (5) Interpretive statements/author speculation—These sentences describe either an author’s broader interpretation of an experimental finding or their speculation on that finding, but do not necessarily provide direct evidence for a GO annotation.

‘The binding site for AR-C155858 involves TMs 7-10 of MCT1, and probably faces the cytosol.’

• (6) Contiguous sentence—These sentences were selected by curators, but only as part of an annotation that required additional sentences that may or may not be directly adjacent to the annotated sentence. In these cases, curators felt that additional information was needed to completely support the GO annotation.

‘These results are in agreement with those obtained for the TIEG3 protein in HeLa and OLI-neu cells [32] and indicate that the Cbt bipartite NLS within the second and third zinc fingers is functional in mammalian cells, suggesting that different nuclear import mechanisms for this protein are being used in Drosophila and mammalian cells.’

• In this case, information about the assay used to determine ‘these results’ is not available in just this sentence alone.

• (7) Sentence was captured—In these cases, the sentence was captured by the curator, but for some, the annotation was for a different gene product than that predicted by the participating teams.

‘Furthermore, in primary macrophages, expression of Fcgr3-rs inhibits Fc receptor-mediated functions, because WKY bone marrow-derived macrophages transduced with Fcgr3-rs had significantly reduced phagocytic activity.’

• (8) Negative result—These sentences describe an experimental finding, but one for which the result is negative, i.e. the gene product is not involved, and thus the sentence would not be annotated for GO.

‘The atnap null mutant and WT plants are developmentally indistinguishable in terms of bolting and flowering times.’

• (9) Mutant background—These sentences describe an experimental finding that does not reflect the wild-type activity of the gene product. This classification is distinct from sentences that describe mutant phenotypes, which are often used to assign GO Biological Process annotations.

‘S5 shows Mad2MDF-2 enrichment on monopolar spindles in the PP1-docking motif mutants.’

• (10) Other—This classification was reserved for sentences that did not readily fit into any of the additional classifications.

‘Arrows indicate the main CSP41a and CSP41b protein species.’

• (11) Unresolved entity—These sentences mentioned entities, e.g. protein complexes, for which curators were not able to assign a specific ID for annotation.

‘It is suggested that CSP41 complexes determine the stability of a distinct set of chloroplast transcripts including rRNAs, such that the absence of CSP41b affects both tar-get transcript stability and chloroplast translational activity.’

• (12) Future experiments—These sentences describe proposed future experiments.

‘Thus, mechanistic insight into the reactions that activate checkpoint signaling at the kinetochore and testing the effect of KNL-1 microtubule binding on these reactions as well as elucidating whether KNL-1 mutants participate in parallel to or in the same pathway as dynein in checkpoint silencing are important future goals.’

The results of the classification analysis are presented in Figure 1. These results indicate that the FP sentences cover a broad range of classifications, but importantly, that only 12% of the FP sentences were classified by curators as completely unrelated to a GO concept. Many of the FP sentences thus contain some element of biology that is relevant to GO annotation, but lack the complete triplet, i.e. an entity, GO term and assay, that is typically required for making a manual experimentally supported GO annotation.

As an additional part of the sentence classification, we also asked curators to indicate which, if any, of the GO triplet was missing from a FP sentence. These analyses indicate some overall trends. For example, many of the sentences that describe previously published results do not indicate the nature of the assay used to determine the experimental findings. In contrast, sentences that describe how or what type of an experiment was performed may include all aspects of the triplet yet lack the actual experimental result that supports an annotation. Likewise, sentences that describe negative results may contain all aspects of the triplet, but the prediction methods failed to discern the lack of association between the gene product and the GO term.

The results of the sentence classification analysis suggest that many of the FP sentences returned by the participating teams have some relevance to GO annotation but either lack one element of the GO annotation triplet or contain all elements of a triplet but fail to correctly discern the actual experimental result. This suggests that further work to refine how evidence sentences are identified and presented to curators may help to improve the utility of text mining for GO annotation. For example, if curation tools can present predicted evidence sentences within the context of the full text of the paper, curators could easily locate those sentences, such as those presented within Results sections, that are most likely to support GO annotations. Additionally, postprocessing of sentences to remove those that contain terms such as ‘not’ or ‘no’ may help to eliminate statements of negative results from consideration.

Further analysis of the content of evidence sentences will hopefully provide valuable feedback to text-mining developers on how to refine their prediction algorithms to improve precision of evidence sentence identification. For example, within a sampling of the largest sentence classification category, ‘Experiment was Performed’, curators marked nearly half of the sentences as containing no GO term. Systematic comparison of these sentences with similarly categorized sentences that did contain a GO term concept may help to improve techniques for GO term recognition.

Additional follow-up analysis may also help annotation groups consider new ways in which to use text-mining results. Text mining for GO annotation might thus expand to include not only predictions for experimentally supported annotations but also predictions for other annotations supported by the text of a paper such as those described from previously published results. GO annotation practice includes an evidence code, Trace-able Author Statement (TAS), that can be used for these types of annotations, so perhaps a new evidence code that indicates a TAS annotation derived from text mining could be developed for such cases.

Individual system descriptions

Each team has agreed to contribute a brief summary of the most notable aspects of their system. In summary, the machine learning approaches performed better than the rule-based approaches in Task A. For example, Team 238 achieved the best performance by using multiple features (bag-of-words, bigram features, section features, topic features, presence of genes) and training a logistic regression model to classify positive vs negative instances of GO evidence sentences.

A variety of methods were attempted for Task B, such as K-nearest-neighbor, pattern matching and information retrieval (IR)-based ranking techniques. Moreover, several participants (Team 183, 238 and 250) used the evidence sentences they retrieved in Task A as input for finding GO terms in Task B. The best performance in Task B was obtained by Team 183’s supervised categorization method, which retrieved most prevalent GO terms among the k most similar instances to the input text in their knowledge base (43).

Team 183: Julien Gobeill, Patrick Ruch (Task A, Task B)

The BiTeM/SIBtex group participated in the first BioCreative campaign (22). We then obtained top competitive results, although for all competing systems, performances were far from being useful for the curation community. At this time, we extracted GO terms from full texts with a locally developed dictionary-based classifier (44). Dictionary-based categorization approaches attempt to exploit lexical similarities between GO terms (descriptions and synonyms) and the input text to be categorized. Such approaches are limited by the complex nature of the GO terms. Identifying GO terms in text is highly challenging, as they often do not appear literally or approximately in text. We have recently reported on GOCat (45, 46), our new machine learning GO classifier. GOCat exploits similarities between an input text and already curated instances contained in a knowledge base to infer a functional profile. GO annotations (GOA) and MEDLINE make it possible to exploit a growing amount of almost 100 000 curated abstracts for populating this knowledge base. Moreover, we showed in (46) that the quality of the GO terms predicted by GOCat continues to improve across the time, thanks to the growing number of high-quality GO terms assignments available in GOA: thus, since 2006, GOCat performances have improved by +50%.

The BioCreative IV Track 4 gave us the opportunity to exploit the GOCat power in a reference challenge. For Task A, we designed a robust state-of-the-art approach, using a naïve Bayes classifier, the official training set and words as features. This approach generally obtained fair results (top performances for high precision systems) and should still benefit from being tuned for this task with the new available benchmark. We also investigated exploiting GeneRIFs for an alternative 40 times bigger training set, but the results were disappointing, probably because of the lack of good-quality negative instances. Then, for Task B, we applied GOCat to the first subtask output and produced three different runs with five, ten or twenty proposed GO terms. These runs outperformed other competing systems both in terms of precision and recall, with performances up to 0.65 for recall with hierarchical metrics. Thanks to BioCreative, we were able to design a complete workflow for curation. Given a gene name and a full text, this system is able to deliver highly relevant GO terms along with a set of evidence sentences. Today, the categorization effectiveness of the tool seems sufficient for being used in real semiautomatic curation workflows, as well as in fully automatic workflow for nonmanually curated biological databases. In particular, GOCat is used to profile PubChem bioassays (47), and by the COMBREX project to normalize functions described in free text formats (48).

Team 220: Anh Tuan Luu, Jung-jae Kim (Task B)

Luu and Kim (49) present a method that is based on the cross products database (50) and combined with a state-of-the-art statistical method based on the bag of words model. They call the GO concepts that are not defined with cross products, ‘primitive concepts’, where the primitive concepts of a GO term are those that are related to the GO term through cross products possibly in an indirect manner. They assume, like the assumption of bag-of-words approach, that if all or most of the primitive concepts of a GO term appear in a small window of text (e.g. sentence), the GO term is likely to be expressed therein. For each GO term and a text, the method first collects all primitive concepts of the GO term and identifies any expression of a primitive concept in the text. It recognizes as expressions of a primitive concept the words that appear frequently in the documents that are known to express the concept (called domain corpus), but not frequently in a representative subset of all documents (called generic corpus). Given a document δ and a primitive concept γ, if the sum of the relative frequency values of the top-K words of the concept found in the text is larger than a threshold θ, we regard the concept as expressed in the document. Finally, a text is considered to express GO term Γ whose cross products definition has n primitive concepts, if this text expresses at least k primitive concepts among the n concepts, where the value of k is dynamically determined using a sigmoid function, depending on n.

Furthermore, the cross products–based method (called XP method) is incorporated with Gaudan’s method (51), which shows a better coverage than the XP method, as follows: For each GO term Γ whose cross products definition has n primitive concepts, if the XP method can find evidence to k primitive concepts (as explained above) in the text zone δ, the combined method calculates the sum of the scores from the two methods. If the sum is greater than a threshold, we assume that δ expresses Γ. If a GO term does not have a cross products definition, we only use the score of the Gaudan’s method. In short, we call the combination method is XP-Gaudan method. The experiment results show that the F-measures of the two individual methods are lower than that of the XP-Gaudan method. The recall of the XP-Gaudan method (21%) is close to the sum of the recall values of the two individual methods (26%), which may mean that the two methods target different sets of GO term occurrences. In other words, the XP method is complementary to the Gaudan’s method in detecting GO terms in text documents.

Team 237: Jung-Hsien Chiang, Yu-De Chen, Chia-Jung Yang (Task A, Task B)

We developed two different methods: a sequential pattern mining algorithm and GREPC (Geneontology concept Recognition by Entity, Pattern and Constrain) for the BioCreative GO track to recognize sentences and GO terms.

In our sequential pattern mining algorithm, the highlight of this method is that it can infer GO term and which gene(s) products the GO term belongs to simultaneously. In this method, each of the generated rules has two classes, one for the inferred GO term and another for the GO term to which the gene(s) products belong to. Besides, each of the rules is learned from data without human intervention. The basic idea of the sequential pattern mining algorithm we used was similar to (52–55). We also used Support and Confidence in association rule learning to measure the rules generated. In this work, the items were terms that appeared in sentences. The different permutations of terms will be considered different patterns because of the spirit of sequential pattern. In the preprocessing, we removed stop words, stemmed the rest and added P.O.S. tagger to each term. Then, we anonymized each of the gene(s) products for generating rules that can be widely used in the situation with different gene product names. For instance, a sentence ‘In vitro, CSC-1 binds directly to BIR-1’ would become ‘vitro_NN __PROTEIN_0__ bind_VBZ directli_RB __PROTEIN_1__ ’. Both the terms ‘__PROTEIN_0__’ and ‘__PROTEIN_1__’ are anonymized gene(s) products. After preprocessing, we thereafter generated rules from the preprocessed sentences. In the instance we mentioned above, we can generate some rules, e.g. ‘__PROTEIN_0__ bind_VBZ __PROTEIN_1__ => GO: 0005515, __PROTEIN_0_’ and ‘__PROTEIN_0__ bind_VBZ __PROTEIN_1__ => GO: 0005515, __PROTEIN_1__’, where the part before the symbol ‘=>’ is the pattern and after the symbol are the classes. The first class GO: 0005515 is the GO ID. The second class represents the GO term belonging to which anonymized gene(s) products. After all rules have been generated, we used those rules to classify sentences in the testing data.

In the GREPC, we indexed the GO concepts based on three divisions: entity, pattern and constrain. We gathered these kinds of information by text mining inside the GO database (56). Within that, we reconstructed the semi-structured name and synonyms for a GO concept into a better-structured synonym matrix. With GREPC, we can find GO terms in a sentence with a higher recall without losing much of the precision.

Team 238: Hongfang Liu, Dongqing Zhu (Task A, Task B)

For Task A, the Mayo Clinic system effectively leveraged the learning from positive and unlabeled data approach (57, 58) to mitigate the constraint of having limited training data. In addition, the system explored multiple features (e.g. unigrams, bigrams, section type, topic, gene presence, etc.) via a logistic regression model to identify GO evidence sentences. The adopted features in their system brought incremental performance gains, which could be informative to the future design of similar classification systems. Their best performing system achieved 0.27 on exact-F1 and 0.387 for overlap-F1, the highest among all participating systems.

For Task B, the Mayo Clinic team designed two different types of systems: (i) the search-based system predicted GO terms based on existing annotations for GO evidences that are of different textual granularities (i.e. full-text articles, abstracts and sentences) and are obtained by using state-of-the-art IR techniques [i.e. a novel application of the idea of distant supervision in information extraction (59)]; (ii) the similarity-based systems assigned GO terms based on the distance between words in sentences and GO terms/synonyms. While the search-based system significantly outperformed the similarity-based system, a more important finding was that the number and the quality of GO evidence sentences used in the distance supervision largely dictates the effectiveness of distant supervision, meaning a large collection of well-annotated, sentence or paragraph level GO evidences is strongly favored by systems using similar approaches.

Team 243: Ehsan Emadzadeh, Graciela Gonzalez (Task B)

The proposed open-IE approach is based on distributional semantic similarity over the Gene Ontology terms. The technique does not require the annotated data for training, which makes it highly generalizable. Our method finds the related gene functions in a sentence based on semantic similarity of the sentence to GO terms. We use the semantic vectors package (60) implementation of latent semantic analysis (LSA) (61) with random indexing (62) to calculate semantic similarities. GO terms’ semantic vectors are created based on the names of the entries in GO; one semantic vector is created for each term in the ontology. Stop words are removed from GO name, and they are generalized by Porter stemming (63).

After creating the GO semantic vectors, the question is to find whether a sentence is related to a gene. We do this by using lexical patterns and generalizing the sentence and the gene symbol (e.g. removing the numbers and nonalphabetic characters). If lexical patterns imply that a sentence is related to a gene, then we calculate semantic similarity of the sentence to all GO terms using the generated semantic vectors. The predicted GO terms for the sentence and the gene are the conjunction of top similar GO terms to the sentence (set G) and top similar GO terms to the related abstract (set D):

A GO term with the highest semantic similarity to the sentence in GeneGO set will be chosen as the final GO annotation for each gene in the sentence. For example, if a sentence top m(=2) similar GO terms are {g5, g10} and the abstract top n(=5) GO terms are {g4, g8, g5, g2, g9}, then the final predicted GO terms for the sentence related to the gene will be {g5}. m and n are tuning parameters that control the precision and the recall. We found that the first sentences of the paragraphs are the most important sentences in terms of information about gene functions, and including all sentences in a paragraph significantly reduced the precision.

Team 250: Yanpeng Li, Hong Yu (Task A, Task B)

For Task A, we built a binary classifier to identify evidence sentences using reference distance estimator (RDE) (64, 65), a recently proposed semi-supervised learning method that learns new features from around 10 million unlabeled sentences. Different from traditional methods for text classification e.g. bag-of-words features with support vector machine (SVM) or logistic regression, our method generates new features using the co-occurrence of existing features in big unlabeled data, thus incorporating richer information to overcome data sparseness and leading to more robust performance. RDE is a simple linear classifier in the form of:

where $x_i$ is the ith example represented by a Boolean vector of $x_{ij}$, j is the index of feature, and r is called a reference feature. The probability of $P(r|j)-P(r)$ can be directly estimated from unlabeled data. In the work (64), we showed in theory and experiment that if r is discriminative to the class label and highly independent with other features, the performance of RDE tends to be close to a classifier trained with infinite labeled data. In our submitted runs, we applied the Algorithm 2.2 presented in the paper (64), which generates new features from multiple RDEs and integrates them in a logistic regression model. For this task, we selected 110 reference features for semi-supervised RDE based on the labeled (training and development sets) and unlabeled data, and then used the output scores of the 110 RDEs leaned from unlabeled data as features (66). Our best submitted run using only bag-of-words features achieved an F-score of 19.3% in exact match and 32.5% in relaxed match, which ranked second place by teams. It is also promising to see that in both development and test sets RDE achieved much better F score and area under curve than SVM and logistic regression (64, 66).

In Task B, we developed an IR-based method to retrieve the GO term most relevant to each evidence sentence using a ranking function that combined cosine similarity and the frequency of GO terms in documents. The ranking function is defined as follows:

where the first part is the cosine similarity of the sentence and GO term, and $count(GOterm)$ is the number of documents related to the GO term in the Gene Ontology Annotation (GOA) databases. Similar to the idea of page rank algorithm in Web search, the GORank function prefers ‘important’ (high-frequency) GO terms, as we found that the occurrence of GO term in documents follows a power law distribution, that is, a small fraction of GO terms appear in a lot of documents, and most GO terms appear rarely. In addition, in order to make use of the information in the annotated sentences to improve the performance, after the ranking, we built a classifier for 12 high-level GO classes trained on labeled sentences to prune the result. A filtering threshold t was defined as the number of t most relevant high-level GO classes to the sentence determined by the classifiers. If the highest ranked GO term by GORank is in the t classes, it will be selected as a positive result. The result of submitted runs showed that the F score increased from 3.6 to 7.8% using these two strategies (66). Our submission as well as post-submission results showed these novel methods were able to achieve the F scores competitive to the top-ranked systems.

Team 264: Jian-Ming Chen, Hong-Jie Dai (Task A)

To efficiently and precisely retrieve GO information from large amount of biomedical resources, we propose a GO evidence sentence retrieval system conducted via combinatorial applications of semantic class and rule patterns to automatically retrieve GO evidence sentences with specific gene mentions from full-length articles. In our approach, the task is divided into two subtasks: (i) candidate GO sentence retrieval, which selects the candidate GO sentences from a given full text, and (ii) gene entity assignment, which assigns relevant gene mentions to a GO evidence sentence.

In this study, sentences containing gene entities or GO terms are considered as potential evidence sentences. Semantic classes including the adopted and rejected class derived from the training set using semantic-orientation point-wise mutual information (SO-PMI) are used for selecting potential sentences and filtering out FP sentences (67). To further maximize the performance of GO evidence sentence retrieval, rule patterns generated by domain experts are defined and applied. For example, if a potential sentence matches the rule pattern ‘[GENE].* lead to .*[GO]’, the sentence is selected again as a GO evidence sentence candidate. After generating the sentence candidates, the process of gene entity assignment is performed to identify probable gene mentions contained within each sentence. In our current implementation, a gene is assigned to the sentence S if the gene is mentioned in S. Otherwise, we identify the gene with the maximum occurrence from retrieved sentences in paragraph P in which S belongs, and assign this gene to sentence S. Alternatively, gene with the maximum occurrence from retrieved sentences in article A is verified and assigned to sentence S.

The performance of our GO evidence sentence retrieval system achieves the highest recall of 0.424 and 0.716 in the exact match and relaxed overlap measure, respectively. However, the inadequate F score of our system suggests that the rule patterns used may decrease the system performance. In the future, the conduction of rule selection in rule pattern generation and co-reference resolution in gene entity assignment will be performed to maximize the overall performance.