UNIT 4 Information Retrieval using NLP

Introduction to Information Retrieval –

An Information Retrieval (IR) system is a software system designed to retrieve relevant
information from a large collection of unstructured or semi-structured data, usually in the
form of text documents, in response to a user query. In the context of Natural Language
Processing (NLP), an IR system deals with text data and aims to understand the user's query
and retrieve relevant documents or information.
Here's the concept of an Information Retrieval system in terms of NLP:
1. Text Data Collection:
 An IR system starts with a collection of text documents. These documents can
be web pages, emails, articles, books, etc.
 In NLP, these documents are typically represented as a corpus, a large
collection of text.
2. Indexing:
 The IR system indexes the documents in the corpus to facilitate efficient
retrieval.
 Each document is analyzed and tokenized into individual words or terms.
 Stop words (common words like "and", "the", etc.) are usually removed.
 Stemming or lemmatization might be applied to reduce words to their base
forms.
 The resulting terms are then stored in an index, along with information about
which documents contain each term.
3. User Query Processing:
 When a user enters a query, the IR system processes the query to understand
the user's information needs.
 The query might be a single word, multiple words, or even a complex question
or phrase.
 NLP techniques are used to parse the query, tokenize it, and extract important
terms or concepts.
4. Retrieval:
 Based on the processed query, the IR system retrieves documents from the
index that are likely to be relevant to the user's query.
 This retrieval is typically done by calculating the relevance scores of
documents with respect to the query.
  Various retrieval models can be used, such as the Vector Space Model (VSM)
or Probabilistic models like Okapi BM25.
5. Ranking:
 The retrieved documents are ranked based on their relevance scores.
 The most relevant documents are usually presented to the user first.
6. Presentation:
 Finally, the relevant documents are presented to the user, often in the form of a
list of titles or snippets.
 In some cases, the system may also highlight the portions of the document that
match the query.
7. Feedback and Iteration:
 Some IR systems support feedback mechanisms where users can provide
feedback on the relevance of the retrieved documents.
 This feedback can be used to improve the system's performance over time.

Vector Space Model –

The Vector Space Model (VSM) is a widely used technique in Information Retrieval (IR) to
represent text documents and queries as vectors in a high-dimensional space. It's based on the
idea that documents and queries can be represented as vectors, and their similarity can be
measured using geometric operations in this vector space. Here's a detailed explanation of the
Vector Space Model and how it works:
1. Representation of Documents and Queries:
In the Vector Space Model, each document and query is represented as a vector in a high-
dimensional space.
 Term Frequency (TF): The most basic approach is to represent each document and
query based on the frequency of terms in them. The 𝑖i-th component of a document's
vector represents the frequency of term 𝑖i in that document.
 Inverse Document Frequency (IDF): To account for the fact that some terms are
more common than others, IDF is often used. The IDF of a term is defined as the
logarithm of the ratio of the total number of documents to the number of documents
containing that term. This helps in giving less weight to common terms and more
weight to rare terms.
 TF-IDF Weighting: The TF-IDF weight of a term in a document is the product of its
term frequency and inverse document frequency. This gives a higher weight to terms
that are frequent in the document but rare in the entire corpus.
2. Construction of the Term-Document Matrix:
Using TF-IDF weighting, we construct a matrix where rows represent terms and columns
represent documents. Each element 𝑎𝑖𝑗 of this matrix represents the TF-IDF weight of term 𝑖
in document 𝑗.
3. Query Processing:
When a user enters a query, it goes through similar processing to represent it as a vector in
the same space.
4. Similarity Measurement:
To measure the similarity between a query vector and document vectors, various similarity
measures can be used. The most common one is cosine similarity.

5. Retrieval:
Once the similarity scores between the query vector and all document vectors are computed,
documents are ranked based on these scores.
Pros:
1. Flexibility:
 VSM can handle various types of text data and can be adapted to different
NLP tasks.
 It's versatile and can accommodate different representations and similarity
measures.
2. Efficiency:
 Once the Term-Document Matrix is constructed, retrieval is efficient.
 Retrieval time is generally proportional to the number of terms in the query,
making it suitable for large-scale retrieval tasks.
3. Simple Implementation:
 The concept of representing documents and queries as vectors is
straightforward and easy to implement.
 It doesn't require complex algorithms or heavy computational resources for
implementation.
4. Interpretability:
 The similarity scores produced by VSM have an intuitive interpretation.
 Users can understand the relevance of retrieved documents based on their
similarity to the query.
 5. Scalability:
 VSM scales well to large collections of documents.
 It's suitable for applications where the corpus contains millions of documents.
6. Customization:
 VSM allows for customization of various parameters such as term weighting
schemes, similarity measures, and dimensionality reduction techniques.
 This customization enables tailoring the model to specific requirements and
datasets.
Cons:
1. Sparse Representation:
 Large vocabularies lead to sparse vectors, where most elements are zeros.
 This sparsity can affect the efficiency of computation and storage.
2. Semantic Gap:
 VSM treats terms as independent and doesn't capture semantic relationships
well.
 It may fail to understand the meaning or context of words, leading to
mismatches between query terms and relevant documents.
3. Curse of Dimensionality:
 In high-dimensional spaces, distances between vectors lose meaning due to the
curse of dimensionality.
 High-dimensional vectors can be computationally expensive to handle and
may require dimensionality reduction techniques.
4. Lack of Context:
 VSM ignores the order and context of words within documents.
 This can lead to inaccurate results, especially in tasks where context is crucial,
such as sentiment analysis or language translation.
5. Need for Preprocessing:
 VSM heavily relies on preprocessing steps such as tokenization, stop word
removal, and stemming.
 Poor preprocessing can lead to suboptimal results and affect the quality of
retrieval.
6. Difficulty with Synonyms and Polysemous Words:
 VSM may struggle with synonyms and polysemous words, as it treats each
term independently.
  Variations in word usage can lead to mismatches between query terms and
relevant documents.

Named Entity Recognition –

Named Entity Recognition (NER) is a task in Natural Language Processing (NLP) that
involves identifying and classifying named entities in text into predefined categories such as
person names, organization names, locations, dates, numerical expressions, and more.
Working -
The named entity recognition process can be broken down into five steps:
Tokenization. Before identifying entities, the text is split into tokens, which can be words,
phrases, or even sentences. For instance, "Steve Jobs co-founded Apple" would be split into
tokens like "Steve", "Jobs", "co-founded", "Apple".
Entity identification. Using various linguistic rules or statistical methods, potential named
entities are detected. This involves recognizing patterns, such as capitalization in names
("Steve Jobs") or specific formats (like dates).
Entity classification. Once entities are identified, they are categorized into predefined classes
such as "Person", "Organization", or "Location". This is often achieved using machine
learning models trained on labeled datasets. For our example, "Steve Jobs" would be
classified as a "Person" and "Apple" as an "Organization".
Contextual analysis. NER systems often consider the surrounding context to improve
accuracy. For instance, in the sentence "Apple released a new iPhone", the context helps the
system recognize "Apple" as an organization rather than a fruit.
Post-processing. After initial recognition and classification, post-processing might be applied
to refine results. This could involve resolving ambiguities, merging multi-token entities, or
using knowledge bases to enhance entity data.
Industry applications of NER
 Customer service. NER models are used in customer service to power chatbots and
organize data related to customer care. For example, ChatGPT responds to user
queries conversationally by identifying relevant entities to determine context. A
customer support system can route users to the appropriate departments by
categorizing their complaints and matching them to resolutions.
 Health care. Medical professionals use NER models to analyze large amounts of
documentation regarding diseases, drugs, and patients. Being able to quickly identify
and extract the most pertinent information from lengthy, unstructured text helps
reduce research time.
 Finance. In the financial field, NER can be used to monitor trends and inform risk
analyses. Aside from financial information such as loans and earnings reports, NER
models can analyze company names and other relevant mentions on social media to
monitor developments that may affect stock prices.
  Entertainment. Recommendation systems such as the ones you see on Netflix,
Spotify, and Amazon are often powered by NER models that analyze your search
history and content you’ve recently interacted with.

Evaluation metrics -
Evaluation metrics for Named Entity Recognition (NER) measure the performance of NER
systems by comparing their predicted named entities to the ground truth (annotated) entities.
Here are some common evaluation metrics used in NER:
1. Precision, Recall, and F1-score:
 Precision: Precision measures the accuracy of the positive predictions made by the
model. It is the ratio of correctly predicted positive entities to the total entities
predicted as positive.

 Recall: Measures the completeness of the predicted entities. It calculates the ratio of
correctly predicted entities to the total number of actual entities.

 F1-score: The harmonic mean of precision and recall, providing a balance between
precision and recall.

2. Accuracy:
  Accuracy: Measures the overall correctness of the predicted entities. It calculates the
ratio of correctly predicted entities to the total number of entities.

3. Entity-Level Metrics:
 Correct Entities (CE): The number of correctly predicted entities.
 Partial Entities (PE): The number of partially overlapping entities (e.g., predicting
"New York City" instead of "New York").
 Missed Entities (ME): The number of ground truth entities that were not predicted.
4. Token-Level Metrics:
 Token-Level Precision: Measures the proportion of correctly predicted tokens among
all tokens predicted as entities.
 Token-Level Recall: Measures the proportion of correctly predicted tokens among all
tokens that should have been predicted as entities.
 Token-Level F1-score: The harmonic mean of token-level precision and recall.
5. CoNLL Evaluation:
 The CoNLL evaluation measures precision, recall, and F1-score at the token level and
takes into account exact entity matching.
 It's commonly used for evaluating NER systems, especially in shared tasks and
competitions.
Example –
let's use a sentence where the prediction can lead to different values for precision, recall,
and F1-score.
Sentence: "Elon Musk is the CEO of Tesla and he lives in Palo Alto, California."
Ground Truth:
 "Elon Musk" - PERSON
 "Tesla" - ORGANIZATION
 "Palo Alto, California" - LOCATION
Predicted:
 "Elon Musk" - PERSON
 "Tesla" - ORGANIZATION
 "Palo Alto" - LOCATION
 Calculating Precision, Recall, and F1-score:
 True Positives (TP):
 "Elon Musk" - Correctly predicted as PERSON
 "Tesla" - Correctly predicted as ORGANIZATION
 False Positives (FP):
 None
 False Negatives (FN):
 "Palo Alto, California" - Missed from predictions

Entity Extraction:
Entity Extraction, also known as Named Entity Recognition (NER), involves identifying and
classifying named entities in text into predefined categories such as person names,
organization names, locations, dates, numerical expressions, and more.
Working of Entity Extraction:
1. Preprocessing:
 Tokenize the input text into words or subwords.
 Remove irrelevant information like punctuation.
2. Feature Extraction:
 Extract relevant features from the text, which may include:
 Word embeddings
 Part-of-speech (POS) tags
  Contextual information
3. Model Prediction:
 Use a pre-trained NER model or train a new model on labeled data.
 For each token in the text, predict its named entity category.
 Many models use a token-level tagging approach, where each token is tagged
with its entity category.
4. Post-processing:
 Refine the predictions to improve accuracy and consistency.
 Resolve conflicts and handle complex cases like nested entities.
Relation Extraction:
Relation Extraction involves identifying and extracting relationships between entities
mentioned in text. These relationships can represent various types of connections between
entities, such as ownership, affiliation, location, etc.
Working of Relation Extraction:
1. Entity Extraction:
 Extract named entities from the text using techniques like NER.
2. Dependency Parsing:
 Analyze the syntactic structure of the sentence to identify relationships
between entities.
 Use techniques like dependency parsing to identify the grammatical
relationships between words in the sentence.
3. Pattern Matching:
 Use predefined patterns or rules to identify specific relationships between
entities.
 For example, a pattern like "X is the CEO of Y" can be used to identify the
CEO relationship between two entities.
4. Supervised Learning:
 Train a supervised machine learning model to predict relationships between
entities based on labeled data.
 Features for the model may include entity types, syntactic features, and
contextual information.
5. Post-processing:
 Refine the predicted relationships to improve accuracy and coherence.
  Resolve conflicts and handle cases where multiple relationships exist between
the same pair of entities.
Example:
Let's consider the sentence: "Elon Musk, the CEO of SpaceX, was born on June 28, 1971, in
Pretoria, South Africa."
Entity Extraction:
 "Elon Musk" - PERSON
 "SpaceX" - ORGANIZATION
 "June 28, 1971" - DATE
 "Pretoria" - LOCATION
 "South Africa" - LOCATION
Relation Extraction:
 CEO of:
 Subject: "Elon Musk"
 Object: "SpaceX"
 Relation: "CEO of"
 Place of Birth:
 Subject: "Elon Musk"
 Object: "Pretoria, South Africa"
 Relation: "Place of Birth"
In summary, Entity Extraction involves identifying named entities in text, while Relation
Extraction involves identifying relationships between these entities.
Reference Resolution:
 Identifies all expressions in a text that refer to the same entity, including pronouns,
proper nouns, and even descriptions.expand_more
 This is a broader category encompassing various ways words can refer to entities.
 Example: In the sentence "Barack Obama, the 44th president of the United States,
delivered a speech. He spoke about the importance of education."
o Here, "Barack Obama" and "he" both refer to the same entity (Barack Obama).
o Reference resolution would identify both mentions.
Coreference Resolution:
  Focuses specifically on resolving pronouns to the entities they refer to within a
text.expand_more
 It's a subcategory of reference resolution that deals with pronouns like "he," "she,"
"it," "they," etc.expand_more
 Example: Consider the sentence "Alice went to the store. She bought some groceries."
o Coreference resolution would identify "Alice" and "she" as referring to the
same entity (Alice).
Key Differences:
 Scope: Reference resolution covers a wider range of expressions, including pronouns,
proper nouns, and descriptions. Coreference resolution is limited to pronouns.
 Focus: Reference resolution aims to find all mentions of the same entity, regardless of
the type of expression. Coreference resolution specifically targets pronouns and who
they refer to.
Here's an analogy: Imagine a party.
 Reference resolution: Identifies everyone at the party, including people with name
tags (proper nouns), those described by their clothes (descriptions), and people you
only know by sight (pronouns).
 Coreference Resolution: Focuses on figuring out who people are referring to when
they use pronouns like "he" or "she" at the party.
Where to use Coreference Resolution –
 Text understanding
 Document summarization
 Information extraction
 Sentiment analysis
 Machine translation

Cross-lingual information retrieval (CLIR)

Cross-lingual information retrieval (CLIR) is the task of retrieving relevant information
written in a different language than the language of the query. In other words, it allows users
to search for information in one language and retrieve documents or information in another
language.

How it works:
 Query Translation: The first step in CLIR is to translate the user's query from the source
language into the target language. This can be done using machine translation techniques.

Document Retrieval: Once the query is translated, the system searches for relevant
documents in the target language using traditional information retrieval methods. This could
involve searching through indexed documents or web pages.

Result Translation: After retrieving relevant documents, the system may translate them
back into the source language for presentation to the user.

Example:

Let's say a user who speaks English wants to find information about "climate change" in
Spanish documents:

Query Translation: The user's query, "climate change," is translated into Spanish as
"cambio climático."

Document Retrieval: The system searches through a collection of Spanish documents (e.g.,
articles, websites) for those containing the term "cambio climático" or related terms.

Result Translation: Once relevant documents are retrieved, the system may translate them
back into English for the user to read and understand.

Use in Natural Language Processing:

Cross-lingual information retrieval is essential in various NLP tasks, including:

Multilingual Search Engines: CLIR allows users to search for information on the web in
languages they may not understand, broadening access to information across linguistic
barriers.

Cross-Lingual Document Classification: CLIR can be used to classify documents written

in different languages into predefined categories, enabling tasks such as sentiment analysis or
topic modeling across languages.
 Machine Translation Evaluation: CLIR is used to evaluate the performance of machine
translation systems by assessing the relevance of translated documents to the original queries.

Cross-Lingual Text Mining: CLIR facilitates mining information from multilingual

sources, helping researchers and organizations gather insights from diverse linguistic
datasets.