Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd Logo
Upload

Welcome to Scribd!

  • 147 views

    Lucene

    Uploaded by

    Arafath Cherukuri
    This document provides an overview and agenda for Project 2 of the CSE 535: Information Retrieval course. It discusses using Apache Lucene to implement different retrieval models, evaluate query processing techniques, and compare results. Key aspects covered include Lucene's indexing and querying capabilities, various IR concepts like retrieval models and evaluation metrics, and applications of Lucene for the project requirements.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd

    Lucene

    Uploaded by

    Arafath Cherukuri
    147 views15 pages
    This document provides an overview and agenda for Project 2 of the CSE 535: Information Retrieval course. It discusses using Apache Lucene to implement different retrieval models, evaluate query processing techniques, and compare results. Key aspects covered include Lucene's indexing and querying capabilities, various IR concepts like retrieval models and evaluation metrics, and applications of Lucene for the project requirements.

    Original Description:

    lucene

    Copyright

    © Attribution Non-Commercial (BY-NC)

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document provides an overview and agenda for Project 2 of the CSE 535: Information Retrieval course. It discusses using Apache Lucene to implement different retrieval models, evaluate query processing techniques, and compare results. Key aspects covered include Lucene's indexing and querying capabilities, various IR concepts like retrieval models and evaluation metrics, and applications of Lucene for the project requirements.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    147 views15 pages

    Lucene

    Uploaded by

    Arafath Cherukuri
    This document provides an overview and agenda for Project 2 of the CSE 535: Information Retrieval course. It discusses using Apache Lucene to implement different retrieval models, evaluate query processing techniques, and compare results. Key aspects covered include Lucene's indexing and querying capabilities, various IR concepts like retrieval models and evaluation metrics, and applications of Lucene for the project requirements.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 15

    CSE 535: INFORMATION RETRIEVAL

    Apache Lucene and Project 2

    Agenda
    ! Walk through for Project 2 ! Lucene ! Introduction and package overview ! Hands on: Simple indexing and querying ! Cook book: Some recipes ! Applications to Project 2 ! Recap of concepts

    PROJECT TWO
    IR models, query processing and evaluation

    The two minute overview


    ! Whats given ! A big collection with queries and relevance judgments ! A program that evaluates results tells you how good you did ! Your task ! Section one: Compare different methods
    ! Do models behave differently for different queries query length, OOV

    terms, etc. ! Can you tweak parameters to perform better? ! Can you simulate Boolean queries?

    ! Section two: Write a query processing module


    ! Read up Lucene query syntax ! Easy: Give different weights to different zones. Trial & error may work as first

    stab ! Harder: Syntactical parsing, NLP processing


    ! Section three: Discuss QP results

    ! Did some models behave better to QP than others? ! Any interesting patterns: Technique wise? Query wise?

    THE THEORY
    What concepts would you need?

    Relevance models
    ! VSM or tf-idf ! Boolean ! BM25 ! LM ! DFR

    Evaluation metrics
    ! Precision, Recall & F-measure single valued ! tp, fp, tn, fn ! Based on rank ! Average precision ! Mean Average Precision: MAP, GM MAP ! Precision @ k ! R-precision ! Reciprocal rank ! Bpref Preference based ! Incremental value? ! DCG ! nDCG

    LUCENE
    A 100% Java Text IR engine

    Introduction
    ! Originally written in 1999 by Doug Cutting ! Part of the Apache Software Foundation family ! Full text indexing and searching ! Flexible, extendable ! Spawned several related projects: ! Nutch: Web crawling + HTML parsing ! Solr, ElasticSearch: Enterprise search servers ! Used by: ! AOL, Apple, CiteSeerX, Eclipse, IBM, JIRA, LinkedIn, Twitter, etc.

    Package overview
    Package analysis Usage Converts text from a Reader into TokenStream. Analyzer combines Tokenizer and TokenFilters to create TokenStream Primarily composed of IndexWriter and IndexReader Data structures to represent queries, an IndexSearcher to search over docs and QueryParsers to convert strings to queries Abstractions for storing persistent data Just a bunch of handy classes

    document Simple Document class that is simply a collection of Fields index search store util

    Getting started
    ! Pre-shipped files: IndexFiles and SearchFiles ! IndexFiles: Used to create index ! java -cp lucene-core.jar:lucene-demo.jar:lucene-analyzerscommon.jar org.apache.lucene.demo.IndexFiles -index <directory> -docs <directory> ! SearchFiles: Use to query index ! java -cp lucene-core.jar:lucene-demo.jar:lucenequeryparser.jar:lucene-analyzers-common.jar org.apache.lucene.demo.SearchFiles

    IndexFiles
    ! Setup Directory, Analyzer, IndexWriter ! Directory dir = FSDirectory.open(new File(indexPath)); // where are you reading files from! ! Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_40); //standard out of box: different analyzers can do different thing! ! IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_40, analyzer);! ! IndexWriter writer = new IndexWriter(dir, iwc); ! ! Construct Document, add Fields and add to writer ! Document doc = new Document();! ! Field pathField = new StringField("path", file.getPath(), Field.Store.YES); //properties for each field ! ! doc.add(pathField);! ! doc.add(new TextField("contents", new BufferedReader(new InputStreamReader(fis, "UTF-8"))));! ! writer.addDocument(doc); //add document to writer!

    SearchFiles
    ! Setup IndexReader and QueryParser ! IndexReader reader = DirectoryReader.open(FSDirectory.open(new File(index))); //Read the physical index! ! IndexSearcher searcher = new IndexSearcher(reader);! ! Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_40);! ! QueryParser parser = new QueryParser(Version.LUCENE_40, field, analyzer); // setup the parser: needs a searcher and analyzer! ! Parse input text into Query ! Query query = parser.parse(line); ! ! Run the search and get results ! TopDocs results = searcher.search(query, 100);! ! ScoreDoc[] hits = results.scoreDocs;!

    Some recipes
    ! Indexing: ! Use different analyzers ! Play around with document and fields ! Searching ! Different query parsers ! Powerful query syntax:
    ! Boolean queries, phrase queries, proximity queries, field boosting etc.

    ! Highlighter ! Highlight passages that correspond to matches

    ! Similarity classes: ! Default (TF-IDF) plus other variants: BM25, LM, DFR ! Defines how relevance ranking is done. ! New in Lucene 4: Per field similarity

    Applications to project 2
    ! The wrapper code makes the corresponding calls to

    different similarity classes. ! Customizing scoring


    ! Subclass the corresponding similarity class ! Overwrite methods as needed ! http://www.lucenetutorial.com/advanced-topics/scoring.html

    ! Query parser: ! Simplest is boost fields: Rank one higher than others ! Machine learned weights? Can treat as a classic regression problem ! Can write more complicated queries:
    ! POS tagging - some tags more important than others ! SVO analysis: Whats the subject? Interchangeable verbs etc.

    You might also like