elasticsearch

Shifa Khan github/shifakhan @findshifa

Why do we need search options?

We need it to make our lives easier Find stuff that is relevant to us Find it faster

How do we add seach features to our projects?

ptions available!

Standard "atabase #uery $ucene libraries Solr Sphin% &lasticsearch

elasticsearch
'''

(opics for today)

Full (e%t Searching *&S(ful &S &lasticsearch in *ails +dvanced features of &S

,ote! &S - &lasticSearch

What is elasticsearch?

Database server Implemented with RESTful HTTP/JSON Easil scalable !hence the name #ased $n %ucene

elasticsearch"

Features of elasticsearch

Sche.a/free *eal/ti.e &asy to e%tend with a plugin syste. for new functionality +uto.atic discovery of peers in a cluster Failover and replication 0o..unity support! 1ultiple clients available in various (ire ge./ ruby client available for +ctive1odel integration

languages for easy integration

(er.inology
*elational database "atabase (able *ow 0olu.n Sche.a
&lasticsearch 2nde% (ype "ocu.ent Field 1apping

2nside &S

" 0/(45&

2,"&3

"ata Structure of &lasticsearch

+n 2nde% can have .ultiple types' " 0/(45& &ach type can have .ultiple docu.ents'

2,"&3
" 0/(45&

&ach docu.ent contains data in JSON for.at

6''7 6''7
"ocu.ent

Searches can also be perfor.ed in .ultiple indices " 0/(45&

2,"&3
" 0/(45&

How does elasticsearch work?

Full te%t searching

2nverted inde%ing

+nalysis

F8$$ (&3( S&+*0H

stars shine sky stars stars sky cloudy sky

9:;

9<;

9=;

F8$$ (&3( S&+*0H

stars shine sky stars stars sky cloudy sky

W *" stars shine sky cloudy

$ 0+(2 , :>< : :><>= =

F8$$ (&3( S&+*0H

stars shine sky stars stars sky cloudy sky

W *" stars shine sky cloudy

$ 0+(2 , :>< : :><>= =

F8$$ (&3( S&+*0H

stars shine sky stars stars sky cloudy sky

W *" stars shine sky

$ 0+(2 , : < : : < =

5 S2(2 , : :>< < = = < :

cloudy

F8$$ (&3( S&+*0H

stars shine sky stars stars sky cloudy sky

W *" stars shine sky

$ 0+(2 , : < : : < =

5 S2(2 , : :>< < = = < : +re these positions in the docu.ent are consecutive ?

cloudy

F8$$ (&3( S&+*0H

stars shine sky stars stars sky cloudy sky

W *" stars shine sky

$ 0+(2 , : < : : < =

5 S2(2 , : :>1 < = 2 < : We find the words in consecutive positions in "ocu.ent <

cloudy

+,+$4S2S

+naly?ing is e%tracting @ter.sA fro. given te%t' 5rocessing natural language to .ake it co.puter searchable'

*uby is a dyna.ic> reflective> general/purpose 5 language that co.bines synta% inspired by 5erl with S.alltalk/like features' *uby was first designed and developed in the .id/<BB:s by 4ukihiro C1at?C 1atsu.oto in Dapan'

Stopwords!/re.oval of words of less

se.antic significance

ruby dyna.ic reflective general purpose oop language co.bine synta% inspire perl s.alltalk feature first design develop .id <BB: yukihiro .at? .atsu.oto japan

*uby is a dyna.ic> reflective> general/purpose 5 language that co.bines synta% inspired by 5erl with S.alltalk/like features' *uby was first designed and developed in the .id/<BB:s by 4ukihiro C1at?C 1atsu.oto in Dapan'

$owercase and punctuation .arks

ruby dyna.ic reflective general purpose oop language co.bine synta% inspire perl s.alltalk feature first design develop .id <BB: yukihiro .at? .atsu.oto japan

*uby is a dyna.ic> reflective> general/purpose 5 language that co.bines synta% inspired by 5erl with S.alltalk/like features' *uby was first designed and developed in the .id/<BB:s by 4ukihiro C1at?C 1atsu.oto in Dapan'

Ste..er!/"eriving root of words

ruby dyna.ic reflective general purpose oop language co.bine synta% inspire perl s.alltalk feature first design develop .id <BB: yukihiro .at? .atsu.oto japan

&S +naly?er

+naly?er! 0onsists of one tokeni?er and .ultiple token filters eg! Whitespace> Snowball>etc

(okeni?er! 2t tokeni?es all words' Splits sentences into individual Eter.sE ' ,gra. and &dge,gra. highly useful for autoco.plete feature' 5ath hierarchy tokeni?ers'

(oken filter! +ctions on tokeni?ed words> basic lowercase to phonetic filters and ste..ers Favailable in .any languagesG

5opular +naly?ers
&S is easy to use' *eady.ade analy?ers for general usage'

Snowball H e%cellent for natural language

Standard tokeni?er> with standard filter> lowercase filter> stop filter> and snowball filter So.e fancy analy?ers! 5attern analy?ers For all the *egular e%pressions guys out there) CtypeC! CpatternC> CpatternC!CIIsJC

How do we access &lasticsearch?

How do we access &lasticsearch?

2ts *&S(ful

How do we access &lasticsearch?

2ts *&S(ful

K&(

5 S(

58(

"&$&(&

58( /inde%/type/id

action?

58( /inde%/type/id

where?

58( /twitterLdevelop.ent/type/id

58( /twitterLdevelop.ent/type/id

what?

58( /twitterLdevelop.ent/tweet/id

58( /twitterLdevelop.ent/tweet/id

which?

58( /twitterLdevelop.ent/tweet/<

curl /358( Elocalhost!B=::/twitterLdevelop.ent/tweet/<E /d E6 CtweetC ! C &lasticsearch is cool) C> Cna.eC ! C1r "eveloperC 7E

curl /358( Elocalhost!B=::/twitterLdevelop.ent/tweet/<E /d E6 CtweetC ! C &lasticsearch is cool) C> Cna.eC ! C1r "eveloperC 7E 6 CLinde%C!CtwitterLdevelop.entC> CLtypeC!CtweetC> CLidC!C<C> CLversionC! <> CokC!true

Si.ilarly>
curl /3K&( Ehttp!//localhost!B=::/twitter/tweet/<E

Si.ilarly>
curl /3K&( Ehttp!//localhost!B=::/twitter/tweet/<E
6 CLinde%C ! CtwitterC> CLtypeC ! CtweetC> CLidC ! C<C> CLsourceC ! 6 CtweetC ! C &lasticsearch is cool) C> Cna.eC ! C1r "eveloperC 7

+lso
K&( /inde%/Lsearch K&( /inde%<>inde%=/Lsearch K&( /indM/Lsearch K&( /inde%/type/Lsearch K&( /inde%/type<>type=/Lsearch K&( /inde%/typeM/Lsearch K&( /Lall/typeM/Lsearch
Search .ultiple or all indices FdatabasesG Search all types Fcolu.nsG

Si.ilarly>
"&$&(&
"elete docu.ent

curl /3"&$&(& Ehttp!//localhost!B=::/twitter/tweet/<E curl /3"&$&(& Ehttp!//localhost!B=::/twitter/E

"elete inde%

curl /3"&$&(& Ehttp!//localhost!B=::/twitter/tweet/LNueryE/d E6 Cter.C ! 6 Cna.eC ! CdeveloperC 7 7E

&lasticsearch in *ails

&S in your *+2$S app

<'2nstall &S fro. website' Start service' ='2n your browser check Ehttp!//localhost!B=::E for confir.ation' O'+dd EtireE to Ke.file' P'2n your .odel file Fsay Echwink'rbE G add/ include (ire!!1odel!!Search include (ire!!1odel!!0allbacks Q'*un rake environ.ent tire!i.port 0$+SS-0hwink R'2n rails console type 0hwink'searchF@worldAG or 0hwink'tire'searchF@worldAG S'*esults)

*eNuest for 1odel'search

curl /3K&( Ehttp!//localhost!B=::/chwinkLdevelop.ent/chwink/Lsearch? /d E6 CNueryC! 6 CNueryLstringC! 6 CNueryC! CworldC 7 7 7E

+utoco.plete feature i.ple.entation

and

So.e e%a.ples of ready.ade and custo. analy?ers

#uerying

1ore types of Nueries and use cases! Faceting ! +llowing .ultiple filters 5agination ! $i.iting per page results Sort ! Sorting results by relevance and scoring Fusing &lasticsearchEs scoring algorith.G

+dvanced features of &lasticsearch

8S5 of &lasticsearch

T2K) "ata +nalysis in *eal (i.e

+uto.atic discovery of peers in a cluster Failover and replication

+uto.atic "iscovery 1odule

,ode <

,ode =

,ode O

+<

T<

T=

T<

+=

+O

0<

+=

+O

+<

+=

T<

+O

T=

+=

T=

+<

T=

Shard

*eplica

When a new node is added'''

+uto.atic "iscovery 1odule

,ode <

,ode =

,ode O

,ode P

+<

T=

0<

T=

+<

T=

+O

+=

+= T< += +O T< +O T< T=

T<

+<

High +vailability
For each inde% you can specify!

,u.ber of shards &ach inde% has fi%ed nu.ber of shards Shards i.prove inde%ing perfor.ance ,u.ber of replicas &ach shard can have :/.any replicas> can be changed dyna.ically *eplicas i.prove search perfor.ance

(ips
1ake sure you .ake separate indices for test and develop.ent environ.ent &g! +dd inde%Lna.e C0hwinkLU6*ails'env7C to your .odel file

"uring tests whenever you save an object .ake sure you add ! 0hwink'tire'inde%'refresh after each test case'

1ake sure you delete and recreate the inde% after tests' &g! 4ou can add this to your *spec configuration file config'afterF!eachG do 0hwink'tire'0hwinkLtest'delete 0hwink'tire'createLelasticsearchLinde% end "ebug using log files ! +dd to (ire config file! logger CtireLU6*ails'env7'logC

*oo. for e%ploration'''

Tig "esk plugin)

5lug/in for analysis of your search data)

*oo. for e%ploration'''

Tig "esk plugin)

5lug/in for analysis of your search data)

*oo. for e%ploration'''

Tig "esk plugin)

5lug/in for analysis of your search data)

,ow you can go ahead and start e%ploring &lasticsearch for yourself))

(hank you)
#uestions are welco.e)