Anwendungsfälle für Elasticsearch JAX 2015

Anwendungsfälle für
Florian Hopf - @fhopf

Installation
# download archive
wget https://download.elasticsearch.org/
elasticsearch/elasticsearch/elasticsearch-1.4.4.zip
# zip is for windows and linux
unzip elasticsearch-1.4.4.zip
# on windows: elasticsearch.bat
elasticsearch-1.4.4/bin/elasticsearch

Zugriff
curl -XGET http://localhost:9200
{
"status" : 200,
"name" : "Anomalito",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.4.4",
"build_hash" :
"c88f77ffc81301dfa9dfd81ca2232f09588bd512",
"build_timestamp" : "2015-02-19T13:05:36Z",
"build_snapshot" : false,
"lucene_version" : "4.10.3"
},
"tagline" : "You Know, for Search"
}

Document
{
"title" : "Anwendungsfälle für Elasticsearch",
"speaker" : "Florian Hopf",
"date" : "2015-04-22T10:30:00.000Z",
"tags" : ["Java", "Lucene"],
"conference" : {
"name" : "JAX",
"city" : "Mainz"
}
}

Speichern
curl -XPOST http://localhost:9200/conferences/talk/
--data-binary @talk-example.json
{
"_index":"conferences",
"_type":"talk",
"_id":"GqjY7l8sTxa3jLaFx67_aw",
"_version":1,
"created":true
}

Speichern
{
"_type":"talk",
"_version":1,
"created":true
}
Index

Speichern
{
"_type":"talk",
"_version":1,
"created":true
}
Type

Lesen
curl -XGET http://localhost:9200/conferences/talk/
GqjY7l8sTxa3jLaFx67_aw?pretty=true
{
"_index" : "conferences",
[...]
"_source":{
"date" : "2015-04-22T10:30:00.000Z",
"conference" : {
"name" : "JAX",
"city" : "Mainz"
}
}
}

Sharding
●
Aufteilen eines Index in mehrere Teile
●
Default: 5 Shards pro Elasticsearch-Index
●
Mehrere ES-Instanzen bilden einen Cluster
●
Automatische Verteilung auf die Knoten

Recap
●
Einfache Speicherung von JSON-Dokumenten
●
Index und Type
●
Sharding für große Datenmengen
●
Verteilung ist First-Class-Citizen

Nutzer
●
Atlassian HipChat
●
http://highscalability.com/blog/2014/1/6/how-hipchat-
●
OpenCrowd
●
https://www.elastic.co/use-cases/open-crowd

Anwendungsfälle für Elasticsearch JAX 2015

Suche per Parameter
curl -XGET "http://localhost:9200/conferences/talk/_search
?q=elasticsearch&pretty=true"
{"took" : 73,
[…]
"hits" : {
[…]
"hits" : [ {
[…]
"_score" : 0.076713204,
"_source":{
[…]
} } ]
}
}

Query-DSL
curl -XGET "http://localhost:9200/conference/talk/_search" -d'
{
"query": {
"filtered": {
"query": {
"match": {
"title": "elasticsearch"
}
},
"filter": {
"term": {
"tags": "java"
}
}
}
}
}'

Query-DSL
client.prepareSearch("conferences")
.setQuery(filteredQuery(
matchQuery("title", "elasticsearch"),
termFilter("tag", "java")))
.execute()
.actionGet();

Sprache
curl -XGET "http://localhost:9200/conferences/talk/_search
?q=title:anwendungsfall&pretty=true"
{
[...]
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

Mapping
curl -XDELETE "http://localhost:9200/conferences/"
curl -XPUT "http://localhost:9200/conferences/“
curl -XPUT "http://localhost:9200/conferences/talk/_mapping" -d'
{
"properties": {
"title": {
"type": "string",
"analyzer": "german"
}
}
}'

Sprache
curl -XGET
"http://localhost:9200/conferences/talk/_search
?q=title:anwendungsfall&pretty=true"
{
[...]
},
"hits" : {
"total" : 1,
[...]
}
}

Was noch?
●
Faceting/Aggregations
●
Suggestions
●
Highlighting
●
Sortierung
●
Pagination
●
...

Recap
●
Ausdrucksstarke Suchen über Query DSL
●
Analyzing und Mapping
●
Lucene bietet viele Search-Features

Users
●
GitHub
●
http://exploringelasticsearch.com/github_interview.html
●
http://www.elasticsearch.org/case-study/github/
●
StackOverflow
●
http://meta.stackexchange.com/questions/160100/a-new-
search-engine-for-stack-exchange
●
http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-
stack-overflow/
●
XING
●
http://www.elasticsearch.org/case-study/xing/

Ergebnislisting
curl -XPOST "http://localhost:9200/conferences/_search
" -d'
{
"filter": {
"term": {
"tags": "java"
}
}
}'

Nutzer
●
Microsoft MSN
●
https://www.elastic.co/elasticon/2015/sf/powerin
g-real-time-search-at-microsoft

Anwendungen
●
Zeige nächste Filiale
●
Filialsuche
●
Sortierung Kleinanzeigen
●
Filterung auf Nähe
●
Social Media-Analysen

Document
{
"date" : "2014-07-17T15:35:00.000Z",
"conference" : {
"name" : "JAX",
"city" : "Mainz",
"coordinates": {
"lon": "8.275626",
"lat": "50.002090"
}
}
}

Mapping
curl -XPUT "http://localhost:9200/conferences/talk/_mapping" -d'
{
"properties": {
[…],
"conference": {
"type": "object",
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}'

Sortierung
curl -XPOST "http://localhost:9200/conferences/_search " -d'
{
"sort" : [
{
"_geo_distance" : {
"conference.coordinates" : {
"lon": 8.403697,
"lat": 49.006616
},
"order" : "asc",
"unit" : "km"
}
}
]
}'

Filterung
curl -XPOST "http://localhost:9200/conferences/_search" -d'
{
"filter": {
"geo_distance": {
"conference.coordinates": {
"lon": 8.403697,
"lat": 49.006616
},
"distance": "200km",
"distance_type": "arc"
}
}
}'

Recap
●
Elasticsearch kann mehr als Volltext
●
Ausgefeilte Geo-Algorithmen
●
Sortierung nach Distanz
●
Filterung nach Distanz oder Bereich
●
Berechnung von Distanz

User
●
FourSquare
●
http://engineering.foursquare.com/2012/08/09/fo
ursquare-now-uses-elastic-search-and-on-a-
related-note-slashem-also-works-with-elastic-
search/
●
Gild
●
http://www.elasticsearch.org/case-study/gild/

Logfile-Analyse
●
Zentralisierung Logs aus Anwendungen
●
Zentralisierung Logs über Maschinen
●
Auch ohne Zugriff
●
Leichte Durchsuchbarkeit
●
Real-Time-Analysis / Visualisierung
●
Daten für alle!

Logfile-Analyse
●
Einlesen
●
Logstash
●
Speicherung
●
Elasticsearch
●
Auswertung
●
Kibana

Logstash-Config
input {
file {
path => "/var/log/apache2/access.log"
}
}
filter {
grok {
match => { message => "%{COMBINEDAPACHELOG}" }
}
}
output {
elasticsearch_http {
host => "localhost"
}
}

Recap
●
Einlesen, Anreichern, Speichern von Logevents
●
Zahlreiche Inputs in Logstash
●
Konsolidierung
●
Zentralisierung
●
Auswertung

User
●
CERN
●
https://medium.com/@ghoranyi/needle-in-a-
haystack-873c97a99983
●
Bloomberg
●
http://www.elasticsearch.org/videos/using-
elasticsearch-logstash-kibana-techologies-
centralized-viewing-logs-bloomberg/

Analytics
●
Auswertung auch großer Datenmengen
●
Social Media
●
Data Warehouse
●
Aggregationen auf Feldern
●
Datenkonsolidierung aus unterschiedlichen
Quellen
●
Visualisierung

Aggregations
curl -XGET "http://localhost:9200/devoxx/tweet/_search" -d'
{
"aggs" : {
"hashtags" : {
"terms" : {
"field" : "hashtag.text"
}
}
}
}'

Aggregations
"aggregations": {
"hashtags": {
"buckets": [
{
"key": "dartlang",
"doc_count": 229
},
{
"key": "java",
"doc_count": 216
},
[...]

Aggregations
"aggs" : {
"hashtags" : {
"terms" : {
"field" : "hashtag.text"
},
"aggs" : {
"hashtagusers" : {
"terms" : {
"field" : "user.screen_name"
}
}
}
}
}

Aggregations
"key": "scala",
"doc_count": 130,
"hashtagusers": {
"buckets": [
{
"key": "jaceklaskowski",
"doc_count": 74
},
{
"key": "ManningBooks",
"doc_count": 3
},
[...]

Recap
●
Auswertung großer Datenmengen
●
Visualisierung
●
Zahlreiche Aggregationen
●
Bucket-Aggregations
●
Metric-Aggregations

User
●
Cogenta
●
https://www.elastic.co/use-cases/cogenta
●
The Guardian
●
http://www.elasticsearch.org/blog/using-
elasticsearch-and-logstash-to-serve-billions-of-
searchable-events-for-customers/
●
http://www.infoq.com/presentations/elasticsearch
-guardian

Vielen
Dank!
@fhopf
mail@florian-hopf.de
http://blog.florian-hopf.de

Images
●
http://www.morguefile.com/archive/display/685952
●
●
●
●
●
●

Anwendungsfälle für Elasticsearch JAX 2015

More Related Content

Anwendungsfälle für Elasticsearch JAX 2015