Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd Logo
Upload

Welcome to Scribd!

  • 343 views

    Introduction To Logstash

    Uploaded by

    Pablo Ernesto Vigneaux Wilton
    Introduction to Logstash use. Logstash is a tool for extracting data and import to Elasticsearch index.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Introduction To Logstash

    Uploaded by

    Pablo Ernesto Vigneaux Wilton
    343 views4 pages
    Introduction to Logstash use. Logstash is a tool for extracting data and import to Elasticsearch index.

    Original Title

    Introduction to Logstash

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Introduction to Logstash use. Logstash is a tool for extracting data and import to Elasticsearch index.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    343 views4 pages

    Introduction To Logstash

    Uploaded by

    Pablo Ernesto Vigneaux Wilton
    Introduction to Logstash use. Logstash is a tool for extracting data and import to Elasticsearch index.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 4

    01/06/2016

    IntroductiontoLogstash

    IntroductiontoLogstash
    TableofContents
    WhatisLogstash?
    Downloading/InstallingLogstash
    RunningLogstash
    Agent
    Web
    Logstasharchitecture
    Simplestconfiguration
    Changingthewaydataisrepresented
    Readinginputfromfilesondisk
    OutputingtoanembeddedElasticsearch
    OutputingtoaseparateElasticsearch
    Addingafilterintothemix
    The"grok"filter
    Combiningeverythingtogether
    TheUIforLogstash

    WhatisLogstash?
    Agenericconceptforreceivingdata,transformingit,andoutputtingit.

    Downloading/InstallingLogstash
    Downloadingisaseasyasgettingitfromhttp://logstash.net,butI'llbeusingversion1.4(whichiscurrentlybetaas
    ofthiswriting)
    Toinstall,untarthepackagesomewhere,orusethe.deb/.rpmrepositoriesforyourrespectiveoperatingsystem.

    RunningLogstash
    Thereare2mainmodesofrunningLogstash.Notethatbothcanberunatonce.

    Agent
    Runningasanagentcollectsinformation,forwardingittothebackend(inourcase,Elasticsearch)

    Web
    RunsthewebUI(knownasKibana)bundledinLogstash

    Logstasharchitecture
    Logstashisacollectionof:
    Inputs
    Codecs
    Filters
    Outputs

    Simplestconfiguration
    Startingwiththesimplestinput,standardin:
    input{
    stdin{}
    }

    Andthesimplestoutput,standardout(nofiltersfornow):
    output{
    stdout{}
    }

    Torunthis,youcando:
    bin/logstashagentflogstashsimple.conf

    Changingthewaydataisrepresented
    Let'schangethecodec(datarepresentation)toprintmoreinformation:
    input{
    stdin{}
    }

    http://writequit.org/articles/logstashintro.html

    1/4

    01/06/2016

    IntroductiontoLogstash
    output{
    stdout{
    codec=>rubydebug
    }
    }

    Readinginputfromfilesondisk
    Thistime,insteadofreadinginfromstdin,readfromafile:
    input{
    file{
    type=>"apache"
    path=>"/Users/hinmanm/introtologstash/example.log"
    }
    }
    output{
    stdout{
    codec=>rubydebug
    }
    }

    OutputingtoanembeddedElasticsearch
    Logstashcanoutputtomanymoreplacesthanjuststdout,itcomeswithelasticsearchasanoutputoptionthatcanrun
    embedded:
    input{
    file{
    type=>"apache"
    path=>"/Users/hinmanm/introtologstash/example.log"
    }
    }
    output{
    stdout{
    codec=>rubydebug
    }
    elasticsearch{
    embedded=>true
    }
    }

    Addafewlogstothefile:
    echo"thisisalogmessageaboutfoo">>example.log
    echo"thisisalogmessageaboutbar">>example.log
    echo"thisisalogmessageaboutbaz">>example.log

    Logstashcreatesanindex,noticethatitcreateditforthedaythiswasrun.Logstashwillcreatedailyindicesby
    default:
    curl'localhost:9200/_cat/health?v'
    echo""
    curl'localhost:9200/_cat/shards?v'

    epochtimestampclusterstatusnode.totalnode.datashardsprireloinitunassign
    139504637202:52:52elasticsearchyellow2155005

    indexshardprirepstatedocsstoreipnode
    logstash2014.03.172pSTARTED099b172.22.255.231MultipleMan
    logstash2014.03.172rUNASSIGNED
    logstash2014.03.170pSTARTED099b172.22.255.231MultipleMan
    logstash2014.03.170rUNASSIGNED
    logstash2014.03.173pSTARTED24.2kb172.22.255.231MultipleMan
    logstash2014.03.173rUNASSIGNED
    logstash2014.03.171pSTARTED13.9kb172.22.255.231MultipleMan
    logstash2014.03.171rUNASSIGNED
    logstash2014.03.174pSTARTED099b172.22.255.231MultipleMan
    logstash2014.03.174rUNASSIGNED

    Andyoucansearchforlogmessages(here'sanexamplequery)
    {
    "query":{
    "simple_query_string":{
    "query":"foo|bar",
    "fields":["message"]
    }
    },
    "size":3
    }

    Andgetbacktheresults:
    HTTP/1.1200OK
    ContentType:application/json;charset=UTF8
    ContentLength:1248

    http://writequit.org/articles/logstashintro.html

    2/4

    01/06/2016

    IntroductiontoLogstash

    {
    "took":76,
    "timed_out":false,
    "_shards":{
    "total":20,
    "successful":20,
    "failed":0
    },
    "hits":{
    "total":4,
    "max_score":0.35355338,
    "hits":[{
    "_index":"logstash2014.03.17",
    "_type":"apache",
    "_id":"q8EqCk2RjWwB70rxz7bw",
    "_score":0.35355338,"_source":{"message":"thisisalogmessageaboutfoo","@version":"1","@timestamp":"20140317T08:52:
    },{
    "_index":"logstash2014.03.17",
    "_type":"apache",
    "_id":"e0KXf2eCQjmm302UB6n60g",
    "_score":0.35355338,"_source":{"message":"thisisalogmessageaboutbar","@version":"1","@timestamp":"20140317T08:52:
    },{
    "_index":"logstash2014.03.13",
    "_type":"apache",
    "_id":"DXwFHMvTTsauxjr9lJ5Xcg",
    "_score":0.25427115,"_source":{"message":"thisisalogmessageaboutbar","@version":"1","@timestamp":"20140313T08:45:
    }]
    }
    }

    OutputingtoaseparateElasticsearch
    Embeddedisgreatfordevelopment,butoutputtingtoadifferentElasticsearchserverisbetterforproduction:
    input{
    file{
    type=>"apache"
    path=>"/Users/hinmanm/introtologstash/example.log"
    }
    }
    output{
    stdout{
    codec=>rubydebug
    }
    elasticsearch{
    host=>"localhost"
    port=>9300
    node_name=>"logstashagent007"
    workers=>2
    }
    }

    Addingafilterintothemix
    Filtersallowyoutomodifyoutput
    Themostusefulisgrok,butlet'sstartwithmutate.Sothestandardinput/outputconfigurationfirst:
    input{
    stdin{}
    file{
    type=>"apache"
    path=>"/Users/hinmanm/introtologstash/example.log"
    }
    }
    output{
    stdout{
    codec=>rubydebug
    }
    }

    Addingthemutatefiltertoaddafieldaswellaslowercasethe"message"field
    filter{
    mutate{
    add_field=>["myhost","Hellofrom%{host}!"]
    lowercase=>["message"]
    }
    }

    The"grok"filter
    Logstash'sarguablymostusefulfilter.~120differentpatternsthatcanbecomibned.
    https://github.com/elasticsearch/logstash/tree/1.4.x/patterns
    https://grokdebug.herokuapp.com/
    Again,standardboilerplate:
    input{

    http://writequit.org/articles/logstashintro.html

    3/4

    01/06/2016

    IntroductiontoLogstash
    stdin{}
    file{
    type=>"apache"
    path=>"/Users/hinmanm/introtologstash/example.log"
    }
    }
    output{
    stdout{
    codec=>rubydebug
    }
    }

    Andthenagrokfiltermeanttomatchthetext"name:John"intheinputs:
    filter{
    grok{
    match=>["message","name:%{WORD:custom_name}"]
    }
    mutate{
    lowercase=>["custom_name"]
    }
    }

    Combiningeverythingtogether
    Readfromafile(thistimeanElasticsearchlogfile),usetheeslogtypewhenputtingthelogmessageinto
    Elasticsearch.OutputwillbewrittentoaseparateElasticsearchclusteratlocalhostonport9300:
    input{
    file{
    type=>"eslog"
    path=>"/Users/hinmanm/introtologstash/es/logs/elasticsearch.log"
    }
    }
    output{
    stdout{
    codec=>rubydebug
    }
    elasticsearch{
    host=>"localhost"
    port=>9300
    }
    }

    ThisexamplefilterwillmatchElasticsearch'slogformat,extracttheusefulpiecesofthelog(time,level,package,
    node_name,andlogmessage).
    Themutatefilterwillthen:
    lowercasetheloglevel(INFO=>info)
    stripthewhitespaceforthepackage("indices.recovery"=>"indices.recovery")
    Additionally,themultilinefilterwillmatchlinesthatlooklikeaJavaException,andcollapsethemintoasingle
    messagefromthepreviousline.
    filter{
    grok{
    match=>["message",
    "^\[%{TIMESTAMP_ISO8601:time}\]\[%{LOGLEVEL:level}.*\]\[%{DATA:package}\]\[%{DATA:node_name}\]%{DATA:logmsg}$"
    }

    mutate{
    lowercase=>["level"]
    strip=>["package"]
    }

    multiline{
    pattern=>"(org\.elasticsearch\.Exception.+|(at.+))"
    what=>"previous"
    }
    }

    TheUIforLogstash
    LogstashbundlesKibana,whichcanbeusedforvisualizingdata,andisaseasyasrunning:
    bin/logstashweb

    orbothatoncewith:
    bin/logstashagentflogstash.confweb

    Author:LeeHinman
    Created:20140319Wed01:07
    Emacs24.3.50.1(Orgmode8.2.5h)
    Validate

    http://writequit.org/articles/logstashintro.html

    4/4

    You might also like