Data Onboarding

Where Do I begin?

Luke Netto | Senior Professional Services Consultant @ Splunk

September 26, 2017 | Washington, DC

 Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
 Who Are You?

▶ You have Splunk installed, either in your datacenter or your laptop

▶ You have data you want to onboard into Splunk
▶ Your data comes from syslog, wineventlog, and/or flat files such as .csv | .log |
.json | .txt
 Who Am I?

▶ 3+ years of Splunk experience

▶ 7+ years of systems engineering
▶ 5+ years of data analytics
▶ systems engineering + data analytics = Splunk
 Agenda

▶ Importance
▶ Splunk Terms/Components
▶ Pre-onboarding/Data Discovery
▶ Splunkbase
▶ Creating your own sourcetype
▶ Onboarding – inputs.conf
▶ Optimizing for performance
▶ Normalizing
 Why Is This Important?

▶ Your organization wants to become data-driven

• Data collection
• Data access
▶ Decisions without quality data is simply guessing
Just Remember
Data preparation is 80%
What Your Executives Want
Splunk Data
Collection
Architecture
Basic Architecture Refresh
How Splunk works at a high level
 What Can Splunk Ingest?
Agent-Less and Forwarder Approach for Flexibility and Optimization
Aggregated/API Data Sources
Heavy Forwarder Local File Monitoring
Universal Forwarder
shell
API

perf syslog
*nix Windows Mainframes TCP/UDP

DevOps, IoT,
Event Logs, Active Directory, OS Stats Wire Data Containers
Unix, Linux and Windows hosts HTTP Event Collector
Splunk Stream
Universal Forwarder (Agentless)
Universal Forwarder or
HTTP Event Collector

syslog hosts
and network devices
Default Fields
Six Things to Get Right at Index Time

Host

Event
Boundary / Source
LineBreaking

Date
Timestamp Sourcetype

Index
 Host

▶ A default field that contains the hostname or IP address of the network device
that generated the event
▶ Use the host field in searches to narrow the search results to events that
originate from a specific device
▶ Allows you to located the originating device
 Source

▶ A default field that identifies the source of an event, that is, where the event
originated
▶ For data monitored from files and directories, the source consists of the full
pathname of the file or directory
• /var/log/messages
• /var/log/messages.1
• /var/log/secure
▶ For network-based sources, the source field consists of the protocol and port
• UDP:514
• TCP:1514
 Sourcetype

▶ A default field that identifies the data structure of an event

▶ The format of the data input from which it originates
• access_combined
• cisco:asa
▶ Determines how Splunk extracts & calculates fields during search time
▶ Use the sourcetype field in searches to find all data of a certain type (as opposed
to all data from a certain source)
▶ Important – syslog, csv, json, xml are not sourcetypes!
 Source vs Sourcetype

▶ Events with the same sourcetype can come from different sources
• /var/log/messages
• /var/log/messages.1
• udp:514
▶ sourcetype=linux_messages_syslog may retrieve events from both of those
sources
 What Happens With Bad Sourcetypes
Same Regex, Same Sourcetype
Cisco Squid
Bluecoat
 Index

▶ The repository for data in Splunk Enterprise

▶ Indexes reside in flat files
▶ Similar to a folder
▶ Used for data access, retention, or organization
 Timestamp

▶ Splunk uses timestamps to:

• correlate events by time
• create the timeline histogram in Splunk Web
• set time ranges for searches
▶ Usually automatic
 Line Break

▶ Allows Splunk to break the incoming stream of bytes into separate events
▶ Supports single-line and multi-line
▶ Splunk can usually do this automatically
Data Discovery
 Find the Data

▶ What is producing the data?

• Appliance
• Application
▶ Where is the data?
• Flat file
• Network/Syslog feed
• REST API
• Database
• Wineventlog
▶ Can you get a sample?
Apps & Add-ons (TA’s)

▶ Your first choice when onboarding

new data
▶ Includes relevant config files
(props/transforms) and ancillary
scripts & binaries
Where do you get Apps? Splunkbase!
What If There Is No App?
Let’s Do It
 inputs.conf
Tells Splunk to monitor a file, directory, listen on a port, etc.

[monitor:///var/log/messages]
[stanza]
host = [tcp://:1514]
index =
sourcetype = [udp://:1514]
source = [WinEventLog://Security]
 Example: With An App

▶ You have an appliance owned by the network team.

▶ The appliance is a Cisco ASA.
▶ Now what?
 Is There An App?
https://splunkbase.splunk.com/app/1620/

Install the TA, typically on your Indexers and Search Heads.

 Let’s Make It Work
Not the best way…
▶ Make an inputs.conf.
[udp://:1515]
index = thenetworkindex
sourcetype = cisco:asa

▶ Where are the other fields?

• They are automatic!
 Why Is It Not the Best Way…

▶ Disruption of Data Collection

▶ Splunk Metadata
▶ Syslog-ng/Rsyslog Flexibility
 The Better Way
Use syslog-ng or rsyslog with a UF.
▶ Use syslog-ng or rsyslog, write the syslog to a file, and allow Splunk to monitor it.
▶ /data/syslog/<sourcetype>/<host>/<host>.log
• /data/syslog/cisco_asa/<host>/<host>.log
• /data/syslog/pan_log/<host>/<host>.log

[monitor:///data/syslog/cisco_asa]
index = thenetworkindex
sourcetype = cisco:asa
host_segment = 4
 A Special Note On Syslog

▶ Syslog is a protocol – not a sourcetype

▶ Syslog typically carries multiple sourcetypes
▶ Best to pre-filter syslog traffic using syslog-ng or rsyslog
▶ Do not send syslog data directly to Splunk over a network port (514)
▶ Use a UF (next slide) or HEC to transport data to Splunk
▶ Remember to rotate your logs
A Recommended Syslog Architectures
 What About CSV, JSON, XML, etc.

▶ Same rule as syslog – these are data formats and carry multiple sourcetypes
Wait…What If There Is No App?
 Example: With No App

▶ You have an application from Super Awesome Apps called Incredible.

▶ The app has several logs it outputs:
• /opt/saa/incredible/useractivity.log
• /opt/saa/incredible/dbactivity.log
• /opt/saa/incredible/webui.log
 We Have This!

[monitor:///opt/saa/incredible/useractivity.log]
index = theindex
sourcetype = ????

▶ UF already knows the hostname, usually.

▶ What is the sourcetype?
 Let’s Talk

▶ Naming format: vendor:product:technology:format

▶ /opt/saa/incredible/useractivity.log
• sourcetype = saa:incredible:useractivity
▶ /opt/saa/incredible/dbactivity.log
• sourcetype = saa:incredible:dbactivity
▶ /opt/saa/incredible/webui.log
• sourcetype = saa:incredible:webui
 With Correct Sourcetypes
Same Regex, Different Sourcetype
Cisco Squid

Bluecoat
 For Performance
Configuring these 6 settings for each sourcetype, on your indexers.
[saa:incredible:useractivity]
TIME_PREFIX = ^
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 30
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%f%z
TRUNCATE = 10000
Data Usability
 Common Information Model (CIM)
http://docs.splunk.com/Documentation/CIM/latest/User/Overview
▶ A way of normalizing your data for maximum efficiency at search time
▶ Splunk Certified TA’s typically include necessary normalizations
▶ Allows end-users to search using common fields such as “user” across many
sourcetypes
▶ Extract your fields, normalize, then tag your data
 CIM Data Models

▶ Alerts ▶ Java Virtual Machines

▶ Application State ▶ Malware
▶ Authentication ▶ Network Resolution (DNS)
▶ Certificates ▶ Network Sessions
▶ Databases ▶ Network Traffic
▶ Data Loss Prevention ▶ Performance
▶ Email ▶ Ticket Management
▶ Interprocess Messaging ▶ Updates
▶ Intrusion Detection ▶ Vulnerabilities
▶ Inventory ▶ Web
 What Do You Do?
Making your data usable.
▶ Already have a proper sourcetype
▶ Extract your fields
▶ Create field aliases
• username AS user
▶ Create calculations
• action=if(action="OK","success","failure")
▶ Create tags
▶ Use the Data Models as a guide
 Recap

▶ Always set a sourcetype!

▶ Don’t use syslog as a sourcetype!
▶ Don’t use csv, json, xml as a sourcetype!
▶ Use Splunkbase as a starting point!
▶ Make your own sourcetype if you must, but use the naming format:
vendor:product:technology:format
▶ Make the data usable – CIM!
 Do You Have Bad Sourcetypes?
inputs.conf without a sourcetype defined
▶ | tstats count where index=* (sourcetype=*-*
OR sourcetype=*too_small) by sourcetype

▶ index=* (sourcetype=*-* OR
sourcetype=*too_small) | stats count by
sourcetype

▶ Don’t make the puppy sad!

 What’s Next?

▶ Splunk Enterprise Data Administration

• https://www.splunk.com/view/SP-CAAAPSE

▶ Go see these sessions….

• Dashboards, Alerting, Reporting and Visualization - What’s New
• Next Generation Dashboards
• Lesser Known Search Commands
• Creating Your Own Splunk Learning Environment
 Resources

▶ https://docs.splunk.com/Documentation/Splunk/latest/Data/Whysourcetypesmatter
▶ https://docs.splunk.com/Documentation/AddOns/released/Overview/Sourcetypes
▶ https://www.splunk.com/blog/2012/08/10/sourcetypes-whats-in-name.html
▶ https://www.splunk.com/blog/2010/02/11/sourcetypes-gone-wild.html
 © 2017 SPLUNK INC.

Don't forget to rate this session in the

.conf2017 mobile app