Search Under the Hood – Lab Guide

Overview
Welcome to the Splunk Education lab environment. These lab exercises will test your knowledge of the Job
Inspector, your ability to optimize searches, the makeresults command, the fieldsummary command and
informational functions.

Scenario
You will use data from the international video game company, Buttercup Games. A list of source types is
provided below.

NOTE: This is a lab environment driven by data generators with obvious limitations. This is not a
production environment. Screenshots approximate what you should see, not the exact output.

Index Type Sourcetype Interesting Fields

web Online sales access_combined action, bytes, categoryId, clientip, itemId,
JSESSIONID, price, productId, product_name,
referer, referer_domain, sale_price, status,
user, useragent

security Active Directory winauthentication_security LogName, SourceName, EventCode, EventType, User

Web server linux_secure action, app, dest, process, src_ip, src_port,

user, vendor_action

sales Business sales_entries AcctCode, CustomerID, TransactionID

Intelligence server

network Web security cisco_wsa_squid action, cs_method, cs_mime_type, cs_url,

appliance data cs_username, sc_bytes, sc_http_status,
sc_result_code, severity, src_ip, status, url,
usage, x_mcafee_virus_name, x_wbrs_score,
x_webcat_code_abbr

Firewall data cisco_firewall bcg_ip, dept, Duration, fname, IP, lname,

location, rfid, splunk_role, splunk_server,
Username
games Game logs SimCubeBeta date_hour, date_mday, date_minute, date_month,
date_second, data_wday, data_year, date_zone,
eventtype, index, linecount, punct,
splunk_server, timeendpos, timestartpos

Lab Connection Info

Access labs using the server URL, user name, and password shown in your lab environment.

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 1
Common Commands and Functions
These commands and statistical functions are commonly used in searches but may not have been explicitly
discussed in the module. Please use this table for quick reference. Click on the hyperlinked SPL to be taken to
the Search Manual for that command or function.
SPL Type Description Example

Sorts results in Sort the first 100 src_ip values in descending order
descending or ascending
sort command
order by a specified field.
| sort 100 -src_ip
Can limit results to a
specific number.

Return events with a count value greater than 30

Filters search results
where command
using eval-expressions.
| where count > 30
Rename SESSIONID to 'The session ID'
Renames one or
rename command
more fields.
| rename SESSIONID as "The session ID"

Remove the host field from the results

Keeps (+) or removes (-)
fields command
fields from search
| fields - host
results.

Calculate the total sales, i.e. the sum of price values

Calculates aggregate
stats command
statistics over the
results set.
| stats sum(price)

Concatenate first_name and last_name values with a

Calculates an expression space to create a field called "full_name"
eval command and puts the resulting
value into a new or
existing field.
| eval full_name=first_name." ".last_name

Output vendorCountry, vendor, and sales values to

table command Returns a table. a table

| table vendorCountry, vendor, sales

Returns the sum of the Calculate the sum of the bytes field
statistical values of a field. Can be
sum() function used with stats,
timechart, and chart
| stats sum(bytes)
commands.
Count all events as "events" and count all events that
Returns the number of contain a value for action as "action"
occurrences of all events
count or statistical
or a specific field. Can | stats count as events,
count() function
be used with stats, count(action) as action
timechart, and chart
commands.

Refer to the Search Reference Manual for a full list of commands and functions.
© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 2
Lab Exercises
Description
Configure the lab environment user account. Then, evaluate lispy expressions and optimize searches, use
the makeresults command to test a regex expression, generate summary statistics with the fieldsummary
command, and use informational functions to gain insights about search results.

Steps
Task 1: Log into Splunk and change the account name and time zone.

Set up your lab environment to fit your time zone. This also allows the
instructor to track your progress and assist you if necessary.
1. Log into your Splunk lab environment using the username and
password provided to you.
2. You may see a pop-up window welcoming you to the lab environment.
You can click Continue to Tour but this is not required. Click Skip to
dismiss the window.
3. Click on the username you logged in with (at the top of the screen) and
then choose Account Settings from the drop-down menu.
After you complete step 6,
4. In the Full name box, enter your first and last name.
you will see your name in
5. Click Save. the web interface.
6. Reload your browser to reflect the recent changes to the interface.
(This area of the web interface will be referred to as user name.)

NOTE: Sometimes there can be delays in executing an action like saving in the UI or returning results
of a search. If you are experiencing a delay, please allow the UI a few minutes to execute
your action.

7. Navigate to user name > Preferences.

8. Choose your local time zone from the Time zone drop-down menu.
9. Click Apply.
10. (Optional) Navigate to user name > Preferences > SPL Editor > Search auto-format and click on the
toggle to activate auto-formatting. Then click Apply. When the pipe character is used in search, the SPL
Editor will automatically begin the pipe on a new line.

Search auto-format disabled (default)

Search auto-format enabled

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 3
Scenario: Search the web server data where the source port is 1062.

Task 2: Find the lispy expression for a search.

11. In the top left corner of Splunk Web, select Apps > Search & Reporting. This sets our app context to the
search app.
12. Execute this search over the Previous month.
index=security sourcetype=linux_secure host=www1 src_port=1062

NOTE: The data generators for the linux_secure sourcetype are random. Therefore, if your search
does not return results for src_port=1062 then you should choose a different value. Run the
search without src_port=1062. Then, find src_port in the Interesting Fields list and click on
a port value. The search will re-execute with the new port value. You will still receive full credit
for saving the L1S1 report even if you have a different value for src_port.

13. Click Job > Inspect Job to open the job inspector. The Search job inspector launches in a new window.
14. On the Search job inspector page, note the number of events returned vs. the number of events
scanned. You should notice that more events are scanned than returned. Next, you will view the lispy
used to retrieve these events.
15. In the Search job inspector, click the search.log link.
16. Using your browser's Find feature, find the lispy expression.
17. Close the search.log window.
18. Save your search as a report with the name L1S1.
a. Click Save As > Report
b. For Title, enter L1S1.
c. Save.
d. You can View your report or exit out of the Your Report Has Been Created window by clicking
the X in the upper-right corner.
e. You can access your saved reports using the Reports tab in the application bar.

Your recently saved L1S1 report will be visible in the Reports tab.

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 4
Scenario: SecOps suspects that some unsafe websites have been accessed by employees. Search for
Cisco Web Security log events that occurred over the previous 30 days with a reputation
score (x_wbrs_score) equal to -6.4.

Task 3: Use the TERM directive to optimize your search so that the number of events scanned is equal
to the number of events returned.

19. From the menu, click Search. (The search box re-initializes and should now be empty.)
20. Execute this search over the Last 30 days:
index=network sourcetype=cisco_wsa_squid x_wbrs_score=-6.4

NOTE: The web-based reputation score (WBRS) assigned to URLs determines the likelihood that the
webpage contains URL-based malware. The Cisco Web Security appliance uses this
information to stop malware attacks before they occur. Scores range from 10.0 to -10.0 and
anything under 6.0 is scanned or blocked.

21. Click Job > Inspect Job to open the Search job inspector. The job inspector launches in a new window.
22. On the Search job inspector main page, note the number of events returned vs. the number of events
scanned. You should notice that more events are scanned than returned. Next, you will view the lispy
used to retrieve these events.
23. In the Job Inspector, click the search.log link.
24. Using your browser's Find feature, find the lispy expression. The value for lispy should look like this:
[ AND 4 6 index::network sourcetype::cisco_wsa_squid ]

Notice that the value on which you were searching, -6.4 is broken into separate terms for the search and
thus returns all events that contain a 6 or a 4.

25. Close the search.log window.

26. The training videos demonstrated a technique to prevent the lispy from breaking up search terms.
Rewrite and run the search using this technique so that Splunk generates a lispy expression that searches
for the number -6.4, not two separate numbers 6 and 4.

27. Click Job > Inspect Job. Note the improvement in the number of events scanned. Is there a difference
between the number of events scanned and the number of events returned?

28. Click the search.log link.

29. Use the browser's Find feature to find the lispy expression. The value for lispy should look like this:
[ AND -6.4 index::network sourcetype::cisco_wsa_squid ]

Notice that this expression is more specific than it was before you edited and re-executed the search.

30. Close the search.log window.

31. Save your search as a report with the name L1S2.

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 5
Scenario: Search for OS login data on Buttercup Games *nix servers (linux_secure) that occurred
during the previous week with a user id (uid) equal to 0.

Task 4: Use the TERM directive to optimize your search so that the number of events scanned is equal
to the number of events returned.

32. From the menu, click Search. (The search box re-initializes and should now be empty.)
33. Execute this search over the Previous week:
index=security sourcetype=linux_secure uid=0
You should see events that include raw data that looks similar to what is shown below. Notice that the
string uid=0 can be found in the event data.

34. Click Job > Inspect Job to open the Job Inspector. What do you notice about the number of events
scanned versus the number of events returned by the search?

35. In the Search job inspector, click the search.log link.

36. Use your browser's Find feature to find the lispy expression. The value for lispy should look like this:
[ AND 0 index::security sourcetype::linux_secure ]

37. Close the search.log window.

38. Rewrite and run a search to produce a lispy that causes fewer events to be scanned from disk.

39. Execute the new search.

40. Click Job > Inspect Job. How has the relationship changed between the number of events scanned and
number of events returned?

41. Click the search.log link.

42. Use the browser's Find feature to find lispy. The value for lispy should look like this:
[ AND index::security sourcetype::linux_secure uid=0 ]
Notice that this expression is more specific than it was before you edited and re-executed the search.

43. Close the search.log window.

44. Save your search as a report with the name L1S3.

Scenario: Search for Cisco Firewall log events that occurred during the previous month, where the
bcg_ip field is equal to 10.1.10.107.

Task 5: Optimize a search that uses a field from a lookup.

45. From the menu, click Search. (The search box re-initializes and should now be empty.)

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 6
46. Execute this search over the Previous month:
index=network sourcetype=cisco_firewall bcg_ip=TERM(10.1.10.107)

NOTE: Please ignore any warnings you see during this step. Expand the time range to All time if your
search does not return results.

47. Click Job > Inspect Job to open the Search job inspector.
48. In the Search job inspector, click the search.log link.
49. Use your browser's Find feature to find lispy. The value for lispy should look like what is shown below.
[ AND index::network sourcetype::cisco_firewall ]

NOTE: 10.1.10.107 does not appear in the lispy and therefore is not used to determine which events
to read from disk.
This value does not appear because the bcg_ip field is obtained using a lookup; it is not
populated directly from the raw event. In this situation, you can examine the data to see what
else can be used to limit the events returned from disk.

50. Rewrite and run a search to produce a lispy that retrieves fewer events from disk and filters results from
a field other than bcg_ip. (Hint: bcg_ip is an alias.)

51. Click Job > Inspect Job to open the Search job inspector.
52. In the Search job inspector, click the search.log link.
53. Use your browser's Find feature to find lispy. The value for lispy should look like what is shown below.
[ AND 10.1.10.107 index::network sourcetype::cisco_firewall ]
54. Why does this lispy contain 10.1.10.107?

55. Save your search as a report with the name L1S4.

Scenario: Create firewall data to test a regex expression with the rex command.

Task 6: Use the makeresults command to create data for testing.

56. From the menu, click Search. (The search box re-initializes and should now be empty.)
57. Write a search that will use the following eval command to create firewall data and run this search over
the Last 24 hours.

| eval raw = "Aug 27 2020 21:10:08 awesome-vpn.buttercupgames.com %ASA-4-113019:

Group = buttercupgames Username = lteng, IP = 10.2.10.44, Session disconnected.
Session type = IPsec, Duration = 8h:8m:25s, Bytes xmt: 18998681, Bytes rcv: 1453738,
Reason: Connection Lost"

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 7
58. Now that you have data to work with, you can test the following rex command. This command should pull
reason description information from the event and populate a column called reason.

| rex field=raw "^(?:[^:\n]*:){8}\s+(?P<reason>.+)"

59. Save your search as a report with the name L1S5.

Scenario: A security operations manager wants to compare summary statistics for the size of
requested objects (sc_bytes) and the threat score of the requested object (x_wbrs_score)
from the web security appliance data over the past 24 hours.

Task 7: Use the fieldsummary command to generate summary statistics on 10 values from 2 fields.

60. From the menu, click Search. (The search box re-initializes and should now be empty.)
61. Pipe the results of this search to the fieldsummary command and run over the Last 24 hours. The
resulting table will display information for every field returned by the search.

index=network sourcetype=cisco_wsa_squid

62. Modify the search to limit results to the top 10 values of sc_bytes and x_wbrs_score.

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 8
63. Save your search as a report with the name L1S6.

Challenge: A security analyst wants to study the variance between bytes consumption and web scores
in the web security appliance data from yesterday.

Task 8: Use the fieldsummary command to calculate statistics on transformed data.

64. From the menu, click Search. (The search box re-initializes and should now be empty.)
65. Search web security appliance data (index=network sourcetype=cisco_wsa_squid) and calculate the
sum of sc_bytes values by x_wbrs_score. Run the following search over Yesterday.

66. Make the resulting statistics table more useful for analysis by using the fieldsummary command to place
the results in a comparison table.

67. Save your search as a report with the name LX1.

Challenge: Count events from the BCG e-commerce system that have and do not have a value for
CustomerID over the last 15 minutes. Count these events by sourcetype and list the unique CustomerID
values present in the events.

Task 9: Complete the search to fulfill the scenario request.

68. From the menu, click Search. (The search box re-initializes and should now be empty.)
69. Complete the missing portions of the search so that the eval command assigns "yes" to events that do not
contain a value for CustomerID and "no" to events that do contain a value for CustomerID. Execute the
search over the Last 15 minutes.

index=sales sourcetype=sales_entries
| eval IsCustomerIDNull = if(???(???),"yes","no")
| stats count(eval(IsCustomerIDNull="yes")) as "Events with null values",
count(eval(IsCustomerIDNull="no")) as "Events without null values",
values(CustomerID) as "CustomerID Values" by sourcetype

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 9
70. Save your search as a report with the name LX2.

© 2022 Splunk Inc. All rights reserved. Search Under the Hood 11 January 2022 10