Beginning Splunk With Docker

Beginning Splunk With Docker - Vincent Sesto
1
2
Contents
Contents 3
Preface 7
What this book covers 7
What you need for this book 8
Who this book is for 8
Conventions 8
Downloading the example code 9
Piracy 9
1 Docker Fundamentals With Splunk 11

Docker Key Terminology 12
Installing Docker 12
Docker Hello World 13
Docker Processes 14
Docker Images 16
Docker Containers 18
Monitoring Our Docker Container 21
Interacting With Your Docker Container 23
Killing Docker Containers 24
Mounting Volumes To Manipulate Configurations 25
Using Data Containers 27
Indexing Container Data 28
Summary 31
2 Building Splunk Images And Using Dockerfiles 35

Introducing Dockerfiles 35
Extending Dockerfiles With Splunk Apps 38
About the Mood Radiator Splunk App 43
Further Extending Our Splunk App 43
Adding Splunk Data For Our App 44
Using Docker Registries 48
Committing Changes To Images 53
More Dockerfile Commands 56
Summary 58
3
3 Building Splunk Environments With Docker Compose 61

Installing Docker Compose 61
Our First Docker Compose File 62
Review Our Docker Commands 63
Extending Docker Compose File 64
Combining Services With Docker Compose 66
Building Splunk Environments 68
Forwarding and HTTP Event Collection 72
Using Configuration Files 72
Persistent Data Storage 74
Summary 77
Appendix 1
The Spluk Web Interface 81

The Splunk Welcome Screen 81
Splunk Administration Menu 82
Adding Data To Splunk 83
Searching Basics In Splunk 84
1. Start With A Search Term, Eg; ERROR 84
2. Wildcards To Expand Your Search Terms 84
4. Look Through Your Extracted Fields 85
5. Add Functions To Provide More Depth 86
Stats 86
Chart 87
Appendix 2
Docker Command Summary 89

Chapter 1 Summary of Commands 89
4
This book has been published in conjunction with Splunk User, Developer,
Administrator Magazine at https://medium.com/splunkuserdeveloperadministrator
5
6
Preface
In a perfect world our dev, test and production environments would be
exactly the same, on immutable infrastructure, allowing us to move our
development process like a production line seamlessly through these
environments and into production ready for our users. If issues or bugs with
our code come up, we pick them up as early as possible to make sure our
customer experience is limited to only our best code possible.
Unfortunately, this isn’t a perfect world and bugs creep into our work, our
development environment comes out of step with our production and test
environment, and sometimes we don’t always ship the best code possible.
Hopefully with this book we will start to look at a way to implement Docker
for our development, test and production environments to hopefully make
all of these as identical as possible.
This book hopes to bring you a unique approach to learning Docker as we
will do this in conjunction to implementing Docker with Splunk.
This book is not aiming to provide you with 10,000 hours or courses for you
to master the subjects outlined but instead provide 20 hours of necessary,
deliberate training with a purpose in mind to give you 80% of the
knowledge needed to be proficient at the subject at hand.
What this book covers

The book is separated into three chapters which will allow you to build on
your knowledge with each chapter, developing further as we move through
the examples:
7
Chapter 1, Docker Fundamentals With Splunk, provides you with the basics
to get started using docker as well as using it with Splunk. We also take a
quick look at the Splunk interface with a quick Splunk 101 tutorial.
Chapter 2, Building Splunk Images and Using Dockerfiles, takes you a step
further in implementing Splunk through Docker by providing an indepth look
at implementing environments with Dockerfiles.
Chapter 3, Building Splunk Environments With Docker Compose, then
gives you your next steps in setting up your Splunk network by using
Docker Compose.
What you need for this book

To be able to work along with the example applications created in this
book, you will need to have the following items available:
· A running version of Docker, preferably on Linux or Mac, but a
majority of the commands will work across in Windows as well
· A modern and stable web browser, such as Chrome or Firefox
· A basic understanding of web technologies as well as being
competent using the Linux or Windows command line
· An Internet connection
Who this book is for

This book is designed to provide an introduction to both Docker and Splunk
and allow the reader to build experience with each chapter to cover more
advanced topics.
Conventions
In this book, you will find a number of text styles that distinguish between
different kinds of information. Here are some examples of these styles and
an explanation of their meaning.
A block of code is set as follows:
1 <dashboard>
2 <label>SimpleXMLDashboard</label>
8
Any command-line input or output is written as follows:
echo "# SplunkAndDocker"
SplunkAndDocker
Above you can see that the command itself is in bold, but the output is not.
New terms and important words are shown in bold.
Downloading the example code

You can download the example code files for this book from the following
location. https://github.com/vincesesto/beginningsplunkwithdocker
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across
all media. We take the protection of our copyright and licenses very
seriously. If you come across any illegal copies of our works in any form on
the Internet, please provide us with the location address or website name
immediately so that we can pursue a remedy.
Please contact us at vince.sesto@gmail.com with a link to the suspected

pirated material.
We appreciate your help in protecting our authors and our ability to bring
you valuable Content.
9
10
1
Docker Fundamentals
With Splunk
I am sure that a lot of people would skim past this book thinking “what’s the
point?”, but for me Docker and Splunk can go hand in hand. It can allow
you to test and develop a Splunk App, installation environment or
configuration change, on your desktop, move it into a test environment and
then release to production. It simplifies the process of creating a consistent
environment and limits the chance of your environments changing through
the development process.
Docker creates everything you need and places it into an isolated

environment or container, but unlike a VM a container doesn’t include the
full operating system, only the libraries required to make your software
work. This makes things a lot smaller, efficient, and easier to transport
around.
In this chapter we are going to get you comfortable with Docker. In this
process we will be using Spunk as our container to give you twice the bang
for your buck.
The aim of this book is to get you working as quickly as possible so, let’s
get started.
11
Docker Key Terminology

I promise this will be quick, but we just need to clarify some terminology
when we are talking about Docker. Which we can hopefully quickly
summarize for you below:
● Registry - This is where we will be downloading our Docker images
from to work with. This will usually be from Docker Hub or from a
private registry.
● Image - This is usually a snapshot of a container that will provide a
particular service within the Docker container. In quite a few of our
examples we will be using Splunk but they can be anything you need,
including Nginx, Proxy, DNS, etc.
● Container - This is the running instance of a Docker image.
From the image above we can see that we use Docker commands from
our host to pull our images from the registry and build and run the
containers on our host, which could be our laptop or development server
or production environment.
Installing Docker
Docker supports a long list of OS’s. For a full list of supported platforms
and how to install. It’s best to go to the official Docker web page and find
your OS. You will also be provided with some in depth instructions on how
to install Docker on your environment at the following url:
https://docs.docker.com/engines/installation/#supported-platforms
12
● Once you have Docker installed and running on your PC, you can
start by checking the version of docker you are using. You will need
to open a command prompt on your PC or host and simply use the --
version option with the Docker command to do that:
docker --version
Docker version 17.05.0-ce, build 89658be
The rate of development for Docker over the past few years has been fast
to say the least. In this book we will do our best to ensure that all the
commands and examples will work with at least Docker version 17 or
more recent versions.
Docker Hello World

If you’ve worked in Technology for long enough, I am sure by now you
have had to perform a “hello world!” to get your progress started.
Fortunately, Docker allows us to do exactly the same thing. At the
command prompt, we will use the run option with the docker command and
call the hello-world image:
docker run hello-world
Unable to find image 'hello-world:latest' locally

latest: Pulling from library/hello-world
78445dd45222: Pull complete
Digest:
sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05
f6d12a1ac8d7
Status: Downloaded newer image for hello-world:latest
Hello from Docker!

This message shows that your installation appears to be
working correctly.
13
To generate this message, Docker took the following steps:

1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from
the Docker Hub.
3. The Docker daemon created a new container from that
image which runs the
executable that produces the output you are currently
reading.
4. The Docker daemon streamed that output to the Docker
client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu

container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free

Docker ID:
https://cloud.docker.com/
For more examples and ideas, visit:

https://docs.docker.com/engine/userguide/
● From the output, it looks like things don’t kick off that well with the
“Unable to find image 'hello-world:latest' locally”. All this is
saying is that if we use the run command, if there are no versions of
the docker image on our host, it will be downloaded from the registry.
● Once the image is downloaded, we can then see “Hello from
Docker!”, this is the start of our output for the running container, it
then goes on to provide us with some useful information on Docker
and some examples to take our knowledge further.
● The service provided by the hello-world image has then completed
and we are then provided with our command prompt.
Docker Processes
From our simple hello-world example above we can see straight away that
the process is complete and the container is no longer running. We can
verify this by using the ps docker option in the command line.
14
● The ps option of the docker command will show all of the running
containers on the host:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS

PORTS NAMES
● If we use the -a or “--all” flag with the ps option, we will be able to

see all of the docker containers that are available on your host even if
they are no longer running. Below we can see out hello-world image
that was run just a moment ago:
docker ps -a
CONTAINER ID IMAGE COMMAND

CREATED STATUS PORTS
NAMES
2d5e1414661e hello-world "/hello" 2
minutes ago Exited (0) 2 minutes ago
relaxed_lovelace
We can see that out container “Exited (0) 2 minutes ago”.
● From the output we can from our ps command, we can use the
CONTAINER ID to view the logs from this container. Simply using the
logs option to see what services and process have run as part of the
container:
docker logs 2d5e1414661e
Our example will simply show the same output that was printed to
screen, but this command is useful for troubleshooting containers that
fail. If a container does not start or run what we hoped it would,
hopefully any errors should come up in our logs.
15
Docker Images
From our hello-world example, we saw “Unable to find image 'hello-
world:latest' locally” so it needed to be pulled from the registry. In this
instance we have used Docker Hub, which is an online registry that you
can use to pull many of the images available to run different services. We
will show you how to create your own images at a later time, but for now we
will be able to get a lot done by using pre compiled containers that run from
Docker Hub. To access this registry, go to the following domain:
https://hub.docker.com/
● Firstly we can see what images we have available on our current host
by running the images command option.
docker images
REPOSITORY TAG IMAGE ID CREATED

SIZE
hello-world latest 48b5124b2768 4 months ago
1.84 kB
We have only pulled one image so far so, hello-world is the only one
visible. As we move along, you will see more images on your host.
● Of course, we can visit a registry like Docker Hub, but from the
command line we can see the type of images available by using the
search command option and specifying the type of image we are
looking for. For our example, let’s search for what we came here for,
and that’s Splunk Docker images.
docker search splunk
NAME DESCRIPTION
STARS OFFICIAL AUTOMATED
splunk/splunk The platform for operational
intelligence 47
16
adobeplatform/docker-splunk Splunk implementation in

Docker 7 [OK]
...
Our search comes up with over 20 images available, so we have cut

down what we have shown above to limit the space needed. There
are a wide range of images on the registry, some are supported and
some not, so be careful and choose wisely. As you can see there are
star ratings that the users can provide for each of the image and
thankfully there is a Docker image supported by Splunk which is first
in our list with the name splunk/splunk.
The cool thing about Docker Hub is that, if you’re interested in a

particular image, the provider will usually give you details on how to
use it and other specific details. For our Splunk image that we will pull
from the registry shortly, you can view the Docker Hub page at the
following domain: https://hub.docker.com/r/splunk/splunk/
● We will be getting familiar with the supported Splunk image, so we

will use the pull command option now to download it from the Docker
registry. This will bring it into our environment ready to be used:
docker pull splunk/splunk:latest
latest: Pulling from splunk/splunk

10a267c67f42: Already exists
cbad09443956: Pull complete
7f825dc4d416: Pull complete
f4c4f2c51818: Pull complete
0e768ea1f6a5: Pull complete
e5d903ddcf19: Pull complete
5ea02cfd6a88: Pull complete
Digest: sha256:e601a0df42730de64e866a66400cca4fc8e…
Status: Downloaded newer image for splunk/splunk:latest
We can see that our command is using :latest which will download
the latest version onto our environment. Without specifying the
17
version, Docker will pull the latest version by default. If we require a

different version, such as an older or knewer unsupported version,
this can be specified here, but you need to know the exact version for
the command to be successful.
Each part of the image the is pulled down is a layer of the container
image. The bigger the image the more layers it will have, and once
we make a change, we will add new layers ourselves.
● If we run the images command option again, we will now see that we
have two images available to be used:
docker images
REPOSITORY TAG IMAGE ID

CREATED SIZE
splunk/splunk latest a5e4afb500c0
2 weeks ago 745 MB
hello-world latest 48b5124b2768
4 months ago 1.84 kB
We can see from our output that we now have two images
downloaded onto our system, one for hello-world and one for
splunk/splunk.
Docker Containers
So we have our Docker container downloaded and ready to use on our
system, so let’s get started using it. Just like we did earlier, we will use the
run command option to start up our Splunk container, but to get the
container running we will need to add a few more option.
● We need to once again open our command line prompt and start with
the following command:
18
docker run -d -e "SPLUNK_START_ARGS=--accept-

license" -e "SPLUNK_USER=root" -p 8000:8000
splunk/splunk
24d10be49a8dd142b5c64312ca487...c4d7f12ba19bf83b117e8c34
The output gives us an identification hash for our container. With the
run command, we have used the following options:
● The -d option allows us to run the container in detached mode.
This means that the container will run in the background of our
system.
● -e allows us to specify environment variables to start our
container with. In the command above, we are specifying the
SPLUNK_START_ARGS to accept the license terms and
conditions and the SPLUNK_USER variable runs the service in
the container as the root user.
● The -p option allows us to map our ports from the running
container, to the host. In our example we are mapping port
8000 to 8000.
● Finally, the splunk/splunk option is the image we are using for
our container, which we pulled from our registry earlier. If we
had not done this, our output would show the image being
downloaded before our container starts up.
● Although it doesn’t look like much is currently happening, we can use

the ps option with the Docker command to see our container running
and available.
docker ps

NAMES
24d10be49a8d splunk/splunk "/sbin/entrypoint...."
3 seconds ago Up 3 seconds 1514/tcp, 8088-
19
8089/tcp, 8191/tcp, 9997/tcp, 0.0.0.0:8000->8000/tcp

kickass_northcutt
● I know you’re still not impressed, but if you open a web browser and
navigate to the following domain, you will see your new Splunk install
ready to be used: http://localhost:8000
If all worked as it should, you will be presented with the first time login
screen and once you have changed your password, you will be
presented with an image similar to the one shown above. A fresh
Splunk install ready to be used.
If we wanted to set up more Splunk containers on the same system, we

could do that, but would need to make sure that they did not conflict with
any other containers running. Specifically, we would need to make sure to
use a different port as 8000 will be assigned to the currently running
container.
20
Monitoring Our Docker Container

We have used the ps command option to view our container and view
containers that are no longer running, but we have a few more options if we
need to gain some further information on what is currently running on our
system.
● There may be a time when we need to get all the relevant details of
our container and the inspect command option gives us a lot of those
details. When we know our container image ID, usually by running
the ps command option, Docker will provide a detailed list of
parameters that the container is being run with:
docker inspect <container_id>
[
{
"Id":
"72769c19065a76270925ca45c7ca5f511fa58caefa47b72a87ecd88d63
c0d166",
"Created": "2017-05-28T21:34:59.900188891Z",
"Path": "/sbin/entrypoint.sh",
"Args": [
"start-service"
],
...
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
}
}
]
21
We have trimmed the output we are displaying a little as the full

output is over 200 lines in length but has a large amount of data that
we will use as we continue with this book.
● As we discussed earlier, by using the logs command option to view

the output of the container and in most circumstances will be related
to the service we are running, as you can see from the output below.
docker logs <container_id>
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to
'/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
...............................+++
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
The Splunk web interface is at http://72769c19065a:8000
Once again we have reduced the output from the logs command, but
you will be able to see that it is generating an RSA key and going
through the first time start up processes.
● The history command option is also a good way to see the command
history you have used with a specific container image:
docker history hello-world
IMAGE CREATED CREATED BY

SIZE COMMENT
48b5124b2768 4 months ago /bin/sh -c #(nop) CMD
["/hello"] 0 B
<missing> 4 months ago /bin/sh -c #(nop) COPY
file:22b680a46dca70... 1.84 kB
22
We have used the hello-world image to view history here to once

again reduce the output. Our Splunk image has a lot more commands
as the image has more layers involved in its creation of the image.
● Finally, we can use the stats Docker command option to provide

helpful statistics on our running containers. If we use it without any
container id specified we will see all of the containers on our system,
otherwise we can limit this to the one container with the following
command.
docker stats <container_id>
Interacting With Your Docker Container

The idea behind Docker is that we want our containers and images to be as
immutable as possible, meaning that services and resources are replaced
instead of changed. We do whatever we can to automate and streamline
our configuration as much as possible and in the coming chapters we will
see how we can do that. In some instances, the best way that we can
troubleshoot or see what is happening on our running container is by
accessing it.
● If we want to attach to a running container to see what is going on,

we can do this by using the attach command option with the specific
container ID we are wanting to access.
docker attach <container_id>
● Another helpful command option we can use is the exec command

option. This allows us to interact directly with the services on the
container, as well as accessing it directly via the command line where
you can see your output directly when you make your commands.
There are two main ways that the exec command can be used, the
first is by running a command directly in the container:
23
docker exec -d <container_id> touch /tmp/test.txt
It’s pretty simple, but the command above will run the “touch
/tmp/test.txt” on the running container and create the file that we
want in the tmp directory. The second way that we could use exec is
by parsing in the -it options to bring you into an interactive shell. We
also need to specify the shell that we would like to use when
interacting with the container:
docker exec -it <container_id> /bin/bash
This then allows you to access the container as you normally would
access a server within the command line.
Killing Docker Containers

We have started things up, so now we can start tearing things down.
Images can accumulate on your system and you may need to clean up
running containers because you would like to restart with different
configurations.
● When we need to stop a running container, we use the kill command

option with the instance number to stop the service and container
from running. If successful, Docker will return the instance number
back to you on the command line.
docker kill <container_id>
● If the container has completed and is still available, we use the rm

command option to remove the container from our system.
docker rm <container_id>
● If you realise the specific image you are using may need to be
updated or you wish to clear up some space on your environment,
24
using the rmi command option will let you remove any specific
images needed.
docker rmi <image_id>
You’ll need to remember that images, networks and volumes cannot be

deleted if there is a container running that still uses these Docker items.
You will need to kill the container from running before you are able to
remove the associated image, network or volume.
Mounting Volumes To Manipulate Configurations

Docker uses the underlying operating system to limit the size of the image
that the container uses. By doing this we can then use files and directories
from our host and mount them onto our container. We can do this by using
the -v or --volume flag in our command options.
● If we needed to mount a directory, we would use the following

command to mount the /tmp/test directory from our host onto the
/home/test directory on our container.
docker run -d -v /tmp/test:/home/test image
We can use this to start configuring our Splunk environment when you
bring up the container. We will do this a lot more efficiently in the next few
chapters but for now we will use the run command option and use a
mounted volume to help change our configuration.
1. First log into your environment at the command prompt and make
sure there are no Splunk containers currently running.
2. For our example we are going to stop Splunk from showing us the
“First Time Login” screen when we open our web browser to access
the web interface. To do this, we need to create an empty file called
25
“.ui_login”, we will create it in the tmp directory, but you can place it
anywhere you have access to it.
touch /tmp/.ui_login
3. No we mount that file in the /opt/splunk/etc/ directory when we run the

Docker container.
docker run -d -v /tmp/.ui_login

/opt/splunk/etc/.ui_login -e "SPLUNK_START_ARGS=--
accept-license" -e "SPLUNK_USER=root" -p 8000:8000
splunk/splunk
This time, instead of being presented with the “First Time Login” screen,
we get a normal login prompt. We still need to enter the admin username
and changeme password to login, but as you can see, this is an easy way
that we can start to change the configuration of our application when we
start it up as a Docker container.
Remember that we have only mounted one file, if we mounted the entire
directory, all of the files in the directory would be added to your container,
which could cause problems if there are files in there that are not needed.
26
Using Data Containers

As things change with our containers and images, we can commit these
changes to our image for later use, but in some instances we may want to
use that data across multiple containers. We saw in the previous section
that we can use volume mounted files to achieve this, but one way that I
like to keep and store data is by using a designated container for your data.
In the next example we will copy the data from our Splunk container
running, create a data container with it and then restart our Splunk
container using that data.
1. If you haven’t already, log into your environment and access the
command line.
2. If you don’t have a Splunk container running, start on up now with the
following command:

splunk/splunk
3. Once you have this running and you have logged in, we can take a
copy of the running configuration files located in the /opt/splunk/etc/
directory with the docker cp command.
docker cp <container_id>:/opt/splunk/etc/
/tmp/data-store/
4. We can now create a container to store that data for our splunk
environment. With the create Docker command option.
docker create -v /tmp/data-store/etc/ --name data-

store busybox /bin/true
27
In our example we are calling our new container data-store and using
the busybox image to reduce the amount of space that the container
will take up.
5. We can now need to stop our Splunk container with the kill command
option and then start it up again, this time we will be using the --
volumes-from flag to use our data store container:
docker run -d --volumes-from data-store -e

"SPLUNK_START_ARGS=--accept-license" -e
"SPLUNK_USER=root" -p 8000:8000 splunk/splunk
Although we needed to kill the running container, if we needed to

make future changes or enhancements, we could simply use the
restart option that would update and restart the image for us as listed
below:
docker restart <container_id>
I personally like using data containers because it allows us to package up

our data and move it around like we would a container image and
sometimes this can be missed when using data volumes. Over the next few
chapters we will be using both to allow you to make up your own mind.
Indexing Container Data

This is probably what you actually want to use Splunk for, and it’s taken us
awhile to get to this point, so I am glad you’ve stuck with us. In the next
section we will set up our Splunk server and then get some data indexing
so we can start to search on it.
1. We should already be in our development environment and if not get

back in there and go to the command line.
28
2. As we did previously we need to start up a Splunk server instance,

with this time we will also enable port 8088 as well as port 8000 in the
command below.

license" -e "SPLUNK_USER=root" -p 8000:8000 -p
8088:8088 splunk/splunk
3. We are now going to use a newer feature of Splunk to allow logging

of HTTP Events. To do this, we will need to log on to the web
interface.
4. Once logged in click on the Settings menu and select Add Data.
5. Then click on the Monitor icon and select HTTP Event Collector.
6. You will then be presented with the screen below where you can
enter a name for your token that you will assign to your logging
instance. As you can see our token is called DockerToken and I have
left all other options blank. You can also select the defaults for the
following screen as well.
7. Once complete you will be assigned a token value, with the one that I
created was 85CD93B0-6C66-4AD1-ACC3-65180B5ADE1E
29
8. Now we can run a different type of service and have these logs
moved into our Splunk instance. For our example, we are going to set
up a basic nginx web server.
docker run -d -p 80:80 --log-driver=splunk --log-

opt splunk-token=85CD93B0-6C66-4AD1-ACC3-
65180B5ADE1E--log-opt splunk-
url=https://172.17.0.2:8088 --log-opt splunk-
insecureskipverify=true nginx
You will notice that we are running the new container and exposing
port 80, but there are other command line arguments that are specific
to Splunk.
● --log-driver=splunk is pretty self-explanatory, and tells our
new container that it will be using Splunk.
● All other options are prefixed with --log-opt, such as splunk-
token which provides the token that we have just created on
our Splunk server.
● splunk-url is the domain or IP address and port that we are
using to connect to our Splunk server.
● And finally we set splunk-insecureskipverify to true to skip
some extra security controls that are not needed at the
moment.
9. If all goes well, we should have an Nginx Docker container running

and our logs should also be getting sent through to our Splunk
instance and being indexed. If we open a web browser, we should
see the Nginx welcome screen displayed just like the image below.
30
We can now move to our Splunk instance and verify that we are
receiving data and indexing it. Open the search interface and search
for the HTTP Event token with the following search query.
source=”http:DockerToken”
As you can see below, we now have data available to searching in

our Splunk instance.
Summary
Hopefully we have been able to keep your attention and keep things
moving at a pretty fast pace. There is a lot that we have gone through to
31
get you started with the basics of Docker and Splunk and in our next
chapter we are going to move further with both and start to streamline our
automation with Dockerfiles. Below is a summary of all the commands used
in the chapter for you to refer to if you need them.
32
33
34
2
Building Splunk Images
And Using Dockerfiles
We’re going to move into automating our processes by starting to use
Dockerfiles with our Splunk Docker projects. Dockerfiles allow us to move
to specifying a lot of the information and configurations that we need to set
up our images in a state that we would like them to look before we run
them. This means that we can limit the amount of information we need to
present as command line options and have our services in a more reusable
state.
In the following chapter we are going to continue using Docker but get
straight into using Dockerfiles to help us with creating our images. We will
also look at setting up and running a Splunk App on our container and
moving our images to Docker Hub so we can allow other people to use
them.
Introducing Dockerfiles
In its simplest terms, a Dockerfile is a recipe of steps on how you would
define your Docker image. We take a plain text file called “Dockerfile” and
list all of the details and commands of our image, which we then use the
build Docker command option to build that into an image which we can
then run on our system.
1. Log onto your environment and make sure that we have no Splunk
containers running.
35
2. Create a directory that you can set up your Docker project and then
change into it. In our example, we will call it DockerSplunk.
mkdir DockerSplunk; cd DockerSplunk
3. As we did in the previous chapter we added a blank .ui_login file to

bypass the “First Time Login” screen and will do the same here.
Either use a text editor or if you are in a Unix based command line,
use the touch command to create it within our DockerSplunk
directory.
touch .ui_login
4. Open your favorite text editor and create a new file in the
DockerSplunk directory called Dockerfile and add the following lines
of code.
1 FROM splunk/splunk:latest
2 MAINTAINER vince.sesto@gmail.com
3
4 # Set up environment variables
5 ENV SPLUNK_START_ARGS --accept-license
6 ENV SPLUNK_USER root
7
8 # Copy ui_login to stop the first time login
screen
9 COPY .ui_login /opt/splunk/etc/.ui_login
Each line of the code of the Dockerfile is explained below:

● The first line starts with the FROM instruction telling us which
image to use.
● Line two then tells us who the MAINTAINER of that Dockerfile,
in this case it’s me and I have added my email address, incase
there are us any support needed.
36
● We leave some white space for line three and then we have a
comment in line four using the hash character, similar to other
scripting languages.
● Line five and six then define our environment variables the
same way that we did in our command line in chapter one, but
in the Dockerfile we use the ENV instruction.
● After some more white space and another comment and then in
line nine we then use COPY to add in the empty .ui_login file
that we just created into the /opt/splunk/etc/ directory of our
new container image.
Make sure you don’t include the line numbers at the start of each line.
This is just to help us refer back to them at a later time.
5. We can then build our Docker image using the build Docker option.
Below we are also going to label the image with the -t flag and
specify the name “splunkhost”. We also use the period(.) to specify
that the Dockerfile in our current directory be used for the build.
docker build -t splunkhost .
Sending build context to Docker daemon 415.1 MB

Step 1/5 : FROM splunk/splunk:latest
---> a5e4afb500c0
Step 2/5 : MAINTAINER vince.sesto@gmail.com
---> Using cache
---> 743bbd852e17
Step 3/5 : ENV SPLUNK_START_ARGS --accept-license
---> Using cache
---> fd1ef2d4beb4
Step 4/5 : ENV SPLUNK_USER root
---> Using cache
---> 893381a437ba
Step 5/5 : COPY .ui_login /opt/splunk/etc/.ui_login
---> 308ddaf400bc
Removing intermediate container ccd9712db80a
Successfully built 308ddaf400bc
37
The Splunk image we are using is already on my system so your

build output may show it pulling the image from the Docker registry in
Step 1. As you can also see that for each instruction or command we
have provided in our Dockerfile, the output is described as a “Step”
and if our build failed on a specific step, the build would stop and fail
compilation.
6. We can now use our run Docker command to run the image that we
have just created, but unlike on previous instances we can now limit
the number of command line options needed to run the container:
docker run -d -p 8000:8000 splunkhost
In this instance, we have only needed to specify the -d option to run

the container in detached mode, -p to setup port forwarding and
splunkhost is to make sure we use the image we have just created.
Extending Dockerfiles With Splunk Apps

In the world of Splunk a segregated container of dashboards, panels and
data that can be packaged up for a specific use case or client is referred to
as a Splunk App. We can use Splunk Apps to make our environment
immediately useful, especially to non-technical users who are not familiar
with the technology as you can provide all of the information they need in a
segregated area where they don’t need to know anything else.
This is also a perfect use case for extending our Dockerfile. In the next few
pages we are going to keep moving with our Docker image and create a
basic Splunk App that we will be able to develop and test using Docker as
the medium to speed things up and enhance our development experience.
38
This book is not designed to give you an in depth knowledge of Splunk

App Development, but if you are interested in learning more about this
subject, the following link is a book that I have written on the subject that
may be able to give you the information you need.
https://www.packtpub.com/big-data-and-business-
intelligence/learning-splunk-web-framework
We will get started now with the basic shell of our Splunk App, and
remember that all of the code can be downloaded from the GitHub repo at
the following location:
https://github.com/vincesesto/beginningsplunkwithdocker.git
1. Once again log onto your system and access the DockerSplunk
directory that we created previously.
2. In the directory we are going to create the bare minimum to create a

Splunk App. We create our App directory called mood_radiator and
create the following directory structure. If we use the tree Unix based
command, it will look similar to the following.
tree
.
├── Dockerfile
└── mood_radiator
├── default
└── metadata
mkdir -p mood_radiator/default;
mkdir -p mood_radiator/metadata
We will give you some more details on what the “Mood Radiator”
Splunk App, but for now, continue to create the basic shell of our
app.
39
3. We will now create our app.conf file in the mood_radiator/default/

directory, so open your text editor and enter the following:
1 #
2 # Splunk app configuration file
3 #
4
5 [install]
6 is_configured = 0
7
8 [ui]
9 is_visible = 1
10 label = Mood Radiator
11
12 [launcher]
13 author = <enter_your_name>
14 description =
15 version = 1.0
The code above is simply adding in our labels for our app, naming it,
showing who the author and giving a version.
4. Next set up the metadata for our Splunk app by creating the
default.meta file in the mood_radiator/metadata/ directory.
1
2 # Application-level permissions
3 []
4 access = read : [ * ], write : [ admin, power ]
5
6 ### EVENT TYPES
7 [eventtypes]
8 export = system
9
10 ### PROPS
40
11 [props]
12 export = system
13
14 ### TRANSFORMS
15 [transforms]
16 export = system
17
18 ### LOOKUPS
19 [lookups]
20 export = system
21
22 ### VIEWSTATES
23 [viewstates]
24 access = read : [ * ], write : [ * ]
25 export = system
This is where we set up some of the permissions of the new app and
what users are able to do.
5. We have now set up the shell of our new Splunk App and we can
now add the details to our Dockerfile to make sure that our new
container build with it. Open the Dockerfile in your text editor and add
the next two lines to the bottom of the the file.
11 # Creating a new app

12 COPY mood_radiator/
/opt/splunk/etc/apps/mood_radiator/
We are not doing anything we haven’t done previously. We are

simply using the COPY command to add the new directory structure,
config and metadata that we have just created.
6. Just as we did earlier, we can run the build Docker command again
to build the container with the new features.
41
Sending build context to Docker daemon 6.144 kB

Step 1/6 : FROM splunk/splunk:latest
---> a5e4afb500c0
Step 2/6 : MAINTAINER vince.sesto@gmail.com
---> Using cache
---> 743bbd852e17
Step 3/6 : ENV SPLUNK_START_ARGS --accept-license
---> Using cache
---> fd1ef2d4beb4
Step 4/6 : ENV SPLUNK_USER root
---> Using cache
---> 893381a437ba
Step 5/6 : COPY .ui_login /opt/splunk/etc/.ui_login
---> Using cache
---> 308ddaf400bc
Step 6/6 : COPY mood_radiator/
---> 281d09473180
Removing intermediate container 26650061b9d3
Successfully built 281d09473180
Our image builds and as you can see from the output above we now
see an extra step that we did not see before:
Step 6/6 : COPY mood_radiator/

7. As long as there are no other splunkhost containers running and our

8000 port is not being used we can run our docker command again to
see the results of our work.
The image below shows what our hard work has given us. Not much
but in left menu we can now see our new Splunk App called “Mood
Radiator” listed in there.
42
We have done all of this in a limited amount of time and with less that fifty
lines of code.
About the Mood Radiator Splunk App

The app was something that I created a few years ago and now resides
on the Splunkbase at the following location:
https://splunkbase.splunk.com/app/1366/#/overview
The Mood Radiator Splunk App was set up as a way to gauge team and
developer sentiment towards their work or a specific project.
Further Extending Our Splunk App

We can keep moving with our Splunk image and add some more
functionality through our Dockerfile. If we use the RUN instruction, we can
run shell commands as we set up our container. In the following example,
we will use it to install the vim application onto our container, in case we
need to look through some files at a later time.
1. Once again start by logging onto your system and access the
DockerSplunk directory that we created previously.
2. Open the Dockerfile in your text editor and add the next two lines to
the bottom of the the file.
43
14 # In case we need it
15 RUN apt-get update && apt-get install -y vim
Once again, it’s not too complicated where we can see that we have
added in a command and then by using RUN, we perform and apt-get
update and if that is successful, we then perform an apt-get install -y
vim.
3. Just as we did earlier, we can run the build Docker command again to
build the container with the new features.
…
Step 7/7 : RUN apt-get update && apt-get install -y vim
---> Running in 6bf027880045
Get:1 http://security.debian.org jessie/updates InRelease
[63.1 kB]
Get:2 http://security.debian.org jessie/updates/main amd64
Packages [519 kB]
…
Running hooks in /etc/ca-certificates/update.d....done.
---> d280b9580a21
Removing intermediate container 6bf027880045
Successfully built d280b9580a21
Once again, we have reduced the output that is displayed to the screen as
there is a lot. With our new step added, we can see that the apt-get update
and then install is run by Docker to build our new image.
Adding Splunk Data To Our App

Splunk is not much use without data in there. For our new Splunk app we
can add some simple log data as well as our configurations to have the
data indexed and stored in a specific source.
44
1. We should still be in our development system so access the

DockerSplunk so we can start to make some minor changes.
2. If you have not downloaded all of the sample data already, do so

now. You will notice that in the mood_radiator directory for chapter2,
there is a log file called showMood.log
3. Make sure that the sample data is located in the mood_radiator

base directory.
4. We can now set up our inputs.conf file which will set up the
sourcetype and location so Splunk will know where to index the data.
With your text editor save the following text.
1
[monitor:///opt/splunk/etc/apps/mood_radiator/showMood.
log]
2 disabled = false
3 sourcetype = moodLog
Make sure to save this file as inputs.conf in the

mood_radiator/default/ directory.
5. The props.conf file can also be added to the

mood_radiator/default/ directory. Save the following file as
props.conf to make sure that the data will be indexed correctly.
1 [moodLog]
2 DATETIME_CONFIG =
3 NO_BINARY_CHECK = true
4 category = Custom
5 pulldown_type = true
6. While we are here, we might as well add in the main dashboard that
uses the data that we are about to index. First we need to add the
45
navigation xml file to make sure that our new dashboard loads when
we enter the app. First you will need to make sure the nav director is
available.
mkdir -p mood_radiator/default/data/ui/nav/
7. We can now add in a new navigation xml. Open your text editor and
save the following text as default.xml in the
mood_radiator/default/data/ui/nav/ directory.
1 <nav>
2 <view name="flashtimeline"/>
3 <view name="MoodRadiator" default='true' />
4 <view name="information" />
5 </nav>
This is simply saying that it will search for a dashboard named

MoodRadiator and display that as default.
8. Now create a views directory for all our dashboards to now live.
mkdir -p mood_radiator/default/data/ui/views/
9. The actual dashboard code is over 100 lines in length so we will not
go through all of the text, but copy the dashboard from our GitHub
repository and add it to the mood_radiator/default/data/ui/views/
directory. It should look similar to the following condensed code.
1 <dashboard>
2 <label>MoodRadiator</label>
3 <row>
4 <panel>
5 <chart>
6 <title>Your Mood</title>
7 <search>
46
8 <query>sourcetype=moodLog |
…
111 </chart>
112 </panel>
113 </row>
114 </dashboard>
As we did earlier, we can run the build Docker command again to

build the container with the new features.
10. As long as there are no other splunkhost containers running

and our 8000 port is not being used we can run our docker command
again to see the results of our work.
If all went to plan you should see the following dashboard presented
when we load the Mood Radiator Splunk App.
47
By utilizing a service like GitHub, it will mean that we can perform daily
builds with our code to make sure that everything is working as it should.
We make changes, commit and then we are able to build the code into an
environment that is closely matched to our production environment to
make that the changes taking place are not causing any issues with our
code.
Using Docker Registries

We have talked about registries for some time now, but the cool thing is
that a registry like Docker Hub is free and you are able to push your images
up to this service to both store your complete images as well as allow other
users to learn from your work and even use them if they wish.
Although you can create your own registry that is not available to others in
a dedicated environment or on AWS, we will not cover this aspect but
instead show you how to utilize Docker Hub for your images.
48
1. Start by opening a web browser and accessing the Docker Hub web
page at the following url: https://hub.docker.com
2. If you don’t have an account already, create one. Don’t worry, unless
you need to really use it for business purposes, the service is free
and you should still be able to do quite a bit with a free account.
3. Once you are logged in, you will see a big blue button with “Create
Repository” on it. We could simply store our image in our base
repository, but if we create a new repo it will be nicer to show other
users. Click on Create Repository and you will be presented with a
similar for as the one below.
4. As you can see from the image above, I have added the repository
name of “dockersplunkapps” and provided a brief description. I am
49
also going to leave the visibility as public, but that is your choice.
Then click on create.
5. You new repository should now be displaying with basic information

that you can customise to your liking. You will notice that it also has a
“Docker Pull Command” that you can use to pull down your image,
but first we need to push something for people to consume.
6. By now we should have some Docker images available to push up to

our new repository. Access the command line on your development
system and using the images Docker command, you should see
what you have available.
docker images

CREATED SIZE
splunkhost latest 7241fea52165 11
minutes ago 744 MB
splunk/splunk latest cf104961b12b 6
hours ago 744 MB
As you can see above, we have two images, splunkhost is the image
that we have just created and the other is the original Splunk image
that we created it from.
7. We want to push the splunkhost image to our repository, but first we

need to login to Docker Hub through the command line. We need to
set our environment variable first that our login will use to recognise
who we are.
export DOCKER_ID_USER=”dockerhub_login_name”
The “dockerhub_login_name” will be the same as the one you use

to log on to Docker Hub from a web browser.
50
8. Then we use the login Docker command to login via the command
line.
docker login
Login with your Docker ID to push and pull images from
Docker Hub. If you don't have a Docker ID, head over to
https://hub.docker.com to create one.
Username (dockerhub_login_name):
Password:
Login Succeeded
9. We need to now tag our image that we wish to push to the repository.
This is so Docker Hub will know where to store it.
docker tag splunkhost

$DOCKER_ID_USER/dockersplunkapps
As you can see we have used the splunkhost image and named it the
same as the repository that we created.
10. We can also see the tag that we have made by using the
images command again.
docker images
REPOSITORY TAG
IMAGE ID CREATED SIZE
splunkhost latest
7241fea52165 22 minutes ago 744 MB
vincesesto/dockersplunkapps latest
7241fea52165 22 minutes ago 744 MB
splunk/splunk latest
cf104961b12b 6 hours ago
Compared to what we had previously, we now have an extra image with

the repository named of vincesesto/dockersplunkapps. It is tagged as
latest, but if we needed to tag it with a different version or the like, we
51
would simply add that to the end of our command, eg:

docker tag splunkhost $DOCKER_ID_USER/dockersplunkapps:ver0.1
11. We can now push our image to Docker Hub with the following
command.
docker push vincesesto/dockersplunkapps
The push refers to a repository

[docker.io/vincesesto/dockersplunkapps]
266d35cfbbae: Mounted from vincesesto/splunk_mood_app
4daa6604d0eb: Mounted from vincesesto/splunk_mood_app
ea82aeba03b5: Mounted from vincesesto/splunk_mood_app
992efd726e67: Mounted from vincesesto/splunk_mood_app
f285c4523707: Pushed
68c9733ab9ce: Mounted from vincesesto/splunk_mood_app
85ee2a79a86d: Mounted from vincesesto/splunk_mood_app
fdf64660979b: Mounted from vincesesto/splunk_mood_app
007ab444b234: Pushed
latest: digest:
sha256:c6ef92f1366ab00da121b21b15cb2bee60cfddb582b7102261b0
3d153847f86e size: 2205
If all goes well, we should now see the new image pushed to our
repository. To verify that this has been successful, access the web
interface again for your repository and click on the Tags tab, which should
display that you have recently pushed a new image.
52
Committing Changes To Images

For now, we have been making the changes to our code and then building
our image with this changed code. If there is something that you need to do
and you are not sure of how to do it, you can perform the change on the
running container, like installing a package or making a change to an
application and then commit the changes.
We are going to demonstrate this with a more lightweight image as it will

save us some time and space. The busybox image is just perfect for the
next example where we will make a change to our image, save it to a file,
remove all our images from our system and then load it from the file to
make sure we still have our changes.
1. We will need to pull down the busybox image from Docker Hub and
this can be done as usual with the run Docker option.
docker run -dt busybox
2. I previously cleaned out all of the containers in my system so when I

run a ps Docker command option, I am only seeing my new busybox
container running
docker ps
53

NAMES
e897169ed57b busybox "sh" 16
seconds ago Up 15 seconds
clever_haibt
3. We can now access the running container with the exec Docker
command option and interact with the environment as we would a
normal server.
docker exec -it e897169ed57b sh
4. We are going to make a simple change in our container to create a

text file as listed below.
ls; echo "Test" > test.txt; ls
bin dev etc home proc root sys tmp usr var
bin dev etc home proc root

sys test.txt tmp usr var
As you can see the second ls command is now showing an extra file
named test.txt.
5. Exit the container by using the exit command.
6. We can now commit these changes to another container with the

commit Docker command option and using the container id number.
docker commit <container_id> busychanges
7. If we view the containers on our system, we should now see a new

container with the name of busychanges.
54
docker images
CREATED SIZE
busychanges latest ab6db1a204f3 4
seconds ago 1.11 MB
busybox latest c75bebcdd211 3
weeks ago 1.11 MB
8. We saw how we can move our image into a repository, but what you
can also do is copy it to a file and then restore from that file. We copy
our image to a file with the save Docker command option.
docker save busychanges > busychanges.tar
9. To test this out, we can now kill all our running containers and
remove all of the images.
docker kill $(docker ps -q); docker rm -f $(docker

ps -a -q) docker rmi -f $(docker images -q)
In the above command we can kill all of the image id’s for the running
docker containers, with the ps -a -q, and then do the same with our
images with the images -q command.
10. Now that we have no more images left on our system, we can
use our busychanges.tar archive with the load Docker command
option to add it back into our system.
docker load < busychanges.tar

a94434c756c1: Loading layer [=============>] 3.584 kB/3.584
kB
Loaded image: busychanges:latest
docker images
CREATED SIZE
55
busychanges latest ab6db1a204f3

About a minute ago 1.11 MB
You can now see that once loaded back into our system, the
busychanges image is now available again for us to create a
container with the changes that we made earlier.
More Dockerfile Commands

We have had a pretty quick run through Dockerfiles in the above examples,
but you need to know that there are more Dockerfile instructions besides
the FROM, MAINTAINER, ENV, COPY and RUN that we have been
using so far. Hopefully the next few pages will fill in some of the blanks.
This book tries to help you learn by using examples, so by now you
should be aware of the fact the basic format of the Dockerfile is:
# Comment
INSTRUCTION argument
The following pages are a few of some of the more common Dockerfile
instructions that you will need to know.
● You can define what you would like your escape character to be by
using the # escape= instruction at the start of your Dockerfile. In the
example below, we set our escape value to the slash character.
# escape=\
● Not to be added as part of our Dockerfile, but part of the directory that
we set up our Dockerfile in, the .dockerignore can add a list of files to
ignore when running the COPY or ADD instructions in the Dockerfile.
● Similar to the RUN instruction is the CMD instruction where you

provide a default command for the running container. Only one can
be added in the Dockerfile and if there are more that one, only the
56
last CMD instruction is run. This command can be overridden with a

command line option when you run the container. As you can see
below, the format is to have your command in square brackets,
separated by commas.
CMD [“executable”, “param1”, “param2”]
● The LABEL instruction add tags or metadata to your docker

containers, in a key value format as listed below.
LABEL “label1”=”value1”
● The EXPOSE instruction informs Docker that the container listens on

a specified port, but does not make the port accessible to the host,
you still need to use the -p flag when you run it.
● Similar to COPY, the ADD instruction adds data from source to

destination on the image.
● ENTRYPOINT allows you to specify the default executable to run

when your container runs, which cannot be overridden by a command
line option. Just like CMD you can specify the command in square
brackets, but you can also define the executable as below.
ENTRYPOINT executable param1 param2
● The VOLUME instruction creates a mount point for your container

and provides a name and marks it as holding an externally mounted
volume from the host
● If you need to perform a RUN, CMD or ENTRYPOINT as a specific

user or UID, you can do this with the USER instruction.
USER splunk
57
● You can change the directory that your RUN, CMD or ENTRYPOINT
instructions run from by using the WORKDIR instruction.
WORKDIR /opt/splunk/
Summary
This time we have been using Dockerfiles to create our Docker images and
we have introduced you to a lot of the functionality that Dockerfiles can
bring to your system and set up. We have looked at how we can use
Dockerfiles to create and configure Splunk apps and have extended our
app and also included some test data that we have indexed in our
container. We also have a look at the ways that we can package our
containers for other people to use, both with the commit Docker command
option and by using Docker Hub.
In our next chapter we are going to continue to work with our environment
but we are going to start to use Docker Compose to help automate our
process.
58
This book has been published in conjunction with Splunk User, Developer, Administrator
Magazine at https://medium.com/splunkuserdeveloperadministrator
59
60
3
Building Splunk Environments
With Docker Compose
We’ve covered a lot of ground and we’re doing our best to build on our knowledge that
each chapter has provided us. We have moved from single commands, to Dockerfiles to
build and run images and now we are going to use Docker Compose to streamline the
process even further and allow us to define multi container Docker applications. In the
next section we are going to define our Splunk environment and then start all of the
services running with one command.
The cool thing about Docker Compose is that all of the work we have done so far is not
wasted. We will use the Dockefiles that we have been working with to incorporate them
into our new environment. Let’s not waste any more time and get started.
Installing Docker Compose

Just because we have Docker installed on our environment, does not
necessarily mean we would have Docker Compose. To verify the
application is installed run the following command from your command line
docker-compose --version
docker-compose version 1.13.0, build 1719ceb
For a full list of supported platforms and how to install. It’s best to go to the
official Docker Compose web page and find your OS. You will also be
provided with some in depth instructions on how to install it on your
environment at the following url.
https://docs.docker.com/compose/install
61
As with the version of Docker, we will do our best to make sure that the
work we do is as current as possible with all commands and examples
working with Docker Compose version 1.13.0 or more recent versions.
Our First Docker Compose File

We can use our existing Dockerfiles in conjunction with our first compose
file to show exactly how we use Docker Compose to define how we build
and run our Docker images.
We use Docker Compose by creating a file named:
docker-compose.yml
This is a YAML format file that is used to provide our instructions on how
we want to set up our environment. So let’s not get delayed with theory and
make a start.
1. Log back into your system that you were working and we will continue
to work from the DockerSplunk directory that we have already been
using in the previous chapters.
2. Using your favorite text editor we are going create our new compose
file in the DockerSplunk directory as we said earlier, called docker-
compose.yml.
3. There will not be too much to our first compose file, but enter in the
following six lines of code:
1 version: '3'
2 services:
3 splunkhost:
4 build: .
5 ports:
62
6 - "8000:8000"
● Line one will specify a number which corresponds to the

version of Docker Compose we are using.
● Line two then starts to define the service we want to create,
with the next line letting us call it “splunkhost”.
● Line four tells us that all the information on how we need to
build this service is located in our current directory.
● Lastly, we then specify that we want to open ports 8000 and
then map it to our host port 8000 as we normally would with our
run Docker command option.
4. All we now need to do is run Docker Compose to both build and

then run our services that we have defined with the following
command.
docker-compose up -d
Here we use the up option to create and build our container and the -
d option to run it in detached mode.
You will notice that the output is slightly different to when we ran our other
commands but by the end of it you will have your Splunk server running as
we did from our Dockerfile in the last chapter. If we run the ps Docker
command, we should see our running server and we should also be able to
log in.
Review Our Docker Commands

I wanted to take a quick step back to show the difference in the methods
we have used to run our Docker Containers.
When we first started our work, we needed to specify everything that we

needed to in our command line arguments to have our environment run.
63

splunk/splunk
We were then able to make things a little more complex by adding in a

Dockerfile to define a lot of these arguments and configurations, we then
built from the Dockerfile and then ran the image that we created from that
Dockerfile.
Now we have defined all of the build and run options in either our
Dockerfile and now in our docker-compose.yml file, so all we are now doing
is running the following.
Extending Docker Compose File

In our example above we’ve been able to use the Dockerfile and images
that we have built in our previous chapters and work. But what if we were
starting from scratch, would we need to do all of this work and create our
Dockerfile only to then create our docker-compose.yml file? Of course we
wouldn’t. Docker Compose allows us to start from scratch building our
services and defining all of the points as we would in a Dockerfile.
In our next piece of work, we should look at how we can use our docker-
compose.yml file to define our environment from scratch.
1. If you’re not logged into your environment, start by logging in and

make sure you are in our DockerSplunk directory to continue working.
2. Open the docker-compose.yml file with your text editor.
64
3. We are going to remove lines 3 to 6(4 lines in total), and then replace
them with the following code.
3 splunkserver:
4 hostname: splunkserver
5 image: splunk/splunk
6 environment:
7 SPLUNK_START_ARGS: --accept-license --
answer-yes
8 SPLUNK_ENABLE_LISTEN: 9997
9 SPLUNK_USER: root
10 ports:
11 - "8000:8000"
12 - "9997:9997"
13 - "8088:8088"
14
By now, I think you would be able to walk through all of this and know
exactly what it does, but just to be safe:
● We are naming our service and host splunkserver in lines three
and four.
● In line five we get to define the image that we will be using,
just as we would use the FROM instruction in a Dockerfile.
● Lines six to nine are then defining the environment variables
for our host, alternatively using ENV in a Dockerfile.
● Lastly lines ten to fourteen let us bind and map all of the ports
that we need to open up and have available for our service to
work.
4. Just as we did previously, all we now need to do is run Docker

Compose to both build and then run our services that we have
defined with the up command option.
65
There will be one main difference that you need to remember when
the container starts up is that we have not set up a .ui_login file so
you will be presented with the “First Time Login” screen when you
open it in a browser.
Combining Services With Docker Compose

I’m not sure you are convinced of the benefit of using Docker Compose and
to be honest the power of using Docker Compose is more apparent when
you start to set up multiple servers on the one compose file. From our first
chapter, we will set up an Nginx server that loads data into our Splunk
server. In the following example, we are going to do that to show you that
we define both of these environments from the one compose file.
1. If you aren’t logged into your environment, start by logging in and

make sure you are in our DockerSplunk directory to continue working.
2. Let’s also start up our Splunk server so we can set up our HTTP
Event Collector.
3. If you can’t remember how to, go back to first chapter and set up
HTTP Event Collector as we specified in the “Indexing Container
Data” section.
4. Run the inspect Docker command option and view the network
details of the running Docker container. To limit the amount of output
we see, we can use the --format flag and specifiy the exact
information we need.
docker inspect --format='{{range

.NetworkSettings.Networks}}{{.IPAddress}}{{end}}'
$(docker ps -q)
66
5. Once we have our Splunk Token and the IP address that the Splunk
server is running on, open the docker-compose.yml file with your text
editor and move to the bottom of the file and add in the following lines
of code.
15 nginx:
16 hostname: webserver
17 image: nginx
18 ports:
19 - "80:80"
20 logging:
21 driver: splunk
22 options:
23 splunk-url: https://172.18.0.2:8088
24 splunk-token: 85CD93B0-6C66-
4AD1-ACC3-65180B5ADE1E
25 splunk-insecureskipverify: 'true'
6. We can now run our compose command and in this instance, we are
just going to specify the new Nginx service we have added.
docker-compose up -d nginx
7. If you open your web browser and access http://localhost you

should get the Nginx welcome screen as you did in the earlier
chapters. We can also open up our search screen in Splunk and run
the following search to look for our new HTTP Events.
source=”http:DockerToken”
By now I am hoping that you can see the power of what you can do with
Docker Compose. Especially if you are familiar with Splunk environments,
the first thought that has probably popped into your head is that you should
be able to configure a fully clustered environment from Docker Compose.
67
Well the good news is that you can, but it may be a little more advanced for
this tutorial, but hopefully we will get you on your way in the next few pages
to help you achieve something like that.
If you haven't heard of the YAML file format before, it’s a human-readable
data serialization language, which is commonly used for configuration
files, but could be used anywhere data is being stored. The main thing
you need to know is that each item is differentiated by the whitespace or
indentation between them. We have done our best to make sure that all
the examples show the correct format for the values, but this can get
mixed up in the publication process.
If you are in doubt at all, please make sure you download the examples
from Github as these will have the correct and working format.
Building Splunk Environments

If you’ve built a Splunk environment before you may know how complex
they can get. This book doesn’t have enough time to go through all of the
intricacies of setting up a complete clustered Splunk environment using
Docker, but we can start to lay the foundations for you.
So in the following example, we are going to set up a Splunk forwarder that

will work with our Splunk server, adding in some extra features along the
way.
1. We need to get back into our environment where we were building

our hosts and start by killing all running containers and removing any
images.
2. Use your text editor and once again we are going to edit our docker-
compose.xml file.
3. Let’s remove our Nginx host for now, so remove all of the commands
after line 14.
68
4. We will first add some more instructions to our Splunk server. Add the
text below to give the container a label and to set up a network that it
can use.
15 labels:
16 splunk.env: "splunkserver"
17 networks:
18 splunk:
19 aliases:
20 - splunkserver
5. We will now set up a Splunk forwarder below our server so we can

forward data into the environment without having to move it directly to
the indexer. Add the following lines of code after the Splunk server.
21
22 splunkforwarder:
23 image: splunk/universalforwarder
24 hostname: splunkforwarder
25 environment:
26 SPLUNK_START_ARGS: --accept-license --
answer-yes
27 SPLUNK_FORWARD_SERVER: splunkserver:9997
28 SPLUNK_USER: root
This is the first half of our forwarder set up, and is very similar to our
server details. Here we name the service, provide a hostname, image
and environment variables. As you can see in line 27, we specify
where our data will be forwarded to using the hostname value that we
specified for the server, with no need for the actual IP address.
6. Now place in the second part of the forwarder code to specify the
labels and the network details for the host.
69
29 labels:
30 splunk.env: "splunkforwarder"
31 networks:
32 splunk:
33 aliases:
34 - splunkforwarder
7. We have been using the network instructions for both the server and
the forwarder. These will not work without setting up the basic
network labels for it to use. So, finally add the following lines to our
compose file to complete the network.
35
36 networks:
37 splunk:
8. We can now start our environment up again, by using the up directive

for the docker-compose command.
9. If you now look to see what Docker containers are running, we will be
able to see two containers running and nicely named:
 dockersplunk_splunkserver_1
 dockersplunk_splunkforwarder_1
docker ps
NAMES
bf1bd894e56c splunk/splunk
"/sbin/entrypoint...." 7 seconds ago Up 6 seconds
0.0.0.0:8000->8000/tcp, 1514/tcp, 8089/tcp, 0.0.0.0:8088-
>8088/tcp, 8191/tcp, 0.0.0.0:9997->9997/tcp
dockersplunk_splunkserver_1
70
09ff4b23560f splunk/universalforwarder
"/sbin/entrypoint...." 7 seconds ago Up 6 seconds
1514/tcp, 8088-8089/tcp
dockersplunk_splunkforwarder_1
10. We can double check to make sure the servers are all working
correctly by logging onto our Splunk server and going to the search
screen. Splunk keeps an internal index of all data moving between
hosts in an environment. We can see this by entering the following
into the search bar.
index="_internal"
11. On the left of the search screen, Splunk will provide you with
selected fields that you can search against. If you click on the host
field, you will notice that there are two entries, one for each of our
new containers.
71
Forwarding and HTTP Event Collection

In our earlier example we used the HTTP Event Collector feature of Splunk
to log the data from our Nginx host. We have done a lot of work to
configure our Docker images, so I am sure you are wondering why we don’t
simply install a forwarder onto our container. The first reason is that we
want to limit the size of the containers that we are creating as well as
limiting the number of services that are running on each of the containers. It
would be hopeful that we would only be running one service on each
container.
Secondly, the log files for our containers are usually not being stored on
our containers as this would also increase the size of our image. Instead
the logs will be stored on the host file system under the directory
/var/lib/docker/<container_id>/<container_id>-json.log. This is why we
need to specify a change to our log driver as we did in our previous
examples with Nginx.
Using Configuration Files

Just as we were using a Dockerfile in conjunction with Docker Compose,
there are other options to limit the amount of information included in your
compose file and move it somewhere else. Docker Compose allows you to
specify an env_file that includes all the relevant environment variables and
in some instances provides the ability to perform command as you will see
below.
So let’s add one to our compose file for the work that we are currently
doing. We will be able to move some of the code away from our compose
file and onto a configuration file.
1. We should still be in our directory where our docker-compose.yml file

is, so open it up with your text editor.
72
2. Currently lines seven to ten outline all our environment variables for
our Splunk server. We are going to replace these with an env_file. So
start by removing those four lines.
3. Next add the following two lines in their place.
7 env_file:
8 - splunkserver.env_file
4. Save the file, and the Splunk service should look similar to the lines
below.
4 splunkserver:
5 image: splunk/splunk
6 hostname: splunkserver
7 env_file:
8 - splunkserver.env_file
9 ports:
10 - "8000:8000"
11 - "9998:9998"
12 - "8088:8088"
13 labels:
14 splunk.env: "splunkserver"
15 networks:
16 splunk:
17 aliases:
18 - splunkserver
5. We now need to set up a file named splunkserver.env_file that will

now include all of our environment variables and configurations.
Open your text editor again and place the following text in it.
1 SPLUNK_START_ARGS=--accept-license --answer-yes
--no-prompt
73
2 SPLUNK_BEFORE_START_CMD_1=version
$SPLUNK_START_ARGS
3 SPLUNK_BEFORE_START_CMD_2=cmd python
/opt/splunk/bin/splunk_setup.py --configure
6. Save the file as splunkserver.env_file.
7. If you run your Docker Compose command to get it running again,

you will be able to have the container up and running and using the
env_file as set out in the compose file.
Persistent Data Storage

We are approaching the end of this chapter and we are going to touch
quickly on persistent data storage using volumes. We have been using
version: “3” as our docker-compose.yml file and since the introduction of
this version the creation of data containers has changed a little. In the
following example, we will add a data container to our compose file and
ensure that the volumes can be mounted on the Splunk forwarder so we
can then start to store our configuration data.
1. Start by removing all of the current running containers, images and

stopped containers.
2. We should still be in our directory where our docker-compose.yml file

is, so open it up again with your text editor.
3. We first need to make some changes to our Splunk forwarder

service. Start by adding in the following four lines to the end of the
service, which will mount the data volume on our forwarder as well as
stating that the forwarder cannot be created until the datavolume
service is created. This uses the depends_on instruction to ensure
that it is created first.
74
35 volumes:
36 - datavolume:/opt/splunk/etc
37 depends_on:
38 - datavolume
4. Then move below the forwarder and we need to now set up the
datavolume service with the following code.
39
40 datavolume:
41 image: busybox
42 volumes:
43 - datavolume:/opt/splunk/etc
This is simply creating a basic container with the busybox image and
setting up the volume that the forwarder will use.
5. As of version “3” of Docker Compose we now need to declare the

volume as well.
44
45 volumes:
46 datavolume:
6. Let’s start things up again and run docker-compose again.
7. Once the compose command has completed processing, a search of

the running processes will show that only the forwarder and the
server are running, but when we list all of the volumes running on our
host we will see the following.
docker volume ls
75
DRIVER VOLUME NAME

local
6bd40264e733c0460b4ae875d…79dd07575875a4c93673b12
local
8e4fdb60a7e5da23c02ea98ec…319440678ec76c28712ea5f
local
a3e70981049e8655ccf358c62…c72d7283hc89e30793a2c2d
local dockersplunk_datavolume
This is showing our new dockersplunk volume and three separate

volumes to account for the logs of the three other services.
8. Again we can use the exec Docker command option to create some
files on our forwarder.
docker exec -d ebc3bc0bf05b touch

/opt/splunk/etc/test.txt
9. If we now kill the running forwarder, remove the stopped container

and remove the forwarder image, we will be able to see if our data is
staying persistent.
docker kill <forwarder_container_id>

docker rm <forwarder_container_id>
docker rmi <forwarder_image_id>
10. If we now run the compose command, we will see that only the
forwarder needing to be started up.
11. Once the forwarder is back up again, we can now access it and
see if our test file has remained on the file system.
docker exec -it ebc3bc0bf05b /bin/bash; ls -l

-rw-r--r-- 1 root splunk 0 Jul 4 01:45

76
Although this is a very simply example, it means that we can start to use
persistent storage and containers for our configurations and if we needed
to, our indexed data.
Summary
Our work with Docker Compose has been brief but I am hoping it has been
worthwhile. We have seen how we can set up basic, one container
environments from scratch and by using existing Dockerfiles. We then
moved onto setting up Splunk environments with forwarders and Nginx
hosts.
Finally we took a look at moving some of the complexity away from our
compose file by using dedicated environment files as well as moving
persistent data onto volumes.
I hope you have been able to create a solid foundation of knowledge for
both Splunk and Docker even though we have only spent a short time
working. We worked hard on the first chapter to create and configure your
Docker containers from the command line, and worked on basic
configuration of the Splunk environment. We then started working with
Dockerfiles in chapter 2. This enabled us to further configure our Docker
containers and allow us to set up Splunk app and even index some sample
data that we were able to test our Splunk app with. Finally, we took a look
at Docker Compose and showed how we can use existing Dockerfiles to
work with the compose file, as well as setting an environment up from
scratch. We also looked at networks and keeping persistent data with
volumes.
Although we are at the end of this book, please note that there are two
appendix providing information on basic Splunk web interface usage,
including Splunk query language, and there is also a summary of all the
Docker commands used through this book.
77
78
79
80
Appendix 1
The Splunk
Web Interface
The following appendix is a guide to getting started with the Splunk Web
interface and tries to give you a quick rundown of how to administer and
search within the environment.
The Splunk Welcome Screen

This overview is designed to be as quick as possible to get you working
with minimal fuss. There are more in depth articles and tutorials on the web
if you need them, but let’s get started with the welcome screen that you are
presented with upon login.
1. This is the Splunk welcome icon, if you want to return to the welcome
screen at any time, click on this icon.
2. This allows you to manage and configure your Splunk Apps. It also
allows you to install and search for new apps available.
81
3. The App menu displays all of the installed apps you have on your
environment and allows you to move them around to fit your needs.
4. This is a collapsible panel which provides useful tutorials and
information to new users.
5. Currently empty, but this is where you can display your default
dashboard.
6. This is your menu bar providing you with different menus allowing
you to manage and configure your environment.
Splunk Administration Menu

In the previous image this is number six and provides you with a majority of
the control of your environment and application.
● On the image above it mentions “Administrator” but this is relevant to

your user and allows you to manage your own account and logout.
● Messages are general information and warnings issued about the
environment to users.
● Settings provides the main control and a bulk of your configuration
for your environment. It will be different depending on the level of
access you have on your environment. As an Administrator account
we can see all of the options above which provide us with the ability
82
to configure how our data is presented, how it is indexed and

inputted, specific system level configurations and access for users.
● The Activity drop down menu gives you a quick listing of running
jobs and searches.
● Help provides useful links to Splunk tutorials and help pages.
● The Find menu provides a quick search option to find specific items
that may be on your environment like apps, dashboards or data.
Adding Data To Splunk

Splunk give you three main ways to add data to your environment. You can
upload data directly and monitor specific log files and directories, you can
monitor specific ports like we have been doing in this book, and you can
receive data from a forwarder.
One really great thing about Splunk is that, when you are using the web
interface all of the forms are self explanatory and easy to use.
● Upload allows you to search for specific files and directories to index
data once or monitor the files or directories and update your index as
they update.
● Monitor controls streaming data coming into your system such as the
HTTP Event Collector we have been working with in this book, UDP
or TCP data coming into the environment, as well as collecting data
using scripts for API’s or databases.
● Forward allows you enable and configure forwarders to send data
into your environment.
83
Searching Basics In Splunk

The Splunk query language is a powerful tool to help you interpret, analyze
and present your data. It can also be one of the main reasons why people
are put off using Splunk at all. Although it can be complex and sometimes
overly complicated you don’t need to be a power user to start getting useful
information from the search bar.
1. Start With A Search Term, Eg; ERROR

Back in the day, if you didn’t know the index name you wanted to search
against, your searches would result in a blank green screen. But now all
you need is a term to search against and you can start to find useful
information. As an example let’s use the term “ERROR”. This will now
provide any log file that has this term inside it.
2. Wildcards To Expand Your Search Terms

If you are using a version of Splunk earlier than version 6.3, you’ll still need
to add an index for the term to be searched against but this is where you
can start to use wildcards. For example; our original search for ERROR
could now run as “index=* ERROR*”. This will search across all our
indexes and look for and log files with the term ERROR, ERRORS or even
ERRORnotanothererror.
3. Narrow Down Your Searches With Date And Time Range

For our first two examples we’ve used, we’d be searching across all the
data that your Splunk environment would have available in its index. This
could result in a very long search or a huge amount of data in your resulting
report. To narrow our search down further, the search interface allows you
to add a date or time range to narrow things down further. These time and
date values can be added directly into your the search bar but for now, to
the right of the search bar we are able to select from a drop down list of
available preset time and date ranges, or narrow things down even further.
84
4. Look Through Your Extracted Fields

Just as we have preset values for date and time, Splunk will also try and
provide useful information on your indexed data. When your data is
indexed, Splunk will perform field extraction across log files and return
useful information that can be used when searching and analysing data. On
the left of your search screen you will see a list of the more important and
hopefully relevant fields extracting relating to your search. When you click
on this data, Splunk will also give you a summary breakdown of the data
which can also help narrow things down further.
85
5. Add Functions To Provide More Depth

Use functions to present and chart your data. The search bar provides
useful hints on functions as you’re typing to help you complete your
function. Combine the search terms you are looking for with functions to
help transform and present your data. The three main functions we can
start with to help present our data are:
Stats
The stats function is used to aggregate statistics over the search results we
have previously found. For example, if we wanted to see the number of
errors by host, we would pipe our outputted results into | stats count by
host…the complete search would look similar to the search below:
index=* ERROR* | stats count by host
86
Chart
The chart function is considered a transformation command as it returns its
results in a table format. Much like stats these values can then be used in a
chart or visualization.
index=* ERROR* | chart values(process) by date_minute
Timechart
The timechart function allows you to create a time series chart with the
defined table of statistics. Even though you can create these charts with the
87
chart function, timechart is specifically designed to manage data and times

and the amount of time that each data point is spanned.
index=* ERROR* | timechart span=1h count by source
88
Appendix 2
Docker Command
Summary
Chapter 1 Summary of Commands
Display the Version of Docker running on your system.
docker --version
Run the basic hello-world Docker service.

docker run hello-world
Search for a Docker image or type that you need from Docker Hub.
docker search <image_name>
Pull the latest stable version of your required image.

docker pull <image_name>:latest
Display all running Docker containers.

docker ps
Display all running and stopped Docker containers.

docker ps -a
Running Splunk as a Docker container with environment variables

set.
splunk/splunk
89
Show all relevant information for a Docker container.

docker inspect <container_id>
View the logs of a Docker container.

docker logs <container_id>
Show the history of a Docker image.

docker history <image_name>
Access the shell of a running container.

docker attach <container_id>
Perform the touch command on a running container using exec.

docker exec -d <container_id> touch /tmp/test.txt
Access the bash shell of a running container.

docker exec -it <container_id> /bin/bash

Build your image from a Dockerfile and give it a name.
docker build -t <image_name> .
Run a Docker container in detached mode exposing port 8000.

docker run -d -p 8000:8000 <image_name>
View all Docker images available on your system.

docker images
Set your environment variables for your Docker User.

export DOCKER_ID_USER=”dockerhub_login_name”
Log in to Docker Hub.

docker login
Tag your image on your local host ready to push to Docker Hub.
90
docker tag <image_host_id>

$DOCKER_ID_USER/<docker_hub_name>
Push your new image to Docker Hub.

docker push $DOCKER_ID_USER/<docker_hub_name>
Commit changes from a running container to a new image.

docker commit <container_id> <committed_image_name>
Save a running container to an archive file.

docker save <container_id> > <saved_file_name>.tar
Kill all running containers, remove all stopped containers and remove
all Docker images on your system.
docker kill $(docker ps -q); docker rm -f $(docker
ps -a -q) docker rmi -f $(docker images -q)
Load a Docker image from an archived file.

docker load < <saved_file_name>.tar

Check the version of Docker Compose you are using.
docker-compose --version
Use Docker Compose to build and run your compose file in detached
mode.
Only start up one specific service in your compose file.

docker-compose up -d <service>
List all the current volumes on your system.

docker volume ls
91
92

Beginning Splunk With Docker

Uploaded by

Copyright:

Available Formats

Beginning Splunk With Docker

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Beginning Splunk With Docker

Uploaded by

Copyright:

Available Formats

Beginning Splunk With Docker - Vincent Sesto

1 Docker Fundamentals With Splunk 11

2 Building Splunk Images And Using Dockerfiles 35

3 Building Splunk Environments With Docker Compose 61

The Spluk Web Interface 81

Docker Command Summary 89

What this book covers

What you need for this book

Who this book is for

A block of code is set as follows:

Any command-line input or output is written as follows:

echo "# SplunkAndDocker"

New terms and important words are shown in bold.

Downloading the example code

Please contact us at vince.sesto@gmail.com with a link to the suspected

Docker creates everything you need and places it into an isolated

Docker Key Terminology

Docker version 17.05.0-ce, build 89658be

Docker Hello World

docker run hello-world

Unable to find image 'hello-world:latest' locally

Hello from Docker!

To generate this message, Docker took the following steps:

To try something more ambitious, you can run an Ubuntu

Share images, automate workflows, and more with a free

For more examples and ideas, visit:

CONTAINER ID IMAGE COMMAND CREATED STATUS

● If we use the -a or “--all” flag with the ps option, we will be able to

CONTAINER ID IMAGE COMMAND

We can see that out container “Exited (0) 2 minutes ago”.

docker logs 2d5e1414661e

REPOSITORY TAG IMAGE ID CREATED

docker search splunk

adobeplatform/docker-splunk Splunk implementation in

Our search comes up with over 20 images available, so we have cut

The cool thing about Docker Hub is that, if you’re interested in a

● We will be getting familiar with the supported Splunk image, so we

docker pull splunk/splunk:latest

latest: Pulling from splunk/splunk

version, Docker will pull the latest version by default. If we require a

REPOSITORY TAG IMAGE ID

docker run -d -e "SPLUNK_START_ARGS=--accept-

● Although it doesn’t look like much is currently happening, we can use

CONTAINER ID IMAGE COMMAND

8089/tcp, 8191/tcp, 9997/tcp, 0.0.0.0:8000->8000/tcp

If we wanted to set up more Splunk containers on the same system, we

Monitoring Our Docker Container

docker inspect <container_id>

We have trimmed the output we are displaying a little as the full

● As we discussed earlier, by using the logs command option to view

docker logs <container_id>

The Splunk web interface is at http://72769c19065a:8000

docker history hello-world

IMAGE CREATED CREATED BY

We have used the hello-world image to view history here to once

● Finally, we can use the stats Docker command option to provide

docker stats <container_id>

Interacting With Your Docker Container

● If we want to attach to a running container to see what is going on,

docker attach <container_id>