Artillery API
Artillery API
Artillery API
===========================
Introduction to Artillery
Artillery is a powerful load testing tool to test WebSocket and http based
applications.
In this course you will learn about load testing, writing sample scripts in
artillery, handling http requests and writing data driven scripts.
You will also see various debugging options and integration with Jenkins and Faker.
Performance testing
Performance testing determines the speed at which a system performs under a
particular load.
It helps us check the speed, stability, and scalability of the system.
Benefits of performance testing:
Improves user experience on websites and application.
Checks new release readiness for deployment to production.
Ensures high level of resilience and performance of the application.
Performance Terminologies
Let us understand a few commonly used terminologies in performance testing.
Resource Utilization
Response Time
Throughput
Number in System
Utilization
Visit Counts
Flow balance
Little’s Law
Application of Little’s Law
Resource Utilization
Resource utilization is the cost in terms of system resources. The primary
resources are CPU, memory, disk I/O, and network I/O.
Response Time
Response time is the time difference between entry time and exit time.
Response Time = Exit Time – Entry Time
Response time can be ,
Time to service a web request
Time to service a database transaction
Time spent in disk subsystem
Response time Metrics could be
Average Response Time
Variance in Response Time
Throughput is the number of requests that can be served by an application per unit
time.
It can vary depending on the load.
It is typically measured in terms of requests per second.
Throughput = Number of Departures Per Unit Time
Number of System
The number of entities in a system at any given time is the difference between the
total number of arrivals request and the total number of departure requests.
It can be any of the following:
Number of concurrent users in a web system
Number of customers at a bank
Number of tellers in a bank
Average number in system is an average over time
Utilization
Resource Utilization is the fraction of time a resource is busy to the total
measurement interval. Utilization is by definition a time average.
Utilization = Total Busy Time / Total Measurement Interval
Types of Utilization:
CPU Utilization
Disk Utilization
Network Utilization
Little's law
Little's law is the most used queuing theory.
Average number of users in a stable system(N) is arrived when the average time(W)
spent by a user in the system is multiplied by the average arrival rate( λ) of the
users.
Littles Law will be N = λ*W
Little's Law(Cont..)
For Web/App and Database server:
Little's Law N = Throughput * (Response time + Think Time).
Throughput is the exit rate or a number of request process per unit.
Response time is the time spent by the user in the system. It includes both the
wait and service time.
The Little's law assumes a stable system.
Throughput Computation
Let us understand this law with another example.
You have 5 shoppers with only one product in their shopping basket .
Each shopper spends 1 min for billing .
So, Throughput will be 1 Transaction per min.
If the same number of users carry more products in their shopping basket and spend
2 mins at the billing counter, then Throughput will be 0.5 transactions per min.
Note: Wait time is negligible.
Introduction to Artillery
Artillery is powerful load testing tool which helps developers,
Develop easy-to-use applications.
Build application faster .
Make application more scalable and resilient
Artillery Features
The following are the key features of Artillery:
Multiple protocols - Can perform load test for HTTP, WebSocket, and socket.io
application .
Scenarios - Can be used to test multiple test steps scenarios interaction in API or
web application .
Performance metrics - Provides details of performance metrics for latency, requests
per second, concurrency and throughput.
Scriptable - Can write custom script using JavaScript to perform any action .
High performance - Can generate serious load even on modest hardware.
Integrations - Can integrate it will Datadog, Librato, InfluxDB, and statsd .
Extensible- Can generate custom reporting plugins and protocol engines .
HTML reporting - Can generate report in html form .
Parameterization with CSV files.
We can get Response Time using -- Difference between Exit Time and Entry Time.
Stress testing means verifying application’s stability and capabilities -- False
Which of the following is correct about Little’s law? -- Both the below options are
correct
It is a product of average arrival rate and average time spent by the user in the
system
Sum of response time and Think time multiplied with Throughput will give the number
of system.
Install Artillery
The following are the pre-requisites to install artillery in your local system:
Node.js pre-installed as artillery work on Node.js
Version of Node.js should be v6+
Follow these steps to install artillery:
Enter npm install -g artillery in your terminal.
npm stands for Node Package Manager.
-g defines global installation.
Artillery Playground
In this course, we will use Katacoda playground to install artillery. Follow these
steps:
Launch Katacoda with Node.js v6+ preinstalled.
Type npm install artillery in terminal.
We do not use -g in the installation command as it is not being installed globally.
Verify the artillery installation using the command artillery dino. Here dino
stands for the dinosaur. If dinosaur image appears, it means artillery is
successfully installed.
Help Command
To begin with, let us understand the help command and its functionality in
artillery.
artillery -help
Lists all the commands available along with its options. Some of the commands are :
artillery run - To run a script
artillery quick - To run a quick test without test script
artillery report - To create a report from a JSON file created by artillery while
running the test
artillery convert <file> - To convert JSON to YAML and vice Versa
Quick Test Command
Let us execute our first test .
Quick test command is used to run a test without the test script.
Here is the scenario:
You want to pump a quick load on mercury site with 10 concurrent virtual users for
50 times to GET a request.
artillery quick -c 10 -n 50 https://reqres.in/api/users/2
where -c is the count of virtual users to perform load test and -n is the number of
requests each user sends.
A report for each user along with final report will be auto generated after
execution.
config:
target: 'https://reqres.in'
phases:
- duration: 1
arrivalRate: 10
scenarios:
- name:
- loop:
-get:
url: "/"
Here,
arrivalRate holds the number of concurrent users which is 10
URL for testing is set in target
Additionally, action is set to get for the target url
config:
target: 'https://reqres.in/'
phases:
- duration: 60
arrivalRate: 20
First Script - Scenario Section
Let us move to the scenarios section of the script.
User can interact with APIs using get,post,put,patch and delete methods. This will
be covered in upcoming topics.
For the current scenario, let us have the user perform get action on the API.
scenarios:
- flow:
- get:
url: "/api/users/2"
First Script
So we are done with our scripting. Let us include config and scenarios script
together in sample.yml.
config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- get:
url: "/api/users/2"
More on Config
Let's take a deeper look at the Config section. Following are some of the actions
that can be included:
Environments
Environment action can define the target URL for different environments. That is, a
dev URL could be different from a stage or a production URL.
config:
target: "https://reqres.in/"
phases:
-
duration: 10
arrivalRate: 1
environments:
production:
target: "http://dummy.prod:40403"
staging:
target: "http://dummy.127.0.0.1:3002"
phases:
-
duration: 20
arrivalRate: 1
Phases
It defines the load or wait time. It has a ramp-up phase followed by a slightly
intense phase. Here is how the phase script would look like.
phases:
-
duration: 300
arrivalRate: 5
name: "Warm-up"
#Create 5 virtual users coming at an
average of 5 minutes of interval.
-
pause: 10
#It will wait for 10 seconds
-
duration: 60
arrivalCount: 30
#Create 30 virtual user in duration of 60
seconds.
-
duration: 600
arrivalRate: 50
name: "High load phase"
#We can name the arrival rate
Here,
arrivalRate - defines the arrival of virtual user for the set duration
arrivalCount - defines the number of users to create over a period of time.
pause - wait time
More on Scenarios
A scenario is a series of steps that need to be run sequentially. It represents the
series of calls generated by a simulated user.
Action in scenarios are
name - name of the scenario
weight- priority of the scenarios. For example, if there are 3 scenarios with
weight as 1,2 and 6.
priority for scenario 1 will be 1/9 or 11.11%
priority for scenario 2 will be 2/9 or 22.22%
priority for scenario 3 will be 6/9 or 66.66%
flow - Operates on GET and POST requests. You can use think step to pause the
execution.
If we have defined the priority of execution of three phases as 2,3,7, which of the
following is correct? -- None of the above as none is matching 27.27%
What does think argument stand for in artillery script? -- Both options are correct
Time difference between start and end of the request.
Will pause the next response for the defined duration of time.
Priority of the scenarios are defined in -- Weight
arrivalCount is defined as the arrival of the virtual users for the duration of
time. -- False
Target is defined in the scenarios section. -- False
HTTP Action
Let us take a look at the list of HTTP Actions:
GET
POST
PUT
DELETE
config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 10
scenarios:
- flow:
- get:
url: "/api/users/2"
Go ahead and save your test script. Example: sampletest.yml.
To run the script use
artillery run sampletest.yml
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
scenarios:
- flow:
- put:
url: "/api/users/2"
json:
job: "SE"
Let's execute and check the output.
artillery run sample.yml
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
scenarios:
- flow:
- delete:
url: "/api/users/2"
Now execute the script
artillery run sample.yml
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
variables:
id:
- "1"
- "2"
- "3"
- "4"
Here, name of the variable is id. Please note that the variable would have multiple
values and not a single value.
scenarios:
- flow:
- get:
url: "/api/users/{{ id }}"
Here we have defined variables (id) in curly braces like {{ variable }}.
Now you can execute the script.
artillery run sample.yml
1,Artist
2,Doctor
3,Engineer
4,Politician
5,Architect
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
payload:
fields:
- "id"
- "job"
scenarios:
- flow:
- post:
url: "/api/users/{{ id }}"
json:
job: "{{ job }}"
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
payload:
path: "./data.csv"
fields:
- "id"
- "job"
scenarios:
- flow:
- post:
url: "/api/users/{{ id }}"
json:
job: "{{ job }}"
1 config:
2 target: 'https://reqres.in'
3 - phases:
4 - duration: 60
5 arrivalRate: 10
6 scenarios:
7 - flow:
8 - get:
9 url: "/api/users/2"
Reading Reports
After every execution, reports are generated. Let us take a look at these reports,
a sample of which is shown below.
Scenarios launched: 5
Scenarios completed: 5
Requests completed: 58
RPS sent: 0.86
Request latency:
min: 102.4
max: 3067.5
median: 325.5
p95: 2118.5
p99: 3020
Scenario duration:
min: 56745.4
max: 67339.1
median: 59275.6
p95: NaN
p99: NaN
Codes:
200: 58
You will notice that a logfile like artillery_report_xxxxxxx_xxxxxx.json is
created.
Understanding Report
Let us try to understand this report.
Scenarios Launched - Number of virtual users created in the preceding time.
Scenarios Completed - Number of virtual users who have completed the scenarios in
the preceding time. Please note, it is the number of completed sessions and not the
number of sessions started and completed in a time interval .
Requests Completed - Number of HTTP requests and responses or websocket messages
sent.
RPS sent - Average number of requests per second completed in the preceding time or
throughout the test.
Understanding Report
-Request latency - It is in milliseconds, and p95 and p99 values are the 95th and
99th percentile values (a request latency p99 value of 500ms means that 99 out of
100 requests took 500ms or less to complete)
Scenario duration - similar to Request latency but the value changes
Codes- count of HTTP response codes
If there is a NaN reported as value anywhere, it means there were not enough
responses to calculate the percentile.
Whenever the execution of the script is successful, it results in code being
reported as 200.
In case of any errors like socket timeouts, it results in an error in the report.
config:
target: "https://reqres.in"
http:
timeout: 10
Notice config.http.timeout in config section. timeout: 10 means that a response
should be sent within 10 seconds. Else, the request will abort.
config:
target: "https://reqres.in"
http:
pool: 15
Here, pool value 15 means, all the http requests from all the virtual users will be
sent over the same 15 connections.
Logging
Scenario: Consider you want to print value(s) on the console. Let's say you want to
print the Id generated by setting user name and job.
You can do this by passing the value to log.
Sample:
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
capture:
json: "$.id"
as: "id"
- log: "Id value: {{ id }}"
Seting Header
Scenario: Consider that you need to set a header for the script.
You can do this using headers in scenario section as shown here:
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- get:
url: "/api/users"
headers:
X-My-Header: "2343"
scenarios:
-
flow:
-
loop:
-
get:
url: "/api/users/2"
count: 100
In case the count in the loop is removed or omitted, the script would run for
infinitely.
loop can be an array and so any number of requests can be defined.
Socket.io scripts:
Socket.io
Socket.io is a method to "push" messages from the client to the server and vice
versa using symmetric syntax.
It is a javascript library which is used for making a real-time application such as
chat messenger and online games.
There are very few load testing tool in market that supports socket.io. Artillery
is one of the few.
Let us play around with socketio and artillery using a sample chat application. You
could alternately search for socket.io online and play try it.
Socket.io Testing
Scenario: Consider that you want to perform heavy load testing on a chat
application .
The load test could be for various reasons:
To check the traffic the app can handle.
To test if you can scale application performance by adding more app servers behind
the load balancer .
To perform CPU profiling in order to improve runtime performance.
Using Artillery, one can send data to the server and optionally wait for and verify
responses that come back.
To enable socket.io engine, set the engine attribute to socketio
To perform HTTP request with socket.io, emit action needs to be called.
Understanding Socket.io
To understand about socket.io, click on sample chat application. Open Developer
tools -> networks->WS option.
Now Refresh the URL to see some socket responses coming up in the name section. You
will see many such responses appearing. Click on any of the responses to understand
the different emits needed for the script.
Type your name in the text box. You will find the green highlighted response which
is the request made by typing the name.
config:
target: "https://socketio-chat.now.sh"
phases:
- duration: 10
arrivalRate: 1
Here, arrivalRate as 1 means calling only one user at each time interval.
Socket.io Script
So, the scenarios section of the script will look like this:
scenarios:
- name: "Our First Artillery script"
weight: 15
engine: "socketio"
flow:
- get:
url: "/"
- emit:
channel: "add user"
data: "Ranjan"
- think: 5
- emit:
channel: "new message"
data: "Hello"
- think: 60
Here Weight is the number of times each scenario will be executed across the total
number of virtual users.
Go ahead and combine config and scenarios section scripts and execute the script.
artillery run sample.yml
Datadriven Approach
In our previous example, we hard-coded the username or message in the script.
However, this is not a good practice.
In real-time scenarios, there could be multiple users typing different messages.
Let us learn how to create random users and random messages.
Artillery has one built-in function to generate random alphanumeric string
$randomString().
To generate random messages, small JavaScript can be written as shown.
'use strict';
module.exports = {
setMessage: setMessage
};
const MESSAGES = [
'Do you know Play application',
'I really love reading on play',
'Play app is not about learning it is about playing with course',
'It has games as well',
'It is the best learning app and you get coupons and gifts with knowledge',
'Learn and earn on play',
'I am getting addicted to PLAY',
'Love play'
];
config:
target: "https://socketio-chat.now.sh"
phases:
- duration: 10
arrivalRate: 5
variables:
greeting: ["hello", "namaste", "Vanakkam", "Kon'nichiwa", "Bonjour", "hola"]
processor: "./functions.js"
scenarios:
- name: "data driven in socketio"
weight: 10
engine: "socketio"
flow:
- get:
url: "/"
- emit:
channel: "add user"
data: "Ranjan-{{ $randomString() }}"
- emit:
channel: "new message"
data: "{{ greeting }}"
- loop:
- function: "setMessage"
- emit:
channel: "new message"
data: "{{ message }}"
- think: 10
count: 10
- think: 60
WebSocket
WebSocket stabilizes connection between the user's browser and a server.
Let us understand this concept with a few samples:
Consider a chat application like Sametime. Messages appear in the chat box without
manually refreshing the app. This is due to WebSocket.
Likewise, a live match score or points table auto-updates on the web page without
any manual refresh. This is due to WebSocket.
WebSocket Script
We have seen scripting process of HTTP and socketio. Scripting for WebSocket is no
different. The additional input to note is
Set engine value as ws
Actions supported in WebSocket are sent and think(wait time)
Here's the script.
config:
target: "wss://echo.websocket.org"
phases:
- duration: 10
arrivalRate: 5
scenarios:
- name: "First websocket Script"
engine: "ws"
flow:
- send: "Check"
- think: 1
- send: "world"
Here, engine value is set as ws. Texts like "world" are set with a wait time of 1
second before sending the second text.
Inline Variables
You have already learnt how to use an inline variable in HTTP request. The same can
be done for WebSocket too.
config:
target: "wss://echo.websocket.org"
phases:
- duration: 6
arrivalRate: 2
variables:
text:
- "Check"
- "World"
scenarios:
- name: "First websocket Script"
engine: "ws"
flow:
- send: "{{text}}"
- think: 1
- send: "{{text}}"
Execute the script and check the result. You may try Data-Driven approach too in
the similar fashion.
Debugging Profile
Debug environment can be created in the script itself. For this, change the config
scenario sections and create only one virtual user.
Let us try the Script.
config:
target: "https://reqres.in"
environments:
debug:
phases:
- duration: 5
arrivalCount: 1
scenarios:
- name: "Debug Scenario"
flow:
- get:
url: "/api/users/2"
Notice environment -> debug -> phases -> duration and arrivalCount.
Let us execute the script with debug option.
artillery run -e debug sample.yml
Faker.js
Now you know how to generate random data for input to simulate real time scenario.
Likewise, you can use a separate data sheet as an input too. However, this may be
little cumbersome process.
How about having a ready-made API that provides these random data which can be
simply called in the script.
Faker is a node API that solves our purpose. As the name suggests, Faker creates
random fake data for our use.
Artillery script
Let us consider a sample post request for registration. Here, two inputs are needed
for registration in the reqres API.
email
password
config:
target: "https://reqres.in"
phases:
- duration: 60
arrivalRate: 20
scenarios:
- name: "Register a user"
flow:
- post:
url: "/api/register"
json:
email: "ranjan@play.me"
password: "FrscoPlay"
This will be the base script while integrating faker.
Jenkins
Configuring Jenkins
To try out Jenkins integration with artillery in the local machine. The following
steps are involved for configuration:
Install Jenkins in your local machine.
Start Jenkins server. Use the command Jenkins to start the server. To run Jenkins
server on a specific port, say 9090, use Jenkins --httpPort=9090.
Please note, by default, Jenkins server will start on 8080 port. In case 8080 port
is already taken by other application server, you may use 9090 port.
Login to Jenkins.
Click on Manage Jenkins->Select Manage Plugins.
Click on the Available tab.
Enter NodeJs in the Filter search box and click.
Select Nodejs, click on the checkbox and install. If NodeJs is already installed,
it will be in Installed tab.
You can verify your installation from Installed tab, search for Nodejs in installed
tab.
Click on Manage Jenkins -> Global Tool Configuration.
Scroll down to Nodejs and click on the NodeJs installations.. button.
Enter the Node name.
Select the node version you want to use. Ensure you select the latest version of
the nodejs.
Select Global npm packages to install so that we are using artillery to be
installed using npm package. Enter artillery.
Enter the Global npm packages refresh hours.
Click on Apply and the save.
Artillery Practicals:
---------------------
Create a user:
--------------
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Harley"
job: "Software Engineer"
Log:
-----
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
capture:
json: "$.id"
as: "id"
- log: "Id value: {{ id }}"
Loop:
-----
config:
target: 'https://reqres.in'
phases:
- duration: 1
arrivalRate: 10
scenarios:
- name:
- loop:
-get:
url: "/"
Timeout:
--------
config:
target: "https://reqres.in"
http:
timeout: 10
Rampup:
-------
phases:
-
duration: 300
arrivalRate: 5
name: "Warm-up"
#Create 5 virtual users coming at an
average of 5 minutes of interval.
-
pause: 10
#It will wait for 10 seconds
-
duration: 60
arrivalCount: 30
#Create 30 virtual user in duration of 60
seconds.
-
duration: 600
arrivalRate: 50
name: "High load phase"
#We can name the arrival rate
config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- get:
url: "/api/users/3"
Question: Create ramp up arrival rate from 10 to 20 over 2 minutes to create user
account. Define the variable request URL, name and job in the Inline variable.
Use Reqres application as base and request URL.
config:
target: 'https://reqres.in'
phases:
- duration: 120
arrivalRate: 10
- duration: 120
arrivalRate: 20
scenarios:
- flow:
- put:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- put:
url: "/api/users/2"
json:
name: "Ranjan"
job: "Artist"
Question: Create ramp up arrival rate from 10 to 20 over 1 minute to create the
user account. Define the variable request URL, name, and job in the Inline
variable. Search the newly created user using ID. Use Reqres application as base
and request URL.
config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 10
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- put:
url: "/api/users/2"
json:
name: "Ranjan"
job: "Artist"
- post:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
capture:
json: "$.id"
as: "id"
- get:
url: "{{ id }}"
--------------------------------
config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 10
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
capture:
json: "$.id"
as: "id"