Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                

Artillery API

Download as txt, pdf, or txt
Download as txt, pdf, or txt
You are on page 1of 25

Artillery - API Freighting:

===========================

Introduction to Artillery
Artillery is a powerful load testing tool to test WebSocket and http based
applications.
In this course you will learn about load testing, writing sample scripts in
artillery, handling http requests and writing data driven scripts.
You will also see various debugging options and integration with Jenkins and Faker.

Performance testing
Performance testing determines the speed at which a system performs under a
particular load.
It helps us check the speed, stability, and scalability of the system.
Benefits of performance testing:
Improves user experience on websites and application.
Checks new release readiness for deployment to production.
Ensures high level of resilience and performance of the application.

Roles and Importance of Performance


Various project stakeholders are concerned about the application performance in
different ways.
An architect would need to balance scalability and performance compared to other
quality services.
A developer should know when to start performance testing and the level of software
optimization needed.
A tester should validate the ability of the application to support the expected
workload .
An administrator should check for application compliance to service level
agreements and create effective growth plans.
A project sponsor would be interested in the performance of the application with
lower total cost of ownership of the software that the organization creates.

Reactive Vs Proactive Approach


Often, performance testing is neglected until the system testing is completed. In
some cases it is even done post initial deployment. Issues related performance, if
found later, may be critical to fix with a simple solution such as - adding some
hardware.
In Reactive Performance approach,
A poorly designed system cannot be fixed to perform better
Hardware expenses may go high
Total cost of ownership may increase
In Proactive performance approach,
The team would know where to focus on optimizing effort
The need to tune and redesign can be reduced thus saving costs
The need for frequent hardware upgrade may be reduced with less expensive hardware
It reduces operational costs
One must adopt proactive performance approach while building applications.

Performance Testing Types


Different types of performance testing:
Load Testing - Checks application behavior underpeak ornormal load condition.
Stress Testing - Validates application behavior when you push load beyond peak or
normal load condition .
Spike testing - Verifies applications stability by varying degrees of load in
varying time periods .
Endurance testing - Verifies application stability with a significant load extended
over a significant period of time.
Scalability testing - Verifies the application's capability to scale
Volume Testing - Verifies the application with a huge volume of data, also called
as flood testing.

Performance Terminologies
Let us understand a few commonly used terminologies in performance testing.
Resource Utilization
Response Time
Throughput
Number in System
Utilization
Visit Counts
Flow balance
Little’s Law
Application of Little’s Law

Resource Utilization
Resource utilization is the cost in terms of system resources. The primary
resources are CPU, memory, disk I/O, and network I/O.

Response Time
Response time is the time difference between entry time and exit time.
Response Time = Exit Time – Entry Time
Response time can be ,
Time to service a web request
Time to service a database transaction
Time spent in disk subsystem
Response time Metrics could be
Average Response Time
Variance in Response Time

Throughput is the number of requests that can be served by an application per unit
time.
It can vary depending on the load.
It is typically measured in terms of requests per second.
Throughput = Number of Departures Per Unit Time

Number of System
The number of entities in a system at any given time is the difference between the
total number of arrivals request and the total number of departure requests.
It can be any of the following:
Number of concurrent users in a web system
Number of customers at a bank
Number of tellers in a bank
Average number in system is an average over time

Utilization
Resource Utilization is the fraction of time a resource is busy to the total
measurement interval. Utilization is by definition a time average.
Utilization = Total Busy Time / Total Measurement Interval
Types of Utilization:
CPU Utilization
Disk Utilization
Network Utilization

Little's law
Little's law is the most used queuing theory.
Average number of users in a stable system(N) is arrived when the average time(W)
spent by a user in the system is multiplied by the average arrival rate( λ) of the
users.
Littles Law will be N = λ*W

Little's Law Example


Let’s understand this law with an example.
Rohan is a performance engineer for a movie streaming application. When he reviewed
the application response there were two findings.
Every hours, 200 movie watchers register for the streaming app.
Each user spend 30 minutes watching a specific movie.
Now, Rohan is interested in finding the number of movie watchers in the system at
any point of time.
Here is the data point:
Wait time (W) = 30 minutes = 30/60 Hrs
User arrival per hour λ = 200/hr
So number of movie watchers in the system will be:
N = λ*W = (30 * 200)/60 = 100 .
According to Little’s law, even though 200 movie watchers enroll per hour and spend
30 minutes on a movie, there are only 100 movie watchers in the system at one point
of time. 99 users are in the queue and 1 is being served by the system.

Little's Law(Cont..)
For Web/App and Database server:
Little's Law N = Throughput * (Response time + Think Time).
Throughput is the exit rate or a number of request process per unit.
Response time is the time spent by the user in the system. It includes both the
wait and service time.
The Little's law assumes a stable system.

Throughput Computation
Let us understand this law with another example.
You have 5 shoppers with only one product in their shopping basket .
Each shopper spends 1 min for billing .
So, Throughput will be 1 Transaction per min.
If the same number of users carry more products in their shopping basket and spend
2 mins at the billing counter, then Throughput will be 0.5 transactions per min.
Note: Wait time is negligible.

Introduction to Artillery
Artillery is powerful load testing tool which helps developers,
Develop easy-to-use applications.
Build application faster .
Make application more scalable and resilient

Artillery Features
The following are the key features of Artillery:
Multiple protocols - Can perform load test for HTTP, WebSocket, and socket.io
application .
Scenarios - Can be used to test multiple test steps scenarios interaction in API or
web application .
Performance metrics - Provides details of performance metrics for latency, requests
per second, concurrency and throughput.
Scriptable - Can write custom script using JavaScript to perform any action .
High performance - Can generate serious load even on modest hardware.
Integrations - Can integrate it will Datadog, Librato, InfluxDB, and statsd .
Extensible- Can generate custom reporting plugins and protocol engines .
HTML reporting - Can generate report in html form .
Parameterization with CSV files.

How Artillery Models Load


Virtual users arrive at the chosen APIs for testing.
Each user will execute a scenario, which is a sequence of steps to interact with
API.
Users then arrive in the phase.
Every virtual user is independent from other virtual users.
Load test duration is the time taken by the user to perform in the phase duration.
You will learn more about how a scenario or phase look like in upcoming cards.

Realistic Load Modeling


In the realistic load model,
Each virtual user will keep hitting the requested API irrespective of the scenario
completion status of the others virtual users.
The individual virtual user will execute the scenarios sequentially, waiting for a
response before continuing with the other scenarios.
Each user will be mapped to a new TCP connection.

We can get Response Time using -- Difference between Exit Time and Entry Time.
Stress testing means verifying application’s stability and capabilities -- False
Which of the following is correct about Little’s law? -- Both the below options are
correct
It is a product of average arrival rate and average time spent by the user in the
system
Sum of response time and Think time multiplied with Throughput will give the number
of system.

Install Artillery
The following are the pre-requisites to install artillery in your local system:
Node.js pre-installed as artillery work on Node.js
Version of Node.js should be v6+
Follow these steps to install artillery:
Enter npm install -g artillery in your terminal.
npm stands for Node Package Manager.
-g defines global installation.

Artillery Playground
In this course, we will use Katacoda playground to install artillery. Follow these
steps:
Launch Katacoda with Node.js v6+ preinstalled.
Type npm install artillery in terminal.
We do not use -g in the installation command as it is not being installed globally.
Verify the artillery installation using the command artillery dino. Here dino
stands for the dinosaur. If dinosaur image appears, it means artillery is
successfully installed.

Verify Artillery Installation


In this section, we will learn to verify artillery installation using the command.
Enter the command artillery dino
Where dino stands for the dinosaur.
If dinosaur image appears then we have successfully installed artillery.

Help Command
To begin with, let us understand the help command and its functionality in
artillery.
artillery -help
Lists all the commands available along with its options. Some of the commands are :
artillery run - To run a script
artillery quick - To run a quick test without test script
artillery report - To create a report from a JSON file created by artillery while
running the test
artillery convert <file> - To convert JSON to YAML and vice Versa
Quick Test Command
Let us execute our first test .
Quick test command is used to run a test without the test script.
Here is the scenario:
You want to pump a quick load on mercury site with 10 concurrent virtual users for
50 times to GET a request.
artillery quick -c 10 -n 50 https://reqres.in/api/users/2
where -c is the count of virtual users to perform load test and -n is the number of
requests each user sends.
A report for each user along with final report will be auto generated after
execution.

What is the help command for artillery? -- artillery -help


What is the command to verify artillery? -- artillery dino
Artillery can be installed on Nodejs v3. -- False

Quick Test Using Scripts


Let us consider another example.
Consider artillery quick -c 10 -n 50 https://reqres.in/ as the quick test command.
Let us try to convert this into script. Hang on to learn more about this script in
detail a little later.

config:
target: 'https://reqres.in'
phases:
- duration: 1
arrivalRate: 10
scenarios:
- name:
- loop:
-get:
url: "/"
Here,
arrivalRate holds the number of concurrent users which is 10
URL for testing is set in target
Additionally, action is set to get for the target url

How to Write Test Script?


Artillery test script has two sections.
Configuration section
- `target` - API for test
- `phase` - load phase definition
Scenarios section
- Holds how the user interacts

First Script - Config Section


To write the test script, we will be using reqres fake API provider. Scripts are
written in .yml file. So go ahead and do touch sample.yml or vi sample.yml.
In Config section, let us define the target and phase
target: pass the value of the URL to hit
phase: define the duration and number of virtual users.
Our script for config will look like this:

config:
target: 'https://reqres.in/'
phases:
- duration: 60
arrivalRate: 20
First Script - Scenario Section
Let us move to the scenarios section of the script.
User can interact with APIs using get,post,put,patch and delete methods. This will
be covered in upcoming topics.
For the current scenario, let us have the user perform get action on the API.

scenarios:
- flow:
- get:
url: "/api/users/2"

First Script
So we are done with our scripting. Let us include config and scenarios script
together in sample.yml.

config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- get:
url: "/api/users/2"

Our final script will look like this.


To run this script, use
artillery run sample.yml
Isn't this quite easy to write?

More on Config
Let's take a deeper look at the Config section. Following are some of the actions
that can be included:

target - holds the base URL for testing .


environments - defines the list of environments and associated targets.
phases - defines the recurrence of the request and the duration of the time.
payload - imports data.
defaults - sets default headers applicable to all the HTTP requests.
timeout - defines wait time for the server before responding.

Environments
Environment action can define the target URL for different environments. That is, a
dev URL could be different from a stage or a production URL.

config:
target: "https://reqres.in/"
phases:
-
duration: 10
arrivalRate: 1
environments:
production:
target: "http://dummy.prod:40403"
staging:
target: "http://dummy.127.0.0.1:3002"
phases:
-
duration: 20
arrivalRate: 1

To execute the script for a particular environment, use -e argument followed by


environment name.
artillery run -e production sample.yml
Please note, the above code snippet will fail to execute since the URLs are
invalid.

Phases
It defines the load or wait time. It has a ramp-up phase followed by a slightly
intense phase. Here is how the phase script would look like.

phases:
-
duration: 300
arrivalRate: 5
name: "Warm-up"
#Create 5 virtual users coming at an
average of 5 minutes of interval.
-
pause: 10
#It will wait for 10 seconds
-
duration: 60
arrivalCount: 30
#Create 30 virtual user in duration of 60
seconds.
-
duration: 600
arrivalRate: 50
name: "High load phase"
#We can name the arrival rate
Here,
arrivalRate - defines the arrival of virtual user for the set duration
arrivalCount - defines the number of users to create over a period of time.
pause - wait time

More on Scenarios
A scenario is a series of steps that need to be run sequentially. It represents the
series of calls generated by a simulated user.
Action in scenarios are
name - name of the scenario
weight- priority of the scenarios. For example, if there are 3 scenarios with
weight as 1,2 and 6.
priority for scenario 1 will be 1/9 or 11.11%
priority for scenario 2 will be 2/9 or 22.22%
priority for scenario 3 will be 6/9 or 66.66%
flow - Operates on GET and POST requests. You can use think step to pause the
execution.

If we have defined the priority of execution of three phases as 2,3,7, which of the
following is correct? -- None of the above as none is matching 27.27%
What does think argument stand for in artillery script? -- Both options are correct
Time difference between start and end of the request.
Will pause the next response for the defined duration of time.
Priority of the scenarios are defined in -- Weight
arrivalCount is defined as the arrival of the virtual users for the duration of
time. -- False
Target is defined in the scenarios section. -- False

HTTP Action
Let us take a look at the list of HTTP Actions:
GET
POST
PUT
DELETE

HTTP GET Request


GET requests data from specified resources.
Let us take a scenario where you want to list the user details from page 2.
Base URL would be https://reqres.in.
User location would be /api/users.
To find user detail from page 2, call the query string represented as ? and use
page=2 after query string.
So final URL for the request will look like /api/users?page=2.
Here, Base URL will be the same. However, Request URL will change depending on the
request.
Once the Get request for the first user is successful, a response of 200 (means OK)
will be received along with listing the user details of page 2.

HTTP GET Request Example


Scenario: To write a script where 10 virtual users request for user details from
page 2 of base URL https://reqres.in/ lasting for 60 seconds`.
So, the need is 10 virtual users, duration is 60 seconds and request URL will be
/api/users?page=2.
So, your code would look like this:

config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 10
scenarios:
- flow:
- get:
url: "/api/users/2"
Go ahead and save your test script. Example: sampletest.yml.
To run the script use
artillery run sampletest.yml

HTTP POST Request


HTTP POST request offers data to be prepared for a detailed resource.
Some examples where POST will be used are
The user wants to register for a site
The user wants to login to the site
The user wants to create an another user's information

HTTP POST Request Example


Let us try this example
Scenario: User wants 5 virtual users to create a new user account that lasts for 50
seconds. Name and job is mandatory for the user account.
Given
Base URL as https://reqres.in/
Request URL as /api/users
Script looks like this:
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Harley"
job: "Software Engineer"
Execute this code using
artillery run sample.yml
where sample.yml contains the above mentioned code.

HTTP PUT request


Now, let us understand put request first before moving to scripting.
PUT request allows user to update API data.
For example, the job description of a user gets updated from the front-end
application. This will in turn trigger the PUT request on the API at the back-end.

PUT Request Example


Scenario: User wants 15 virtual users to perform update job request for userid 2
that lasts for 50 seconds.
Given
Base URL as https://reqres.in/
Request URL as /api/users/2
Script for Config and Scenario Sections:

config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
scenarios:
- flow:
- put:
url: "/api/users/2"
json:
job: "SE"
Let's execute and check the output.
artillery run sample.yml

HTTP DELETE request


We have covered PUT, POST and GET requests. Let's proceed with DELETE.
DELETE removes data from the API or database.
For example: A user removes some data entry from the front end, this triggers a
delete action on the API in the backend.

DELETE Request Example


Scenario: User wants 15 virtual users to perform delete request for userid 2 that
lasts for 50 seconds.
The code to achieve the same would be

config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
scenarios:
- flow:
- delete:
url: "/api/users/2"
Now execute the script
artillery run sample.yml

What is the success response for creating a new account? -- 202


Get request is used for -- Listing all the users
Post request is used for -- Creating data
Which request is similar to patch request? -- PUT
Creating user script:
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Ranjan"
email: “ranjan@play.me”

Inline Variables in Scripts


Till now, we hard-coded values to get results. From now, let's call values from the
data sheet or by the defined variable.
To start with, variables are defined in config section. Sample code would be:

config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
variables:
id:
- "1"
- "2"
- "3"
- "4"

Here, name of the variable is id. Please note that the variable would have multiple
values and not a single value.

Inline Variables in Scripts Contd...


Let's call the defined variable in the scenarios section. A sample is as shown:

scenarios:
- flow:
- get:
url: "/api/users/{{ id }}"

Here we have defined variables (id) in curly braces like {{ variable }}.
Now you can execute the script.
artillery run sample.yml

Variables from Files


Let us use data-driven variables in the script.
For this, create a data.csv file with some sample data as follows:

1,Artist
2,Doctor
3,Engineer
4,Politician
5,Architect

Variables from File Example


Scenario: Search a user by Id and update the job type.
Here is the code for config and scenario section:

config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
payload:
fields:
- "id"
- "job"
scenarios:
- flow:
- post:
url: "/api/users/{{ id }}"
json:
job: "{{ job }}"

You would notice job and Id in the payload fields.


While executing this script, provide the data path as argument.
artillery run sample.yml -p data.csv
where -p stands for set payload

Variables from File


The second approach for providing datasheet as the variable is to include path in
config.
The script would like this:

config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 15
payload:
path: "./data.csv"
fields:
- "id"
- "job"
scenarios:
- flow:
- post:
url: "/api/users/{{ id }}"
json:
job: "{{ job }}"

To execute the script, -p option is now not needed.


You can execute it by calling
artillery run sample.yml

Defined variable are called inside -- {{ variable }}


If the path of the variable is not defined in the script, how you can run the file?
-- artillery run sample.yml -p data.csv
Following artillery script will error out. Which line is erroneous? -- Line number
3 (Reason : Intendation is not followed)

1 config:
2 target: 'https://reqres.in'
3 - phases:
4 - duration: 60
5 arrivalRate: 10
6 scenarios:
7 - flow:
8 - get:
9 url: "/api/users/2"

Reading Reports

After every execution, reports are generated. Let us take a look at these reports,
a sample of which is shown below.
Scenarios launched: 5
Scenarios completed: 5
Requests completed: 58
RPS sent: 0.86
Request latency:
min: 102.4
max: 3067.5
median: 325.5
p95: 2118.5
p99: 3020
Scenario duration:
min: 56745.4
max: 67339.1
median: 59275.6
p95: NaN
p99: NaN
Codes:
200: 58
You will notice that a logfile like artillery_report_xxxxxxx_xxxxxx.json is
created.

Understanding Report
Let us try to understand this report.
Scenarios Launched - Number of virtual users created in the preceding time.
Scenarios Completed - Number of virtual users who have completed the scenarios in
the preceding time. Please note, it is the number of completed sessions and not the
number of sessions started and completed in a time interval .
Requests Completed - Number of HTTP requests and responses or websocket messages
sent.
RPS sent - Average number of requests per second completed in the preceding time or
throughout the test.

Understanding Report
-Request latency - It is in milliseconds, and p95 and p99 values are the 95th and
99th percentile values (a request latency p99 value of 500ms means that 99 out of
100 requests took 500ms or less to complete)
Scenario duration - similar to Request latency but the value changes
Codes- count of HTTP response codes
If there is a NaN reported as value anywhere, it means there were not enough
responses to calculate the percentile.
Whenever the execution of the script is successful, it results in code being
reported as 200.
In case of any errors like socket timeouts, it results in an error in the report.

Viewing JSON Reports


To view JSON report, jq must be installed in the machine.
jq command is used to slice, filter, map and transform structured data with ease.
Usage: jq .aggregate artillery_report_xxxxxxxx_xxxxxx.json.
Replace artillery_report_xxxxxxxx_xxxxxx.json with the json file that gets created
after execution.
Please note, in our playground on katacoda, jq is not installed and hence you may
encounter an error saying bash: jq: command not found.

Converting JSON Report to HTML


Let us convert our auto generated JSON report file to HTML. To do so, you can run
the following:
artillery report artillery_report_xxxxxx_xxxxxx.json
Replace artillery_report_xxxxxx_xxxxxx.json with the json file created at the time
of execution.

What is the command to convert json report to html? -- artillery report


sample_report.json
What does NaN (Not a Number) stand for -- There was not sufficient response for
this action.
If sample_report.json is the report generated, how can the report be viewed using
json file? -- jq .aggregate sample_report.json

Advanced Scripting Concepts:


Request Timeout
Scenario: Sometimes, the script may take a long time to execute due to longer
response time. The execution may then abort with ETIMEDOUT error.
To handle this, you have to set a request timeout. The timeout value can be
incremental or decremental.

config:
target: "https://reqres.in"
http:
timeout: 10
Notice config.http.timeout in config section. timeout: 10 means that a response
should be sent within 10 seconds. Else, the request will abort.

Fixed Connection pool


Scenario: Now you know that by default Artillery will open a new connection for
each new virtual user. Now, what if a fixed number of connections have to reused by
calling the same virtual user(s) instead of new virtual user.
You can do this by looping the value in config.http.pool
The config script for this would be

config:
target: "https://reqres.in"
http:
pool: 15

Here, pool value 15 means, all the http requests from all the virtual users will be
sent over the same 15 connections.

Logging
Scenario: Consider you want to print value(s) on the console. Let's say you want to
print the Id generated by setting user name and job.
You can do this by passing the value to log.
Sample:

log: "Id value: {{ id }}"


This piece of code will print the Id value.

config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
capture:
json: "$.id"
as: "id"
- log: "Id value: {{ id }}"

Seting Header
Scenario: Consider that you need to set a header for the script.
You can do this using headers in scenario section as shown here:

config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- get:
url: "/api/users"
headers:
X-My-Header: "2343"

Looping Through a Number of Requests


Scenarios: Consider you want each virtual user to perform get request 100 times
For this, you need to loop the get request 100 times for each virtual user. You can
do this by passing the get request in the loop constant in scenarios section of the
script as shown.

scenarios:
-
flow:
-
loop:
-
get:
url: "/api/users/2"
count: 100

In case the count in the loop is removed or omitted, the script would run for
infinitely.
loop can be an array and so any number of requests can be defined.

Where can we set a Header? -- We can set a header in scenario section


How to add pool count? -- config.http.pool
Which is the correct syntax for log? -- Following one is correct
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "ranjan"
job: "programmer"
capture:
json: "$.id"
as: "id"
- log: "Id value: {{ id }}"
In the following code snippet, how do we set the timeout? -- config.http.timeout
How to add loop count? -- scenarios.flow.count

Socket.io scripts:
Socket.io
Socket.io is a method to "push" messages from the client to the server and vice
versa using symmetric syntax.
It is a javascript library which is used for making a real-time application such as
chat messenger and online games.
There are very few load testing tool in market that supports socket.io. Artillery
is one of the few.
Let us play around with socketio and artillery using a sample chat application. You
could alternately search for socket.io online and play try it.

Socket.io Testing
Scenario: Consider that you want to perform heavy load testing on a chat
application .
The load test could be for various reasons:
To check the traffic the app can handle.
To test if you can scale application performance by adding more app servers behind
the load balancer .
To perform CPU profiling in order to improve runtime performance.
Using Artillery, one can send data to the server and optionally wait for and verify
responses that come back.
To enable socket.io engine, set the engine attribute to socketio
To perform HTTP request with socket.io, emit action needs to be called.

Understanding Socket.io
To understand about socket.io, click on sample chat application. Open Developer
tools -> networks->WS option.
Now Refresh the URL to see some socket responses coming up in the name section. You
will see many such responses appearing. Click on any of the responses to understand
the different emits needed for the script.
Type your name in the text box. You will find the green highlighted response which
is the request made by typing the name.

["add user", "ranjan"]


0:"add user"
1:"ranjan"
So, channel value will be add user.
data value is ranjan.

Socket.io - Config Section


Scenario: User has to join the chat. User has to enter the name then the text.
Please note that a user can join only once in the chat, so you will need to send
the variable name each and every time.
So, you have to write two scenarios, similar to the ones in HTTP responses.
Config Section:
There is no extra input needed in the script.

config:
target: "https://socketio-chat.now.sh"
phases:
- duration: 10
arrivalRate: 1
Here, arrivalRate as 1 means calling only one user at each time interval.

Socket.io - Scenario Section


Scenarios section:
Let us take a look at this:
Set engine value to be socketio
Call get request to load the URL
To enter the value, call the emit and provide channel and data values
So, enter the username first and wait for the user to reach the chat page, then
type the message to be entered.

Socket.io Script
So, the scenarios section of the script will look like this:

scenarios:
- name: "Our First Artillery script"
weight: 15
engine: "socketio"
flow:
- get:
url: "/"
- emit:
channel: "add user"
data: "Ranjan"
- think: 5
- emit:
channel: "new message"
data: "Hello"
- think: 60

Here Weight is the number of times each scenario will be executed across the total
number of virtual users.
Go ahead and combine config and scenarios section scripts and execute the script.
artillery run sample.yml

Datadriven Approach
In our previous example, we hard-coded the username or message in the script.
However, this is not a good practice.
In real-time scenarios, there could be multiple users typing different messages.
Let us learn how to create random users and random messages.
Artillery has one built-in function to generate random alphanumeric string
$randomString().
To generate random messages, small JavaScript can be written as shown.

Script for Random Message

'use strict';

module.exports = {
setMessage: setMessage
};

const MESSAGES = [
'Do you know Play application',
'I really love reading on play',
'Play app is not about learning it is about playing with course',
'It has games as well',
'It is the best learning app and you get coupons and gifts with knowledge',
'Learn and earn on play',
'I am getting addicted to PLAY',
'Love play'
];

function setMessage(context, events, done) {


// this will pick a message randomly
const index = Math.floor(Math.random() * MESSAGES.length);
// make it available to templates as "message"
context.vars.message = MESSAGES[index];
return done();
}

Save this file as .js, say function.js.

Socket.io Script with Random Data


Here is the script for setting different usernames, with random greeting messages.
Notice, the JavaScript being called in processor.

config:
target: "https://socketio-chat.now.sh"
phases:
- duration: 10
arrivalRate: 5
variables:
greeting: ["hello", "namaste", "Vanakkam", "Kon'nichiwa", "Bonjour", "hola"]
processor: "./functions.js"
scenarios:
- name: "data driven in socketio"
weight: 10
engine: "socketio"
flow:
- get:
url: "/"
- emit:
channel: "add user"
data: "Ranjan-{{ $randomString() }}"
- emit:
channel: "new message"
data: "{{ greeting }}"
- loop:
- function: "setMessage"
- emit:
channel: "new message"
data: "{{ message }}"
- think: 10
count: 10
- think: 60

To send message in socket io, what command should we use? -- emit


Where can we define your JavaScript file to get the random data? --
config.processor

WebSocket
WebSocket stabilizes connection between the user's browser and a server.
Let us understand this concept with a few samples:
Consider a chat application like Sametime. Messages appear in the chat box without
manually refreshing the app. This is due to WebSocket.
Likewise, a live match score or points table auto-updates on the web page without
any manual refresh. This is due to WebSocket.

WebSocket Script
We have seen scripting process of HTTP and socketio. Scripting for WebSocket is no
different. The additional input to note is
Set engine value as ws
Actions supported in WebSocket are sent and think(wait time)
Here's the script.

config:
target: "wss://echo.websocket.org"
phases:
- duration: 10
arrivalRate: 5
scenarios:
- name: "First websocket Script"
engine: "ws"
flow:
- send: "Check"
- think: 1
- send: "world"

Here, engine value is set as ws. Texts like "world" are set with a wait time of 1
second before sending the second text.

Inline Variables
You have already learnt how to use an inline variable in HTTP request. The same can
be done for WebSocket too.

config:
target: "wss://echo.websocket.org"
phases:
- duration: 6
arrivalRate: 2
variables:
text:
- "Check"
- "World"
scenarios:
- name: "First websocket Script"
engine: "ws"
flow:
- send: "{{text}}"
- think: 1
- send: "{{text}}"
Execute the script and check the result. You may try Data-Driven approach too in
the similar fashion.

Debugging HTTP Scenario


To debug the artillery script, use DEBUG. Using DEBUG=http prints all the requests
and errors.
Syntax - DEBUG=http artillery run sample.yaml.
Similary, using DEBUG=http:response prints all the responses.
Syntax - DEBUG=http:response artillery run sample.yml
Using http:capture, prints the captured operation.
Syntax - DEBUG=http:capture artillery run sample.yml.
Multiple debugging options can also be set together.
Syntax - DEBUG=http,http:response artillery run sample.yml

Debugging Socket.io and WebSocket Scenario


Debugging Socket.io
Syntax - DEBUG:socketio artillery run sample.yml.
Debugging WebSocket
Syntax - DEBUG:ws artillery run sample.yml

Debugging Profile
Debug environment can be created in the script itself. For this, change the config
scenario sections and create only one virtual user.
Let us try the Script.

config:
target: "https://reqres.in"
environments:
debug:
phases:
- duration: 5
arrivalCount: 1
scenarios:
- name: "Debug Scenario"
flow:
- get:
url: "/api/users/2"
Notice environment -> debug -> phases -> duration and arrivalCount.
Let us execute the script with debug option.
artillery run -e debug sample.yml

How to debug web socket? -- DEBUG:ws artillery run sample.yml


The command to debug http -- DEBUG:http artillery run sample.yml
The command to debug http response -- DEBUG:http:responce artillery run sample.yml

Faker.js
Now you know how to generate random data for input to simulate real time scenario.
Likewise, you can use a separate data sheet as an input too. However, this may be
little cumbersome process.
How about having a ready-made API that provides these random data which can be
simply called in the script.
Faker is a node API that solves our purpose. As the name suggests, Faker creates
random fake data for our use.

Artillery with Faker


Scenario: Consider that you need to perform load testing for registration page that
takes inputs as first name, last name, email id, phone number, country, state and
more.
We have seen two ways to define data and use it.
Inline data-driven approach
Files data-driven approach
However, in both the methods, random data has to be prepared as input.
Let us try using faker to create data for us. Widely used data is already defined
in faker. Calling Faker function is all that is needed.
Faker link can be visited to understand and play around with data.

Artillery script
Let us consider a sample post request for registration. Here, two inputs are needed
for registration in the reqres API.
email
password

config:
target: "https://reqres.in"
phases:
- duration: 60
arrivalRate: 20
scenarios:
- name: "Register a user"
flow:
- post:
url: "/api/register"
json:
email: "ranjan@play.me"
password: "FrscoPlay"
This will be the base script while integrating faker.

Data Driven from Faker


Let us list out the changes needed in the artillery script to support faker.
Install faker npm install faker
Remove all the hard coded data with the variables.
Create a JavaScript folder where we will call the data from faker.
Save the data in .js format.
In config call processor to load the JavaScript folder that was created earlier.
Call the function generateRandomData which we have defined in the .js file.
Log the data in the logger to check the value.
Let us see these steps in action.

Jenkins

Jenkins is used for continuous integration. For a detailed learning on Jenkins,


kindly visit our Continuous Integration with Jenkins course.
For those who are familiar with Jenkins, let us dive into Jenkins and artillery
integration.
Prerequisite
Jenkins should be installed in your local machine. We will be using a playground in
this course.
Artillery works on nodejs. So we need a plugin nodejs to be installed in the
Jenkins.
Configure your Jenkins with the nodejs.
Now you're good to go!

Configuring Jenkins
To try out Jenkins integration with artillery in the local machine. The following
steps are involved for configuration:
Install Jenkins in your local machine.
Start Jenkins server. Use the command Jenkins to start the server. To run Jenkins
server on a specific port, say 9090, use Jenkins --httpPort=9090.
Please note, by default, Jenkins server will start on 8080 port. In case 8080 port
is already taken by other application server, you may use 9090 port.
Login to Jenkins.
Click on Manage Jenkins->Select Manage Plugins.
Click on the Available tab.
Enter NodeJs in the Filter search box and click.
Select Nodejs, click on the checkbox and install. If NodeJs is already installed,
it will be in Installed tab.
You can verify your installation from Installed tab, search for Nodejs in installed
tab.
Click on Manage Jenkins -> Global Tool Configuration.
Scroll down to Nodejs and click on the NodeJs installations.. button.
Enter the Node name.
Select the node version you want to use. Ensure you select the latest version of
the nodejs.
Select Global npm packages to install so that we are using artillery to be
installed using npm package. Enter artillery.
Enter the Global npm packages refresh hours.
Click on Apply and the save.

Workspace for Artillery


Let us create the workspace for artillery.
Click on New Item.
Enter an Item name.
Select the project type as Freestyle project and click on Ok button to confirm.
It is time to setup our artillery script for execution.
If you have your project in Git, you can set the git repo location. Else you can
set path from the local system.
Click Advance button from General tab -> check in Use custom workspace.
Enter Directory location.
Enter file name in Display Name.
Scroll to Build option and select Execute shell from drop down.
Enter the command needed for execution of the artillery project.
npm -version
npm install artillery
artillery run sample.yml
Click on Apply and Save.

Execution of Artillery Project


We have configured and set up the artillery workspace. Let us execute the
workspace.
Click on the workspace.
Select Build Now tab.
You can see the left-hand side in Build Queue the project is ready for execution.
Click on the execution build.
Click on the Console Output to see the execution response.
We have covered step-by-step configuration, project creation and execution from
Jenkins for artillery project in the upcoming video.

Artillery - Course Summary


Hope you have enjoyed learning Artillery. You should be now familiar with the
following:
Artillery commands
Basic scripting in artillery
Data driven scripting
Writing scripts to get, post, put and delete data using HTTP actions
Testing Socket.io and websocket based applications
Debugging artillery scripts
Integrating artillery scripts with Jenkins and Faker API

What argument is used for environment associated target? -- -e argument


Which request is similar to patch request? -- PUT
Inline variable is defined under -- config.variables
Target is defined in the scenarios section. -- False
What does Scenarios launch stand for -- The number of virtual user generated in the
defined time.
Phase of the script defines the API that we want to test -- False
Response time -- All of the options are correct

Artillery Practicals:
---------------------

Create a user:
--------------
config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Harley"
job: "Software Engineer"

Log:
-----

config:
target: 'https://reqres.in'
phases:
- duration: 50
arrivalRate: 5
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
capture:
json: "$.id"
as: "id"
- log: "Id value: {{ id }}"

Loop:
-----

config:
target: 'https://reqres.in'
phases:
- duration: 1
arrivalRate: 10
scenarios:
- name:
- loop:
-get:
url: "/"

Timeout:
--------
config:
target: "https://reqres.in"
http:
timeout: 10

Rampup:
-------
phases:
-
duration: 300
arrivalRate: 5
name: "Warm-up"
#Create 5 virtual users coming at an
average of 5 minutes of interval.
-
pause: 10
#It will wait for 10 seconds
-
duration: 60
arrivalCount: 30
#Create 30 virtual user in duration of 60
seconds.
-
duration: 600
arrivalRate: 50
name: "High load phase"
#We can name the arrival rate

Successful updation with inline parameters:


------------------------------------------
cat test.yml
config:
target: 'https://reqres.in'
phases:
- duration: 120
arrivalRate: 10
- duration: 120
arrivalRate: 20
scenarios:
- flow:
- put:
url: "/api/users/2"
json:
job: "SE"

Question: Create 20 virtual users in 60 seconds (one every 2 seconds on average) to


perform GET request to search user ID 3. Each user should perform this action 10
times.

config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- get:
url: "/api/users/3"

Question: Create 20 virtual users in 60 seconds (one every 3 seconds on average) to


perform PUT request for updating name and job for ID 2.

Question: Create ramp up arrival rate from 10 to 20 over 2 minutes to create user
account. Define the variable request URL, name and job in the Inline variable.
Use Reqres application as base and request URL.

config:
target: 'https://reqres.in'
phases:
- duration: 120
arrivalRate: 10
- duration: 120
arrivalRate: 20
scenarios:
- flow:
- put:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"

Question: Create 20 virtual users in 60 seconds (one every 3 seconds on average) to


perform PUT request for updating name and job for ID 2.

config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- put:
url: "/api/users/2"
json:
name: "Ranjan"
job: "Artist"

Question: Create ramp up arrival rate from 10 to 20 over 1 minute to create the
user account. Define the variable request URL, name, and job in the Inline
variable. Search the newly created user using ID. Use Reqres application as base
and request URL.

config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 10
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- put:
url: "/api/users/2"
json:
name: "Ranjan"
job: "Artist"
- post:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
capture:
json: "$.id"
as: "id"
- get:
url: "{{ id }}"

--------------------------------

config:
target: 'https://reqres.in'
phases:
- duration: 60
arrivalRate: 10
- duration: 60
arrivalRate: 20
scenarios:
- flow:
- post:
url: "/api/users"
json:
name: "Ranjan"
job: "Artist"
capture:
json: "$.id"
as: "id"

You might also like