Testing in the new age of DevOps

TESTING IN THE NEW AGE OF
DEVOPS
Moataz Mahmoud Ahmed
Junior QC developer at xWare

AGENDA
• Introduction to DevOps.
• Glossary.
• Continuous testing.
• The DevOps lifecycle.
• Where does QA fit in DevOps.
• Test-Driven Development (TDD).
• References.

INTRODUCTION TO DEVOPS.
• DevOps is a clipped compound of
"software DEVelopment" and "information
technology OPerationS“.
• DevOps is a term used to refer to a set of
practices that emphasize the collaboration and
communication of both software
developers and information technology (IT)
professionals while automating the process
of software delivery (deployment) and
infrastructure changes.

INTRODUCTION TO DEVOPS CONTD.
• Because DevOps is a cultural shift and collaboration between development, operations
and testing, there is no single "DevOps tool": it is rather a set which is called "DevOps
toolchain", consisting of multiple tools.
• Generally, DevOps tools fit into one or more of these categories, which is reflective of
key aspects of the software development and delivery process:
1. Code — Code development and review, version control tools, code merging.
2. Build — Continuous integration tools, build status.
3. Test — Continuous testing tools that provide feedback on business risks.
4. Package — Artifact repository, application pre-deployment staging.
5. Release — Change management, release approvals, release automation.
6. Configure — Infrastructure configuration and management, Infrastructure as Code tools.
7. Monitor — Applications performance monitoring, end–user experience.

• There are many problems DevOps is trying to solve which are:
1. Shift Operations or IT tasks earlier in the process.
2. To leverage a bunch of newer tools and technologies to automate operations
tasks, like provisioning and migrating code, leveraging these tools.
3. Have everyone on the product team communicate and collaborate much more
and earlier.
4. To have the product be ready to be deployed—in a consistent state of readiness
so that the business can decide when new functionality goes to customers. And
ultimately, not be held back by Dev. or Ops being unstable and not prepared to
go with whatever is read.

• The specific goals of DevOps span the entire delivery pipeline. They include
improved deployment frequency, which can lead to:
1. Faster time to market.
• It means faster time to recovery (in the event of a new release crashing or otherwise
disabling the current system).
• It saves your time to work on other project though you can gain more profit.
2. Lower failure rate of new releases.
3. Shortened lead time between fixes.
4. Better quality of the final products.
5. As a result, gain customer satisfaction.

INFRASTRUCTURE AS CODE (IAC)
• It is the process of managing and provisioning computing infrastructure
(processes, bare-metal servers, virtual servers, etc.) and their configuration
through machine processable definition files, rather than physical
hardware configuration or the use of interactive configuration tools.
• The definition files may be in a version control system. This has been
achieved previously through either scripts or declarative definitions, rather
than manual processes, but developments as specifically titled “IaC” are
now focused on the declarative approaches. Infrastructure as Code
approaches have become increasingly widespread with the adoption of
cloud computing and Infrastructure as a Service (IaaS). IaC supports IaaS,
but should not be confused with it.

INFRASTRUCTURE AS A SERVICE
(IAAS)
• It is a standardized, highly automated offering, where
compute resources, complemented by storage and
networking capabilities are owned and hosted by a service
provider and offered to customers on-demand. Customers
are able to self-provision this infrastructure, using a Web-
based graphical user interface that serves as an IT operations
management console for the overall environment. API
access to the infrastructure may also be offered as an
option.

ENVIRONMENT AS A SERVICE
(EAAS)
• Environment-as-a-Service, often referred to as IaaS Plus,
extends the traditional IaaS ecosystem into the application
development space. Here advanced automation is layered
on top of the existing IaaS instance to not only configure
servers for a particular application, but also to deploy and
test all the other components needed to run a given
application.

CONTINUOUS DEVELOPMENT
• It’s a software development practice in which every code change
goes through the entire pipeline and is put into production,
automatically, resulting in many production deployments every
day.
• With Continuous Delivery your software is always release-ready,
yet the timing of when to push it into production is a business
decision, and so the final deployment is a manual step.
• Continuous Deployment mandates Continuous Delivery, but the
opposite is not required.

CONTINUOUS INTEGRATION
• It’s a software engineering practice in which the changes made by
developers to working copies of code are added to the mainline code base
on a frequent basis, and immediately tested.
• The goal is to provide rapid feedback so that, if a defect is introduced into
the mainline, it can be identified quickly and corrected as soon as possible.
• In the end, well implemented CI reduces the cost of software development
and helps speed time to market.
• What’s this step is doing is it integrates every single changed component at
this build in the whole system.

CONTINUOUS DELIVERY
• Is a software engineering approach in which teams produce software in short
cycles, ensuring that the software can be reliably released at any time.
• It aims at building, testing, and releasing software faster and more frequently.
• What’s this step is doing is confirming that the integrated components in the
previous stage are ready to be tested now. This is done by running unit tests,
collecting the release notes and other documents, versioning the new build,
and some other activities.
• There are many well known tools like Docker, Jenkins, Puppet, and Vagrant
are often used and frequently referenced in DevOps tooling.

CONTINUOUS TESTING
• It is the process of executing automated tests as part of the software
delivery pipeline to obtain immediate feedback on the business risks
associated with a software release candidate.
• Because CI detects deficiencies early on in development, defects are
typically smaller, less complex, and easier to resolve.
• Once the integration is done, they would do unit tests on the integrated
code. They may run other tests such as white box security tests, code
performance tests, etc. All needed tests are running here.
• In the next section of this presentation, there are much more details
about continuous testing.

CONTINUOUS MONITORING
• In Production, the Ops team manages and ensures that the application is
performing as desired and the environment is stable via Continuous Monitoring.
While the Ops teams have their own tools to monitor their environments and
systems, they need to ensure that the applications are performing at optimal
levels – down to levels lower than system monitoring tools would allow.
• This requires that Ops teams use tools that can monitor application performance
and issues.
• It may also require that they work with Dev. to build self monitoring or analytics
gathering capabilities right into the applications being built. This would allow for
true end-to-end monitoring – continuously.

Testing in the new age of DevOps

CONTINUOUS TESTING CONTD.
• When I think of Continuous Testing I
think of the Lean principle, quality
at every step. When developers
commit new code, test it! When the
product gets integrated, test it!
When the product gets moved to
any new environment, like the test
environment, staging environment
or production, test it!

• There are three important parts here:
1. The build delivered by Development.
2. The deployment on infrastructure handled by OPS/IT.
3. The customers who use the apps/system.
• Test teams must:
- Have strong knowledge of these three parts.
- Know the goals of testing at each part.
- Be able to design great tests for each part.
- Have the ability to automate these tests.

• Tests can be designed and built for successful continuous integration first, and expand that
to continuous testing by knowing the right things to do, staying focused on a few key
points and most importantly, automating as smartly as possible.
• DevOps for Testers is focusing on continuous testing.
• When people describe DevOps as a “shift left,” one reference is to running performance
tests early and not waiting until the end of development.
• This “shift left” will uncover bugs earlier which will save time and cost for your company.
• This is an important example because it illustrates the many pieces of DevOps.
• The production environment can be virtualized and spun up at any moment with a VM or
in the cloud and have full performance tests run early in development.

• Your team has to be great at Agile. Scrum butts and Agile Falls will fail in
DevOps.
• A quick, Continuous Testing task overview is:
1. Start with an automated smoke test. Move these into CI build process tool.
2. Build bigger regression suites. Move these into CI build process tool.
3. Grow in levels of awesomeness of CI; Run smoke and/or regression on multiple VMs.
4. Easy and effective reporting back to the organization.
5. Use containers and/or virtualization for data and full production like environments.
6. Distribute automated tests into different suites with varying goals on different
environments. Use VMs for various environments to grow automation, coverage, speed,
monitoring and feedback to the team.

THE DEVOPS LIFECYCLE.
In order for DevOps to be successful, Development, Operations and the Test team must align their duties.

THE DEVOPS LIFECYCLE CONTD.
Finally, here is the full DevOps lifecycle.

WHERE DOES QZ FIT IN DEVOPS?
• DevOps QA is about preventing defects, not finding them:
• QA takes a critical role in this organizational structure because they have the visibility and
the directive to push code out when it is working, and roll it back
when it is not.
• This has several implications:
• QA owns continuous improvement and quality tracking across the entire development cycle.
• Test are code.
• Anything that can be automated, should be automated.
• Testers don’t just find bugs. They look for any opportunity to improve repeatability and
predictability of them.
• Beyond functional testing: automation for load testing, stress testing, and performance
testing.

WHAT YOU NEED TO CONSIDER BEFORE
YOU JUMP TO CONTINUOUS TESTING…
1. Consider the platforms and devices.
• What are the platforms and devices that each of your products run over?
2. Consider the environment.
• Consider the environment that will be in the customer side and try to virtualize it.
3. Consider the tool.
• There are many available tools. You should consider a tool that’s scalable, and can be run on a variety
of machines and environments.
4. Consider the number of tests.
• It’s better to cover more scenarios in less number of tests to save time in recording and running them.

HOW TO MAKE THE JUMP TO DEVOPS.
1. Automate more, automate faster.
• You need to automate not only your testing, but other tasks that were done by other teams. And yes,
automate faster to save the cost and raise the quality.
2. Build a solid framework.
• You will need a framework with reusable and low maintenance functions.
3. Choose effective methods.
• Automating traditional test scripts doesn’t work anymore. Consider Action based testing, test-driven
development, and A/B testing.
• Once you have efficient Agile practices, good low-maintenance test automation and Continuous
Integration processes in place, then Continuous Testing and DevOps is the next step in achieving hyper
efficiency.

TEST-DRIVEN DEVELOPMENT (TDD)
• Test-driven development (TDD) is a software development
process that relies on the repetition of a very short development
cycle: requirements are turned into very specific test cases, then the
software is improved to pass the new tests, ONLY.
• The most important concept here is that testing team starts its work
before the development. And it’s the true meaning of “shift left” in
continuous testing.
• This approach prevents software development that allows software to
be added which is not proven to meet requirements.

TEST-DRIVEN DEVELOPMENT LIFECYCLE
1. Add a test: Each new feature begins with that the developer writes a test that defines a function or
improvements of a function, which should be very concise.
2. Run this test: At the first run, this test will fail due to there are no code to make it pass! So now it’s
the time to write the code to pass this failed test.
3. Write the code: The main target at this step is to write code which passes the test. No matter how
elegant or efficient it’s. Just pass the test and check this step as completed.
4. Re-run the previously failed test: Now we will re-run the failed test. If it’s passed so you have
done. If not, keep looping between steps #3 and #4 until it’s done.
5. Refactor code: It’s the time now to refactor your code now. The previously written code passed
the test successfully, but it may not be the most efficient code.

TEST-DRIVEN DEVELOPMENT LIFECYCLE CONTD.
6. Add new tests: You can now move on and write new test for a new feature or requirement.
7. Run the new tests: This test will fail expectedly, so you move on to write some code to
make it pass. If these new tests pass from the first run, so there is a design problem here!
This means that you built the features which make it pass in the previous step and this is
out of the right flow.
8. Write more code: Write code to pass the new tests.
9. Run all the tests including the new ones: The new tests should pass now. If not keep
writing code until it pass. All the old tests should pass also, it’s for the sake of integration.
If not, stop working and fix the failed tests.
10. Refactor code.
Repeat…

Traditional development model Test driven development model

NOTES ABOUT TDD
• The size of the steps should always be small, with as few as 1 to 10 edits
between each test run.
• If new code does not rapidly satisfy a new test, or other tests fail unexpectedly,
the programmer should undo or revert in preference to excessive debugging.
• Continuous integration helps here by providing revertible checkpoints.
• When using external libraries, there is some reason to believe that the library is
buggy or is not sufficiently feature-complete to serve all the needs of the
software under development.
• This process is also called Red-Green factorization. Red refers to the failed tests,
green refers to the passed ones, and factorization refers to the process of
refactoring the code each time it passes the tests.

NOTES ABOUT TDD CONTD.
• Some tend to see TDD as a game of ping pong (or table tennis). The game is
very fast. They tend not to spend more than a minute on either side of the table
(test and implementation). Write a short test and run it (ping), write the
implementation and run all tests (pong), write another test (ping), write
implementation of that test (pong), refactor and confirm that all tests are
passing (score), repeat. Ping, pong, ping, pong, ping, pong, score, serve again.
Do not try to make the perfect code. Instead, try to keep the ball rolling until
you think that the time is right to score (refactor).
• Very useful tool when working in the TDD fashion are watchers. They are
frameworks or tools that are executed before we start working and are
watching for any change in the code. When such a change is detected, all the
tests are run.

NOTES ABOUT TDD CONTD.
• Only trust those tests which you saw in both cases (fail and pass).
• The most important part of TDD is the middle D. You let the
tests drive you. The tests tell you what to do, what to do next,
when you are done. They tell you what the API is going to be, what
the design is.
• This is important: TDD is not about writing tests first. There are
plenty of projects that write tests first but don't practice TDD.
Writing tests first is simply a prerequisite for being able to let the
tests drive the development.

USE TESTS AS DOCUMENTATION
• Another very useful side effect of TDD (and well structured tests in general) is
documentation.
• In most cases, it is much easier to find out what the code does by looking at
tests than the implementation itself.
• Additional benefit that other types of documentation cannot provide is that
tests are never outdated.
• If there is any discrepancy between tests and the implementation code, tests
fail. Failed tests mean inaccurate documentation.
• If your tests aren’t that so clear or your product have much details which are
needed to be translated from code to words, you can use tests as main
headlines for the documentation.

REFERENCES
• Test-Driven Development article: (Wikipedia).
• DevOps article: (Wikipedia).
• TDD testers.
• Technology Conversation
• Intro to TDD
• JavaScript testing:
• Tests as documentation:
• Testing in the new age of DevOps.

Testing in the new age of DevOps

More Related Content

Testing in the new age of DevOps