Testing Techniques in Agile Frameworks: A study of

User Acceptance Testing

Raghav Rathi
raghav17083@iiitd.ac.in
Indraprastha Institute of Information Technology

Abstract:​ ​Assuring the quality of the software is a crucial part of the software development process.
Testing is usually considered the last part of the software development life cycle (SDLC) and, therefore,
not considered very important. Testing is an inevitable part of the SDLC and is now being adopted with
newer and efficient techniques and methodologies. This paper firstly focuses on the importance of testing
in SDLC and then discusses the existing testing paradigms. Further, we talk about the enhancements
automation has brought to the world of testing and quality assurance. Today, the entire world is shifting
gradually towards agile development, as it provides faster and accurate software as described by the
users by involving them in the process. The paper also focuses on aspects of testing supported in agile
frameworks, like TDD, BDD, and ATDD. Then we build our discussion on user acceptance techniques in
ATDD in an agile environment.

Introduction:
Testing is a vital phase in the software development cycle in which several evaluation criteria map against
the system to evaluate if the system meets the specified requirements or not. In broader terms, it allows
the verification and validation of user-specified needs and requirements. This process involves finding
bugs, errors, or missing requirements in the software developed. This activity is an evaluation that gives
stakeholders objective knowledge of the product quality. Since testing at any stage can be a
resource-intensive process, the developers must try to minimize a large scale project into manageable test
cases.
Studies[1][2] suggest that there is an evident relationship between the cost of testing and errors in the
system; hence, it is necessary to make wise decisions while minimizing the cost of testing such that bugs
in the systems can be circumvented. According to ​Figure 1[1]​, the number of bugs and the cost of testing
follow an inverse trajectory. Hence, it is essential to conduct just the optimal minimum number of tests
and reduce the cost without performing over-testing [2].
The process of testing involves three steps, which are Unit testing, Integration Testing, and System
testing. Every software development project is divided into several modules, where each module in itself
is a complete component or a unit. Once such a module is completed, it is tested by a software developer,
or a quality assurance engineer to ensure that the module works as expected. As defined in [3], this step is
called Unit testing. Once, the modules of a single software system have been independently developed.
Another round of testing is done after merging these independent modules as their integration can give
rise to errors. This step is called integration testing. Integration testing also ensures that the integration of
modules does not interfere with the working of other modules. System testing is done as a final testing
step at the end of the completely developed software [1]-[3].
In this paper, we discuss the testing cycle, which involves many phases, from Test planning to Test
results. Further, we discuss the various aspects of this test cycle, which includes the existing testing
methods like software test life cycle (STLC) and software release life cycle (SRLC).
We try to assimilate these thoughts to apply these concepts in the frameworks supported by agile
development techniques. We discuss the usages of Test Driven Development(TDD), Acceptance Test
Driven Development(ATDD) and Behavioural Driven Development(BDD) and draw comparisons
between them [].
Further, we also have a discussion over a new and robust technique of testing that most agilist are
adopting called acceptance testing, which is also called beta testing in which test cases are built so as to
conduct formal testing to determine whether the system satisfies the acceptance criteria for the user's
needs and requirements. Acceptance testing can subsequently be divided into many types of which User
Acceptance testing (UAT) and Operational Acceptance Testing (OAT) are the most prominent. The final
part of the paper discusses the scope of user acceptance testing in newer frameworks of software
development, like the agile framework and compares it with the traditional SDLC approach. Agile testing
practices are developed because an agile environment requires a more dynamic decision making for the
development of test cases because of its iterative and incremental nature, and requirements of users may
evolve over time.

Existing testing methodologies

A typical testing cycle involves many phases. The major step involved is the commencement of test cases
for modules to be tested. There are majorly three techniques for designing these test cases which are
namely, white box testing, black box testing, and grey box testing [4]. White box testing is the most
transparent form of testing in which not only the functionalities but also the internal composition of the
system is tested [2],[4]. White box testing ensures that the functionalities and consistencies of the system
remains secure and protected and hence it is also called security testing. Because, even if the module wise
structure is checked, it becomes necessary for the tester to have good programming skills and know
data-structures [2] which makes this technique quite complex [5]. The other type of technique to build the
test cases is black box technique, which involves the testing of only the functionalities of the software
without going into the internal structure. It is the simplest method of building the test cases [5].

The software industries have primarily been using well defined software testing paradigms in the past,
some of which are:

● Software Testing Life cycle: An STLC process starts with the review of requirements and
analysing them based on usability and quality assurance criterias. All the conflicts are resolved by
the testing team and then the results are fed to the next phase which is Test planning.
Test planning involves the development of a testing architecture upon which the entire process of
testing is based on. It provides documentation for the testing procedure.
Test planning is followed by Test case development in which development teams create test cases
according to the requirements analysed in the first step. Test case development is also called the
test designing phase [6] .
The next phase is the Test execution phase in which test cases are executed according to the test
plan developed in the previous phases. If any test case fails upon execution it means that the
requirements are not satisfied or the software is not bug free. The report from this execution task
is fed to the last phase which is the Test reporting phase, in which the reports regarding any error
or bug are then submitted to the development team for further analysis [6].

● Further testing processes are involved after the completion of STLC. These testing processes
include Alpha and Beta testing. In Alpha testing the entire testing procedure is carried out at the
developer’s side as a first stage testing of the application. The testing is done using white-box
testing as it involves testing at integration or system level. Hence, the developers who perform
these testing must be adept at knowing the architecture of the system [7].
On the other hand, beta testing comes after alpha testing, as this testing is done by the users after
the release of a beta version of the product.These users who are also known as beta testers check
the system for any bugs, or errors in the released version and therefore, perform a formal user
acceptance testing on this prototype version. The feedback from the users are then incorporated in
the system. Alpha and beta testing are part of the Software Release Cycle [7], [8] .

Over the years, many enhancements have been done to these primitive testing methods for making the
testing phase efficient and effective. Testing automation is one such enhancement being embedded in
various aspects of testing. Automation in testing is a sort after the procedure, as it saves the cost and time
of the testing done manually. In STLC, Testing Automation also allows a vigorous comparison between
the actual results and the expected results as it tests the software based on some predefined benchmarks.
Hence, entirely automated testing suites will lead to an overall sophisticated testing procedure as they
essentially foreshorten the margin for human error. For example, regression testing checks if the code
runs without any bugs after it fixes the error. Regression testing is typically manual work, and it can take
a lot of time to check every module of the software. Hence, an automated regression testing suite is used,
which monitors the code quickly and swiftly, and decreases the bug ratio significantly. [8]
The testing environment is known as the testing framework, where all the execution is done, and feedback
reports are incorporated. Usually, these testing frameworks are built with aspect-orientation; that is, each
module in the software has a separate testing framework, which maintains independence between various
modules. Such a type of framework is called a Modular testing framework. This framework helps in
easier maintainability of such modular test suites. A significant drawback of using this framework is that
they are at every modular level data-dependent; hence these frameworks embed the data. Therefore any
modification to the data will lead to changes in the test framework. This limitation of the modular
structure led to the development of the Data-Driven framework, in which different files store the testing
data and testing benchmarks. [2],[8]

One of the most critical developments in testing frameworks is assimilating the agile lifecycle in the
testing cycle by repeatedly carrying out testing in various phases of software development. In turn, these
testing cycles are short and quick to absorb recurring changes in user requirements. Research is being
done to make frameworks in an agile environment more efficient and adaptive to modifications in the
requirements and specifications. [8]
In the following sections, we discuss the testing in agile frameworks in more detail and then develop an
analysis of how user acceptance testing can be carried out in an agile model.

Discussion
Context

Over many years, the agile framework has established itself as a robust methodology for software
development, because it provides space for the development team to respond to any unpredictability.
Hence, it has become essential to adopt testing techniques in this evolving paradigm of software
development. Some commonly used agile practices are TDD, BDD, and ATDD, which were introduced
by extreme programming (XP), an agile framework [10]. All the three testing methods are unit testing
techniques which deliver small sets of results. These are described as follows [13]:

1. Test-Driven Development​: TDD is an evolutionary method which uses automated testing, that
combines the test-first development in which quality assurance teams build test cases before-hand
so that the software is designed in accordance with these pre-built tests and refactoring. The idea
is to make a functional test case fail before writing a function for that test case, after which it
should pass consistently [9]. Refactoring is the process of making changes iteratively so that the
additional features pass the test cases. TDD is a low-level technique that focuses on developing
unit testing from the developer’s perspective.
TDD involves the following steps [9]:
● Study the user story and create a unit test that is in accordance with that user-story. This
test must fail, as no application is built yet to fulfill the test-case.
 ● Develop an application by implementing the necessary features.
● Run the application on the predefined test cases on which the application previously
failed, to check if it fulfils the requirements presently.
● The code is now refactored into a more user-friendly and deliverable software.

2. Behaviour Driven Development​: BDD is similar to TDD but involves all the relevant
stakeholders instead of just the developers. For example, some business-minded stakeholders may
want to include test-cases that have business aspects attached to them. Software teams no longer
work on the technical definition of “test” but rather formulate the “behavior” of the software.
Focusing on the “behavioral” aspects rather than the “implementation” aspect gives a clear
understanding of what the user expects from the system. This “behavior” is explained in simple
language; hence, it allows for a broader audience to understand and tell their requirements to the
testers. The necessity of BDD arises when defects or limitations of the software are when the
developers misunderstand the actual conditions of the users. Thus, BDD plays a vital role in
defining the acceptance criteria of the software that the developers will create. Developers in
BDD need to be adequately skilled as they have to convert the non-technically defined behaviors
of the software to technically sophisticated software as stated in [9]. If the intended users or
customers conduct the tests, then it is said to be user acceptance testing [8], [9].

3. Acceptance Test-Driven Development​: This is a TDD development technique in which clients,

developers, and testers come together to discover requirements by collaboration. The main
objective is to incorporate user-defined requirements into the system on a “just in time” basis.
ATDD in ideology is very similar to BDD, as they both focus on aspects laid out by the
customers. While BDD involves tests which are heavily user-focused, ATDD is used to create
tests by the developers by keeping in mind the acceptance criteria of the users [13].
In context, ATDD lies somewhere between TDD and BDD, as the tests are completely developed
by the developers like in TDD but also incorporates the behaviour expected by the system by
setting certain user-acceptance criterias like in BDD [9].

Agile driven Acceptance Testing

As we have reviewed in the earlier sections, there is a massive paradigm shift from traditional software
development techniques to newer agile frameworks. Even in testing, the approach is to follow the agile
principle; that is, developers and stakeholders should work together at each and every step of the
development process. Conducting acceptance testing in just the final phase is a bad practice, as the motive
of UAT is to ensure that the software fits the clients' requirements and business needs. As these needs are
ever-evolving, testing must be done simultaneously with requirements engineering. While the developers
write the unit tests, acceptance tests are similarly communicated directly by the end-users. The most
commonly used form of communication is through defining user-stories [11], [14]. Acceptance criteria
delivered with user-stories, help developers design and fine-tune these functional tests. There exists a
project management dashboard which monitors the progress of agile processes. It computes the
knowledge metrics that takes into consideration knowledge metrics like user-stories with the previously
defined acceptance criterias, sets of test cases and functional attributes of the system provided using UML
diagrams. [14]
Studies have shown that significant failures of software development projects are due to the lack of
involvement of users. Hence, a planned process is required to emphasize the needs of the users from their
perspective. In such processes, the developers first understand the requirements and perform system
testing, and then the end-users perform acceptance testing in the user-environment. Acceptance testing
involves both the functional and non-functional aspects of testing. Non-functional testing requires
checking the compatibility of the software, its vulnerabilities, and the security of the system. While the
functional elements check the fitness of the software in terms of the stated requirements. Users do not
usually participate in the non-functional testing but engage with the developers in functional testing.
Face-to face interactions with users as shown in Padmini et al.[11] observe that most of the users in the
UAT team have limited technical knowledge with regards to testing types, defect identification and
analysing test cases. Conducting workshops to enhance their capabilities, and make them aware about the
agile practices have made it possible to enhance their value addition.
Padmini et al.[11] have also suggested the need for a tailored framework for UAT based on scrum agile
framework, where every team provides deliverables such as defect logs, reports reviewing execution of
tests, etc. Studies also showed meetings are organised daily to monitor the testing status, similar to scrum.
Sprint planning sessions are conducted to identify and create scenarios for UAT executions, and
prioritizing the requirements [15]. A separate UAT review session is also usually created to ensure the
proper functioning of the UAT weekly, and to resolve any issue between the users and developers [11].

Conclusion
We have seen how important testing can be in SDLC, and why there is an urgent need for
user-involvement at every step of the software development. With the latest but limited trends in the
software industry, agile methodologies are quickly taking over traditional waterfall based models where
testing was the final phase. In agile, software development has become more responsive to user feedback
and testing has become synonymous to requirements engineering. We discussed the traditional techniques
in testing like STLC and then discussed the enhancements like automation in testing. The paper also
discussed the various frameworks in agile related testing such as TDD, BDD and ATDD, and how they
have evolved as per the requirements. Finally we reviewed how acceptance testing is done in agile
frameworks, what difficulties the users face while being a part of the team carrying out UAT, and how
they can be overcome by imbibing scrum like architecture.
References Cited:
[1].P. Ron. Software testing. Vol. 2. Indianapolis: Sam’s, 2001

[2].Jamil, Abid & Arif, Muhammad & Abubakar, Normi & Ahmad, Akhlaq. (2016). Software
Testing Techniques: A Literature Review. 177-182. 10.1109/ICT4M.2016.045.

[3].K. Bogdan. “Automated software test data generation”. Software Engineering, IEEE
Transactions on 16.8 (1990): 870-879.

[4].J.Irena. “Software Testing Methods and Techniques”, 2008, pp. 30-35

[5].E. F. Miller, “Introduction to Software Testing Technology”, Software Testing & Validation
Techniques, IEEE, 1981, pp. 4-16

[6].M. Shaw, “Prospects for an engineering discipline of software,” IEEE Software, November
1990, pp.15-24

[7].Luo, Lu, and Carnegie, "Software Testing Techniques- Technology Maturation and Research
Strategies’, Institute for Software Research International-Carnegie Mellon University,
Pittsburgh, Technical Report, 2010.

[8].A. Memon, “A Uniform Representation of Hybrid Criteria for Regression Testing”,

Transactions on Software Engineering (TSE), 2013.

[9].Shweta Kumar, Sanjeev Bansal; “Comparative study of Test-Driven development with

Traditional Techniques”; (IJSCE); ISSN: 2231-2307, Volume-3, Issue-1, March 2016.

[10].Moe, Myint. (2019). Comparative Study of Test-Driven Development TDD, Behavior-Driven

Development BDD and Acceptance Test–Driven Development ATDD. International Journal of
Trend in Scientific Research and Development. Volume-3. 231-234. 10.31142/ijtsrd23698.

[11].Padmini, K.V.J. & Perera, Indika & Bandara, Dilum. (2016). Applying agile practices to
avoid chaos in User Acceptance Testing: A case study. 96-101.
10.1109/MERCon.2016.7480122.

[12].M. S. Sharmila and E. Ramadevi. "Analysis of performance testing on web application."

International Journal of Advanced

[13].Gil, Cesar & Diaz, Jorge & Orozco Bohorquez, Mario & Morales, Roberto & Hoz, Alexis &
Hoz, Eduardo. (2016). Agile testing practices in software quality: State of the art review. Journal
of Theoretical and Applied Information Technology. 92. 28 - 36.
[14].Asha, N. & Mani, P.. (2018). Knowledge-based acceptance test driven agile approach for
quality software development. International Journal of Recent Technology and Engineering. 7.
196-202.

[15].S. Paul and K. J. Singh, “Be agile: Project development with scrum framework,” J. of
Theoretical and Applied Information Technology, vol. 40, no.1, June 2012.