E-ffective Testing for E-commerce

By Angelina Samaroo, Steve Allott and Brian Hambling, ImagoQA, UK

Reviewer 1 overall comments:
1. there needs to be a conceptual flow for this paper. Too many unrelated thoughts are mingled into a
sentence/para.
2. lot of reference is being made to the’ application’ , an applications in the strictest sense for e-
commerce is a misnomer because in most of the cases there is none. The authors need to clarify
this term both as it applies to this paper and e-commerce in general. This is in addition to defining
the legacy applications in general.
3. There need to be references to any inferences and assumptions made in the paper.
4. benchmarking examples and other examples for metrics will aid better in reader comprehension.
5. Need a decent ending like a summary or conclusion with any appendixes and /or references.

Introduction

What is e-commerce? For the purposes of this paper, e-commerce (also known as e-
business) is defined as the software and business processes required to allow businesses
to operate solely or primarily using digital data flows. E-commerce is often associated
with web technology and is commonly transacted via web portals, but e-commerce is
much more than the provision of a web page as the customer interface. The creation of
integrated business processes (Enterprise Resource Planning), the integration of
collections of disparate software applications, each designed to facilitate a different
aspect of the business (Enterprise Application Integration), the extension of software and
business processes to embrace transactions with suppliers’ systems (Supply Chain
Management), the need for increased security for transactions over public networks, and
the potential volume demand at e-commerce sites all provide new and unique challenges
to the e-commerce development community – challenges which will require novel and
innovatory solutions and which will need thorough testing before they are allowed to go
live.

Why is testing important in the e-commerce environment? The first and primary reason
is because e-commerce is, by its very nature, business critical. In the third quarter of
1998, Dell’s e-commerce site exceeded $10 million in daily sales; the E*Trade site
currently exceeds 52, 000 transactions per day, giving a cost of one-day failure of around
$800,000; and the travel industry in Europe will be worth $2 billion by 2002, according
to Datamonitor. The immediacy of the customer, with its implied promise of rapid
delivery at competitive prices, and the sheer accessibility of the web, all combine to
create potentially massive demand on web sites and portals.

The second reason is that e-commerce is a massive and growing market place but one
which requires large up-front investment to enter successfully. There are already 5.8
million web sites worldwide, 2.5 million of which have been created this year (1999).
The International Data Corporation (IDC) estimates that the e-commerce market will

1
grow from over $5billion in 1998 to $1trillion in 2003. The average cost of development
of an e-commerce site is $1 million, says the Gartner Group, and will increase by 25%
annually over the next 2 years.

The third reason is because the history of e-commerce development has been littered with
expensive failures, at least some of which could have been avoided by better testing
before the site was opened to the general public. (In e-commerce terms, ‘the site’ means
the entire architecture from suppliers through back-end systems and front-end systems to
the customers; it typically includes Intranet, Internet and extranet applications as well as
legacy systems and third party middleware).

The Testing Challenges

Business Issues

A successful e-commerce application is:

 Usable. Problems with user interfaces lose clients.

 Secure. Privacy, access control, authentication, integrity and non-repudiation are big
issues.
 Scaleable. Success will bring increasing demand.
 Reliable. Failure is unthinkable for a business critical system.
 Maintainable. High rates of change are fundamental to e-commerce.
 Highly available. Downtime is too expensive to tolerate.

These characteristics relate in part to the web technology that usually underlies e-
commerce applications, but they are also dependent on effective integration and effective
back-end applications. E-commerce integrates high value, high risk, high performance
business critical systems, and it is these characteristics that must dominate the approach
to testing because it is these characteristics that determine the success of e-commerce at
the business level.

Technical Issues

The development process for e-commerce has unique characteristics and some associated
risks. It is generally recognised that a ‘web year’ is about 2 months long. In other words,
a credible update strategy would need to generate e-commerce site updates roughly
monthly. For this reason, Rapid Application Development (RAD) techniques
predominate in the e-commerce environment, and in some cases development is even
done directly in a production environment rather than in a separate development
environment. RAD techniques are not new, and it is generally agreed that they work best
where functionality is visible to the user – so web site development would seem to be an
ideal application area. Unfortunately, though, other aspects of e-commerce are at least as
important as the front-end. The end-to-end integration of business processes and the
consequent severe constraints placed on intermediate processes make them less than ideal
application areas for RAD.

2
These changes increase risk and create new challenges for testers, because time pressures
militate against spending a longer time testing sites before they are released. At the same
time, the technical environment of front-end systems is changing very rapidly, so change
is imposed on e-commerce sites even when the site itself is not changing. This requires
more regression testing than would be expected in a conventional application to ensure
that the site continues to function acceptably after changes to browsers, search engines
and portals. New issues have also come to the fore for testers, notably security of
transactions and the performance of web sites under heavy load conditions.

If we consider an e-commerce site as made up of a front end (the human-computer

interface), a back end (the software applications underlying the key business processes)
and some middleware (the integrating software to link all the relevant software
applications), we can consider each component in isolation.

Front End Systems

Static Testing. The front end of an e-commerce site is usually a web site that needs
testing in its own right. The site must be syntactically correct, which is a fairly
straightforward issue, but it must also offer an acceptable level of service on one or more
platforms, and have portability between chosen platforms. It should be tested against a
variety of browsers, to ensure that images seen across browsers are of the same quality.
Usability is a key issue and testing must adopt a user perspective. For example, the
functionality of buttons on a screen may be acceptable in isolation, but can a user
navigate around the site easily and does information printed from the site look good on
the page when printed? It is also important to gain confidence in the security of the site.
Many of these tests can be automated by creating and running a file of typical user
interactions – useful for regression testing and to save time in checking basic
functionality.

Dynamic Testing. Applications attached to an e-commerce site, either by CGI

programming or server extensions, will need to be tested by creating scenarios that
generate calls to these attached applications, for example by requiring database searches.
The services offered to customers must be systematically explored, including the
turnaround time for each service and the overall server response. This, too, must be
exercised across alternative platforms, browsers and network connections. E-commerce
applications are essentially transaction-oriented, based on key business processes, and
will require effective interfacing between intranet-based and extranet-based applications.

Back End Systems

The back end of e-commerce systems will typically include ERP and database
applications. Back end testing, therefore, is about business application testing and does
not pose any new or poorly understood problems from a business perspective, but there
are potential new technical problems, such as server load balancing. Fortunately, client-
server system testing has taught the testing community many valuable lessons that can be

3
applied in this situation. What is essential, however, is to apply the key front end testing
scenarios to the back end systems. In other words, the back end systems should be driven
by the same real transactions and data that will be used in front end testing. The back end
may well prove to be a bottleneck for user services, so performance under load and
scalability are key issues to be addressed. Security is an issue in its own right, but also
has potential to impact on performance.

Middleware and Integration

Integration is the key to e-commerce. In order to build an e-commerce application, one or

more of the following components are usually integrated:

 Database Server
 Server-side application scripts/programs
 Application server
 HTML forms for user interface
 Application scripts on the client
 Payment server
 Scripts/programs to integrate with legacy back-end systems

The process of developing an e-commerce site is significantly different from developing

a web site – commerce adds extra levels of complexity. One highly complex feature is
that of integration.

If an application is being built that uses a database server, web server and payment server
from different vendors, there is considerable effort involved in networking these
components, understanding connectivity-related issues and integrating them into a single
development (executable) environment. If legacy code is involved, this adds a new
dimension to the problem, since time will need to be invested in understanding the
interfaces to the legacy code, and the likely impact of any changes.

It is also crucial to keep in mind the steep learning curve associated with cutting-edge
technologies. Keeping pace with the latest versions of the development tools and
products to be integrated, their compatibility with the previous versions, and investigating
all the new features for building optimal solutions for performance can be a daunting
task. Also, since e-commerce applications on the web are a relatively new phenomenon,
there are unlikely to be any metrics on similar projects to help with project planning and
development.

The maintenance tasks of installing and upgrading applications can also become very
involved, since they demand expertise in:

 Database administration.
 Web server administration.
 Payment server administration.
 Administration of any other special tools that have been integrated into the site.

4
Technical support should also be borne in mind.

Correctly functioning back-end and front-end systems offer no guarantees of reliable

overall functionality or performance. End-to-end testing of complete integrated
architectures, using realistic transactions, is an essential component.

Ten Key Principles of Effective E-Commerce Testing

Over the decades since Information Technology (IT) became a major factor in business
life, problems and challenges such as those now faced by the e-commerce community
have been met and solved. Key testing principles have emerged and these can be
successfully applied to the e-commerce situation.

Principle 1. Testing is a risk management process. The most important lesson we

have learned about software testing is that it is one of the best mechanisms we have for
managing the risk to businesses of unsuccessful IT applications. Effective testing adopts
a strategy that is tailored to the type of application or service being tested, the business
value of the application or service, and the risks that would accompany its failure. The
detailed planning of the testing and the design of the tests can then be conformed by the
strategy into a business-focused activity that adds real business value and provides some
objective assessment of risk at each stage of the development process. Plans should
include measures of risk and value and incorporate testing and other quality-related
activities that ensure development is properly focused on achieving maximum value with
minimum risk. Real projects may not achieve everything that is planned, but the metrics
will at least enable us to decide whether it would be wise to release an application for live
use.

Principle 2. Know the value of the applications being tested. To manage risk
effectively, we must know the business value of success as well as the cost of failure.
The business community must be involved in setting values on which the risk assessment
can be based and committed to delivering an agreed level of quality.

Principle 3. Set clear testing objectives and criteria for successful completion
(including test coverage measures). When testing an e-commerce site, it would be very
easy for the testing to degenerate into surfing, due to the ease of searching related sites or
another totally unrelated site. This is why the test programme must be properly planned,
with test scripts giving precise instructions and expected results. There will also need to
be some cross-referencing back to the requirements and objectives, so that some
assessment can be made of how many of the requirements have been tested at any given
time. Criteria for successful completion are based on delivering enough business value,
testing enough of the requirements to be confident of the most important behaviour of the
site, and minimising the risk of a significant failure. These criteria – which should be
agreed with the business community - give us the critical evidence that we need in
deciding readiness to make the site accessible to customers.

5
Principle 4. Create an effective test environment. It would be very expensive to
create a completely representative test environment for e-commerce, given the variety of
platforms and the use of the Internet as a communications medium. Cross-platform
testing is, naturally, an important part of testing any multi-platform software application.
In the case of e-commerce, the term ‘cross-platform’ must also extend to include ‘cross-
browser’. In order to ensure that a site loads and functions properly from all supported
platforms, as much stress and load testing as possible should be performed. As an
absolute minimum, several people should be able to log into the site and access it
concurrently, from a mixture of the browsers and platforms supported. The goal of stress
and load testing, however, is to subject the site to representative usage levels. It would,
therefore, be beneficial to use automated tools, such as Segue’s SilkPerformer or Mercury
Interactive’s LoadRunner, for performance/load testing.

Principle 5. Test as early as possible in the development cycle. It is already well

understood and accepted in the software engineering community that the earlier faults are
detected, the cheaper the cost of rectification. In the case of an e-commerce site, a fault
found after shipping will have been detected as a failure of the site by the marketplace,
which is potentially as large as the number of Internet users. This has the added
complication of loss of interest and possibly the loss of customer loyalty, as well as the
immediate cost of fixing the fault. The fact that e-commerce development is rapid and
often based on changing requirements makes early testing difficult, but testing strategies
have been developed by the RAD community, and these can be mobilised for support.
Perhaps the most important idea in RAD is the joint development team, allowing users to
interact with the developers and validate product behaviour continuously from the
beginning of the development process. RAD utilises product prototypes, developed in a
series of strictly controlled ‘timeboxes’ – fixed periods of time during which the
prototype can be developed and tested – to ensure that product development does not drift
from its original objectives. This style of web development makes testing an integral part
of the development process and enhances risk management throughout the development
cycle.

Principle 6. User Acceptance Testing (UAT). The client or ultimate owner of the e-
commerce site should perform field testing and acceptance testing, with involvement
from the provider where needed, at the end of the development process. Even if RAD is
used with its continuous user testing approach, there are some attributes of an e-
commerce site that will not be easy (or even possible, in some cases) to validate in this
way. Some form of final testing that can address issues such as performance and security
needs to be included as a final confirmation that the site will perform well with typical
user interactions. Where RAD is not used, the scope of the provider’s internal testing
coverage and user acceptance testing coverage should be defined early in the project
development lifecycle (in the Test Plan) and revisited as the project nears completion, to
assure continued alignment of goals and responsibilities. UAT, however, should not be
seen as a beta-testing activity, delegated to users in the field before formal release. E-
commerce users are becoming increasingly intolerant of poor sites, and technical issues
related to functionality, performance or reliability have been cited as primary reasons
why customers have abandoned sites. Early exposure of users to sites with problems

6
increases the probability that they will find the site unacceptable, even if developers
continue to improve the site during beta testing.

Principle 7. Regression testing. Applications that change need regression testing to

confirm that changes did not have unintended effects, so this must be a major feature of
any e-commerce testing strategy. Web-based applications that reference external links
need regular regression testing, even if their functionality does not change, because the
environment is changing continuously. Wherever possible, regression testing should be
automated, in order to minimise the impact on the test schedule.

Principle 8. Automate as much as possible. This is a risky principle because test

automation is fraught with difficulties. It has been said that a fool with a tool is still a
fool, and that the outcome of automating an unstable process is faster chaos, and both of
these are true. Nevertheless, the chances of getting adequate testing done in the tight
time scales for an e-commerce project and without automation are extremely slim. The
key is to take testing processes sufficiently seriously that you document them and control
them so that automation becomes a feasible option – then you select, purchase and install
the tools. It will not be quick or cheap – but it might just avoid a very expensive failure.

Principle 9. Capture test incidents and use them to manage risk at release time. A
test incident is any discrepancy between the expected and actual results of a test. Only
some test incidents will relate to actual faults; some will be caused by incorrect test
scripts, misunderstandings or deliberate changes to system functionality. All incidents
found must be recorded via an incident management system (IMS), which can then be
used to ascertain what faults are outstanding in the system and what the risks of release
might be. Outstanding incidents can be one of the completion criteria that we apply, so
the ability to track and evaluate the importance of incidents is crucial to the management
of testing.

Principle 10. Manage change properly to avoid undoing all the testing effort.
Things change quickly and often in an e-commerce development and management of
change can be a bottleneck, but there is little point in testing one version of a software
application and then shipping a different version; not only is the testing effort wasted, but
the risk is not reduced either. Configuration Management tools, such as PVCS and
ClearCase, can help to minimise the overheads of change management, but the discipline
is the most important thing.

Conclusions

E-commerce is both familiar and novel. Some of the technology is relatively novel, and
the application of that technology to a complete business is certainly novel, but the
problems of creating business processes to operate a business in a wholly new
environment overshadow all of that novelty with some familiar and intractable problems.
Paradoxically, it is in the more familiar areas of the technology that the most serious
problems arise, because the emergence of e-commerce has placed new and challenging

7
requirements on this relatively old technology that was designed for a quite different
purpose.

Testing is crucial to e-commerce because e-commerce sites are both business critical and
highly visible to their users; any failure can be immediately expensive in terms of lost
revenue and even more expensive in the longer term if disaffected users seek alternative
sites. Yet the time pressures in the e-commerce world militate against the thorough
testing usually associated with business criticality, so a new approach is needed to enable
testing to be integrated into the development process and to ensure that testing does not
present a significant time burden.

The very familiarity of much of the technology means that tried and true mechanisms will
either be suitable or can be modified to fit. Rapid Applications Development (RAD), in
particular, suggests some promising approaches. Like most new ventures, though, e-
commerce must find its own way and establish its own methods. In this paper we have
suggested some testing principles that have stood the test of time and intermingled them
with some lessons learned from similarly challenging development environments to give
e-commerce testers a staring point for their journey of discovery.