Testing Rapidly Changing Applications With Self-Testing Object-Oriented Selenium Infrastructure

WRITING
WELL-ABSTRACTED
SELF-TESTING
AUTOMATION
ON TOP OF JELL-O®

DAN CUELLAR
LEAD SOFTWARE ENGINEER
ZOOSK - QA TEAM

TERMINOLOGY
Test or Automation Script
a series of steps and expected output

Hook or Element
a means of identifying a piece of UI (think the By class in
WebDriver)

Controller
a program which executes basic commands (think WebDriver or
SeleniumRC Server)

PHILOSOPHY
• Selenium is a controller program, it exists to perform actions
on a platform (web browsers)
• Click a link
• Read some text
• Libraries exist to coordinate these small actions into larger
logical actions on a particular platform
• Sign In
• Send A Message
• Get All Error Messages
• Interfaces exist to remove the platform from the context of
libraries
• Automation scripts should coordinate the highest level of
abstraction whenever possible to maintain agility and
robustness

A BIT ABOUT ZOOSK
• Zoosk is a romantic social network with tens of millions of users that
operates in 25 languages and in over 70 countries.
• Products include:
• web site
• mobile web site
• Facebook app
• iPhone app
• iPad app
• Android app
• Adobe Air app

• The development pace is rapid, features are churned out in around 2
weeks time (including testing)
• Product management is always running experiments and adjusting
plans based on data from instrumentation
• Developer to quality assurance ratio hovers somewhere around 10:1

SO WHAT DO YOU
MEAN BY JELL-O®
Jell-O [jel-oh]
Site wide A/B tests where ½ of users will see a complete site
redesign and the other ½ will use the site as it was before.
Facebook app users can also split between different A and B
designs at times.
Each A & B may have its own JS, CSS, and business logic to go
along with it.
Not to mention that different versions of the API were being used
by the web, Facebook app, and mobile apps.
All the while, different features were being enabled and disabled
for different subsets of the A/B test groups

Testing Rapidly Changing Applications With Self-Testing Object-Oriented Selenium Infrastructure

WHAT CAN GO WRONG?
• General
• Not enough information is retrieved to reproduce the failure
• Elements
• The type can change (e.g. dropdown becomes a checkbox)
• An element can change location
• An element can have multiple ways of identifying itself depending
on an A/B test
• Business Logic
• Logic can change for one platform
• Logic can change for some users and not others (A/B test)
• Logic can change for all platforms

HOW DO WE COMBAT THESE
PROBLEMS?
• General
• Not enough information is retrieved to reproduce the failure
• Separate test actions from controller code, add abundant
logging, and unify the patterns by which common
operations are accomplished (e.g. selecting from
dropdown) Don’t access selenium directly in any scripts.
Create a wrapper that logs everything and performs basic
checks. (e.g. is the choice available on the dropdown)
• If libraries are used then you can log just about everything
possible every time without having to re-paste all the
logging code

PROBLEMS? (CONT.)
• Elements
• The type can change
• Abstracting away all uses of the item into a library
• An element can change location
• Centralize all hooks into the UI, you can change it once
and it will be fixed for all
• Using reflection, you can verify that all of your hooks are
functional
• An element can have multiple ways of identifying itself
depending on an A/B test
• ID’s are great, but you can also use an XPath Or
expression in your centralized hook

PROBLEMS? (CONT.)
• Business Logic
• Logic can change for one platform
• Update the library for that platform
• Logic can change for some users and not others (A/B test)
• Detect the A/B test in the library
• Logic can change for all platforms
• Generic Libraries or test scripts which coordinate platform
specific libraries are a great place to address this

HOW DO YOU DEAL
WITH THAT AGAIN?
• Separate logical actions From calls to controller (in this case
Selenium) code into libraries
• Centralize your site model and logic, so that all automation
runs through one place. When you have to fix something, you
can fix it once for everything.
• Write code that can test the test automation. Investigating test
script failures can be costly, but if you test code can test itself
you will be able to pick out certain kinds of problems easily.
• Write platform-agnostic test scripts when possible. This leaves
you with one copy of site-logic and test-logic for all platforms.
When things change, you’ll be happy its in one place.

SEPARATION OF CONTROL
FROM SCRIPTING
• An extra layer is great to log information and perform additional
actions that Selenium can’t assume it needs to do
• We use a layer between that wraps logging, error
handling, and some other things on top of most WebDriver
commands
Test Script •IsSignedIn()

•Log(“Retrieving Cookie Auth”);
Action •var authCookie = GetCookie(auth);
•Log(authCookie);
Library •return authCookie != null;

•Log (“Getting Cookies”);
•var c = Driver.Manage.().Cookies;
WebDriver •Log(c);
•var a = Driver.Manage().Cookies[“auth”];
Wrapper •Log(“Cookie value: “ + a.ToString());
return a;

CENTRALIZING YOUR SITE
MODEL FOR SELF -TESTING
• All hooks (means by which we retrieve WebElements) are stored
in a single class, namespace, or something similar
• Elements are stored lazily; the means to retrieve them are
stored, but the actual WebElement is not retrieved until run-time.
We use classes that wrap calls to By or methods which will return
a WebElements
• Hooks are grouped relevantly and each group has private self-test
setup methods which will navigate the client (e.g. selenium) to
where the hooks exist
• Reflection is used to iterate through all of these groups and run
the private self-test setup methods and then validate the
functionality of the elements
• Annotations (Java) or Attributes (C#) are used to exclude
elements from the tests

ABOUT INTROSPECTION
& REFLECTION

Introspection (Java) and Reflection (C#) is the process by which a
computer program can observe, do type introspection, and modify
its own structure and behavior at runtime.
In the example to follow we store information about how to use
various hooks (references to pieces of user interface) in the code
so that later at runtime we can use the information to determine
how to test our library of hooks.

CENTRALIZED HOOKS
[C#]
namespace Automation.Hooks.Web
{
public static class LoginPage
{
public static WebHook EmailField = By.Id(“email”);
public static WebHook PasswordField = By.Id(“password”);
public static WebHook SubmitButton = By.Id(“submit”);
[OmitFromSelfTest]
public static WebHook Spinner = By.Xpath(“//div[@class=’wait’]”);
private static void _StartSelfTest()
{
_SelfTestSeleniumInterface.Open(“http://www.zoosk.com”)
}
}
}

SIMPLE SELF-TEST
[PseudoCode]

// iterate in depth-first-search order
foreach(class c in Automation.Hooks.Web)
{
if c.hasMethod(“StartSelfTest”)
c.executeMethod(“StartSelfTest”);
foreach(WebHook h in c)
Log.Verify(IsElementPresent(h));

if c.hasMethod(“StopSelfTest”)
c.executeMethod(“StopSelfTest”);
}

MORE ADVANCED
TECHNIQUES
• Subclass WebHook to things like TextBoxHook and have the
test check for reading and writing from the text field using
Selenium
• Do the same for Links, Images, etc. and write custom test
methods
• Randomize the order of which elements are tested to check for
robustness
• Add random data to the tests, e.g. type a random string into
the text box.
• Write custom test methods for structures that require it

CONSOLIDATING AND ABSTRACTING
PRODUCT LEVEL ACTIONS
• Product Level Actions (e.g. Sign In To The Site) are stored in
libraries, so that there exists one good copy that everyone can
use
• The libraries implement generic interfaces so the platform can
be abstracted away (i.e. the native iPhone app and the website
both have libraries with a signup function that conforms to an
interface that can be used in any test script)
• If the process changes for a particular platform, you can
update it in a single place
• If the process change across all platforms, you can use
platform agnostic (generic) libraries and update the logic in a
single place

ABSTRACT LIBRARIES
[C#]
namespace Automation.Libraries
{
public SeleniumActionLibrary WebSignUpLib : ISignUpLib
{
public void SignIn(string email, string password)
{
if (IsSignedIn())
return;
if (!Driver.Url.contains(“login.php”);
Driver.Navigate().GoToUrl(“https://www.zoosk.com/login.php”);
Hooks.Web.LoginPage.EmailField.type(email);
Hooks.Web.LoginPage.PasswordField.type(password);
Hooks.Web.LoginPage.SubmitButton.click();
WaitForElementToNotExist(Hooks.Web.LoginPage.Spinner);
}
}
}

LIBRARY INTERFACE
[C#]
namespace Automation.Libraries
{
interface ISignUpLib
{
void SignIn(string email, string password);
bool IsSignedIn(string email, string password);
void SignOut();
}
}

PLATFORM AGNOSTIC
AUTOMATION SCRIPTS
• Many tests that need to be run are logically identical
across all platforms.
• Deactivate an account, attempt to sign in, expect failure
• Send a message, verify it shows up in sent items
• Take down some web service, verify an error is produced
• Implemented these tests at an abstract level allows you to
have one test for all platforms
• When the logic changes, the logic only exists one place in
your automation code

AN ABSTRACTED
TEST
[C#]
namespace Automation.Tests
{
public static class SignInWithDeactivatedAccountTest: Test
{
public static void Run()
{
AutomationClient [] clients = new AutomationClient[] {
new WebClient(), new iPhoneClient() }
var account = APIUtility.Accounts.CreateTestAccount();
APIUtility.Accounts.Deactivate(account);
foreach(AutomationClient client in clients)
{
client.SignUpInterface.SignIn(account.email, acount.password);
Log.VerifyFalse(!client.IsSignedIn());
}}}}

CONCLUSION
Developers and Designers frequently have to be able to adapt to
rapidly changing code, automation writers can be just as nimble.

If you design and abstract your code well, you can build
smaller, more robust and agile automation scripts.

If anyone is interested in taking up work on bringing iOS support
to Selenium… Call Me.

MORE INFO
Dan Cuellar
Lead Software Engineer – QA Team
danc@zoosk.com
(Email me to retrieve related source code available for sharing)

See Also
Self-Generating Test Artifacts for Selenium WebDriver
Tue 12:20PM – Track B – Marcus Merrell
Building Quality on Foundations of Mud
Tue 2:40PM – Track A – Kristian Rosenwold
The Restless Are Going Native
Wed 2:20PM – Track A – Dante Briones

Testing Rapidly Changing Applications With Self-Testing Object-Oriented Selenium Infrastructure

More Related Content

Testing Rapidly Changing Applications With Self-Testing Object-Oriented Selenium Infrastructure