Automation Project Management and Automation Architecture

By
M.N. Alam
IMI Systems Inc.
Dallas, Tx

Introduction
Numerous white papers and articles are written on test automation. Most are well
researched and present a wide range of views about the benefits of using automation.
Some of the articles are the collection of individual experiences (some are more painful
than others) to make the automation work for them. In this paper, I would like to discuss
automation from project management and architecture standpoint and hope this may help
others to establish a process that fit their testing environment.

Since a great deal of important information is already available on cost/benefit analysis

of automation, I will not attempt to engage in further discussion on this. But I will say
that it is very important to do a cost/benefit analysis of the test cases that require
automating. James Hancock wrote: “When to Automate Testing - A Cost-Benefit
Analysis” which is an excellent article on this subject.

Along with the cost/benefit analysis, detail planning must be completed. I concur with
the experts that a great deal of planning is necessary before jumping onto this
"automation boat". If omitted, the testing journey could be very rough and possibly
cause the fatality of the project. I was involved in a project where a good deal of time
was spent debugging existing test scripts because minor changes were made to the
Application Under Test (AUT). A small change in the application should not require a
great deal of maintenance. I like to discuss the followings and will try to shed some light
on the subject.

- Automation Project Management

- Automation Architecture

Automation Project Management

Since automation is like a development environment, it should be treated like one.
Keeping this in mind, let’s discuss some of the criteria of putting together an automation
project team. At this point, let’s assume that the cost/benefit analysis is completed and
the pay back period has been determined. Since this is a Quality Assurance (QA) project,
it is recommended that the project leader comes from QA or understands QA and
automation. I will discuss later in this section the danger (so to speak) of having a project
leader with very little or no knowledge of Q.A. ( particularly test automation). Once the
project leader is selected, the selections of team members follow.
Below are typical skills necessary for the members of an automation team.

- Development (Programming) skill

- Knowledge of the test script language (familiar with C syntax if you are using Mercury
automated test tools WinRunner and XRunner; familiar with VB syntax if you are
using SQA suite products)
- Ability to write and execute a test plan.

Let’s talk about these skill sets in some detail. Many test automation tool vendors will
like you to believe that using their tool is so easy that a person with no programming
skill should be able to do automation in no time. They are of course referring to
Record/Playback feature of their test tool. In our discussion, let’s assume that
Record/Playback is not a viable option of automation and I think we all know why (lack
of reusable code). Record/Playback is acceptable if you are planning on using the script
for a very short period of time. I personally think that it is nice to have this feature, but I
will not use it unless I have exhausted all other means. This leaves us with the option to
write test scripts programmatically. Familiarity with the test script language is also
important. There is a learning curve for someone who does not have a prior experience
with a particular test tool. Finally, an automation engineer will have to have prior
experience of writing and executing test cases, because automation is no more than
automating the existing manual test cases.

It is also important to select an automation lead person from the members. A lead person should
have prior experience with the tool. If any of the members do not have prior experience, then my
suggestion would be to hire a consultant who has an extensive scripting experience (no
Record/Playback) with that particular test tool. A word of caution while selecting a consultant to
be your automation lead - ask a lot of questions about their previous projects. For example, what
were some of the problems they had to face and how did they get around it; what kind (if any) of
architecture was in place? If you don’t have enough knowledge about the product or automation,
get someone who does, because this lead person can make or break your project. It is a good idea
to hire someone from a reputable consulting firm even if it costs more money. You probably
discovered during the cost/benefit analysis that the biggest part of the automation cost is labor
and not the product. It is worth paying big bucks to consultants to set up automation the right
way. The alternative is poor setup with no architecture and no standards, which causes a lot
maintenance headache down the road, turnover in the automation team, and eventually the
abandonment of the automation. This is one of the dangers of having a project leader with little or
no knowledge of test automation.

After selecting the project leader and the team members, our team is in place. At this
point we are ready to do the design.
Automation Architecture
“Improving the Maintainability of Automated Test Suites” by Cem Kaner is an excellent
article where he talks about two different architectures, Data-driven architecture and
Framework-based architecture as two viable options for automation. My design supports,
for most part, the Framework-based architecture. The followings are the components of
this architecture,

- Main Routine
- Environment Setup
- Screens Navigation
- Test Data

Main Routine

I like to call this the “Engine” of the architecture, because it drives the body. Here the
engineers will organize the steps of execution. I will come back to this and explain what I
mean by organization of steps. Here you can do the necessary environment setups, open
output file/files, load and unload compiled modules. At the end, close the output
file/files.

Environment Setup

In this section, you can set up the necessary system variables (speed, fonts etc.).
Declaring global variables can also be a part of the environment set up. You can also call
Common_Lib (a common library of reusable functions) in this section. The functions in
the common library are reused over and over again by everybody in the automation team
throughout the existence of automation. This can be dynamic and additional user defined
functions can be added to the library as needed.

Example of reusable functions:

- Opening and closing of the output files

- Drag and drop of an object (location parameter driven - change the parameter according
to the location of the object need to be dragged and dropped)
- Status Check (location parameter driven - change the parameter according to the
location of the status bar on the screen)
- Single mouse Click (location parameter driven - change the parameter according to the
location of the objects need to be clicked)
- Double mouse click (location parameter driven - change the parameter according to the
location of the objects need to be double clicked).

Screens Navigation
 The purpose of this section is to navigate through a screen using a number of user-
defined functions. It can be done many different ways and it is up to the engineers to
decide the best way for their environment. In most cases, engineers write one function
per screen. I prefer writing a number of small and simple functions for each field on the
screen instead of writing one big function per screen.

For example, let’s assume we have a screen called “New Customer” with the following
edit fields and push buttons:

- Customer name
- Street address
- City
- State
- Zip
- Telephone number
- Enter
- Cancel

I would write functions for each edit field and a single mouse click function for push
buttons to navigate through the “New Customer” screen.

Below are some examples (Test script language for WinRunner and XRunner),

# Inserts customer name

function Cust_Name(in cust_name)
{
Type_Text(cust_name);
return;
}

# Inserts customer address(Street, City, state and zip)

function Cust_Street(in st_addr)
{
Type_Text(st_addr);
return;
}

function Cust_City(in city)

{
Type_Text(city);
return;
}

function Cust_State(in state)

{
Type_Text(state);
return;
}

function Cust_Zip(in zip)

{
Type_Text(zip);
return;
}

# Inserts customer telephone number

function Tel_Phone(in ph_no)
{
Type_Text(ph_no);
return;
}

# Clicks button (Right or left mouse click)

function Click_Button(in loc1, in loc2, in loc3, in button)
{
if (!button)
button = 0;
move_locator_abs (loc1, loc2, loc3);
if (button==0)
click("Left");
else if (button==1)
click("Right");
return;
}

There are two advantages to use this over writing one function per screen. First, these are
small and simple which makes maintenance easy and quick. But the biggest advantage is
the code is reusable. If you have other screens that require customer name, address and
telephone number, you should be able to use the same code with different test data.

Test data

This is a separate test script contains test data which are used to populate the edit fields.
These are one-dimensional arrays. Below are some examples:

public test_data [] ={
"Cust_Name" = {"John Doe"},
"Cust_Street" = {"1234 Main street"},
"Cust_City" = { "Dallas"},
"Cust_State" = {"Tx"},
"Cust_Zip" = {"75240"},
"Tel_Phone" = {"972-221-1212"}
};
You may have a small or a large number of test data depending upon the nature of the
test you are performing. It is better to have a large number of test data than a large
number of test scripts because it is easier to maintain test data than test scripts.

Automation is easy if you have to navigate through the same screen with different test
data. In our example above, we can easily automate a test case that checks to see the
maximum number of customers can be created (boundary test). In this case we will use
the same code over and over with a different set of test data. But in most cases, it
requires the use of different set of code with a different set of test data. This is where the
organization of steps comes in.

The following scenario may help you to understand,

Let’s say in order to automate a test case (create a customer) you need to navigate
through three different screens with different edit fields and they are in following order:

- New Customer (Customer name, Street address, City, State, Zip, Telephone)
- Contact Information (Contact name, Street address, City, State, Zip, Telephone)
- Bank information (Bank name, Street address, City, State, Zip, Telephone)

First of all we need a number of Test Data for these edit fields. We will follow above
example (one-dimensional arrays) to set up the test data.
We then need to write a number of user defined functions to navigate through these
screens (Screens Navigation). You may want to either write a single function for each
screen or write a number of small functions for each edit field in the screen. I found the
later more useful.
Test data and all the functions written in Screens Navigation are compiled modules. The
advantage of using compiled modules over regular test scripts is that execution of
compiled modules is faster. But loading too many compiled modules also causes
performance degradation. So, it is a good idea to unload each compiled module after the
execution of that module.
The organization is usually done in Main Routine. In Main Routine we will,
- Call the test script to set up the environment
- Load compiled modules
- Open output files
- Call the first test script (let’s call it ‘Create_Customer’)
If no error run New_Customer(1)  indexed (pointing to a particular set of test data)
from ‘Create_Customer’ test script.
If no error run Contact_Information(2)  indexed (pointing to another set of test data)
from ‘New_Customer’ test script.
If no error run Bank_Information(3)  indexed (pointing to another set of test data)
from ‘Contact_Information’ test script.
- Unload Common library
- Unload compiled modules
- Close output files

. One could make the following arrangement in Main Routine,

- Call the test script to set up the environment
- Load compiled modules
- Open output files
- Call the first test script (Create_Customer)
- If no error
- {
Run New_Customer(1)
If no error
{
Run Contact_Information(2)
If no error
{
Run Bank_Information(3)
}
}
}

- Unload Common library

- Unload compiled modules
- Close output files

In both cases if there is an error, print the appropriate error message to the output file and
exit out of that particular routine. While writing results to an output file, it probably is a
good idea to have enough information (eg: Test case number, Test case objective,
Objective of a particular step etc.) so that in case of a “Fail” status you will know exactly
where to look in your test scripts. This helps a great deal in debugging the test scripts.

Conclusion
If this seems like a lot of work, it is. It takes time to set up architecture, but once the
standard and the architecture is in place, the subsequent development time is
considerably shorter. This architecture may not suit every environment, because it
requires test engineers to have some degree of programming skill. I don’t know any
effective automation process that does not require some degree of programming skill.
The best part of this architecture is more reusability and less maintenance. Reusability
and maintainability go hand in hand. The more code can be reused the less code need to
be written and less the code, less the maintenance. One thing to remember is that
reusability and complexity (complexity of code) also go hand in hand. The more
reusability you want the more complex code you need to write. It is not always easy to
maintain a complex set of code when you don’t have skilled personnel. The reality is you
are probably not going to find a large number of testers who have sophisticated
programming skill. So, it is safe to suggest writing simple code such that a person with
little programming skill can maintain these scripts fairly easily.

NOTE: I have written this paper based on my experience with a number of automation
projects I was associated with. There is no one way to do automation. Every project is
different and every place has their own way of doing things. I have found this particular
architecture seems to work provided you have skilled personnel and the management
who understands and supports the process. It is not to say that other form of architecture
or Record and Playback will not work. No matter what form of architecture you decide
to work with make sure that most everybody in the group is comfortable with it, because
it is the people (not the tool[tool helps]) makes a product as good as it can be(or in most
cases as market allows it to be).