Section 4

Regression Testing
By
Syed Usman Ahmed
 Lessons in Section 4
• What is regression testing?
• Automating regression tests.
• Process of automating regression test.
• Run all regression test
• Improving the regression test
• Bug Fixing
 What is regression testing?
Regression testing is testing done to check that a system update does not re-
introduce errors that have been corrected earlier.
All – or almost all – regression tests aim at checking
• Functionality – black box tests.
• Architecture – grey box tests
 What is regression testing?
Since they are supposed to test all functionality and all previously done changes,
regression tests are usually large.
Thus, regression testing needs automatic
• Execution – no human intervention
• Checking. Leaving the checking to developers will not work.
 What is regression testing?
Since they are supposed to test all functionality and all previously done changes,
regression tests are usually large.
Thus, regression testing needs automatic
• Execution – no human intervention
• Checking. Leaving the checking to developers will not work.
 Automating regression tests
We face the same challenge when doing automating regression test as we face
when doing automatic test checking in general:
Which parts of the output should be checked against the oracle?
This question gets more important as we need to have more version of the
same test due to system variability.

*Please refer lesson-1 to get more information on oracle.

 Automating regression tests
Simple but annoying – and some times expensive – problems are e.g.
• Use of date in the test output
• Changes in number of blanks or line shifts
• Other format changes
• Changes in lead texts
 Automating regression tests
A simple solution seems to be to use assertions. This is not a good alternative
for the following reasons: The assertion will have to be
• Inside the system all the time – extra space and execution time
• Only in the test version – one extra version per variation to maintain.
 Automating regression tests
The answer to the question on the previous slide is to parameterize the test
results.
Instead of using a complete output file as the oracle we use:
• A tool to extract the relevant info from the output file
• Compare the extracted info with the info stored in the oracle
Process for automating regression tests
Build results
Test for version x
data

Build tests
for version x

Run tests
Compare Verdict
for version x
 Run all regression tests
Since a regression test is large, it is always a need to identify which parts of the
regression tests need to be run after a change – e.g. an error correction.
Thus, traceability – which components are used to realize which functionality
or requirement – is important info for two reasons:
• Save execution time. Fast and cheap hardware is, however, steadily reducing
this need.
• Know which tests to change when we change functionality.
 Improving the regression test
Once we have a regression test, it is important to update it each time we
• Fix a bug
• Add, change or remove functionality
• Change platform

If we create variants of the system, we also need to create parallel variants of

the regression test suite.
 Bug fixing
When somebody reports a bug, the first activity is to reproduce it – preferably
with the simplest input sequence possible.
When the bug is reproduced, we need to make a new entry in the regression
test. This entry will be one of the following
• The original reported input sequence
• The simplest possible input sequence
 Bug fixing
• The original reported input sequence
• Pro: is the real thing and will tell us something about how the system is used in real life.
• Con: can be large, complex and difficult to understand
• The simplest possible input sequence
• Pro: is (usually) simple to understand
• Con: is an artificial input sequence and has no info beside being able to reproduce a
reported bug
Thanks!!