Test

Automation
Fundamentals

Mikó Szilárd
2016
EPAM

University Material 2
Blue-chip clients rely on EPAM

University Material 3
SCHEDULE

9.12 Intro Monday 12:15

9.19 Unit testing 1
9.26 Unit testing 2
4 homeworks
10.03 Continuous integration 1
10.10 Continuous integration 2
10.17 Selenium WebDriver 1
10.24 Selenium WebDriver 2
10.31
11.07 Selenium WebDriver 3
11.14 Selenium WebDriver 4
11.21 Mobile Testing 1
11.28 Mobile Testing 2
12.05 Mobile Testing 3

University Material 4
Agenda

1 Introduction

2 Approaches

3 Tools and frameworks

4 Test Execution

University Material 5
Introduction

1 History

2 Motives

3 Advantages

4 Best Practices

5 Expectations and Reality

6 Two disciplines

University Material 6
 Feedback Control Mechanism

University Material 7
 40s – Concept of Automation

University Material 8
 Future of
Automation

University Material 9
 Motives

300
Million

250
Source lines of code
history
200

150

100

50

0
1980 1990 2000 2010 2020
Car Operating system

University Material 10
Motives

RISK

University Material 11
Motives

University Material 12
 Repeatable &
schedulable

Fast test
execution
Reliable

Increased
functional
coverage
Automatic
reporting
University Material 13
Best Practices

Automation is recommended for

• Static and Repetitive testing

• Data driven testing

• High complexity testing

• Load and Performance testing

• Smoke testing

• Regression testing

University Material 14
Best Practices

Automation is NOT recommended for

• One-time, ad-hoc or exploratory testing

• Usability or look-and-feel testing

• Testing a feature under development

• Testing 3rd party services

• Cases where time and cost investment is too high

University Material 15
 Expectations and Reality

• Automate at once • Automate gradually

• Cut testing budget • Increase test coverage

• Reduce testing time • Reinvest time savings

• Automate everything • Keep coverage and quality

• Every team can use automation tools • Skills and training needed for success

University Material 16
Two disciplines

• Test and automation are two different disciplines

Test Automation

Application expertise Development expertise

What to test How to automate

Test cases Test code

University Material 17
Approaches

1 Linear scripting – Record and Playback

2 Structured scripting - Library Architecture

3 Data Driven Testing

4 Keyword/Table Driven Testing

5 Hybrid Testing

6 Test Driven Development

7 Acceptance Test Driven Development

8 Behavior Driven Development

University Material 18
 Linear scripting – Record and Playback

• Manual recording of each test step • Playing recorded test in the future

• Two types of verification • Differences reported as errors

– Insert verification checkpoints

– Automatic record of all responses
• Simplest approach

University Material 19
Linear scripting – Record and Playback

Advantages

• Little training and setup time

– Technical expertise not mandatory

• Develop new tests on the fly

• Able to capture steps for

– Later diagnosis
– Bug reproduction

• Could help understanding test scripting

University Material 20
Linear scripting – Record and Playback

Disadvantages

• Manual capture is unavoidable

• Hard or impossible to maintain

• Test data is hard coded

• Higher chance of redundancy

• Higher chance of false positives

University Material 21
Linear scripting – Record and Playback

Improvements

• For windowed applications

– Use fixed screen format

– Verify only a selected area

• Ensuring the same input data

– Use a database backup

– Use stubbed data

• Use variables to store dynamic data

University Material 22
 Structured scripting - Library Architecture

• Create an abstraction layer for testing • Use control structures (conditionals, loops)

• Each module or function has its own test • Create a common library from reusable parts

scripts
• Build test cases using test functions

University Material 23
Structured scripting - Library Architecture

Advantages

• Scalable

• High level of code reuse

• Easy to adapt functionality changes

• Lower maintenance cost

University Material 24
Structured scripting - Library Architecture

Disadvantages

• Test data is hard coded

• Technical expertise required

• Longer planning time

• Longer test script preparation

• Deeper knowledge of the application

University Material 25
Data Driven Testing

• Test data shouldn’t be hard coded

• Divide test data and test code

• Storage options

– Database, XML, Excel or CSV file

• Read inputs and expected outputs as key-value pairs

• Run your tests against multiple sets of data

University Material 26
Data Driven Testing

Advantages

• Early test data creation is possible

• Script changes doesn’t affect test data

• One script can run with multiple sets of data

• Manipulating test data could mean

– Adding, removing or modifying test cases

• Lower maintenance costs

University Material 27
Data Driven Testing

Disadvantages

• Technical expertise required

• Longer planning time

• Longer test script preparation

• Test data should be maintained carefully

University Material 28
Keyword/Table Driven Testing

• Store test data and actions

separately from code

• Create keywords for actions

• Create keyword parameters from test data

• Create test cases with a series of

keywords

University Material 29
Keyword/Table Driven Testing

Main process stages

Test preparation
Defining test basis

Test design
Analyzing test basis Test case design Test data design

Manual execution

Automating test execution

Implementing keyword actions

Automated execution

University Material 30
Keyword/Table Driven Testing

Advantages

• Early test design creation is possible

• High level of code reuse

• Independence from test tools

• Readable test cases for stakeholders

• Testing work can be distributed

– Test design needs application expertise
– Implementation needs automation expertise

University Material 31
Keyword/Table Driven Testing

Disadvantages

• Very high initial investment

• High automation expertise required

– Creating and maintaining the framework

• Not feasible for smaller projects

University Material 32
Hybrid Testing

• Combination of other approaches

• Over time it’s a natural evolution step

– Structured data driven testing

– Structured keyword driven testing

• Aims to have all their advantages

• Has most of their disadvantages

– Initial investment and needed expertise are high

University Material 33
“If it's not worth testing, why are you wasting
your time working on it?” - Scott Ambler

University Material 34
Test Driven Development

• Introduced a new evolutionary approach

– Test first development

– Refactoring

• Break down requirements to a test case

• Improve on software only that much to pass the new test

• Refactor design flaws and rerun all tests to catch regression

• When all test passed you can start it over

University Material 35
Test Driven Development

University Material 36
Test Driven Development

• Focuses on developers

• Developers doing TDD need

– Strong understanding of requirements

– Knowledge at least in unit testing

• Good unit test characteristics are

– Runs fast
– Runs in isolation
– Easy to read and understand
– Uses real data
– Represents one step towards your overall goal

University Material 37
Acceptance Test Driven Development

• Extension of test driven development

• Acceptance tests drive the development process

– Creating a layer between requirements and developer tests

– Helping to communicate and understand customer needs

• Through them all stakeholders could be involved

– Customers, Business, Testers, Developers

University Material 38
Instead of „What should I test?”,
you might hear
„What’s the intended behavior?”.
- Neel Lakshminarayan

University Material 39
Behavior Driven Development

• Improvement of acceptance test driven development

• Business goals drive the development process

• Enables a common language between stakeholders

• Through the use of a simple domain-specific language

– Domain specific phrases in English sentences

University Material 40
Behavior Driven Development

• Acceptance tests should be written as a user story

– As a [role]
– I want [feature]
– so that [benefit]

• Acceptance criteria should be written as scenarios

– Given [initial context]
– When [event occurs]
– Then [ensure some outcomes]

University Material 41
Behavior Driven Development

User story example

Title: Customer withdraws cash

As a customer,
I want to withdraw cash from an ATM,
so that I don’t have to wait in line at the bank.

Scenario example

Scenario: Account is in credit

Given the account is in credit
And the card is valid
And the dispenser contains cash
When the customer requests cash
Then ensure the account is debited
And ensure the card is returned
And ensure cash is dispensed

University Material 42
Tools and Frameworks

• Test automation tools can help in automating such task as

– Product installation

– Test data creation

– API interaction

– GUI interaction

– Test execution

– Problem detection

– Defect logging

• They serve as a driving agent for the automation process

University Material 43
• An automation framework is not a • Frameworks put automation
tool which performs a specific task approaches into practice

• It is an infrastructure that provides • Building a framework is a

a unified solution strategic decision
University Material 44
Return on Investment

A profitability measure that evaluates the performance of a

business by dividing net profit by investment.

𝑁𝑒𝑡 𝑝𝑟𝑜𝑓𝑖𝑡 𝐺𝑟𝑜𝑠𝑠 𝑝𝑟𝑜𝑓𝑖𝑡 − 𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡

𝑅𝑂𝐼 = =
𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡 𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡

• Good metric to measure return of money per period

• Could be used as an indicator to compare different alternatives

• However it is risky since a pre calculation is based on assumptions

University Material 45
Test Execution

1 Test Log

2 Error Log

3 Analyzing Results

4 Reporting

5 Metrics

University Material 46
Analyzing Results

• Classify false positives

– Test environment

– Application changes

– Test error

• Avoid reporting the same failure

• Keep on eye on false negatives

– Test execution finishes surprisingly fast

– Verified message appears in a different place

University Material 47
Metrics

University Material 48
Questions

University Material 49