Title/Description: Semi-Automation Flow and Guidelines Original Author: Pragya Dwivedi Date: 05/26/09

Revision History
Issue Date 26-May-09 Rev 1.0 Description of Change Initial version Initiated By Pragya Dwivedi

Starent Networks Corp.

Semi-Auto Flow and Guidelines

Rev1.0

Table of Contents
TABLE OF CONTENTS.................................................................................................2 1 OVERVIEW....................................................................................................................3 2 SEMI-AUTOMATION FLOW DIAGRAM................................................................3 2.1 SEMI-AUTOMATION FOR TEMPLATE-BASED TESTS .................................................................4 2.2 SEMI-AUTOMATION FLOW FOR NON-TEMPLATE-BASED TESTS.................................................5 3 SEMI-AUTOMATION CODING GUIDELINES.......................................................5 4 SEMI-AUTOMATION CODE REVIEW GUIDELINES..........................................6 5 TEST PROCEDURE GUIDELINES............................................................................6

Proprietary and Confidential Information

Page 2 of 7

Starent Networks Corp.

Semi-Auto Flow and Guidelines

Rev1.0

1 Overview
Document describes the semi-automation flow QA group (Pune) should follow. It also enlists Semi-Automation Coding Guidelines and Test Procedure Guidelines.

2 Semi-Automation Flow Diagram

Tests can be categorized as template-based or non-template based. Template-based tests are mostly developed for product features and test procedures exhibit similarity to large extent. Therefore if automated script is developed for few tests; the same can be referred as template by other tests for that feature. Non-template-based tests are unique in nature and are mostly developed for customer reported problems.

Proprietary and Confidential Information

Page 3 of 7

Starent Networks Corp.

Semi-Auto Flow and Guidelines

Rev1.0

2.1 Semi-Automation for template-based tests

1. Module Ready for Semi-Auto 2. Auto Resources Identified 3. Test Owner gives modules overview to Auto Owner

QADB Test Procedures

Provides test templates (Test Owner) Reviews test (Auto Owner)

Test Procedure Guidelines

Change Test (Test Owner)

Yes

Test Procedure change needed? (Auto Owner)

QADB Test Procedures Updated

No

New Infra Support Needed (Auto Owner)

Yes

Generate QPRs for New support and add support (Auto Owner)

Gnats updated

No

Automation Coding Guidelines

create templates for module (Auto Owner)

Auto Repository Updated

Review Automated Test templates (Test Owner and Automation Reviewer)

Modify Automated Template (Auto Owner)

Yes

Automated templates Need change? (Test Owner) No Semi-automate few more tests in module (Test Owner)

Auto Repository Updated

Semi-Automation coding guidelines

Auto Repository Updated

Semi-Automation code review checklist

Review Tests (Auto Owner)

Auto Repository Updated Modification Needed Yes Test Owner Modifies Test

No Proceed with semi-auto of remaining tests

Proprietary and Confidential Information

Page 4 of 7

Starent Networks Corp.

Semi-Auto Flow and Guidelines

Rev1.0

2.2 Semi-Automation flow for non-template-based tests

Test Procedure Guidelines

Test Ready for Semi-Auto

Automation API document

New Infrastructure Support Needed (Test Owner) Yes

Generate QPRs for New support (Test Owner)

Gnats updated

Automation Coding Guidelines

Fix QPRs (Auto Owner)

No

Auto Repository Updated

Re-fix QPRs per comments from test owner (Auto Owner)

No

Fix works per requirements? (Test Owner) Yes

Gnats updated

Semi-Automation coding guidelines

Proceed with semiautomation

Auto Repository Updated

Semi-Automation code review checklist

Review Needed? (Test Owner)

Yes

Get test reviewed by automation and incorporate comments

Auto Repository Updated

No

Semi-Automation of test completed

3 Semi-Automation Coding Guidelines

QA engineers responsible for semi-automation should follow the guidelines below.

Proprietary and Confidential Information

Page 5 of 7

Starent Networks Corp.

Semi-Auto Flow and Guidelines

Rev1.0

1. A semi-automated test should not have any syntax errors. Syntax errors are observed due to
incorrect usage of APIs or TCL commands. This can be avoided by understanding the API usage. 2. Tests should not have any hardcoded IPs or names of test devices, interfaces etc. These should be fetched from rack or configuration file. Most of the ips and device names would be available as public variables in base class. 3. Direct use of runCmds like callgen commands, boxer show commands should be avoided. 4. Tests should not have reference to private data [data in personal clones/homes/localdisks is referred to as private data]. Data like tft/configuration file should be part of BK. 5. Huge configuration in tests should be avoided. It could be part of configuration file. 6. Unwanted configuration should not be present in tests. 7. Configuration, rack files used for test should be shared with automation. 8. Log files generated by executing semi-automated tests should be shared with automation. 9. Unwanted delays in tests should be avoided. If there is a need of hard delays, those should be well documented. 10. Use of dangerous/prohibited commands like deleteContext APIs. Using this API and commands like reload is prohibited as by doing this we re-configure the system afresh and hence automation would loose critical bugs which could be caught otherwise. 11. Tests should not have unwanted callgen arguments. 12. Wherever possible, automation APIs should be used. API document can be referred to fetch correct usage. Automation Engineers should help in locating proper APIs if its not easily visible. 13. Naming convention [same as automation coding guideline] to be followed. 14. Tests should not have monitor started for unnecessary protocols

4 Semi-Automation Code review guidelines

Automation Engineers responsible for semi-automation script reviews should review the code to ensure that semi-automation coding guidelines are followed.

5 Test Procedure guidelines

A comprehensive test procedure makes test automation easier. Feature test owners should follow these guidelines while developing test procedures. Automation engineers should check test procedures against these guidelines while reviewing test procedures. 1. Tests should have correct priority assigned. 2. Procedure should have correct physical/logical configuration. Special setup requirements
should be mentioned explicitly.

3. Procedure should clearly indicate the verification points. Like the stats to be checked, 4.
attributes in monitor dump. CLI/Monitor output recorded during test execution should be indicated in test procedures for reference. Mentions radiuslite/callgen versions to be used. By default all tests should use the latest matching versions but sometimes tests use particular callgen/radiuslite versions because of unavailability of needed support or due to PR against other versions. Manual owner mostly has this information. If this information is reflected in test procedure, engineers wont have to dig much if they face issue simulating the test. Specific versions should be kept in common location.

Proprietary and Confidential Information

Page 6 of 7

Starent Networks Corp.

Semi-Auto Flow and Guidelines

Rev1.0

5. Mentions workaround for any tool related issues. 6. Many times one test combines multiple scenarios. Such tests should be split into multiple tests per scenario. This makes it easy for anyone to debug the test failure and also indicate pass/fail precisely. Else one of the 10 scenarios in tests would fail but whole test would be marked as failed. 7. Tests should mention explicitly applicable version, customers for the test. Applicable version for a test as mentioned in QADB is the version for which test scenario is valid. 8. Test specific configuration should be mentioned explicitly. 9. Pass/fail criterion should match with test purpose and procedure. 10. Procedures should be in-lined with Requirement field (eg PRs). 11. Tests should mention callgen arguments clearly. 12. Tests should be properly formatted. All tests steps should have step numbers indicated.

Proprietary and Confidential Information

Page 7 of 7