Application Lifecycle Management

Best Practice Process Document

Test Management

Scenario / ALM Process

Test Management
Problem Description:
How to define, execute and sign-off tests that are based on new requirements as well as on intended changes and potentially impacted processes
How to ensure that tests are reflecting business requirements, process, functional & technical design How to define test scope How to plan test phases, assign testers How to access and execute tests, document the results and manage defects How to analyze project coverage, test planning status and test execution status

SAP Standards
Test Management

Best Practice Processes:

Lean Test Management with SAP Solution Manager

Core Business Processes are documented with assigned Transactions and test cases in a project to be used to define the test scope for intended change activity. Test Plans and Test Packages are created based on the project . One or more testers are assigned to each Test Package to be able to execute the test and set test status. Test status is evaluated based on Test Package, Plan or Projects. In addition to the lean testing approach attribute-based test planning is possible, test-workflow is realized via Release Status and Test Sequences. Change Impact Analysis with Business Process Change Analyzer (BPCA) is used to optimize test scope for regression tests and automated tests to decrease additional effort for repeating tests. Solution Documentation includes various documents like Business Requirements and Test Objects which are transferred to QC to be treated as Test Scope. Automated Tests are created with SAP Test Acceleration and Optimization (TAO) and updated in combination with Business Process Change Analyzer (BPCA).
2

Advanced Test Management with SAP Solution Manager

Advanced Test Management with integrated SAP Quality Center by HP and SAP TAO

2012 SAP AG. All rights reserved.

Lean Test Management

with SAP Solution Manager
Process Description
Test Scope Definition: The business process structure has been created within a SAP Solution Manager project. At least transactions and test cases (preferred type: Test Document) are assigned to the appropriate structure nodes (in most cases: business process steps). The transactions are assigned as Test Objects to the test cases. This provides a repository of test cases to cover the current project scope as well as for future projects for re-use. It is recommended to check the coverage of business processes by test cases before starting with Test Planning. Test Planning: For each test phase one or more Test Plans are created while selecting test cases from the project to be part of the test plan. Test Packages are created with a subset of test cases to be assigned to the relevant tester(s). Test cases to be executed twice by different testers should be assigned twice to 2 different test packages. It is recommended to check that each test case of the defined scope is assigned to Test Plans and Test Packages before starting test execution. Test Execution: The tester is launching his work list that contains all assigned test packages. If Test Packages are assigned to more than 1 tester, these are handled as a common work list, so relevant test cases are intended to be executed by one of the assigned testers. Detected defects can be addressed via messages using the integrated Service Desk. To get full transparency about test status and related messages it is recommended to frequently perform the related reports during the test phase as well as for sign-off.

2012 SAP AG. All rights reserved.

Lean Test Management

with SAP Solution Manager Test Scope definition
Upload of existing Test Cases

Project
Scenario

Business Process 1

Process Step 1 Process Step 2 Process Step 4

= Transaction

Business Process 2

= Test Case Description

Process Step 1 Process Step 2

2012 SAP AG. All rights reserved.

For new Test Cases a template document can be defined in project standards
4

Lean Test Management

with SAP Solution Manager Test Planning
Project
Scenario

Business Process 1

Test Plan
Test Package 1

Process Step 1 Process Step 2 Process Step 4

Process Step 1 Process Step 2 Process Step 4 Tester 2 Tester 1

Business Process 2 Test Package 2 Process Step 1 Process Step 2 Process Step 1 Process Step 2

Tester 3

2012 SAP AG. All rights reserved.

Lean Test Management

with SAP Solution Manager Test Execution
Test Status

Project
Scenario

Business Process 1

Test Plan
Test Package 1

Process Step 1 Process Step 2 Process Step 4

Process Step 1 Process Step 2 Process Step 4 Tester 2 Tester 1

Business Process 2 Test Package 2 Process Step 1 Process Step 2 Process Step 1 Process Step 2

Tester 3

2012 SAP AG. All rights reserved.

Lean Test Management

with SAP Solution Manager Test Execution
Role

Test Workbench
No

Tester

Yes

Service Desk
Service Desk Employee 1st Level
No Yes

Developer 2nd Level

2012 SAP AG. All rights reserved. 7

Lean Test Management

with SAP Solution Manager Test Reporting
Type of Change New SAP Solution Test Scope Identification Business Blueprint Test Planning Test System Setup SAP Solution Update Change Impact Analysis Assignment of Test Cases to Test Plans and Test Packages ? Test Completeness ? Defect Message resolution ? Test Execution Apply Changes to Production

Sign-Off Test Scope

Sign-Off Test Plans

Sign-Off Test Execution

Coverage of Business Processes by Test Cases ?

Test Status and Progress ? Detected Errors and related messages ?

2012 SAP AG. All rights reserved.

Lean Test Management

with SAP Solution Manager
Advantages
Quick and easy setup, preparation and planning of tests
Test Execution allows guided navigation to application to be tested via Test Object Integrated Service Desk to address and follow-up defects Powerful and flexible reporting on different levels: Project, Test Plans, Test Packages for Test Status and related messages No additional license fees for Test Management Tools needed

Disadvantages
Limitations for larger organizations and projects regarding process variants and high volume test scope Difficulty to analyze impact of change and related test scope for regression test Increasing effort for repeating Tests due to missing test automation Additional effort for coordination and notification of testers during larger cross-organizational scenario tests

Integrated with the ALM process*

Solution Documentation Solution Implementation Upgrade Management Change Control Management IT Service Management

* ALM processes are reflected in scenarios in the Business Process Repository

2012 SAP AG. All rights reserved. 9

Advanced Test Management

with SAP Solution Manager
Process Description
The Lean Test Management approach can be enhanced based on customers needs step by step to the full extent called Advanced Test Management. The following description and slides should be treated as delta information compared to the Lean Test Management approach.
Test Scope Definition: In addition to the lean approach business processes and test cases will be classified via attributes to be able to allow a more focused test planning and reporting for the different types and triggers for tests. This also allows to manage high volume of test cases with reasonable effort. To be able to perform change impact analysis and risk based test scope definition so-called Technical-Bill-Of-Material (TBOM) need to be created for each process step / transaction. Test Planning: Based on the preparation done above the actual test planning can be done much more efficient based on relevant filter criteria. In addition it is possible to use Release Status Schema to allow a phased approach for test planning, execution and related sign-off. This includes notification of assigned tester when starting or ending test phases. For process / scenario based testing that requires the processing of individual test cases in a specific order Test Sequences can be defined to support the workflow and handover via mail notification from one to the other tester. Test Execution: During test execution test coordinators and testers will benefit from using Release Status and Test Sequences. For repeating test activities like regression tests it is highly recommended to setup automated test cases to avoid additional effort for every repetition of tests. Enhanced flexibility and visualization is provided via additional Web-Reports based on SAP BW. These reports can be used to evaluate the current test status, the test progress as well as status and priority of test related messages. Also here the attributes defined during preparation can be used as filter criteria. To fulfill legal requirements like FDA a Test Report can be created for the different milestones to compile all test relevant information per test plan.
2012 SAP AG. All rights reserved. 10

Advanced Test Management

with SAP Solution Manager Test Scope definition (advanced)
Classification of Business Processes via attributes Core: Yes/No Development Layer: Customer, Modification, Enhancement, Standard Customer attributes: e.g. Country, Legal impact (FDA), Responsible Person Classification of Test Cases via attributes Status: In Process / Released Priority Time Required Customer attributes: e.g. Country, Legal impact (FDA), Test Type/Phase, Links: e.g. to Process description, Specification documents for traceability reason Connect Test Case to Test Objects (Transactions) Change Impact Analysis with Business Process Change Analyzer Create TBOM at least for all transactions that are part of the Core Business Processes For intended changes perform Change Impact Analysis to identify running processes that might impacted

2012 SAP AG. All rights reserved.

11

Advanced Test Management

with SAP Solution Manager Test Planning
Business Process Attribute Filter
Allows to filter Test Cases based on Business Process Attributes like Status Team Members Keywords SAP/Customer Attributes

Test Case Attribute Filter

Allows to filter on Test Cases Attributes like: Test Case Type Search Terms Priority Status assigned to the individual Test Case

2012 SAP AG. All rights reserved.

12

Advanced Test Management

with SAP Solution Manager Business Process Change Analyzer
Customizing change Coding change

Planned change

BPCA Execution

BPCA Analysis

Test Plan Generation BPCA Analysis

2012 SAP AG. All rights reserved.

Test Plan generated based on impacted process steps and assigned test cases (manual, automated)
13

Advanced Test Management

with SAP Solution Manager Test Planning with Test Sequences
Standard approach:

Test Plan

Tester 1

Test Package Test Package Test Package

Test Case 1 Test Case 2 Test Case 3 Tester 3 Tester 2

Several Testers (Tester Pool) are assigned to a collection of Test Cases (Test Package)

Sequence approach:

Tester 1

Test Plan

Test Package Test Package Test Package

Create Sales Order Outbound Delivery Tester 2

In addition to standard approach you can assign each Test Case to a Tester and the sequence of test cases can be processed as workflow

Post Goods Issue

Create Billing Doc. Tester 3

Example: As soon Test Case 1 has been processed successfully by Tester 1, Tester 2 will be notified by E-Mail that Test Case 2 is ready to be tested

Benefit:
Less coordination effort during complex scenario tests and accelerated test execution
2012 SAP AG. All rights reserved. 14

Advanced Test Management

with SAP Solution Manager Test Planning with Status Schema
Simplified example to highlight the benefit of Release Status Schema for test plans
Sign-Off Test Scope Sign-Off Test Execution

Phase

Test Planning

Test Execution

After Test

Status

New

Released

Protected

Impact

Test plan is open for changes Test plan is closed for execution

Test plan is closed for changes Test plan is open for execution

Test plan is closed for changes Test plan is closed for execution

Workflow active:

Benefit:

2012 SAP AG. All rights reserved.

Automated E-Mail notification to all Tester that test execution phase is started

Automated E-Mail notification to all Tester that Test execution phase has ended

Allows phased approach with Q-Gates and Sign Off with electronic signature to meet legal requirements
15

Advanced Test Management

with SAP Solution Manager Test Automation with eCATT
Test Automation of SAP Business Processes via eCATT
Business Processes
Sales Order Delivery & Shipment Billing & Payment Receipts Return & Complaint Handling

eCATT Test Automation

Import

eCATT Script 1 Parameter

eCATT Script 2

Parameter

eCATT Script 3

Parameter

eCATT Script 4

included in 1 Test Configuration

Test Data Container SAP System Landscape Landscape 1 (Dev) Landscape 2 (Test) CRM Test 2 R/3 Test 2 ERP Test 2 CRM Dev 1 R/3 Dev 1 ERP Dev 1

Strengths
Testing of complex business process scenarios like Order-to-Cash and Procure-to-Pay
Advanced functionality to test backend business logic (function modules, SOA services, test result verification at SAP table / database level, result comparison with previous runs, ) Central definition via SAP Test Workbench combined with execution in distributed systems Central repository for test data

2012 SAP AG. All rights reserved.

16

Advanced Test Management

with SAP Solution Manager Test Automation with eCATT
Frontend
Web Dynpro SAP GUI

?
SAP CRM, Portal, 3rd applications
via integraded 3rd party test tool

SOA

Backend
Application Server

BC Sets

eCATT

ABAP Objects Function Library & BAPIs Dynpro based UI

Backend
Database Table operations
2012 SAP AG. All rights reserved.

Data Table

Cust. Table
17

Advanced Test Management

with SAP Solution Manager Test Automation with integrated HP QTP
Test Design
Launch eCATT and call HP QTP Record non-SAPGUI application via HP QTP + save script within eCATT

Test Execution
Work Center Test Management User executes assigned Tester Worklist Test Worklist: Execution of eCATT Test Configuration eCATT executes HP QTP script QTP scripts executes non-SAPGUI application eCATT & QTP Test logs
18

Test Workbench: Test result reporting

Manual tests
Automated eCATT / HP QTP tests

2012 SAP AG. All rights reserved.

Test Reporting
Test Report: Documenting Test Coverage and Results
Goal
Meet legal requirement regarding Test Documentation

Scope Provide all test related information in one document:

Project / Solution Test Plan - Responsible Person Test Plan - Overall Results Status - Percentage and absolute view

Related Messages
System(s) under Test Involved Processes Keywords Tester assignment Test Case Description incl. attributes Status history per Test Case

Benefit
Efficient solution to get prepared for audits.
2012 SAP AG. All rights reserved. 19

Advanced Test Management

with SAP Solution Manager
Advantages
Reduced effort and complexity for repeating Test Planning with high volume. Transparency about affected processes and reduced risk for planned changes Reduced effort for regression tests System guided Test Execution with workflow to support complex test scenarios Integrated Service Desk to address and follow-up defects Powerful, flexible and visualized reporting on different levels: Project, Test Plans, Test Packages for Test Status, Test Progress and related messages Better support for internal and external audits and to meet legal requirements No additional license fees for Test Management Tools needed

Disadvantages
Increased effort for initial preparation
Classification of processes and test cases Creation of TBOMs for Change Impact Analysis Setup of Automated Tests

Integrated with the ALM process*

Solution Documentation Solution Implementation Upgrade Management Change Control Management IT Service Management
* ALM processes are reflected in scenarios in the Business Process Repository
2012 SAP AG. All rights reserved. 20

Advanced Test Management

with integrated SAP Quality Center by HP* and SAP TAO*
Process Description
Test Scope Definition: The Advanced Test Management approach with SAP Quality Center integrated into SAP Solution Manager is based on Requirements Descriptions (documents) and Test Objects (Transactions) defined in a SAP Solution Manager project. Typically User Acceptance Tests are referring to the Business Requirements. To be able to manage additional test types like String Test, Integration Test it is recommended to setup additional document types in SAP Solution Manager like Specification and Process description. The assigned Test Objects/Transactions are recommended to be used to trigger Regression Tests. This allows traceability for all types of tests between SAP QC by HP and SAP Solution Manager. To define the Test Scope for a specific test the related documents and transactions need to be assigned to the tab Transfer to SAP QC by HP in the SAP Solution Manager Project. Then the Test Scope can be transferred to SAP QC by HP and appears in the Requirements Module. The requirements appear in the same process oriented structure like in the SAP Solution Manager. Documents are created by default as Business Requirements and Transactions as Test Requirements. New Requirements need to be converted to tests, as most likely no test cases exist. Results of the Change Impact Analysis performed via BPCA can be used to define the test scope for regression tests based on the impacted process steps / transactions but also to identify existing test cases built by SAP TAO that might be affected too and so need to be updated. Test Planning: In the Test Plan Module for the new requirements you can now define the test steps for each test. In case of automated tests the test cases are build based on test components with SAP TAO. Existing Test Cases can be mapped to requirements that are part of the test scope. In the Test Lab Module you then define Test Sets and assign relevant Test Cases from the Test Plan Module. Test Execution: Tests are executed in the Test Labs module and in case of failures defects can be to be reported directly. Test Status as well as Defects are transferred to SAP Solution Manager. Changes on the related Service Desk Message are replicated with the Defect in QC until it is completed and re-test is performed successfully. At the various phases and milestones QC provides adequate reporting capabilities to evaluate coverage, status, progress of test activities as well as defect status. In SAP Solution Manager a report is available to evaluate Test Status for each transferred Business Requirement or Test Object.
2012 SAP AG. All rights reserved.

* additional license required

21

Advanced Test Management

with integrated SAP Quality Center by HP Test

Scope Definition
SAP Solution Manager Project
Scenario
Transfer to SAP QC by HP

Quality Center Project

Requirements Module Requirements - SAP Solution Manager Blueprints Project Name Business Scenarios Business Process
Outbound Delivery

Business Process 1

Outbound Delivery

Business Process Step 1

VL01N VL02N

Process Step 1 Process Step 2 Process Step 4

VL01N

VL02N

Adapter for SAP QC by HP

Business Process 2

Process Step 1 Process Step 2

2012 SAP AG. All rights reserved.

22

Advanced Test Management

with integrated SAP Quality Center by HP Test Planning
SAP Solution Manager Project
Scenario
Transfer to SAP QC by HP

Quality Center Project

Requirements Module Requirements - Solution Manager Blueprints Project Name Business Scenarios Business Process
Outbound Delivery

Business Process 1

Outbound Delivery

Business Process Step 1

VL01N VL02N

Process Step 1 Process Step 2 Process Step 4

VL01N

Test Plan Module

VL02N

Test Plans Adapter for SAP QC by HP

String Test - Delivery

VL01N VL02N

Integration Test
Outbound Delivery

Business Process 2

Process Step 1 Process Step 2

2012 SAP AG. All rights reserved.

23

Advanced Test Management with integrated SAP Quality Center by HP Test Execution
SAP Solution Manager Project
Scenario
Transfer to SAP QC by HP

Quality Center Project

Requirements Module Requirements - Solution Manager Blueprints Project Name Business Scenarios Business Process
Outbound Delivery

Business Process 1

Outbound Delivery

Business Process Step 1

VL01N VL02N

Process Step 1 Process Step 2 Process Step 4

VL01N

Test Plan Module

VL02N

Test Plans Adapter for SAP QC by HP

String Test - Delivery

VL01N VL02N

Integration Test
Outbound Delivery

Business Process 2

Test Lab Module String Test - Delivery

VL01N VL02N

Process Step 1 Process Step 2

-

Integration Test Del.

Outbound Delivery

2012 SAP AG. All rights reserved.

24

Advanced Test Management

with integrated SAP Quality Center by HP
Solution Manager: Application Lifecycle
Change Impact Analysis Build Test Sign Off Deploy Operate Optimize

Requirements

Design

SAP Solution Manager adapter for SAP Quality Center by HP*

Quality Center by HP*

Business / Test Requirements Test Plan Test Composition Test Sets Test Execution Defects

Inspect new SAP Business Processes

Generate new Test Components

Update existing Test Components

Inspect updated SAP Business Processes

SAP TAO: Test Acceleration and Optimization*

2012 SAP AG. All rights reserved.

* additional license required

25

Advanced Test Management

with integrated SAP Quality Center by HP
Advantages
Complete and integrated Application Lifecycle Management using the advanced capabilities of SAP QC by HP and SAP TAO Transparency about affected processes and reduced risk for planned changes Reduced effort for regression tests Reduced effort to create and maintain automated Tests while using SAP TAO Integrated Defect Management in QC with Service Desk in SAP Solution Manager to address and follow-up defects Powerful, flexible and visualized reporting inside QC for: Requirements Coverage, Test Status and Defects

Test Status reporting based on transferred requirements in SAP Solution Manager

Disadvantages
Additional software licenses to be purchased Increased effort for initial preparation
Creation of TBOMs for Change Impact Analysis Setup of Automated Tests

Integrated with the ALM process*

Solution Documentation Solution Implementation Upgrade Management Change Control Management IT Service Management
* ALM processes are reflected in scenarios in the Business Process Repository
2012 SAP AG. All rights reserved. 26

 2012 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase, Inc. Sybase is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C , World Wide Web Consortium, Massachusetts Institute of Technology.

2012 SAP AG. All rights reserved.

27