Quality Management KPI Metric Determination

February 2013

Roadmap for KPI Identification (an example)

Quality Scope

Business Continuity

Area

Automation

Mission

Replace manual
procedures

Challenge

Automation in job scheduling

management

Metric

# manually planned or scheduled jobs

Target

2011 SAP AG. All rights reserved.

KPI

Reduce # manually planned or scheduled

jobs by 25% until end of 2013.

QM-BC: Areas for Improvement

Area

Mission

Automation

Process Quality

Replace manual
procedures

Defined process
KPIs

Proactive
Support

Adjusted and
improved
documentation

2011 SAP AG. All rights reserved.

Managed
Integration

Mainstream new
products /
technologies

Reduced costs
of operations

Efficient
Operations

Increase IT
Value

QM-POI: Areas for Improvement

Area

Solution Transition
Quality

Development
Management

Security

Project Control

Mission

Solution is
maintained up to
date

custom
development
stability

Authorizations
maintained

Managed quality
gates

Changes are
analyzed for
impact

Custom
development
standards

Authorizations
reviewed

Integrated
partners

Tested business
processes

Custom
development
documentation

Managed access
(firewall)

Established
project
methodology

Master Release
Plan

ICC

Communication

Updated footprint

2011 SAP AG. All rights reserved.

QM-BPI: Areas for Improvement

Area

Mission

Continuous
Improvement

Business
Alignment

Innovation

Business
deliverables

Business
feedback

Key Drivers

Quality Awareness

Business signoff

ICC

Communication
Plan

2011 SAP AG. All rights reserved.

QM-SIV: Areas for Improvement

Area

Risk Mitigation

Handover to
Operations

Operations
Planning

Mission

Integration
Validation

Operations
readiness

Operations
procedures

Risk Awareness

Operations
solution review

Documented
business
processes

Operations skills

2011 SAP AG. All rights reserved.

Standard Metrics for BC: Automation

Area

Mission

Challenge
Automation in job scheduling
management

Metric
Ratio of automated versus manually
scheduled jobs (a)
Ratio of resource inolvment in job
scheduling (m)

Replace manual
procedures

Automation in test
management

Coverage of business processes in

test cases (a)

Automation

Coverage of EEM
Proactive
Support

2011 SAP AG. All rights reserved.

Ratio of automated test cases vs.

overall test cases performed (a)

Coverage of Business
Process Monitoring

# EEM locations (a)

Ratio of business processes in

Business Process Monitoring (a)

Standard Metrics for BC: Process Quality

Area

Mission

Challenge

Metric
Coverage of E2E standards (m)

Process Maturity
# of Process maturity assessments
(m)

Defined process
KPIs

Ratio of KPIs per IT process (m)

Process Improvement

Ratio of IT processes with

KPIs (m)
Frequency of adjustments of
process KPIs (m)

Process Quality

Adjusted and
improved
documentation

2011 SAP AG. All rights reserved.

Change completeness Updated

Solution Documentation and
Operation Handbook

% incidents with Priority 1/2 that are

caused by missing or wrong solution
documentation change or operations
handbook (m)

Change completeness - test case

adjustment with change request /
implementation and transport/ E2E
processes

% test cases reviewed or changes

(after identification of affected
ones) (m)

Standard Metrics for BC: Managed Integration

Area

Mission

Challenge

New product incidents

Metric

# incidents with priority 1/2 after go-live

of new products / technologies within
first 3 months (m)
Coverage of product trainings
sessions for end users (m)

New product skills assessed

Managed
Integration

Mainstream new
products /
technologies

Ratio of incidents identified as

training issues (m)
Coverage of product trainings
sessions for IT staff (m)

New organizations reviewed

New tool integration

2011 SAP AG. All rights reserved.

# up-to-date skill reviews (m)

# maturity assessments performed

(m)

# maturity assessments performed

(m)

Standard Metrics for BC: Reduced Costs of Operations

Area

Reduced costs
of operations

Mission

Challenge

Metric

Incident effort reduction

Time for incident resolution (priority

1/2) (a)

Incident solving experts

# experts required to solve critical

incidents within customers
organization (m)

Efficient Operations

IT Value

Reduce TCO

Increase transparency of IT
Value

2011 SAP AG. All rights reserved.

% TCO vs. revenue (m)

Trend in escalation / Days

with an outage (m)

10

Standard Metrics for POI: Solution Transition Quality

Area

Mission

Challenge

Solution is maintained
up to date

Solution Maintenance

Changes are
analyzed for impact

Change impact analysis

Solution Transition
Quality

Metric
distance from your current release, EhP,
and switch (m)

# change impact analysis projects in SAP

Solution Manager (a)

Tested business
processes

Master Release Plan

2011 SAP AG. All rights reserved.

Business processes tested

Master Release Plan

frequency

# critical business processes steps tested

for performance regularly (m)
# of MRP updates documented per annum
(m)
MRP update documented per SAP Release
Strategy update (m)

11

Standard Metrics for POI: Development Management

Area

Mission

Challenge

Metric
% of incidents (prio 1/2 ) caused by
custom developments (m)

Custom development
stability

Upgrade change impact

analysis

Custom code clearing

Development
Management

Custom development
standards

Custom development
documentation

Custom development
standards

Custom development
documentation

Ratio of completed analysis projects in

Upgrade Change Impact Analysis (m)
Ratio of completed analysis projects in
Custom Code Clearing (m)
ratio of incidents caused by custom
development (a)
Level of clear development standards
documented (m)
# development projects without review
of development standard compliance
(m)
% custom developments documented
according to SAP Standard (Sol.
Documentation for Custom
Development) (m)
% of custom code objects not
compliant to documentation standard
(m)

ICC

Reduce modification

% of custom code objects reviewed

incl. documentation standard (m)
% of reduction of unnecessary
modifications (m)

2011 SAP AG. All rights reserved.

12

Standard Metrics for POI: Security

Area

Mission

Challenge
Profile changes

Authorizations
maintained
Security notes

Security

2011 SAP AG. All rights reserved.

Authorizations
reviewed

Security notes

Managed access
(firewall)

Infrastructure assessment

Communication

Alignment Meetings

Updated Footprint

Updated Standards/Policies

Metric
Ratio of profile changes for users in
one year (m)
Distance to implemented security note
from released security note (m)
# incidents caused by not implemented
security notes (m)
% security reviews vs new releases
(m)
# reviews of assigned roles vs. actual
requirements (m)
# infrastructure assessment per annum
(m)

# alignment meetings (m)

Ratio of changes in standards/policies

leading to IT reviews (m)

13

Standard Metrics for POI: Project Control

Area

Mission

Challenge

Quality Gates Defined

Managed quality
gates
Quality Gate Criteria
Project Control

Metric

% projects with quality gates defined

(m)
% of quality gates passed by 1st
attempt (m)
# of criteria defined for each quality
gate (m)
# of quality gates passed by mgmt.
decision (m)
# incidents caused by ignored quality
gates (m)

Integrated partners

Risk Management of
Partners

Alignment meetings/ Participation in

risk mitigation reviews

Established Project
Methodology

Project Methodology
aligned

% projects executed according to

project methodology (m)

2011 SAP AG. All rights reserved.

14

Standard Metrics for BPI: Continuous Improvement

Area

Continuous
Improvement

Mission

Challenge

Metric

Usability - Enhancements

# usability enhancements resulting

from Business Feedback (m)

Usability - Problems

# review of usability with users (m)

Business deliverables
Functionality - Assessments

# SMA per major release (m)

Trend: # prototypes

Functionality - Prototyping
Ratio of prototypes built in sandbox
systems to implemented ones in
Business (m)

2011 SAP AG. All rights reserved.

15

Standard Metrics for BPI: Continuous Improvement

Area

Mission

Challenge

Establish Quality Roles

Continuous
Improvement

Metric

% Completeness of Quality Challenges

(m)

Quality Awareness

Execution of Quality Plan

Measurement frequency of KPIs (m)

Continuous Learning

# Lessons Learned performed with

Business (m)
# Success Stories with Business
communicated in the organization (m)

2011 SAP AG. All rights reserved.

16

Standard Metrics for BPI: Business Alignement

Area

Mission

Challenge

Metric
# offered (service catalog) IT-services

Service Catalog
% usage of offered services (m)

Business feedback

# alignment meetings between Business & IT (m)

Alignment meetings

# action items from alignment meetings between

Business & IT (m)
average duration until completion of action items
(m)

Business
Alignement

Business testing

% test cases performed by Business (m)

Business change
acceptance

% change requests confirmed/accepted by

Business (m)

Business signoff

Communication
Plan

2011 SAP AG. All rights reserved.

Informed business

# newsletters /events o inform business users (m)

17

Standard Metrics for BPI: Innovation

Time for completion of Rapid

prototyping (m)

Innovation

Key Drivers

Speed to Respond to
market opportunity

# projects resulting from Rapid

prototyping (m)

Business Feedback rating (m)

ICC

2011 SAP AG. All rights reserved.

Reduce modification

% of reduction of unnecessary
modifications (m)

18

Standard Metrics for SIV: Risk Mitigation

Area

Mission

Challenge
Integration test without
critical issues

Integration Validation

Risk Mitigation

Volume/load test without

critical issues

Metric
# of new issues of type
functional/technical identified during
integration test phase (m)

# of issues of type ** identified during

performance test phase (m)

Reduced duration of
stabilization period

Days after go live until

stabilization/hypercare is no longer
required (m)

Minimize costs for

stabilization

Stabilization by add.
Resources/incidents or end user (m)

Risk Awareness

Documented
business processes

2011 SAP AG. All rights reserved.

Risk tracking

Frequency of risk reviews (m)

Completeness of solution
documentation

% critical business processes

documented

19

Standard Metrics for SIV: Handover to Operations

Area

Handover to
Operations

Mission

Operations
readiness

Challenge

Completeness of solution
operations documentation

% of changes of operation handbook after

GoLive

SignOff

% of completion SignOffs by Business

Sufficient skills

Monitoring procedures
established

2011 SAP AG. All rights reserved.

Metric

% of training schedules performed

Coverage of automated monitoring for critical

business processes

20

Standard Metrics for SIV: Operations Planning

Area

Mission

Challenge

Metric
Ratio of business processes in
Business Process Monitoring (m)

Business Process
Monitoring
Operations
procedures

Operations
Planning

Operations solution
review

Operations skills

# business processes for which monitoring

objects have been defined and instrumented
(m)

Job Scheduling

# jobs for which monitoring objects have

been defined and instrumented (m)

Operations review

# operations reviews for solutions to be

implemented (SMA approach) (m)

Operations trainings

# operations handbook training / skills

transfer for new topics (m)
% of employees with proven skills (m)

2011 SAP AG. All rights reserved.

21