IT Auditing AC

475
Session 10
Bentley University

18 April 2013

Session 10 Agenda
1. Revisiting Closing the Loop

Framework
2. Measuring IT Audit Performance
3. Application System Audit Planning

John W. Beveridge

Section 1

REVISITING THE CLOSING

THE LOOP FRAMEWORK
John W. Beveridge

Revisiting Closing the Loop

Framework

Why revisit the CTL Framework?

What is the relationship of the

Framework to the Team Project?

How applicable is the CTL Framework to

audits other than IT audits?

Where do IT auditing standards, quality

assurance, and audit risk come into
play?
John Beveridge

Revisiting Closing the Loop

Framework

CTL Framework provides:

structured approach for developing an audit work

program for internal control examinations.

reinforces the value gained by closely linking

control objectives to business objectives, and
controls to control objectives

promotes understanding of the benefit of controls

and having an appropriate mix of controls

Strengthens audit work programs by

distinguishing between controls in place and
controls in effect

Helps in drawing conclusions and developing

audit results John Beveridge

Closing the Loop Framework

CTL is a methodology to:

Define audit objectives and audit criteria in

relation to control objectives and control

practices,
Develop targeted audit steps to meet audit
evidence requirements,
Develop references, or work papers, to help
draw conclusions in line with control
objectives and report audit results.

John Beveridge

CTL Framework Forward &

Back

The idea of Closing the Loop is to

tie in what we learn at each step in
the process and to be able to link
that information backwards and
forwards.

By Look Back, we are referring

back to the prior step as a point of
reference and basis for what we do
in the current step.

F. Develop
Audit Results

E. Build Audit
Steps

D. Define
Audit Criteria

A. Define Control
Objectives

Closing
the
Loop

B. Identifying
Control Criteria

C. Develop
Audit Objectives

Framework Outline
Closing the Loop
Framework
A. Identifying operational and
B.
C.
D.

E.
F.

control objectives
Identifying and classifying
control criteria
Developing audit objectives
Defining audit criteria in
terms of evidence
requirements
Building audit steps
Developing audit results

Framework Outline
Closing the Loop
Framework
Audit
Planning

A. Identifying operational and

B.
C.
D.

E.
F.

control objectives
Identifying and classifying
control criteria
Developing audit objectives
Defining audit criteria in
terms of evidence
requirements
Building audit steps
Developing audit results

A. Defining
Control
Objectives

John Beveridge

F. Develop
Audit Results

E. Build Audit
Steps

D. Define Audit
Criteria

A. Define Control
Objectives

Closing
the
Loop

B. Identifying
Control Criteria

C. Develop Audit
Objectives

A. Defining the Control Objectives

Internal Control
Controls are framed by
what is to be attained
(control objectives) and
the means to attain
those goals (the
controls).
John Beveridge

Control (as defined by

COBIT)
The policies, procedures,
practices and organizational
structures designed to provide
reasonable assurance that
business objectives will be
achieved and that undesired
events will be prevented or
detected and corrected.
John Beveridge

A.

Defining the Control

Objectives

IT Control Objective
A statement of desired
result or purpose to be
achieved by
implementing control
procedures in a
particular IT activity

A. Defining the Control Objectives

Identifying Control
Objectives

Identification of relevant
operational and control
objectives:
First, what needs to be achieved

and avoided.
Based on importance and impact of
risk triage what the control
requirements

Understanding Control
Requirements
The auditors understanding of:
what is important to the business, its
customers, and oversight bodies, and
the reasons for why IT needs to be
controlled

helps one focus on the Control

Objectives and then the controls
needed

John Beveridge

A.

Defining the Control Objectives

Understanding the Control

Environment

Business organization, information

systems, and supporting technology

Documenting the business operations
(internal and external, CSFs, IT
environment)
Identifying the key operational and
control objectives
Assessing entitys ethical climate tone at
the top
Identifying and evaluating the
appropriateness
John
ofBeveridge
internal controls

Identifying
Control
Objectives
To Achieve
Business
Objectives

Mission Statement
Business Objectives
Interviews with Managers
Business White Paper

To Avoid
Risks,
Threats and
Exposures

Risk Assessment
Risk Management Reports
Incident Reports
Security Reports

A. Defining the Control Objectives

Identify Important Information

Attributes
For the Data and Information, identify:
Nature and type of information
Business/process requirements of
information
1.
2.
3.
4.
5.

Identify reliability requirements

Identify relevance requirements
Security (access, change, privacy)
Accessibility
Availability
John Beveridge

Defining the Control

Objectives

Select the control objectives that are

most important in terms of what the
organization needs to achieve and
avoid.
Define and articulate the control
objectives for clear understanding by
auditee and auditor

John Beveridge

B. Identifying Control
Criteria

Building Audit
Criteria
John Beveridge

F. Develop
Audit Results

E. Build Audit
Steps

D. Define Audit
Criteria

A. Define Control
Objectives

Closing
the
Loop

B. Identifying
Control Criteria

C. Develop Audit
Objectives

B. Identifying Control Criteria

Look Back

Look Back to Control Objectives for

our primary point of reference.

Controls are the policies,

procedures, organizational
structures, and mechanisms that
help ensure that the control
objectives are met.
John Beveridge

Controls Link to Control

Objectives
Control
Objective

Controls

Control Evidence
John Beveridge

For Defined Control Objectives,

Where Do We Go to Find the
Controls?

Researching controls
Start with researching the control
objectives or purpose to be
achieved

Suggest a two-tier approach to

follow
1. Obtain control lists from control

models, control guidelines, standards,

etc
2. Research individual controls through

B. Identifying Control Criteria

Complete the Control

Classification

Prepare list of controls (with

sources) for each control objective
Identify for each control the control
category and control type
Populate table with control
classification
Identifying category and type of
control is the first step to ensuring
the mix of controls will be
sufficiently comprehensive
John Beveridge

B. Identifying Control Criteria

Control Classification

Policies (rules of the road)

Procedures (How To do tasks and
activities)
Organizational (structure, span of control,
unity of command, segregation of duties,
job descriptions, job responsibilities, points
of accountability)
Practice (methods)
Mechanisms (software tools, programmed
procedures, etc.)
Documentation (audit trails, systems of
record)
Legal (statutory,
regulatory,
and
John Beveridge

B. Identifying Control Criteria

Classify by Types of Controls

General controls
Application controls
Primary controls
Secondary controls
Preventive controls
Detective Controls
Corrective Controls
Compensating controls

General Controls

General controls are controls which

are over and within the entire data
processing or IT environment,
impacting all or most application
systems.

They are pervasive

Reflect the control culture
Management sensitive
John Beveridge

Application Controls

Application controls are specific

to individual application systems

Primarily applied to input, processing,

and output.
Include data preparation controls,
edit checks, reasonableness limits,
processing controls, restart and
recovery, backup, and output
distribution controls.
John Beveridge

Control Benefit or Failure for Each

Control
Our focus is now on Control
Value
What is the benefit derived from
the control?

In place?
In effect?

What is the impact when the

control is not in place, or in effect?
John Beveridge

Adverse Impact of Absence of Control

Control
Objective
Control
Criteria
Impact of control(s)
not in place
Impact of control(s)
not in effect

B. Identifying Control Criteria

Table for Control Classification

Control Objective:

Physical Security

Control

Control
Category

Type of
Control

Control
Benefits

Adverse Impact
of Control not in
Place/Effect

Lock

Mechanism

Primary
Preventive

Prevent
access

Unauthorized
access gained

Monitor,
Camera

Mechanism

Secondary
Detective

Guard(s
) in
lobby

Organizationa
l

Secondary
Preventive
Detective
Corrective

Unauthorized
access gained

C. Develop Audit
Objectives

A. Define Control
Objectives

F. Develop
Audit Results

E. Build Audit
Steps

D. Define
Audit Criteria

Closing
the
Loop

B. Identifying
Control Criteria

C. Develop Audit
Objectives

C. Develop Audit Objectives

Audit Objectives
Depends on the type of audit
Relate the audit objective to the control

objective.
Link the audit objectives and audit
procedures to the control
objectives and the controls (review
and examination steps) to obtain
sufficient audit evidence to draw
conclusions
John Beveridge

C. Develop Audit Objectives

Audit Objectives

For control examinations, best

phrased when focused on selected
control objectives.

Use standard language to phrase

the audit objective in relation to the
control objective

Determine whether adequate

controls are in effect to provide
reasonable assurance that the
John Beveridge

C. Develop Audit Objectives

Example Audit Objective

If the control objective is to ensure
that all changes are authorized
and tested.
Therefore, the audit objective would
be To determine whether
adequate controls were in effect
to provide reasonable assurance
that all changes are authorized
and tested.

John Beveridge

D. Define Audit Criteria

F. Develop
Audit Results

E. Build Audit
Steps

A. Define Control
Objectives

Closing
the
Loop

D. Define Audit
Criteria

B. Identifying
Control Criteria

C. Develop Audit
Objectives

Developing Audit Criteria in

Terms of Evidence
Requirements
defining evidence

requirements for controls

in place
defining evidence

requirements for controls

in effect
John Beveridge

RISKS

Business
Objectives

To Achieve

To Avoid

Controls
Control in Place
Control in Effect

Process
Audit

RISKS

Business
Objectives

To Achieve

To Avoid

Controls
Control in Place
Control in Effect

John Beveridge

Control
Audit

D. Define Audit Criteria

Control Objective: Safeguarding IT Resources
Control

Evidence that
Control is in
Place

Evidence that Control

is in Effect

Sign-out policy
for notebook
computers

Documented
policy that all
notebook
computers are to
be signed out.
Policy is readily
available in
hardcopy and
online

Understanding of
policy by
management and
staff
Register maintained
of all signed out
notebooks
Documentation of
signatures of parties
to whom notebooks
have been assigned
Reconciliation of
sign-out
documentation to IT
resource inventory

Control Objective
Control Practices
Criteria
Evidence Requirements
Audit Objective
Audit Strategy
Audit Procedures

E. Building Audit Steps

F. Develop
Audit Results

E. Build
Audit Steps

D. Define Audit
Criteria

A. Define Control
Objectives

Closing
the Loop

B. Identifying
Control Criteria

C. Develop Audit
Objectives

E. Building Audit Steps

Developing Audit
Strategy
Determine exactly what evidence

that you need to draw conclusions

(prior CTL step really helps)
Identify and assess the reliability of
the sources of audit evidence (from
people to IT)
Objective: to develop, or review
and amend if necessary, an audit
plan to accomplish the audit
objectives
John Beveridge

E. Building Audit Steps

Development of Audit Work

Program

For IT Control Examinations, need to:

define control objective and identify

relevant controls
identify CSFs and Risk Factors
identify control characteristics including
desired evidence
establish audit objective(s) based on
control objective(s)
develop audit procedures to capture and
analyze evidence
John Beveridge

E. Building Audit Steps

Development of Audit Work

Program
The presence, or absence, of

control evidence becomes our

audit evidence
Need sufficient, competent audit

evidence to serve as a basis for

drawing audit conclusions and
forming an opinion
John Beveridge

E. Building Audit Steps

Focus on Audit Evidence and

Steps to Obtain and Analyze it
Determines:

Skills and knowledge requirements for

the audit team
Whether technical assistance is
needed
Whether software tools are needed
Impact on audit schedule

E. Building Audit Steps

Audit Procedures

Audit steps should cover the full

combination of controls
(appropriate mix of categories and
types)

Should include steps to assess the

presence and effectiveness of
assurance mechanisms
John Beveridge

F. Developing Audit Results

F. Develop
Audit Results

E. Build Audit
Steps

D. Define Audit
Criteria

A. Define Control
Objectives

Closing
the
Loop

B. Identifying
Control Criteria

C. Develop Audit
Objectives

F. Develop Audit Results

Control Conclusions

Review the detailed results by

control area and audit objectives.
Review the audit result tables to
identify which controls were found to
be in place and effect, and which
were not.
Generate control strengths and
weaknesses list by control area.
Draw conclusions by audit area.
John Beveridge

F. Develop Audit Results

Control Assessment
What is the control objective?
What business objective is impacted?
Appropriateness of the stated control?
Identify the type of control (application or general;
primary or secondary; and preventive, detective, or
corrective)
Number of components used to execute the control
and number of subsystems or control objectives
impacted?
Evidence that the control is in effect, or impact that it is
not.

John Beveridge

Controls

Benefit of control(s)
in place

Evidence:
control in
place

Audit Steps

Evidence:
control in
effect

Audit Steps

Impact of control(s)
not in place

Benefit of control(s)
in effect
Impact of control(s)
not in effect

F. Develop Audit Results

Forming Conclusions and

Opinions

Is the combination of controls in

place and in effect to the degree
determined to be sufficient to
provide reasonable assurance
that the control objective will be
met?

Auditors judgment is paramount

John Beveridge

Section 2

MEASURING IT AUDIT
PERFORMANCE
John W. Beveridge

60

IT Audit Performance
How well is IT Audit doing?
Is Is IT Audit doing the Right Things?
Is IT Audit Effective?
Are resources used Effectively and
Efficiently?

Performance Measurement

Process of quantifying the

efficiency and effectiveness of
past action
Metrics are specific
representations of a capacity,
process, or outcome relevant to
performance assessment
A metric or performance measure
should be quantifiable and be
documented

Criteria for Effective

Performance Measures

Is the measure relevant?

Is the measure clearly defined?
Is the data easy to obtain?
Is there a tracking/reporting
system?
If yes, is it easy to access and use?
Can reasonable control/influence be
exercised over performance related
to the measure?

Criteria for Effective

Performance Measures

Does the measure accurately reflect

what is happening in the audit
process?
Does the measure communicate
how we are doing?
Does the measure allow one to
demonstrate progress?
Is the measure useful to whoever
can act on element being measured
to improve performance?

Performance Measurement

What should be subject to

performance measurement?

Audit quality
AIC and staff performance
Report content and clarity
Workpapers
Audit Risk
Compliance with auditing standards
Impact to the organization

What else?

Performance Measurement

Cornerstone to effective
management of audit assignments
Relevance and Reliability of Metrics
Reliability of data related to time
and resources

On-line audit management systems

Work papers
Use of automated workpapers
(TeamMate)
Time and attendance

Performance Measurement

Data that is Usually Recorded

Start date to end field date

End of field date to informal exit date
Informal exit to formal exit date
Formal exit to report issuance date
Estimated completion date
Actual completion date
Number of staff (and who)
Number of audit topic areas
continued

Data that is Usually Recorded

(2)

Number of findings
Number of Audit Follow-up Areas
Audit name and audit number
Total number of field days
Number of Site Visits by Support Manager
Number of Site Visits by Manager
Date that Staff End-of-Job Evaluations are
Completed and Filed with HRD
Date that all Completed (Reviewed and Signed
Off) Hardcopy Work Papers are Filed

Performance Measurement

What do Metrics Tell Us?

Total Field Days

Range
Average Field Days

Number of Audit Topics in Scope

Range of Topics: 2 to 10 topics, etc.

Average Number of Topics: 6.79

Performance Measurement

What do Metrics Tell Us?

Number of Days from end of Field

Work to Issuance of the Report
(elapsed time)

Range
Average Elapsed Days
Adjusted for Work Days

Number of Audit Findings

Range of Topics:
Average Number of Topics:

Performance Measurement

What do Metrics Tell Us?

Number of Field Days per Audit

Topic

Range
Average Days

What other statistics would tell

us the effectiveness and
efficiency of each audit?

Performance Measurement

Develop set of reliable and relevant

metrics and targets
Establish performance measurement
baseline
Evaluate in order to manage
performance
Objective: better understand audit
performance and improve it

Example Administrative
Items

IT Audit Survey Results Form

Control Analysis Form
Control Evidence Sheet
Independence Certification Form
Report Preparation Checklist
Report Changes Tracking Form
Audit Topic Areas Table
IT Audit Administrative Review Form

Section 3

APPLICATION SYSTEM
AUDIT PLANNING
John W. Beveridge

74

Application System
Audits

Application System Audits are one of the

fundamental types of IT audits.
The other types of IT audits are general
control examinations (used to be called
facility audits), system under
development audits, and IT technical
audits.
Application audits, like other types of IT
audits, can include performance and
compliance audit work.

Application Controls

Application controls may be:

Administrative or technical
Manual or programmed
They need to address the integrity,
security, maintainability, and
availability of the system.

Application System Audit

Objectives

May be focused on the reliability,

security and availability of the entire
system
May be tailored to specific functional
areas or operational aspects of the
application

Example: billing module, data input,

change control, protection of PII, etc.

Application System
Audits

Before we gain and record an

understanding of the system, we need to
understand the audit entity in terms of:
Mission and business objectives
Business processes and data and
information requirements
Legal mandates (law, regulation and
contract)
Physical and functional organization
Entitys control environment
IT infrastructure

Application System
Changes?
Is the Business Process subject to
Change?
Are changes in business strategy
occurring, or about to occur?
Is there an effort underway, or planned, to
enhance value chain?
Is there a business process engineering
effort underway, or planned?

Application System Audit Planning

To understand the need for the system, start

with understanding the business and its
customers.
Understand the macro and micro business
environment within which the system
operates.

Gain and record an understanding system

functions
Perform risk assessment
Brainstorm fraud issues
Obtain and review stated controls

Develop proposed scope, objectives and

audit strategy

Application System Audits

As with CTL, start with the operational

and control objectives.

What should the system do what is its

purpose?
Address business process support and
functional attributes (integrity, security and
availability)
As with controls (under CTL) identify
functional value and adverse impact of
functional failure.
What is the evidence of functional success
and failure?

Understanding the System

Understanding the system:

What is the name of the application?

Where does it reside? Applications do not
just sit on mainframes.
With what systems does it interface?
Is it a feeder system?
When and by whom was the system
developed?
How long has the system been in
production?
Legal requirements (regulatory
contractual)?

Understanding the System (2)

Understanding the system

Define process boundaries

Identify mission-critical functions
Assess data sensitivity, integrity and
availability requirements
Identify significant types of transactions
Identify the level of documentation
Identify the required IT infrastructure
needed to support the operation of the
system?

Identifying Who is Involved and

Impacted by the system

View this from a RACI chart perspective

Who needs to be informed regarding the system?

Who needs to be consulted regarding the system?
Who is responsible for the integrity, operation,
security, maintenance and availability of the
system?
Who is accountable for the system?

Who relies on the system?

Who monitors and evaluates the system?
Who measures the performance of the
system?
Who are the primary and secondary users?

Compliance with Laws and

Regulations

Determine whether there are external

compliance requirements:

Identify external requirements

Document pertinent laws and regulations
Assess whether management and the IT
function have considered the relevant external
requirements and the actions taken (policies,
procedures, training, etc.)
Review business department documents that
address adherence to applicable laws and
regulations

Application Documentation

User manuals
Functional specs
Detailed specs
DFDs
Flowcharts
Data dictionaries
System narratives
Policy and procedures
Source documents
Screen formats
System reports

Questions
Questions
Questions

John W. Beveridge

87