Performing An Effective Project Audit by Muema Lombe, 2011

Session #H8
Performing an Effective Project Audit
Thursday, April 21
Thursday 21, 2011
11:30am
Muema Lombe, CRISC, CSSLP, CGEIT, CISA

IT Audit Manager
Endurance Services Ltd.
© 2011 Muema Lombe

Key Points
 Project management methodologies

 The many reasons projects fail
 Project audit roles & responsibilities
 Understanding project risk factors
 Case studies of project audits & sample
findings
d gs
 The key components of a project audit
program
© 2011 Muema Lombe

Project Management Methodologies
 Project Management Framework:

– PMI PMBOK
 Project Management Methodology:
– Pi
Prince2
2
 Project Methodologies
– Waterfall
– Agile
– Scrum
– Etc.
© 2011 Muema Lombe

 Project Management Framework (Tools):

– PMI PMBOK
• A Guide to the Project Management Body of Knowledge
(PMBOK Guide) is a book which presents a set of standard
terminology and guidelines for project management.
• It was first published by the Project Management Institute in
1987.
• PMI gglobal standards p provide g
guidelines,, rules and
characteristics for project management.
• http://www.pmi.org/
© 2011 Muema Lombe

 Project Management Methodology (Steps):

– Prince2
• PRojects IN Controlled Environments (PRINCE) is a project
management method. It covers the management, control and
organisation of a project. "PRINCE2" refers to the second
major version of this method and is a registered trademark of
the Office of Government Commerce (OGC), an independent
office of HM Treasury of the United Kingdom.
• http://www.prince-officialsite.com/home/home.asp
© 2011 Muema Lombe

 Project Methodologies
– Waterfall
• The waterfall model is a sequential design process, often
used in software development processes, in which progress
is seen as flowing steadily downwards (like a waterfall)
through the phases of Conception, Initiation, Analysis,
Design, Construction, Testing and Maintenance.
© 2011 Muema Lombe

The "waterfall model". Progress flows from the top to the bottom, like a waterfall.
© 2011 Muema Lombe

Key Points

findings
d gs
program
© 2011 Muema Lombe

The Many Reasons Project Fail
 Failed project example: Denver Airport Baggage System

 Early Warning Signs of Project Failure
© 2011 Muema Lombe

 Denver Airport Baggage System Case Study
© 2011 Muema Lombe

 Originally billed as the most advanced system in the world, the

baggage handling system at the new Denver International
Airport was to become one of the most notorious examples of
project
j failure. Originally
g y planned to automate the handling
g of
baggage through the entire airport, the system proved to be far
more complex than some had original believed. The problems
building the system resulted in the newly complete airport
sitting idle for 16 months while engineers worked on getting the
baggage system to workwork.
 The delay added approximately $560M USD to the cost of the

airport
© 2011 Muema Lombe

The Denver debacle is a template

p for failure that many
y other projects
p j have followed.
As with so many other failures, Denver suffered from:
1. The underestimation of complexity

ac o
2. A lack of p
planning
a g resulting
esu t g in subseque
subsequentt cchanges
a ges in strategy
st ategy
3. Excessive schedule pressure
4. Lack of due diligence
5. Making firm commitments in the face of massive risks and uncertainty
6 Poor stakeholder management
6.
7. Communications breakdowns
8. People working in silos
9. Poor design
10 Failure to perform risk management
10.
11. Failure to understand the implication change requests might have
12. Lack of management oversight
Source: http://calleam.com/WTPF/wp-content/uploads/articles/DIABaggage.pdf
© 2011 Muema Lombe

Source: EARLY WARNING SIGNS OF IT PROJECT FAILURE: THE

DOMINANT DOZEN by Leon A. Kappelman, Robert McKeeman, and
Lixuan Zhang
© 2011 Muema Lombe
Key Points

findings
d gs
program
© 2011 Muema Lombe

Project Audit Roles & Responsibilities
 IT Audit vs PMO Project Role

 IT Audit Role on Pre/Post Implementation
Reviews
 IT Audit Level of Engagement
© 2011 Muema Lombe

 IT Audit vs PMO Project Role

– PMO Project Role – to monitor and report project status, and
to monitor and report project costs.
– IT Internal Audit Project
j Role – opine
p on the overall control
environment of the new project, system or application by
evaluating, testing, and commenting on the effectiveness of
risk management, control and governance processes
focusingg on technology gy risks.
© 2011 Muema Lombe

 IT Audit Role on Pre/Post Implementation

Reviews
– To ensure new systems, or applications include all
the needed controls and documentation in a
consultative and proactive manner by evaluating,
testing, and commenting on the effectiveness of risk
management, control, and governance processes
focusing on technology risks
risks.
© 2011 Muema Lombe

 IT Audit Level of Engagement

– The level and depth of IT audit involvement in pre
and post implementation reviews is based on the
• p j
project risk assessment,,
• project team's project management experience,
• level of management involvement,
• size and complexity of the initiative, and
• impact on the organi
organization
ation if the initiati
initiative
e is dela
delayed
ed or
unsuccessfully implemented.
© 2011 Muema Lombe

Internal auditors' involvement in an organization's system conversion initiatives can range from minimal involvement (level
1) to extensive audit efforts throughout every phase of the project (level 10).
 Level 1: Audit risk assessment during the project initiation phase.

 Level 2–3: Audit review of documentation and project deliverables.
 Level 4–5: Attend project meetings, conduct some interviews, and produce verbal audit reports.
 Level 6–7: Increased audit efforts, conduct more interviews, and produce formal audit reports.
 Level 8–9: Review all milestones, perform extensive audit tests, and produce formal and comprehensive audit reports.
 Level 10: All of the above efforts.
efforts
Internal auditors should determine their level of involvement and approach during the project's initiation phase. The audit
involvement decision should be based on the project risk assessment, as well as factors such as the project team's
project management experience, level of management involvement, size and complexity of the initiative, and impact
on the organization if the initiative is delayed or unsuccessfully implemented
implemented. The most appropriate audit approach
needs to be defined during the audit project planning phase. Following step 1 of the generic eight-step audit process
will complete the definition of audit's level of involvement. Further adjustment of audit involvement may be required
depending on the results of the project’s efforts and auditors' assessment throughout the project's life cycle. Another
important consideration to discuss with management and the project team is the internal auditors' roles and
responsibilities
p in attending
gpproject
j team meetingsg throughout
g the conversion audit.
Source: IIA.org
© 2011 Muema Lombe

 Reporting
– Once of the level of engagement has been decided, agree on the format,
frequency and distribution of the resulting audit reports
reports. For example:
• DISTRIBUTION: Will reports be distributed to all project participants? Project
sponsor? Project steering committee?;
• FREQUENCY: Will reports be prepared quarterly, monthly, or at the end of the
pre/post
/ t implementation
i l t ti review?
i ?
• FORMAT: Agree on format - will the end result be a format audit report? If a
report, will it be a detailed report? Executive summary? Will the end result be a
memo with observations rather than a report with issues?
© 2011 Muema Lombe

Key Points

findings
d gs
program
© 2011 Muema Lombe

Understanding Project Risk Factors
Source: Project risk factors checklist www.techrepublic.com
© 2011 Muema Lombe

Understanding Project Risk Factors
Source: Project risk factors checklist www.techrepublic.com
© 2011 Muema Lombe

Key Points

findings
d gs
program
© 2011 Muema Lombe

Case Studies of Project Audits & Sample
Findings
 Case #1: User Interface Project

 Case #2: Regulatory Reporting Platform
© 2011 Muema Lombe

Findings – Case #1
 Project: User interface for external corporate

customers to view, filter and report on data.
 Audit Involvement: IT audit participated in
weekly
kl project
j t tteam meetings
ti as an observer,
b
and prepared a quarterly memo summarizing
project status and observations.
 Fi di
Findings:
– Resource Management
– Software Development Process
– Vendor Management
– Scope & Requirements Management
– Business Participation
© 2011 Muema Lombe

Findings – Case #1 (cont’d)
(cont d)
 Findings:
– Resource Management
• Business analysts – shortage of team members with the right skill sets to write
requirements.
• Project managers – team members did not have sufficient experience running
large, complex projects.
• Technical architect – no one with end-to-end responsibility for overall technical
design
• Priorities – ongoing resource conflicts due to competing priorities and lack of
dedicated project resources.
• Offshoring – occurred during critical development time period and disrupted
management’s project focus.
• Roles & responsibilities – given the above issues, the role of team members
was unclear and led to an over-reliance on “management
g byy consensus” during g
the requirements process.
© 2011 Muema Lombe

(cont d)
 Findings:
– Software Development Process
• Project Estimation – continual optimistic estimation of task work efforts, duration
and required resources, leasing to late delivery of code for testing and
compromises in quality of testing.
• Contingency Time – the project schedule did not factor in time for the correction
of functional requirement defects and didn’t include enough time to correct
coding defects.
• Metrics – lack of or inadequate use of metrics to track progress and measure
the quality of deliverables.
• Tools – lack of standardized tolls throughout the project lifecycle (e.g. issue
tracking listing, requirements,etc.).
• Risk Management – insufficient risk mitigation for recurring resource problems
(e.g. enough resources, the right resources, extensive use of consultants,
consultant management, etc.)
© 2011 Muema Lombe

(cont d)
 Findings:
– Vendor Management
• Vendor Selection– insufficient screening of 3rd party software packages.
• Vendor Oversight – inadequate management of consultants assigned critical
development work.
© 2011 Muema Lombe

(cont d)
 Findings:
– Scope & Requirements Management
• Project Scope – the project scope was not realistic for the timeframe requested.
• Requirements expertise – analyst team members lacked adequate training in
requirements development and were not dedicated to the task.
© 2011 Muema Lombe

(cont d)
 Findings:
– Business Participation
• Steering Committee – the steering committee served more as a working group
than a senior management oversight committee. As a result, the key business
stakeholders were not included, making it difficult to insure they received early
notice of emerging problems.
© 2011 Muema Lombe

Findings – Case #2
 Project: Create a single platform to automate

regulatory reporting in 3 European countries.
 Audit Involvement: IT audit participated in the
project
j tb by meeting
ti withith th
the tteam llead
d monthly
thl
and reviewing available project
documentation. IT audit prepared a monthly
memo summarizing project status and
observations.
 Findings:
– Project Management
– Design Documentation
– Resources
– Sli
Slippage
© 2011 Muema Lombe

(cont d)
 Findings:
– Project Management
• Lack of a Project Management Resource - Six months into an 18 month project,
a project manager has not been assigned to the project, as a result, project
monitoring and tracking is not being performed.
© 2011 Muema Lombe

(cont d)
 Findings:
– Design Documentation
• Inadequate Design Documentation - Detailed design documentation was not
prepared. A flowchart served as the sole design document. The flowchart
contained limited details with vague explanation of how to receive data from
source systems and further process the data.
© 2011 Muema Lombe

(cont d)
 Findings:
– Resources
• Inadequate Project Resources – In addition to the lack of a dedicated project
manager, 2 resources are expected to document regulatory reporting
requirements for 3 countries, in addition to their daily responsibilities.
© 2011 Muema Lombe

(cont d)
 Findings:
– Slippage
• Project slippage – IT audit noted 2 weeks of slippage based on the original
project plan. The plan does not include any timing for contingency, as such this
could impact the timely delivery of the project. Additionally, this slippage was
not communicated to the Project Sponsor such that appropriate remedial action
could
ld b
be ttaken.
k
© 2011 Muema Lombe

Key Points

findings
d gs
program
© 2011 Muema Lombe

The Key Components of a Project Audit Program
 Project Sponsorship  Quality Assurance Management

 Project Organization  Procurement Management
 Project Milestones  3rd Party Management
 Scope Measurement  Communication Management
 Project Approach Management  Interdependency Management
 Schedule Management  Change Management
 Cost Management  Transition Management
 Personnel Resource Management  Project Conclusion Management
 Risk Management
© 2011 Muema Lombe

 Project Sponsorship
 This element relates to determining who/whom has approved the project, paying the costs,
showing
h i commitmentit t tto the
th project
j t and
d assigning
i i responsibilities
ibiliti
© 2011 Muema Lombe

 Project Organization (Roles and Responsibilities)

 This element refers to how the team is structured, reporting relationships, liaison roles. It is
hi hl d
highly desirable
i bl that
th t user department
d t t staff
t ff be
b included
i l d d within
ithi th
the project
j t tteam iin order
d tto
ensure that the system fully reflects user needs and to obtain greater user commitment to the
system. Some aspects of the project may be delegated to outside consultants and
contractors. It is essential in each case to specify the tasks and deliverables to be provided
byy each p
party,
y, and the dates byy which theyy must be completed.
p
 Audit should verify the roles and responsibilities have been defined and appropriate tasks
and deliverables have been defined.
© 2011 Muema Lombe

 Project Boundaries and Milestones

 This element is particularly important to define clearly the start and end of the project. The
starting
t ti pointi t is
i usually
ll ddefined
fi d iin tterms off a sett off assumptions
ti th
thatt h
have b
been made,
d and
d
the end may equate to a given set of deliverables, with the addition of specified post-
installation support and maintenance. Key milestones for the project should be established.
These milestones provide the opportunity for:
– Reassessment of the costs to complete and benefits of the project.

– Recommitment by users.
– Reviewing the project budget and plans.
– Demonstrating progress to users and management
management.
 Audit should ask for the time frame of the project. In most instances the project team should
have prepared a Gantt chart to measure project status and help in assisting in the timely
completion of this project
project. This will assist the Auditor to determine when/if to get involved in
each stage of the SDR.
© 2011 Muema Lombe

 Scope Measurement
 The element refers to ensuring the scope and objectives must be defined properly. These
can be
b grouped d under
d ttwo maini h
headings:
di
 The project charters, which defines project goals and contains an outline proposal for the
project. This outlines describes the proposed system and the information it will use. Costs,
benefits, preliminary schedule, and the impact the system will have on the organization and
related systems are major components
components.
 The enterprise description is an important component, especially on larger projects with
many people involved. It lays out the users’ need and environment and relates them to those
of the organization and external factors. Without an enterprise description there is a greater
chance that users’ real needs and the overall requirements of the organization will not be
satisfied.
© 2011 Muema Lombe

 Project Approach Management

 It is highly desirable for the project to be defined formally in a document that describes its
scope and d th
the roles
l off th
those iinvolved.
l d E Experience
i h
has shown
h th
thatt users who
h are preparedd tto
proceed on an informal basis have little understanding of what is involved in implementing
the system.
© 2011 Muema Lombe

 Schedule management
 This element refers to the process of allocating activities (identified in the breakdown and
given estimated times) to members of a project team against a calendar. Considerations for
skill level, project team structure and size, and scheduling and budgeting resources should
be included in the development methodology.
 Audit should ensure the necessary resources are allocated to the project and
time commitments are responsible to complete the job assignments
assignments.
© 2011 Muema Lombe

 Cost Management
 This element relates to ensuring the initial pre-determined costs remain at or below projected
costs.
t Normally
N ll on a llarge scale
l project,
j t a ththreshold
h ld will
ill b
be established,
t bli h d th
thatt will
ill b
be iincluded
l d d
in the project charter, which will state if a project is going to exceed projected costs by a
certain dollar amount or percentage, then the project sponsor should be notified immediately.
In some instances, senior management and approval are required.
© 2011 Muema Lombe

 Personnel Resource Management

 People responsible for controlling the projects have a duty to develop the skills, experience,
andd confidence
fid off th
those who
h are working
ki ffor th
them. Staff
St ff should
h ld bbe given
i workk appropriate
i t tto
their experience, preferable erring on the side of stretching their skills, but with adequate
training and monitoring.
© 2011 Muema Lombe

 Risk Management
 This element relates to risks that could adversely effect the completion of the project within
th given
the i hours
h and/or
d/ titiming.
i Risks
Ri k suchh as a thi
third
d party
t vendor
d going
i outt off b
business,
i
project staff turnover should be evaluated.
© 2011 Muema Lombe

 Quality Assurance Management

 This element relates to the process of ensuring that all deliverables are of required quality
and
d that
th t allll workk has
h been
b carried
i d outt accurately
t l and
d tto an appropriate
i t standard.
t d d QA iis
ongoing effort throughout the development cycle and system use.
© 2011 Muema Lombe

 Procurement Management
 This element relates to ensuring that all the necessary resources, equipment, hardware,
software
ft can be
b obtained
bt i d ffor th
the project
j t tto meett project
j t deadlines.
d dli
© 2011 Muema Lombe

 3rd Party Management

 Consideration should be given to look at contract agreements, maintenance agreements,
ti i off expected
timing t d workk completion,
l ti resource allocation,
ll ti physical ti off 3rd party.
h i l llocation t
Evaluation of due diligence work is imperative at the earliest point possible.
 If the project team is implementing 3rd party software, you should consider where the source
code for the software is stored and if in the contract, we are allowed to obtain the source
code in the event that the 3rd party files for bankruptcy
© 2011 Muema Lombe

 Communication Management
 This element relates to formal organizational arrangements which should be adopted to
f ilit t successful
facilitate f l exploitation
l it ti off iinformation
f ti ttechnology.
h l Th
These iinclude
l d th
the iimportance
t off
having a project “sponsor,” the need for informal and formal user involvement, the role of
corporate information and project steering committees, and the advantages of standardized
communication between these various parties. Historically, many projects have failed
because of inadequate
q communication between MIS and the user departments
p
© 2011 Muema Lombe

 Interdependency Management
 This element relates to other projects that depend on the initial project to succeed or other
projects
j t that
th t mustt completed
l t d prior
i tot this
thi project
j t so this
thi project
j t will
ill succeed.
d The
Th project
j t
team should be aware of all of these projects and just be figured into the timing and
milestone documents.
© 2011 Muema Lombe

 Change Management
 This element relates to maintaining version control over project documents relating
specifically
ifi ll tto th
the iinitial
iti l project.
j t This
Thi would
ld involve
i l any documents
d t that
th t would
ld b
be usedd tto
support the System Development Life Cycle of this project and track the project until
completion.
© 2011 Muema Lombe

 Transition Management
 This element relates to the transition of using the old system vs the new system. Can users,
managementt transition
t iti without
ith t a loss
l off productivity
d ti it from
f the
th personnell using
i ththe systems.
t
© 2011 Muema Lombe

 Project Conclusion and Turnover Management

 If the project was planned correctly and user involvement has been adequate, the installation
off the
th system
t and
d the
th transfer
t f off supportt responsibility
ibilit should
h ld proceed
d smoothly.
thl It iis a
specific responsibility of the project director and project manager to satisfy themselves,
before allowing a system to be installed, that:
– All stages of testing, including acceptance testing, have been completed properly.
– Adequate user procedures have been prepared and the users are trained to the required level and want to work
with
ith the
th system
t positively.
iti l
– The MIS department is prepared to run the system.
– All required controls and security procedures are in place.
© 2011 Muema Lombe

Summary
 METHODOLOGY: Understand the project management
methodology deployed at your organization
 ENGAGEMENT: Co-develop
Co develop the audit method of engagement with
the project (e.g. pre-implementation? Post-implementation? Silent
participant in project team? Other?)
 REPORTING Agree
REPORTING: A on format,
f t frequency
f and
d distribution
di t ib ti off
reporting.
 RISK: Understand project risk factors.
 SCOPE: Define the scope of your audit.
 EXECUTE: Execute and report on the audit.
© 2011 Muema Lombe

Performing An Effective Project Audit by Muema Lombe, 2011

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Performing An Effective Project Audit by Muema Lombe, 2011

Загружено:

Авторское право:

Доступные форматы

Session #H8

Performing an Effective Project Audit

Muema Lombe, CRISC, CSSLP, CGEIT, CISA

© 2011 Muema Lombe

 Project management methodologies

© 2011 Muema Lombe

 Project Management Framework:

© 2011 Muema Lombe

 Project Management Framework (Tools):

© 2011 Muema Lombe

 Project Management Methodology (Steps):

© 2011 Muema Lombe

© 2011 Muema Lombe

© 2011 Muema Lombe

 Project management methodologies

© 2011 Muema Lombe

 Failed project example: Denver Airport Baggage System

© 2011 Muema Lombe

 Denver Airport Baggage System Case Study

© 2011 Muema Lombe

 Denver Airport Baggage System Case Study

 Originally billed as the most advanced system in the world, the

 The delay added approximately $560M USD to the cost of the

© 2011 Muema Lombe

The Denver debacle is a template

1. The underestimation of complexity

© 2011 Muema Lombe

Source: EARLY WARNING SIGNS OF IT PROJECT FAILURE: THE

 Project management methodologies

© 2011 Muema Lombe

 IT Audit vs PMO Project Role

© 2011 Muema Lombe

 IT Audit vs PMO Project Role

© 2011 Muema Lombe

 IT Audit Role on Pre/Post Implementation

© 2011 Muema Lombe

 IT Audit Level of Engagement

© 2011 Muema Lombe

 Level 1: Audit risk assessment during the project initiation phase.

© 2011 Muema Lombe

© 2011 Muema Lombe

 Project management methodologies

© 2011 Muema Lombe

Source: Project risk factors checklist www.techrepublic.com

© 2011 Muema Lombe

Source: Project risk factors checklist www.techrepublic.com

© 2011 Muema Lombe

 Project management methodologies

© 2011 Muema Lombe

 Case #1: User Interface Project

© 2011 Muema Lombe

 Project: User interface for external corporate

© 2011 Muema Lombe

© 2011 Muema Lombe

© 2011 Muema Lombe

© 2011 Muema Lombe

© 2011 Muema Lombe

© 2011 Muema Lombe

 Project: Create a single platform to automate

© 2011 Muema Lombe

© 2011 Muema Lombe

© 2011 Muema Lombe

© 2011 Muema Lombe

© 2011 Muema Lombe