Risk Ranking Audit Nonconformities

by Duke Okes

The Opportunity

It’s a given that not all audit nonconformities (NCs) are equal. Some indicate system weaknesses that
could create life ending outcomes (e.g., hygiene issues in the food or healthcare industries), while others
may simply be a minor documentation error (e.g., a class roster that was not signed by the course
instructor).

Recognizing the relative degree of risk related to each NC allows an organization to better allocate
resources where it makes more sense … another application of the Pareto principle. Even ISO 9001
indicates such when saying in the audit section 9.2.2.e) “take appropriate correction and corrective
action …” and in the nonconformity and corrective action section 10.2.1.b) “evaluate the need for action
…” and 10.2.1 “Corrective actions shall be appropriate to the effects …”

So corrective action might not be warranted for some NCs (e.g., correct the problem only), physical
causes level for others, and root cause level for still others. Risk ranking can also help determine the
relative timing allowed for corrective actions to be carried out as well as who should be involved in the
investigation.

Unfortunately, too many organizations have simply adopted the binary classification of NCs used by
most registrars, Major and Minor. While this is better than simply reporting it as a NC, it still does not
provide much information. Of course, organizations may add other classifications to deal with audit
findings where it is unclear whether the system needs work (Observation) or where there are
Opportunities for improvement, but these do not help in ranking actual NCs.

The introduction of the term “risk-based thinking” in the ISO standard goes beyond simply replacing the
preventive action requirements of the previous edition of the standard. It implies that throughout the
management system risks can/should be considered when making decisions. This also concurs with
senior management thinking that efforts should be placed where greater value can be achieved.

Some Examples

As part of courses on risk-based thinking and risk-based quality audits the author developed a simple NC
ranking system (see Table 1). Such an evaluation could consider risks to product/service quality,
customer satisfaction, regulatory compliance, and/or other objectives or stakeholders.

© 2019 Duke Okes – All Rights Reserved 1

Table 1 – NC Risk Ranking Categories

Risk Level Criteria

Severe impact on cost of quality (COQ) and/or operations,
Critical extreme impact on customers/stakeholders
High impact on COQ and/or operations, negative impact to
Major customers/stakeholders
Slight impact on COQ and/or operations, unlikely impact on
Moderate customers/stakeholders
No measurable impact on COQ, operations or
Minor customer/stakeholders

A Google search uncovered a similar four-level NC rating system used by the Finance Division at the
University of Scranton in Pennsylvania. The levels are Nominal, Notable, Significant and Major, with the
level at which the issues should be resolved ranging from the staff of the department where found to
involvement of Deans, and communications to the Board ranging from not at all to “in a timely manner.”
Seriousness considers the financial impact, whether it involves a violation of laws or regulations, fraud,
reputation and others.

Some of the terms in Table 1 are likely to be interpreted differently by each individual so in risk
management programs a risk appetite table is often used to define risk levels in more specific terms. An
example that could help auditors determine the appropriate level for each NC is shown in Table 2. It
requires a deeper evaluation of the potential impact of the NC on each specific objective.

Table 2 – Risk Ranking further defined

Potential Impact on Potential Impact on Potential Impact on
Level Product Functionality Potential Impact on Operations Costs Stakeholders
Unable to operate or Regulatory sanctions or loss of
Critical Not operational functionality reduced for days $500,000 or more customer
Functionality reduced but Functionality reduced for up to Significant concern of
Major operational 6 hours >$20,000 but < $500,000 stakeholders
Slight imperfections not Functionality reduced for up to
Moderate impacting functionality 1 hour $1000 to $20,000 Slight interest of stakeholders
Minor No impact on product No disruption of operations Less than $1000 Not relevant to stakeholders

A similar concept has been applied to external audits in some industries. In 2012 the Global
Harmonization Task Force released a guidance document (GHTF/SG3/N19:2012) describing a NC grading
system that is being adopted by the Medical Device Single Audit Program (MDSAP). It identifies
processes within ISO 13485 that are likely to have a direct impact on device safety and performance
(e.g., product realization) versus that that would have an indirect impact (e.g., documentation). It also
considers whether the NC is a first occurrence or a repeat of a NC found during recent previous audits.
It then uses a 2x2 matrix of Occurrence and Impact to score the NC as 1, 2, 3 or 4. Additions to the score
(called escalations) can also be made if the process is not adequately documented or if a nonconforming

© 2019 Duke Okes – All Rights Reserved 2

device has been released to the market. AS 9101 for the aerospace industry has a 3x3 process
evaluation matrix that considers compliance versus results, and designates a score between 1 and 5.

Quality audit managers who want to develop their own process-risk-focused ranking system might want
to perform a risk assessment of processes in the QMS. Such an evaluation can also help prioritize other
elements of the audit plan (ISO 9001:2015 indicates that audits should “take into consideration the
importance of the processes concerned, …). Table 3 is a partial example, which demonstrates that for
this organization some processes inherently have greater risk, meaning that they not only should be
audited more frequently, but also that NCs in these processes have greater risk. As ISO 9001 points out,
organizational context has a big impact on risk (in this organization all raw materials were supplied by
the customer, which means Purchasing had little impact on quality performance). Such an assessment
would ideally be conducted with input from process owners.

Table 3 – Assessment of Risks for QMS Processes

TYPES OF RISK
QMS Process Delivery Regulatory
Product Function Performance Compliance Cost of Quality
Probability

Probability

Probability

Probability
Impact

Impact

Impact

Impact
Risk

Risk

Risk

Risk
Customer contracting/ordering 1 4 H 3 4 H 2 4 H 1 1 L
Product design 4 5 H 1 2 M 4 4 H 5 5 H
Process design 2 4 H 3 3 M 1 3 M 5 5 H
Purchasing 2 1 L 2 1 L 2 2 M 2 2 M
Order fulfillment
Calibration
Nonconforming material
Document control
Internal audit
…

Aggregating Risks

Of course, looking at single NCs may also cause the organization to not see the big picture. A way to
aggregate risks might then add additional value. This is often done by department or process (see
“Internal Audit Scorecards” in Okes, 2017), but a better way might be to show a matrix of all NCs found
during the audit and the degree to which each aligns to objectives (see Table 4). In this case rather than
using words a number is assigned to each level using a nonlinear scale of 1, 3, 7 and 10. This helps
better differentiate when the number of potential levels is low. Note that NC#s 2 & 4 are potentially

© 2019 Duke Okes – All Rights Reserved 3

more impactful, and customer satisfaction and COQ are the greatest overall risks related to this
combination of NCs.

Table 4 – Relationship of Risks of Individual NCs to Objectives

Objectives

Customer
NCs Product Quality Satisfaction Cost of Quality Total
1 1 1 1 3
2 3 6 10 19
3 1 3 3 7
4 6 6 1 13
5 3 1 3 7
6 1 3 1 5
Total 15 20 19 54

Another factor that could be considered for each NC is velocity. That is, if the potential impact on the
objective turns into an actual impact, how long is it likely to take for it to show up? For example, will the
impact on product functional performance show up at a final test station? Will it only show up after the
customer purchases the car? Or will it be years before the degradation is evident?

Cautions

Internal financial auditors (often called GRC auditors – Governance, Risk & Compliance) have typically
rated either the entire audit or each specific finding according to risk. Richard Chambers of the Institute
of Internal Auditors (IIA, 2017) indicates that this helps draw the attention of the board, as well as that
of executive management. However, he also cautions about potential downsides:

 Process owners may take the ratings personally, especially if their performance reviews are
impacted by audit ratings
 The ratings add time to generating the audit report
 Ratings can draw attention to some factors, with other factors perhaps being ignored or
downplayed

Given that there are likely to be GRC, environmental, occupational health & safety, IT and other audits
conducted in the organization, it would then be useful for quality auditors to consider how NCs are
handled in these audits. After all, a fully integrated management system would include a fully
integrated audit function. But if this integration has not already occurred, each group should at least be
studying/benchmarking the others.

Summary

ISO 31000:2018 defined risk as the “effect of uncertainty on objectives.” When an audit is conducted it
is evaluating the controls that have been put in place to reduce risk, and if those controls are not

© 2019 Duke Okes – All Rights Reserved 4

operating properly a NC is the immediate result. However, it is ultimately the objectives (quality
objectives, organizational objectives, etc.) that is the primary concern, and ranking NCs according to risk
then helps the organization better understand the relative degree of risks identified and how to respond
accordingly.

The process of risk ranking and reporting NCs also needs to have its own controls, and it is hoped that
the ideas presented herein will be useful in helping the reader evaluate the process at his/her own
organization, as well as suppliers, customers and/or clients.

References

For the GHTF guidance document see:

https://www.fda.gov/downloads/MedicalDevices/InternationalPrograms/MDSAPPilot/UCM468937.pdf

For the University of Scranton ranking method see: http://www.scranton.edu/finance/internal-

audit/auditor-ratings.shtml

For Chambers’ IIA article see: https://iaonline.theiia.org/blogs/chambers/2017/Pages/Ratings-in-Audit-

Reports-Lights-or-Lightning-Rods.aspx

For Okes’ book on Internal Audits see: https://www.amazon.com/Musings-Internal-Quality-Audits-

Duke/dp/087389958X/ref=sr_1_10?s=books&ie=UTF8&qid=1497558274&sr=1-10

Bio

Duke Okes has been in private practice for 34 years as a trainer, consultant, writer and speaker on
quality management topics. His book titled “Musings on Internal Quality Audits: Having a Greater
Impact” was published by ASQ Quality Press in 2017. He is an ASQ Fellow and holds certifications as a
CMQ/OE, CQE and CQA.

© 2019 Duke Okes – All Rights Reserved 5