Академический Документы
Профессиональный Документы
Культура Документы
RECORD OF AMENDMENTS
FOREWORD
This Manual is prepared by the International Business Aviation Council (IBAC) and is issued under the
authority of the IS-BAO Standards Board.
The Internal Audit Manual has been developed for the assistance of operators who have implemented, or
are in the process of implementing, the IS-BAO - an International Standards for Business Aircraft
Operations. It is designed to provide the operator with a systematic approach to assessing their
conformity with the provisions of the IS-BAO. It should be of use to operators in determining their
preparedness for an IS-BAO registration audit and for the development of a process for ongoing
assessment of the appropriateness and effectiveness of the safety management activities.
The audit procedures were developed in recognition of proven industrial management auditing concepts.
The procedures will ensure that audits are completed consistently and in accordance with a systematic,
objective and proven process.
TABLE OF CONTENTS
Record of Amendments ..............................................................................................................i
List of Effective Pages ................................................................................................................ ii
Foreword ....................................................................................................................................v
Table of Contents ...................................................................................................................... vi
1.0 Purpose ........................................................................................................................ 1
1.1 Purpose Statement ...................................................................................................... 1
1.2 Structure of the Manual ................................................................................................ 1
1.3 Definitions and Audit Terminology ............................................................................... 1
2.0 IS-BAO – an International Standard for Business Aircraft Operations ........................ 3
2.1 Background .................................................................................................................. 3
2.2 IS-BAO Standards Board ............................................................................................. 3
2.3 Keeping the IS-BAO Current ........................................................................................ 3
3.0 IS-BAO Internal Audits ................................................................................................. 4
3.1 Internal Audit Programs ............................................................................................... 5
3.2 Audit Objectives and Scope ......................................................................................... 5
3.3 Audit Principles ............................................................................................................ 6
4.0 Audit Procedures .......................................................................................................... 9
4.1 The Audit Standard ...................................................................................................... 9
4.2 Audit Opening Meeting................................................................................................. 9
4.3 Audit Protocols ........................................................................................................... 10
4.4 Using the Audit Protocols ........................................................................................... 10
4.5 Conducting the Audit .................................................................................................. 10
4.6 Determining if an Audit is Successful......................................................................... 11
4.7 Judgement and Decision Making ............................................................................... 11
4.8 Closing Meeting ......................................................................................................... 12
5.0 Evaluating the Operator’s SMS .................................................................................. 13
5.1 What is a Safety Management System (SMS)? ........................................................ 13
5.2 Measuring Safety Performance ................................................................................. 13
5.3 Stages in the Development of a Safety Management System .................................. 14
5.4 Determining the Objectives of the Evaluation ............................................................ 14
5.5 Preparations to Evaluate the SMS ............................................................................. 15
5.6 Scoping the Evaluation .............................................................................................. 16
5.7 Measuring Soundness, Appropriateness and Effectiveness ..................................... 16
6.0 Preparing Audit Findings ............................................................................................ 21
7.0 Audit Reports ............................................................................................................. 27
8.0 Audit Forms ................................................................................................................ 29
Appendix A - Auditing Aircraft Management Companies ........................................................ 89
1.0 PURPOSE
1.1 Purpose Statement
The Internal Audit Manual provides guidance operators for the conduct of internal audits pursuant
to the IS-BAO standards established by the IS-BAO Standards Board.
The IS-BAO Internal Audit Manual consists primarily of standards and protocols for the conduct of
IS-BAO audits. Explanatory and guidance material is included where considered necessary for
clarity and guidance.
The following definitions and explanatory material apply to the IS-BAO auditing processes.
Random sample: sample drawn in such a manner that every possible item has an equal chance to be
selected.
Remedial Action a plan to resolve a non-conformity that describes the actions that will be taken and
Plan: the time frame for completing them.
Risk: the consequence of a hazard, measured in terms of severity and likelihood.
Safety: the state in which the risk of harm or damage is limited to an acceptable level as
low as is reasonably practical.
Sampling: the inspection of a representative portion of a particular characteristic to produce a
statistically meaningful assessment of the whole.
Satellite/Additional an airport or heliport other than the operator/ management company’s main base at
Operating Base which one or more aircraft and personnel are based and managed by the
operator/management company.
Standard: established criteria used as a basis for measuring an operator’s level of conformity.
Verification: the independent review, inspection, examination, measurement, testing, checking,
observation, and monitoring to establish and document that products, processes,
practices, services and documents conform to specified standards. This includes
evaluating the effectiveness of management systems.
Working papers: all documents required by the auditor or audit team to plan and implement the audit.
2.1 Background
2.1.1 After a two and one half year development program the IS-BAO - an International Standards for
Business Aircraft Operations was launched in May 2002. It has been widely accepted and is
becoming the “Gold Standard” for business aviation flight departments. Flight departments who
have been involved in implementing the IS-BAO or have already implemented it have expressed
an interest in having an internal audit manual available. At their October 2003 meeting, the IS-
BAO Standards Board agreed to produce and publish such a document.
2.1.2 In 2010 and 2011 IBAC worked with the Helicopter Association International (HAI), the British
Helicopter Association (BHA) and the European Helicopter Association (EHA) to adapt the IS-
BAO to include helicopter operations. Pursuant to the agreement under which this project was
undertaken, the HAI developed a set of helicopter mission specific standards (HMSS), to
augment the IS-BAO. Operators who have implemented the IS-BAO can undergo an IS-BAO
audit and become registered as being in conformance with the IS-BAO. They then can undergo
an audit relative to their specific mission and also become registered as being in conformance
with those standards.
2.2.1 The IS-BAO Standards Board has been delegated responsibility by the IBAC Governing Board for
the maintenance of the IS-BAO standards and for overseeing the auditing process.
2.2.2 The Standards Board consists of members nominated by the IBAC Members associations and
the IS-BAO partner helicopter industry associations. At least three members must be aircraft
operators, one must be involved in the helicopter industry and one must have aircraft
maintenance expertise.
2.2.3 The Standards Board meets at least once per year to consider proposed changes to the IS-BAO,
the Audit Procedures Manual (for auditors) and the Internal Audit Manual. Professional standards
services are provided by the IBAC Secretariat through the IS-BAO Program Director who will
assemble and provide an analysis of recommendations for change.
Proposals for change to the IS-BAO can come from many sources. In essence, all
recommendations will be assessed by the IBAC Director, IS-BAO Program and any proposals
that will enhance the efficiency or effectiveness of the IS-BAO, will be submitted to the Standards
Board for consideration.
3.1.1 An internal audit program is not a specified requirement of the IS-BAO. However, operators may
desire to conduct internal audits to determine their preparedness for an IS-BAO registration audit
or as a means to assess the appropriateness and effectiveness of their safety management
activities on an ongoing basis. An internal audit program can be an effective element of the
continuous improvement through ongoing safety management activities that is a key aspect of the
IS-BAO. Operators may wish to develop an annual internal audit plan to manage their internal
audit program. Such a plan could be composed of a series of audits, each of which would
address one or more elements of the IS-BAO standards. An example of an annual audit plan is
presented in Figure 3.1. It is recommended that personnel involved in doing internal audits have
basic auditor training. One such source is ISO 9000:2000 internal audit training that is provided
by many ISO 9000:2000 registrars. Some registrars offer internal auditor training on-line. Civil
aviation authorities or the operator’s IS-BAO auditor may also be a source of auditor training.
3.1.2 Operators should carefully consider how they are going to conduct their internal audit. A first
principle is that the auditor, or audit team leader, should not have any management
responsibilities for the area being audited. In the case where more than one person is involved in
the audit it is logical to have team members from the area being audited who would act as subject
matter experts, but if possible the team leader should be from outside of the work unit being
audited. In small organizations the manager would probably task a staff member with no
managerial responsibility to conduct the audit. Other options may be to draw auditors or audit
team leaders from the parent company or from another operator on a reciprocal basis.
3.1.3 While the manager of the operation should retain responsibility for the internal audit program the
responsibility to act as convening manager for organizing and administering it can be assigned to
a member of the organization. If this is the case the accountabilities of the convening manager
should be clearly identified.
The objectives and scope of an internal audit should be clearly established prior to
commencement of the audit. The objectives may be:
a. to determine that the operator is in conformity with the IS-BAO standards;
b. to observe and assess the organization’s adherence to the recommended practices
contained in the IS-BAO;
c. to assist the organization identify any safety deficiencies; and
d. to assess the appropriateness and effectiveness of the organization’s safety management
activities.
The scope of the audit will be dependant upon the audit objectives. If the objective is to
determine the organization’s conformity with the IS-BAO standards and assess adherence to the
recommended practices, the scope should include audit of all of the elements of the protocol
contained in Chapter 8. However, should the audit be part of the organization’s ongoing audit
program, the objective may be to focus on the safety management activities or to assess only a
portion of the elements of the IS-BAO standards. In that case the scope should be modified
accordingly. The manager of the operation or other manager, who convenes the internal audit,
should carefully consider the audit objectives and scope and ensure that the audit team leader
and members fully understand them.
3.3.1 In conducting an internal audit the audit team should follow the following audit principles:
a. Transparency and disclosure;
b. Timeliness;
c. Systematic, consistent and objective;
d. Fairness; and
e. Quality.
The philosophy of “no secrets” is very important during the audit and in the reporting process. By
openly discussing concerns or apparent shortcoming, as soon as they are identified, the auditor
will provide the convening manager with the opportunity to provide additional information or in
some cases take immediate action to resolve the issue. Audit results including all identified non-
conformities and findings, will be provided to the convening manager at the end of the audit
through a verbal debriefing and subsequent provision of a written report. The written report must
be consistent with the verbal debriefing.
3.3.3 Timeliness
The audit report will be produced and provided by the auditor to the convening manager on a
timely basis.
Audits will be conducted in a systematic, consistent and objective manner. There should be no
variation in the scope, depth and quality of the audits from that which was agreed between the
auditor and/or audit team and the convening manager. However, it must be recognized that the
depth of investigation may have to be modified during the course of the audit in order that the
auditor can accurately identify the cause(s) of non-conformities.
3.3.5 Fairness
Audits should to be conducted in a manner such that operator personnel involved in the audit are
given every opportunity to monitor, comment and respond. As previously noted such
opportunities should be provided as soon as a non-conformity is identified or suspected.
3.3.6 Quality
Audits will be conducted in recognition of quality audit concepts including the use of auditors who
are trained to carry out the task. The four fundamental rules for auditing should be applied:
a. Audits provide information for decisions.
b. Auditors are qualified to perform their tasks.
c. Measurements are taken against defined requirements.
d. Conclusions are based on facts.
IS-BAO Element Auditor(s) Jan Feb Mar Apr May June July Aug Sept Oct Nov Dec
9. Aircraft Maintenance EG HG IA AC
13. OSH HG IA AC
14. TDG HG IA AC
15. Security HG IA AC
4.1.1 The basic standard to which the audit should be conducted is the IS-BAO. However, the IS-BAO
provides considerable latitude for the operator to develop programs and procedures that are
suited to the size and complexity of the individual operation and to publish them in company
manuals and directives. While a small operation, such as one aircraft and less than five people,
still must meet the standards specified in the IS-BAO they would be expected to use much more
rudimentary programs, systems and procedures than a large operation. In conducting the audit
the auditor must not only determine if these programs and procedures are in conformity with the
requirements of the IS-BAO that are applicable to the category of aircraft operated (aeroplanes or
helicopters), but that they are appropriate to the size and complexity of the operation and that it is
being conducted in conformity with the company manuals and directives as they relate to IS-BAO
requirements.
4.1.2 The operator’s safety management system (SMS) is the cornerstone of the IS-BAO. Hence, it is
a key item of the audit. An evaluation of the operator’s SMS will include the related standards
that are an integral part of their SMS. These related standards include:
organization structure,
duties and responsibility of personnel,
training and proficiency programs,
flight operations procedures and processes,
standard operating procedures,
maintenance control system, etc.
In a registration audit all of these components of the operator’s SMS will be audited to the level
required to determine that it is appropriate and effective. For initial audits and other Stage One
audits, as well as the SMS evaluation the auditor must complete all of the remaining Audit
Protocols contained in chapter 8. When using the audit protocols in chapter 8 it must be
understood that they only paraphrase the standards, and as such, the standards in the IS-BAO
must be considered when conducting the audit.
4.1.3 When conducting Stage Two or Stage Three audits the evaluation of the effectiveness of the
SMS will involve in-depth examination of the systems, procedures and processes related to the
standards that are an integral part of their SMS, such as those noted in Section 4.2.2, a thru f, to
ensure they are appropriate and effective in managing safety risks and that they meet the related
IS-BAO requirements. In that case the detailed protocol for chapter 3 (SMS) will be the primary
audit document and the results of the SMS evaluation should be fully documented in that
protocol. It will be supplemented by the detailed protocols for chapters 4 thru 15. The auditor will
determine whether the detailed protocols for each chapter (4 thru 15) will be audited in-depth or
checked for basic conformance. Although sampling is allowable at the discretion of the auditor, at
least 50% of the protocols for each chapter (4 thru 15) must be assessed to substantiate that the
operator is in conformity with the requirements of the IS-BAO.
4.1.4 For internal audits the convening manager will determine the objective and scope of the audit.
Therefore, the internal auditor or audit team will conduct the audit and apply the appropriate
standards accordingly.
The auditor or audit team leader should arrange for an Audit Opening Meeting with the convening
manager and involved operator personnel. The agenda for the meeting should include:
a. Introductions if the audit team are not known to all of the operator personnel involved;
The Audit Protocols (checklists) have been prepared to guide the auditor through the process of
assessing whether or not the organization is in conformity with the standards specified in the IS-
BAO. However, for internal audits it may be advisable for the convening manager of the audit
team to develop their own checklist that pertain to their individual operation and issues that are to
be addressed. In any event, auditors should complete the parts of any protocol or checklist used
and review them with the convening manager as part of the Audit Closing Meeting.
4.4.1 The Detailed IS-BAO Audit Protocol in section 8.2 follows the structure of the IS-BAO. This
protocol is used to do an in-depth assessment of one or more IS-BAO element. When this
protocol is used the auditor should determine and indicate whether the operator is in conformity
or not, with each of the referenced requirements. If that requirement is not applicable to the
operator check N/A. The IS-BAO contains some standards and recommended practices that
apply to all operators and others that apply only to either fixed wing operators or rotary wing
operators. Some of the requirements refer specifically to aeroplanes or helicopters and as well
the aeroplane specific requirements have an “A” suffix to the element of item number and the
helicopter specific items have an “H” suffix. Also, some of the general provisions would not be
applicable where the operator does not engage in that activity. Therefore, auditors must ensure
that the audit report accurately reflects the type and nature of the operations conducted and the
operator’s conformance with the appropriate IS-BAO requirements.
4.4.2 If an item was not audited, the auditor should leave the three columns blank and note in the
Remarks column the reason why it was not audited. The right hand column is used to record
remarks related to the effectiveness of the operator’s programs, systems and procedures and
objective evidence of any non-conformities. Comments relating to recommended practices, good
aviation safety practices or activities that are especially well done may also be recorded. These
are items that can add value for the operator. At this point it is not necessary to determine
whether there are minor or major non-conformities. That can be done when all instances of non-
conformity and the related objective evidence have been analysed and, if appropriate, Findings
are made. In such cases and Audit Finding Forms would then be prepared. See Chapter 6 -
Preparing Audit Findings.
4.4.3 When a detailed protocol (8.2) item includes sub-elements, the auditor shall indicate their
assessment of each of these sub-elements with a “Y, N, or NA” adjacent to each item, with
appropriate references and remarks associated with the same.
4.5.1 An IS-BAO audit should be conducted as a systems audit. A system is a group of processes all
working together to achieve a common goal. As such, the audit will examine the operator’s safety
management system, programs, procedures, and processes and assess how they are integrated
to ensure conformity with the IS-BAO and effectively manage the safety of the operation.
Therefore, the audit will be conducted at a higher level than a process or product audit. The audit
will be conducted by collecting objective evidence through observations, interviews and
examination of documents. This evidence will then be compared with the requirements of the IS-
BAO and analyzed. If there are shortcomings, that information will be used to develop
observations or statements of non-conformity and if necessary, findings which identify the root
cause(s) of the problems. It must be remembered that non-conformities and findings must
describe the problem, not the symptoms of the problem.
January 1, 2014 page - 10 © IBAC All rights reserved
IS-BAO Internal Audit Manual International Business Aviation Council (IBAC)
1
Figure 4.1 General Model for Auditing
4.5.2 While gathering objective evidence the auditor should ensure that they interview a variety of
operator personnel as well as management personnel. This will help to determine the level of
knowledge and understanding for key IS-BAO program elements throughout the organization and
to form a complete picture of the organization and its activities
4.5.3 While this process will facilitate an effective audit for a variety of types of operations Appendix A
presents some considerations for the audit of aircraft management companies which may be
involved in commercial and non-commercial operations or non-commercial managed aircraft
operations.
4.6.1 As the IS-BAO is based on industry best practices and many of the standards are performance
based, that is they identify “what” must be achieved not “how” it is to be done, a considerable
amount of judgement must be used in determining if an audit is successful. In general terms, the
audit is considered successful if the operator is in conformity with the standards contained in the
IS-BAO in that there are no major non-conformities.
4.6.2 A minor non-conformity would be one that does not constitute a significant risk to safety. It may
be an isolated instance of non-conformity, a lack, or error in documentation of a process or
procedure that was correctly executed, or a similar instance. A major non-conformity is when
there is a pattern of instances of minor non-conformities or a non-conformity with a standard
specified in the IS-BAO that in its self constitutes a significant risk to the safety of the operation.
4.7.1 As the IS-BAO provides considerable latitude for the operator to develop programmes and
procedures that are suited to the individual operation and to publish them in manuals and
directives, a high degree of judgement by the auditor is required when determining Non-
1
Dennis R. Arter, Quality Audits for Improved Performance (3rd ed. Milwaukee: Quality Press, 2003) p. 5.
January 1, 2014 page - 11 © IBAC All rights reserved
IS-BAO Internal Audit Manual International Business Aviation Council (IBAC)
conformity. As the cornerstone of the IS-BAO is the operator’s safety management system, it
should be key in making that decision. There would be a major non-conformity if there is clear
evidence that:
a. the operator has not correctly identified the safety-risks associated with the operation; or
b. effective mitigation has not been developed and applied for a safety-risk assessed in the
Operator Safety-Risk Profile as “Medium” or higher.
4.7.2 After that point, the key consideration is the effectiveness of the process or procedure that the
operator has developed and applied to mitigate the safety-risks inherent in the operation.
Consideration of the “company culture” and the “hazard identification and tracking system” may
be of assistance in making determinations. The company culture should be to effectively involve
all operator personnel in the organization’s safety management system. The hazard identification
and tracking system should actively involve all operator personnel, and management should
analyse observations, develop and implement appropriate solutions and track their effectiveness.
If there is clear evidence that these two key components of the operator safety management
system are in place, considerable latitude in application of IS-BAO standards is appropriate.
Chapter 5 provides more information on evaluation of the operator’s SMS and taking into
consideration the stage of development that it is at.
4.7.3 Of course, it is a given that the process and procedures used by the operator must be in
conformity with the civil aviation rules of the State of Registry and for commercial operations the
State of the Operator.
4.8.2 The purpose of the Closing Meeting is to communicate to the convening manager and other
appropriate personnel, the results of the audit and to do so immediately after completion of the
audit activities.
4.8.3 The Closing Meeting should consist of a discussion of the audit findings, observations and
results. The Closing Meeting should be organized in accordance with the audit protocols used
covering each module in turn.
4.8.4 The format and major headings for the Closing Meeting should be as follows:
a. objective and scope of the audit;
b. summary of the audit procedures, (including a list of those interviewed); and
c. audit findings and observations.
5.1.1 A Safety Management System (SMS) is a process to explicitly identify, manage and measure the
safety risks that inevitably occur in all aviation operations. It is the cornerstone of the
performance based IS-BAO and it includes the key elements of the IS-BAO standards such as:
organization structure,
duties and responsibility of personnel,
training and proficiency programs,
flight operations procedures and processes,
standard operating procedures,
maintenance control system, etc.
Therefore, it must be kept in mind when discussing the operator’s SMS that these key elements
are an integral part of it.
5.1.2 The IS-BAO has been designed so operators can customize their safety management activities to
suit the size and nature of their operation and their specific operating conditions. The operator’s
safety-risk profile or the methods each organization uses to identify and manage safety-risks are
particularly important when the auditor is planning and conducting the evaluation, and when
determining the relative significance of findings. This chapter of the Internal Audit Manual has
been developed to assist auditors in dealing with the range of differences that may result from the
application of these performance based standards in a broad range of specific situations.
5.2.1 The goal of the SMS evaluation is to determine whether the operator is managing the safety-risks
of the operation to a level as low as reasonably practical. This is done by assessing the
soundness, appropriateness and effectiveness of the operator’s safety management activities.
Figure 1.1 refers.
5.2.2 Although auditors will explicitly evaluate the SMS, they will, at the same time, gauge and
comment on aspects of the operator’s safety culture. An operator’s safety performance is the
sum of the achievements of its formal SMS and the attitudes and values that influence the
behaviour of everyone in the company. The guidelines for evaluating the effectiveness of an
SMS, described in section 5.8, assess indicators of safety culture.
The implementation and operation of an SMS takes time, even for mature aviation organizations.
Therefore, the auditor must determine a reasonable level of performance that can be expected
when evaluating the SMS. Figure 5.2 illustrates the different stages of an SMS and guidelines
are provided in section 5.4 for determining the stage of development of the operator’s SMS.
2
Figure 5.2 Stages of Maturing of an SMS
5.4.1 As noted earlier, the maturity of a Safety Management System will vary from operator to operator,
and within the same operation over time. The auditor must take this into account before
conducting an evaluation. It is counter-productive to measure an SMS that is just being
implemented in the same way as one that has been in operation for years. What is as “low as
reasonably practical” will differ from one operator to another and evolve with the maturity of the
operator’s SMS. Yet evaluation processes must be fair, and findings consistently derived.
Therefore, an auditor’s first step before each evaluation is to determine, with the operator, the
objective of that evaluation.
5.4.2 The stages of SMS development described below are illustrated in Figure 5.2.
5.4.3 Stage One – The SMS is Documented, Approved, Resourced and Being Implemented
A Stage One Evaluation is conducted:
a. For initial applicants for IS-BAO registration;
b. For a renewal audit if considerable weaknesses were identified in the previous evaluation; or
c. As a benchmark audit in situations such as after a change in management or after significant
organizational change.
The objectives of a Stage One Evaluation are to confirm that:
a. The necessary SMS infrastructure is in-place; and
b. Past and planned safety management activities are appropriately targeted.
Consequently, the evaluation focuses mainly on assessing the soundness and appropriateness of
the SMS. Auditors should plan approximately 75% of their activities to be spent measuring the
soundness of the SMS, and approximately 25% for assessing appropriateness. Guidance is
provided in section 5.8.1.
2
Adapted from B. Frohlich, “Performance Measurement of Safety Management Systems” in Safety Performance
Measurement, edited by J. van Steen, European Process Centre, Institute of Chemical Engineers, Rugby, UK, 1996.
January 1, 2014 page - 14 © IBAC All rights reserved
IS-BAO Internal Audit Manual International Business Aviation Council (IBAC)
A Stage Two Evaluation is conducted of an operator that has successfully completed either a
Stage One or Stage Two Evaluation.
The objectives of a Stage Two Evaluation are to confirm that:
a. Safety management activities are appropriately targeted; and
b. Safety-risks are being effectively managed.
Consequently, a Stage Two renewal audit focuses on assessing the appropriateness and
effectiveness of the SMS, and only samples indicators of SMS soundness.
Auditors should plan approximately 30% of their activities to measure soundness, and
concentrate their attentions on areas that were previously weak or on areas that are particularly
important (e.g. safety profile, reporting program, hazard tracking process). They should plan to
focus the remaining 70% of their activities on assessing appropriateness and effectiveness.
The proportion may favour appropriateness for companies still implementing their SMS (e.g. 50%
appropriateness and 20% effectiveness) and effectiveness for more mature SMS (e.g. 20%
appropriateness and 50% effectiveness). Guidance is contained in section 5.8.2.
5.4.5 Stage Three – SMS Sustained and Supported by an Ongoing Improvement Process
A Stage Three Evaluation is conducted during an evaluation of an operator that has successfully
completed at least one renewal audit. The objectives of a Stage Three Evaluation are to confirm
that:
a. Safety management activities are fully integrated into the operator’s business; and
b. A positive safety culture is being sustained.
Consequently, a Stage Three Evaluation primarily assesses the effectiveness of the SMS, and
only samples indicators of SMS soundness and appropriateness. Auditors should plan to spend
approximately 75% of their activities measuring effectiveness, and only assess soundness and
appropriateness where necessary to understand the causes of potential deficiencies identified in
effectiveness. Guidance is included in section 5.8.3.
The convening manager and the auditor should work together to ensure that the objective and
scope adequately consider the stage of maturity of the operator’s SMS before the
commencement of the audit.
SUMMARY
Determine the objectives of the Safety Management System evaluation
Stage One Initial applicants
Renewal audit (previously weak performance)
As requested or required
Note: Structure the evaluation to reflect the agreement made with the operator.
5.6.1 As noted earlier, safety performance is best if the SMS is sound, safety activities are appropriate,
and the actions of executives, managers and staff consistently and effectively reduce safety-risks
to a level as low as reasonably practical. For this reason, the SMS evaluation framework is
designed to measure indicators of soundness, appropriateness and effectiveness.
5.6.2 Because operations vary in size and complexity, the aircraft flown, the airspace and airports
used, the geographic and meteorological conditions encountered, and the experience and skills
of the managers and staff employed; optimal safety management varies from operator to
operator. The auditor’s evaluation must consistently and fairly measure the safety performance
of each operator, despite the differences.
5.7.1 Soundness is measured to determine whether the operator has the necessary foundation for
proactive safety management. It is also the first general diagnosis of the operator’s safety health,
and therefore its safety culture. Most of the indicators of soundness can generally be found by
reviewing formal documents, files and procedures. The information can be subsequently verified
by interviews or observations. When assessing soundness, the auditor seeks to answer the
question:
“Is the infrastructure in place to reduce safety-risks to a level as low as reasonably
practical?”
5.7.3 Effectiveness is measured to determine if the desired results are being achieved.
Circumstances embedded in human, operational, and organizational factors will conspire to
impede safety performance, even in companies with a well-managed, well-targeted SMS. The
indicators of effectiveness are generally found when conducting interviews, or when observing
the operation (after which the observations are validated by follow-up interviews, or by examining
files or documents). When assessing effectiveness, the auditor seeks to answer the question:
“Is everybody effectively engaged in, and capable of, reducing safety risks to a level as
low as reasonably practical?”
The evaluation is conducted by assessing each of the SMS requirements specified in the IS-BAO
for soundness, appropriateness, and effectiveness utilizing the Chapter 3 of the Detailed ISBAO
Audit Protocols in section 8.2 of this APM.
Auditors should customize their evaluation to suit the operation being examined. For example a
small operation with one or two people would be expected to have a basic SMS, while the SMS of
a large operation would be expected to be much more intricate.
The aim of the Stage One evaluation is to confirm that the SMS is documented, approved,
resourced and being implemented. The auditor examines indicators of the soundness of the SMS
foundation, and of the appropriateness of the operator’s safety management activities.
The SMS is considered sound if the operator has the required components of an SMS in place
based on the ICAO framework of 4 components and 12 elements. The auditor determines the
level of detail required to evaluate soundness. Section 6.3 contains guidelines for determining the
relative importance of the deficiencies that might be identified.
Appropriateness is assessed for all 12 elements of the SMS. Although effectiveness is not
required to be assessed for a stage 1 evaluation, the auditor may do so if the SMS maturity
allows and if it provides value to the operator.
The auditor should concentrate on assessing the appropriateness and effectiveness of the
operator’s safety management activities utilizing the detailed protocols. Although the auditor is
expected to validate soundness for renewal audits, an in-depth evaluation of soundness is only
required for those areas that have changed since the previous evaluation, or which warrant re-
examination. For example, if there has been a significant change in any of the following areas, a
more in-depth examination of the related SMS elements, to include soundness, would be
appropriate:
Structure of the organization
Roles of key managers
Staffing levels
Aircraft type
Routes, or
Missions.
In cases where there has not been significant change since the previous evaluation, the auditor
may wish to examine, in more depth, the components that were either weak in the previous
assessment, or which are key to effective safety performance.
The on-going improvement process should include regular evaluation of safety management
activities in relation to stated SMS objectives plus the evaluation of the effectiveness of risk
controls. It also should include a formal process to identify the causes of sub-standard
performance of the SMS, determine the implications of sub-standard performance of the SMS in
operations, and eliminate or mitigate such causes.
For Stage Three evaluations, auditors are encouraged to conduct an in-depth evaluation of
effectiveness utilizing the detailed protocols along with the following performance indicator
examples. The possible methods by which the information can be obtained are provided in
brackets with the following abbreviations employed: doc. – manuals, formal documents,
procedures, files, etc; int. – interviews with managers, staff or stakeholders; obs. – on-site or in-
flight observations; exp. – the experience of the auditor.
a. Are comments on the efficacy of the safety policy and related policies actively solicited
from all stakeholders, managers and staff? (doc., int., obs., exp.)
b. Are the policy/policies periodically and thoroughly reviewed and when appropriate,
revised? (doc., int., obs., exp.)
c. Is top management fully aware of the human and organizational factors that endanger the
operation and do they continuously make decisions mindful of those factors? (doc., int.,
obs., exp.)
d. Does management drive the SMS process and vigorously seek excellence? (int.)
e. Are responsibilities for safety and contributing to the SMS effectiveness fully imbedded
throughout all levels of the organization? (doc., int.)
f. Do the appraisals of personnel at all levels include related SMS targets (activities and
results) and is exemplary behaviour rewarded? (doc., int.)
g. Are safety management activities being continually monitored, evaluated and improved
when opportunities are identified? (doc., int.)
h. Have managers and staff clearly demonstrated that they are capable of performing their
roles to proactively manage safety? (obs., exp.
i. Is there evidence of the documentation evolving as the other elements of the SMS
matured? (doc., int.)
2. Safety Risk Management
a. Are the policies and procedures to mitigate safety-risks periodically reviewed for
applicability and effectiveness and modified when found wanting. (int., obs., doc.)
b. Does the mitigation account for predictable human, operational and organizational
limitations? (int., obs., exp.)
c. Are risk assessment and management fully integrated into all operations and
maintenance activities? (int., obs., doc.)
d. Is the information indicating the need to change respected and valued? (doc., int., obs.)
e. Are long and short-term plans to improve safety performance driven by multi-level
analyses of the safety-risk management database and integrated with business planning.
(int., obs., doc.)
f. Is there solid evidence that managers and staff are fully engage in and committed to,
enhancing safety performance? (int., obs., doc., exp.)
g. Is their consistent feedback that has been effective in ensuring the participation of
managers and employees? (doc. int., obs.)
h. Is the information used not only to enhance day-to-day safety, but also to improve long-
term safety management? (doc., int.)
i. Are the results of safety management activities evident in the operations manual and its
amendments? (obs., doc., int.)
j. Are there effective processes to ensure that any amendments to the Operations Manual
that result from safety management activities are made in a timely manner? (obs., doc.,
int.)
k. Are employees highly motivated to report occurrences and does the environment exist
where they are fully prepared to make such reports when it involves admitting that they
made mistakes? (int.)
l. Are occurrences analyzed and the lessons learned integrated into the appropriate
programs and procedures? (doc., int.)
3. Safety Assurance
4. Safety Promotion
a. Is there an established effective process to ensure that the results of SMS activities are
fully integrated into the appropriate training programs? (int., doc.)
b. Are employees highly motivated to report apparent safety deficiencies and fully confident
that there will not be retribution? (int.)
c. Are there well established processes to expeditiously evaluate safety information and
integrate it into safety management activities? (doc., int. obs., exp)
5.9 Reserved
6.1 Introduction
Findings are significant issues which require urgent action by the operator's to correct. Findings may
relate to one or more non-conformities but should be sufficiently serious to require urgent action. As
previously noted, a minor non-conformity would be one that does not constitute a significant risk to safety.
It may be an isolated instance of non-conformity, a lack, or error in documentation of a process or
procedure that was correctly executed, or a similar instance. A major non-conformity is when there is a
pattern of instances of minor non-conformity or a non-conformity that in itself constitutes a significant risk
to the safety of the operation.
6.2.1 Findings must be significant issues that need the operator’s attention. They must relate to
standards specified in the IS-BAO, must be substantiated by factual information and in most
cases should be validated by objective evidence of different forms or sources of information (for
instance, corroborating information from: different documents; documents and observations; or
documents, interviews and observations). Findings of an IS-BAO audit must address the degree
to which safety risks are being managed to a level “as low as reasonably practical” (i.e. the
objective of all evaluations of SMS). As such, the findings should always relate to management
of the safety-risks, not to the operational indicators of deficient safety management.
Consequently, the findings are developed after the components of the SMS and related elements
have been assessed (as described in chapter 5). Auditors must take the information from their
assessments, and use logic and experience to determine the cause(s) of the problem. It is these
cause(s) that make up the findings. It is important that the auditor address the disease not the
symptoms of the disease.
6.2.3 It is therefore important that indications of non-conformity and the related objective
evidence collected in the course of the audit be carefully analysed to determine the
cause(s) of the problem. When the cause has been identified it should be presented as a
Finding with the non-conformities and other objective evidence, used as supporting material.
If it is determined that one or more minor non-conformities are simply slips, lapses or omissions
that can easily be corrected and there is no significant underlying problem, rather than making a
Finding the non-conformity should be noted in the Analysis of Non-conformities section of the
Audit Protocol along with the comment, “ACTION REQUIRED”. “Full Conformity” should not be
selected on the Audit Report Form if there are any instances of non-conformities, regardless of
how they are documented, i.e. on the Finding forms or in Analysis of Non-conformities block.
6.2.4 A major non-conformity, by definition, will be sufficiently critical that a Finding must be made.
However, the same degree of analysis must be undertaken to identify the cause(s) of the
problem.
6.2.5 Findings are to be recorded on the Audit Finding Form that is contained in section 8.3.
6.2.6 It should be noted that a finding cannot be based on non-conformity with a Recommended
Practice contained in the IS-BAO, but that fact can be used in conjunction with other non-
conformity information to substantiate a conclusion.
Auditors must be able to consistently determine the criticality of their findings so operators can
plan appropriate remedial action. Criticality is determined by judging the scope of the finding (i.e.
by determining whether the consequence is far reaching), and by assessing its impact on the
safety of the operation. The measure differs from an Initial and a Renewal audit.
An IS-BAO audit that is conducted of an operator where their SMS is at a first stage of
development and implementation will focus on the degree to which the SMS and other IS-BAO
requirements are documented, approved, resourced and is being implemented, so that safety
risks are reduced to a level as low as reasonably practical. Therefore, critical findings would be
those that indicated deficiencies in documentation, resources and implementation, and
particularly those that could seriously impede a sound SMS, or could lead to unsafe operational
practices. Examples of critical deficiencies in the SMS structure might include:
a. A deficient document management system;
b. An absence of planning or coordinated activity to implement the programs, procedures or
SMS; and
c. An absence of resources (time, money, skills) to implement or operate the programs,
procedures or SMS
Examples of deficiencies that could lead to unsafe practices might include:
a. A poorly constructed hazard identification and risk management process or Safety-Risk
Profile;
b. Poorly developed or implemented mitigation (programs, procedures and manuals) from a
sound Safety-Risk Profile; and
c. Weak management oversight.
Generally, the findings of a Stage One Evaluation are important because deficiencies identified
have the potential to be broad in scope and far-reaching in consequence. Weaknesses in the
foundation of the operators programs, procedures, manuals or SMS suggest:
a. The operator will be unable to proactively and consistently manage and measure its safety
performance; and
b. There may be a lack of understanding, or a lack of commitment to appropriate and effective
safety management.
An IS-BAO audit that is conducted of an operator where their SMS as at the second stage of
development and implementation will focus on the degree to which the operators systems and
procedures, including SMS, are functioning and the results are being measured so that systems
and procedures can be improved when required and safety-risks can be continuously reduced to
a level as low as reasonably practical. Therefore, critical deficiencies would be those that
indicate:
a. There are inconsistencies in adherence to operational, maintenance or safety management
procedures, including during periods of change; and
b. Feedback from hazard reports, safety committees, audits or risk assessments is either not
being obtained, or not being employed to update operator programs, procedures and the risk
profile, or to improve the related mitigation (e.g. procedures, training, equipment, etc.).
The auditor must take this information and determine the cause(s) before making the finding(s).
Deficiencies may indicate a need to make findings on a lack of understanding of human or
organizational factors; a lack of understanding of, or commitment to, proactive safety
management; or conflicting resource requirements, amongst others.
An IS-BAO audit that is conducted of an operator where their SMS as at the third stage of
development and implementation will focus on determining whether the SMS is sustained and
supported by an ongoing improvement process. This ongoing improvement process should be
evident in the operator programs, procedures, manuals and SMS processes and procedures so
that safety risks are continuously reduced to a level as low as reasonably practical.
7.1.1 Audit reports are a consistent and objective reflection of the results of the audit and are to adhere
to the reporting principles established in this Manual.
7.1.4 Operator conformance with the IS-BAO standards is recorded in the audit protocol; however this
does not inform the operator, audit review group or subsequent auditors about the level of
conformity and the maturity of the operator’s systems, programs and procedures. The spaces
provided for remarks and observations within the protocols should be used for this purpose. The
auditor shall provide a comment or reason for any protocol element indicated as “NA”.
7.1.5 While broad, general observations help to better understand the operation, comments regarding
specific elements of the protocol will provide the operator with reinforcement on those done well
and serve as insights to improve areas requiring emphasis. Specific comments also provide audit
reviewers with valuable insights into the operator's program and how the standards are being
applied. Qualitative comments/observations about levels of implementation, program acceptance
and understanding by operator personnel, especially for the SMS, are helpful for all concerned.
Training, standardization levels, corporate culture and record keeping are also areas of interest.
7.2.1 As explained in section 4.9 an audit is considered successful if the operator is in conformity with
the standards contained in the IS-BAO in that there is no major non-conformity. When
considering most of the IS-BAO requirements this will be very straightforward, however, when
determining if there are any minor or major non-conformities related to the operator’s SMS, it is
important to carefully consider the stage of development and implementation that it is at. In
general terms, for an initial IS-BAO audit where the SMS is in the first stage of development and
implementation, the auditor will assess the findings and conclude the evaluation by determining
whether the operator’s SMS is:
7.2.2 If an auditor finds that one or more of the above-listed criteria have not been met, then the auditor
will conclude that the operator’s safety performance is unsatisfactory. Such a conclusion must
be supported with a major non-conformity finding that explicitly identifies the risk to safety. In
doing so the critical deficiencies (as described in 6.3.1 and 6.3.2, respectively) must be
highlighted to explain the auditor’s conclusion, so that the operator can develop an action plan
that will lead to rectification of the deficiency and a successful audit.
7.3.1 A copy of the IS-BAO Audit Report Form is provided in section 8.1. A copy of all IS-BAO Audit
Finding Forms and the other completed Audit Protocols that were used in the audit should be
attached to the Audit Report Form.
Stage 2 SMS is functioning and results are being measured; Safety risks are effectively managed;
Safety management activities are appropriately targeted.
Summary of Audit Including Overall Assessment of the Appropriateness and Effectiveness of the
Operator’s SMS and Other Management System Controls
Additional Comments
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
6.4.6 Reserved
6.4.7 Does the operator have a policy and
procedures for operating in known or
expected icing conditions appropriate
to the aircraft icing certification and
equipment?
6.4.8H Do the operators of helicopters have
VFR weather limits for both day and
night operations that take into
account the nature of the operations
being conducted and the operating
environment?
6.5 All Weather Operations
6.5.1 Does the operator have procedures
that prohibit conducting an
instrument approach or departures
below standard Category I weather
minima unless all equipment, training
and operating requirements and
regulatory requirements have been
met?
6.5.2 If the operator has authority to
conduct CAT II & III ops are:
a. there approved Category II or III
operating procedures in the
company operations manual,
b. the flight crew trained and
certified to conduct Category II or
III instrument approaches,
c. the aircraft equipped and
approved for Category II or III
operations?
6.6 CNS Requirements
6.6.1 Is there a process to ensure that
prior to operations in airspace where
special CNS requirements exist such
as Performance Based Navigation
(PBN) Specifications, Minimum
Navigation Performance
Specification (MNPS), Reduced
Vertical Separation Minimums
(RVSM), Controller Pilot Data Link
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
Note 1: It is the owner's/lessee's/operator's (as applicable) responsibility to take all appropriate actions to
ensure adequate oversight of the contracted CAMO for the continued airworthiness of its aircraft/fleet.
See IG 9.1.2 for more information.
Note 2: The CAMO oversight process should be integrated into the compliance monitoring system
required by IS-BAO 3.3.1.
9.1.3 Does the operator have a written
description of its maintenance control
system in the company operations
manual or maintenance manual?
9.1.4 Does the maintenance control system documentation contain at least the following information:
a. Where maintenance functions
have been assigned:
i. The position or title of the person
to whom functions have been
assigned?
ii. A description of the functions and
scope of work that have been
assigned to each position, person
or organization?
iii. Where necessary for clarity, a
chart depicting the distribution of
functions and lines of authority (if
not depicted in the organizational
chart)?
b. For elementary work or preventive maintenance and aircraft servicing:
i. Identification of those standards or
maintenance data (aircraft
manufacturer’s, CAA’s or other) to
be used?
ii. The procedures to confirm that
regulatory information and
technical data appropriate to the
work performed are used;
iii. Details of the methods used to
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
Analysis of Non-Conformities/Findings
a. Weather Briefing
b. NOTAMs
2. Flight Planning
a. Route Analysis
b. Fuel Consumption
c. Alternates
a. Fuelling Procedures
b. Load Security
c. Ground Handling
d. Aircraft Parking
5. Pre-Flight
a. External Inspection
c. Emergency Drills
7. Pre-Start
a. Engine handling
b. ATC Procedures
c. Noise Abatement
d. Lookout
e. Checks
f. Radio Procedures
a. En-route Comm
b. Navigation
c. Flight Management
a. Planning
b. Descent
c. Final Approach
15. Shutdown
2. MEL
4. Aircraft Log
5. Maintenance Release
6. Aircraft Equipment
7. Emergency Equipment
Analysis of Non-Conformities/Findings
DETAILS OF FINDING [SHOW CAUSE/EFFECT/CRITICALITY IAW APM SECTIONS 6.2 & 6.3]
OBJECTIVE EVIDENCE
IS-BAO registration has been available to and used by a number of aircraft management companies over
the life of the program. However, the addition of larger companies offering aircraft management services
for non-commercial operations has introduced elements that must be considered during the audit. Among
them are:
Company size – Management companies comprising a large number of operators providing both private
and commercial operations are becoming more common. Taken together these operations may
encompass scores or even hundreds of aircraft/personnel.
Multiple bases – Few management companies provide services to clients at a single base, with the norm
being from two to sometimes more than 20 bases.
Changing constituency – Clients enter into or terminate management contracts at will, creating a
constantly changing workforce and aircraft roster.
Mission diversity – Clients inevitably desire different frequency, type and levels of service.
While the structure of IS-BAO is designed to accommodate the aircraft management company concept
the variables mentioned above must be considered when conducting an audit. These are not changes to
the methodology set forth in the IS-BAO Audit Procedures Manual, but points of emphasis to be
addressed when conducting a management company audit:
Management Contract
Does the management contract clearly define the roles and accountabilities of each party and
require compliance with stated procedures and provisions?
SMS
How is the SMS managed at the management company and within each of the client locations?
How is the SMS integrated between the management company and its clients?
Multiple bases
Are there methods for monitoring performance and conformance of all operating bases?
How does the management company ensure that all operations are in compliance with applicable
State of registry regulations and in conformance with IS-BAO standards and procedures?
Are actions taken to rectify deficiencies noted during the internal evaluation process and conduct
post implementation reviews?
Sampling
Personnel at a minimum of 10% of the management company clients should be interviewed to
ensure adequate knowledge of and conformance with IS-BAO standards.
Audit Report
The audit report for a management company providing comprehensive services should:
Clearly indicate the size and scope of operations and level of services provided.
Comment on all of the features listed in the above bulleted statements.