International Journal for Quality in Health Care 2000; Volume 12, Number 3: pp.

231–238

Methods for external evaluation of health

care institutions
LLUÍS BOHIGAS1 AND CATHERINE HEATON2
1
Director, Avedis Donabedian Foundation, Barcelona, Spain and 2External Peer Review Techniques, Bristol, UK

Abstract
Objective. To compare the methods used by external evaluators of health care institutions in Europe.
Methods. A common framework for analysis was designed by the authors and shared among the members of the ExPeRT
Project. Each member prepared a description of a model and the results were compared in two workshops.
Results. Programmes share similarities in the methods used, but they differ in the focus and purpose of the evaluation.
Differences in focus included whether a part or the whole of the institution is analysed and whether the review is patient
or system centred. Different purposes of the programmes are reflected in the emphasis and use of the methodological tools:
for instance, the appeal system is used only in the programmes that provide a certificate to the institution audited.
Keywords: external evaluation, health care institutions

The External Peer Review Techniques project (ExPeRT)
began, in 1996, to catalogue the range of programmes offering
external evaluation of health care facilities in the European
Union. These programmes fell within four main categories
or approaches to the external evaluation of health care
institutions: accreditation, certification by the International
Organization for Standardisation (ISO. In Greek ‘iso’ means
equal), award-seeking [European Foundation for Quality
Management Award (EFQM)], and professional peer review
or ‘visitatie’ (Dutch for ‘visitation’ or ‘peer review’). These
different approaches are presented in the paper by Shaw in
this issue [1].
The ExPeRT project analysed the strategic and the op-
erative aspects of these four approaches. The strategic aspects
are presented in other articles in this issue; this article is
devoted to studying the operational activities undertaken by
the four approaches to gather information and to make
decisions about the institutions evaluated. Three crucial ac-
tivities in the operations of external evaluators are [2]:
• the development of standards;
• the selection, training and monitoring of evaluators;
• the evaluation process.
This article will focus on the second and third activities. The
selection, training, and monitoring of evaluators comprises all
the activities undertaken to prepare peer reviewers to gather
information about the quality of the institution evaluated.
The evaluation process consists of all activities that the
evaluator performs in order to give an informed opinion
about the quality and the performance of the reviewed
institution.
Each of the two activities analysed is comprised of a
set of processes. This article is structured following these
processes. For each process, its content will be explained
and information from the four different evaluation pro-
grammes will be presented. The order of presentation will
be accreditation, ISO, EFQM and visitatie. Not every process
applies to each evaluation method.
Methods
The comparison was made by means of a protocol that was
circulated among the partners of the ExPeRT project [1].
Each partner prepared a summary of one programme. These
summaries were presented and discussed at a seminar in
Budapest, 6 October 1998, and a draft comparative paper
was analysed at a seminar in Stockholm, 24 May 1999.
The partners reviewed the different systems. Lluı́s Bohigas,
Avedis Donabedian Foundation (Spain), prepared the analysis
of accreditation programmes; Catherine Heaton, ExPeRT
(UK), analysed ISO certifiers; Iris Blomberg, National Board
of Health and Welfare (Sweden), studied the European
Foundation of Quality Management; Jacqueline van Bat-
enburg and Bertjan Koekenbier of the National Organization

Address reprint requests to Lluı́s Bohigas, Director, Avedis Donabedian Foundation, Provença 293, 08037 Barcelona, Spain.
E-mail: fad@sct.ictnet.es

 2000 International Society for Quality in Health Care and Oxford University Press 231
L. Bohigas and C. Heaton
for Quality Assurance in Hospitals (CBO, The Netherlands)
studied visitatie. In the analysis of accreditation programmes
data was used not only from European accreditation pro-
grammes, but also from the programmes of the Joint Com-
mission on Accreditation of Healthcare Organizations
(JCAHO, USA), the Canadian Council on Health Services
Accreditation (CCHSA), the Australian Council on Healthcare
Standards (ACHS) and the New Zealand Council on Health-
care Standards (NZCHS).
Selection and training of evaluators
The management of evaluators comprises two basic processes:
selection and training.
Selection
In accreditation, the main criterion for recruiting surveyors
is experience in the health care sector within the defined
professions of doctor, nurse and administrator or chief ex-
ecutive. All accreditors require a minimum period of ex-
perience in senior managerial positions that varies between
2 and 5 years. The accreditors that employ volunteers prefer
their surveyors to be professionals currently practising in a
health care facility: JCAHO requires nurses and administrators
to hold a Masters degree; CCHSA requires its surveyors to
be employed in an accredited institution; ACHS requires
knowledge of the Australian health care system, good in-
terpersonal skills and commitment to ACHS accreditation;
NZCHS requires knowledge and experience in continuous
quality improvement [3].
The ISO norm EN 45012 states that the certification body
shall ‘employ a sufficient number of personnel having the
necessary education, training, technical knowledge and ex-
perience for performing certification/registration functions
relating to the type, range and volume of work performed’.
This norm does not require certification bodies to employ
‘peer reviewers’ for the health care sector. When assessing
health care facilities, many certification bodies in Europe use
auditors with some health care experience, making a certain
level of relevant expertise mandatory. These bodies add more
specific requirements to their auditor selection criteria, for
example: knowledge and experience working in quality man-
agement; understanding of certification schemes and a mini-
mum of 20 days auditing experience; relevant health care
qualifications, including medical, nursing or health care man-
agement; and seniority in the health care sector. Criteria
for recruiting auditors include attributes such as maturity,
judgement and understanding, analytical skills; general edu-
cation and language fluency; management capabilities; training
and evaluation ‘to the extent necessary to ensure their com-
petence in the skills required for carrying out and managing
audits’; at least 4 years full-time appropriate practical work-
place experience; at least 2 years in quality assurance activities;
and at least 20 days and four audits as a trainee auditor,
including documentation review, actual audit activities and
audit reporting.
232
EFQM assessors include some academics and quality pro-
fessionals but the majority are drawn from the ranks of
experienced managers currently practising in Europe.
Visitatie schemes include the following selection criteria
for potential visitors (different scientific associations may
vary). A visitor:
• has to be a registered specialist for at least 5 years;
• should be independent of the clinical staff being sur-
veyed.
In particular the visitor should not:
• be working in the same region as the clinical staff
under review;
• be the former trainer of one of the departmental
clinical staff members being surveyed;
• be the former promoter of one of the departmental
clinical staff members being surveyed.
Training
In accreditation all surveyors undertake training at the be-
ginning of their surveyor careers. Most accreditors require
2–4 days of initial training, an exception being JCAHO which
requires surveyors to undertake 15 days of orientation and
training. Thereafter surveyors receive ongoing updates and
education of between 1 and 5 days per year. The training
methodologies are always participatory. The content of the
training usually includes the following areas: standards know-
ledge; survey processes; communication; interviewing; and
report writing. Most accreditors use the observation of real
surveys as a part of the training.
ISO requires auditors to be trained and assessed by ex-
ternally recognized training bodies, who approve auditor
training programmes and officially certify auditors who have
passed certified courses. Some certification bodies run their
own training programmes. Most use auditors who have
completed approved training courses and have been certified
by the International Register of Certificated Auditors (IRCA)
or an equivalent organization. IRCA-approved training
courses are required to last for 36 hours (soon 40 hours) and
cover all procedures relating to the audit of facilities to ISO
9000 standards. Training courses are usually residential, spread
over 5 days and include role-play, lectures and, in some cases,
a live audit. In addition, IRCA auditors are required to meet
certain academic qualifications and have experience in a
working quality and audit environment, in order to become
certified. ISO standards require simply that an auditor’s ability
to meet ISO requirements is judged by an evaluation panel
chaired by a qualified auditor and including members selected
by internal audits, customer audits and independent third
party audits. Selection of the auditor or audit team operates
under the general guidelines of ISO 10011.
In visitatie the visitors receive 1 day’s training from CBO
on procedures, attitudes and techniques. CBO continues to
coach them through their first visit with specific personal
feedback.

The evaluation process
The evaluation process usually comprises three phases:
• survey;
• evaluation;
• between surveys.
The survey is the primary fact-finding phase. It usually takes
the form of a team of professionals visiting the institution and
writing a report of their findings. The second phase is
contingent on the survey: based on the report, the evaluator
uses criteria and scoring to grade and eventually give an
award or a certificate to the evaluated institution. The third
phase occurs between the survey or certification and the next
evaluation [4].
Survey
The survey phase can be divided into three subphases, namely:
preparation; survey; report.
Preparation
The preparation for the survey comprises all activities carried
out by the accreditor and the institution to prepare for the
survey evaluation.
Five elements are present in the preparation phase, although
not all systems apply all processes:
• questionnaire;
• self-assessment by the institution being evaluated;
• composition of the evaluation team;
• agenda or audit plan;
• pre-survey activities.
Questionnaire. The voluntary nature of the evaluation pro-
cess is common to all four models, so the onus is on the
health care institution to request an evaluation. The evaluator’s
initial response often takes the form of a questionnaire aimed
at identifying the institution’s eligibility. This questionnaire is
sometimes considered a contract between the institution and
the evaluator, and includes notification of the evaluation fees
and other legal regulations, such as confidentiality of the
data. The demographic and biographic data obtained through
the questionnaire are usually the basis for planning the size
of the evaluation team, the number of days, etc.
Self-assessment by the institution under evaluation. All
evaluators suggest that the institution undertake a self-assess-
ment. How this self-assessment is used varies not only
between the four models but also between programmes
within the models.
Some accreditation programmes use the institution’s self-
evaluation to guide the surveyors in their assessment. The
institution may be asked to state or grade its compliance to
a set of explicit standards; the surveyors then verify the
institution’s results and comment on the differences.
External evaluation of health care
Having registered with an ISO accredited certification/
registration body and determined the need, purpose and
general scope of the audit the client institution is encouraged
to show evidence of self-preparation in some of the following
ways:
• documentation;
• internal audits;
• management review;
• pre-audit by an external organization.
Within certain guidelines, ISO allows the content of pre-
audit documentation submitted to the certification body to
be specified by the client. This should include information
on general features and resources of the institution; a copy
of the institution’s quality manual; general information on
the quality systems in place and a description of those to be
assessed.
EFQM promotes ‘self-assessment’, believing the process
to be a catalyst for driving business improvement. The EFQM
definition of self-assessment is as follows:
(i) Self-assessment is a comprehensive, systematic and
regular review of an organization’s activities and
results referenced against the EFQM model of busi-
ness excellence.
(ii) The self-assessment process allows the organization
to discern clearly its strengths and areas in which
improvements can be made and culminates in
planned improvement actions that are then mon-
itored for progress.
When applying to EFQM, the health care service is asked
to provide data about the activities generally derived from
their self-assessment. This information must be closely aligned
with EFQM’s nine award assessment criteria.
Clinical departments participating in a peer review (visitatie)
scheme are also asked to complete a self-assessment ques-
tionnaire prior to the visit. This questionnaire addresses the
organizational aspects of professional performance, giving
the ad hoc visitation committee an opportunity to select and
discuss key quality issues with the practitioners and other
staff members before the evaluation visit.
Composition of the evaluation team. Also common to the
evaluation process of all four approaches is the use of an
auditor or team of reviewers. This team is composed of
professionals trained to evaluate the quality of service pro-
vided by the institution. The size of the evaluation team
depends on the size, the nature and complexity of the health
care organization being surveyed.
In the accreditation field, the core survey team consists of
a senior executive physician, nurse and administrator, usually
led by one surveyor, chosen on the basis of seniority. As
organizations increase in size so the survey team gets larger;
for example, a small community hospital may be reviewed
by a team of two surveyors, a larger acute hospital by a team
of four. The composition varies according to the accrediting
body; some prefer nurse–manager teams; others doctor–nurse
233
L. Bohigas and C. Heaton
teams. Most accrediting bodies prefer surveyors who are
generalists (rather than specialists) with specific experience
and expertise in defined settings such as acute care, community
care, ambulatory care.
Agenda/audit plan. External evaluators usually define the
general structure of the on-site evaluation using a timetable,
agenda or audit plan. The generally accepted philosophy is
that the institution should know what is being evaluated and
when, rather than receiving a surprise visit. Within limits, the
evaluators usually have the autonomy to ask about or to visit
other areas besides those planned.
A crucial aspect is the duration of the survey, since this is
the key variable to the cost of the evaluation process. The
duration of the survey in the accreditation field varies ac-
cording to the size, nature and complexity of the health care
organization being surveyed. This may range between 1 day
for a small community health centre to 10 days for a large
regional health organization with multiple sites. The number
of beds is a common measure of size, although some ac-
creditors look at the institution’s activities (such as visits,
length of stay) to determine the complexity and consequently
the length of the evaluation.
ISO’s audit plan, produced by the certification body and
agreed by the client, identifies the objectives and scope of the
audit; audit team members; personnel with key responsibilities
within the facility; reference documentation; the language of
the audit; date; time; duration; meeting schedule; con-
fidentiality requirements; and report deadline and distribution.
In visitatie, the practitioner being visited is responsible for the
agenda of the survey. He or she decides whom the visitation
team will interview. The visit usually lasts 1 day, but it can
be between 0.5 and 2 days depending on the number of
locations and/or practitioners being visited.
Pre-survey activities. Pre-survey activities vary considerable
between the four models. Some programmes offer visits prior
to the formal evaluation, others allocate personnel to guide
the institution through the process, whilst others engage in
discussions between the evaluating body and the health care
service. The completion of a self-assessment and audit plan
is considered sufficient by some programmes.
Some accreditation programmes allocate a client or service
manager to the institution to facilitate the complexities of
pre-survey activities. How important this person is to the
accreditation process varies from programme to programme.
The Health Quality Service (UK) uses this position to instruct
the institution about the preparation phase, to be present
during the survey and to write the report. Other programmes
do not use survey managers. The Hospital Accreditation
Programme (UK) offers facilities the opportunity to par-
ticipate in a pre-survey which takes place 3 months before
the full survey. This involves a 1-day survey by a single
surveyor, who assesses the facility against the standards, and
offers verbal (rather than written) recommendations and
guidance in preparation for the full survey. The surveyor
who undertakes the pre-survey is not included in the full
accreditation survey team.
The primary activity undertaken by the ISO certification
234
body in collaboration with their client is to determine the
scope of the audit and make an initial review, to ensure that
the client is sufficiently prepared for the audit. This usually
takes the form of a visit to the ExPeRT by the lead surveyor.
Some organizations, for example Lloyds Register, divide
these responsibilities between the business manager, who
determines the size and agrees to the scope of the audit, and
the lead auditor, who ensures the facility is ready and discusses
the audit schedule. ISO specifies that the final decision about
which functions and locations are to be audited rests with
the client, but should be reached with the assistance of the
lead auditor.
The length of time taken for an ISO audit depends on
the number of employees, location/sites and complexity of
the service. Based on these factors, ISO Guide 62 outlines
the time requirement for each audit. Certification bodies are
free to allocate the time requirements between as many or
as few auditors as they feel necessary. For example, SGS in
The Netherlands stipulates that a minimum of two auditors
must audit a health care facility. Others bodies may use only
a single auditor.
Survey/audit
The purpose of the survey/audit is to gather information in
order to write the report. The procedures and methods used
by the evaluators during the survey/audit are central to the
operations of the evaluating body. Documentation of these
methods enables evaluations to achieve a level of consistency.
Elements used in accreditation surveys usually include:
• review of documentation;
• interviews;
• sample of medical records and other types of records;
• visits-observations.
All accreditors offer feedback at the end of the survey,
to rehearse the formal written report and to verify their
observations. The Health Quality Service survey includes a
night visit, in order to find out information about what
happens in an important and usually neglected part of the
hospital activity.
Consistency among surveyors is a crucial item of the
accreditor’s performance. A programme’s reputation depends
upon objectivity rather than the subjective individual judge-
ment of each survey team member. Protocols and training
are the main tools to ensure surveyors’ consistency. During
the survey some accreditors require surveyors to reach con-
sensus in grading the standards. Another way to gain this
consistency is to allocate the task of editing and compiling
the report to one person, the lead surveyor or the client
manager, together with the central administration of the
accreditation programme to maintain internal consistency and
compliance with defined rules for reports.
The ISO audit begins with an opening meeting to review
and clarify the scope and the objectives of the audit; introduce
the audit team to senior management; and summarize the
procedures to be used in the audit. During the audit, evidence

is collected through interviews and the examination of docu-
mentation and activities, as well as observation. The closing
meeting enables the audit team to present their observations
and conclusions prior to writing the report.
In visitatie, during the collegial review, peers evaluate the
circumstances under which clinical practice takes place by:
(i) Documentation: prior to the survey the physicians
are asked to inform the ad hoc visitation team about the
availability and status of guidelines, patient medical
records etc.
(ii) Observations: part of the survey is to check whether
guidelines are available, readable and consistent;
(iii) Structured and non-structured interviews: intra- and
inter-disciplinary collaboration, evaluation of patient
satisfaction and treatment outcomes.
Collecting information by interview is not limited to the
physicians under review. Uniformity in medical performance,
or the use of guidelines in medical practice, is also discussed
with other professionals working closely with the physicians
being visited (i.e. the nursing staff, the chairman of the
medical staff, assistants, residents, colleagues from other
specialities, etc.). There is a feedback session at the end of
each survey by the ad hoc visitation team. Through feedback
on their performance and by formulating suggestions for
improvement, visitatie supports physicians in their efforts to
improve patient care.
Report
Each model emphasizes the importance of a report. The
evaluation report will be analysed in three parts: report
content; grading of standards; report writing.
Report content. Accrediting bodies produce reports dem-
onstrating compliance and non-compliance with explicit
standards. Although some accreditors produce negative re-
ports, including only areas of non-compliance, the favoured
method is to balance recommendations for improvement with
positive comments on the good practices found. Accreditation
reports usually contain a text summary as well as numerical
grading with appropriate comment, measured against the
standards.
The report resulting from an ISO 9000 audit contains the
details included in the audit plan; documentation against
which the assessment was made; ‘observations of non-con-
formities’ or areas which did not comply with the agreed
quality system standards, protocols and procedures; and the
audit team’s judgement of the level of compliance. As well
as simply stating non-compliance, as ISO standards require,
most certification bodies include a balance of positive and
negative findings. In The Netherlands, non-conformities re-
ported are based on the nine aspects of the HKZ (the
Dutch harmonization model). The lead auditor also has the
responsibility for submitting the report to the client and
ExPeRT who may then agree to distribute it more widely.
‘The audit is complete upon submission of the audit report
to the client’. Any corrective action or re-audit is agreed
between the ExPeRT and the auditing organization.
External evaluation of health care
EFQM’s feedback report is prepared by a team of in-
dependent assessors, who are senior managers and experts
from across Europe. The report provides a list of strengths
and areas for improvement under each criterion addressed
in the application. The assessor’s scoring profile is also given
together with comparative scoring of other applicants for the
award. After submission of the report to the institution, the
senior assessor attends a meeting to discuss the content of
the feedback report.
The visitatie report consists of a description of the clinical
department, positive and negative findings and re-
commendations with suggestions for improvement.
Grading of standards. Most programmes for the external
evaluation of health care services grade their review findings.
Accrediting bodies do not simply record compliance/non-
compliance but expect the surveyors to grade the degree
of compliance. Grading systems vary from accreditor to
accreditor and may be numerical (e.g. 1–5) or descriptive, (e.g.
minimal, partial, substantial or non-compliance). Institutions
may even be asked to grade themselves, the surveyors check-
ing these grades during the survey and commenting in cases
of divergence.
An ISO non-compliance is graded major or minor, ac-
cording to its impact on the quality of the final product or
process.
EFQM assessors use a defined scoring process to allocate
points to each assessment criteria. The factors used to evaluate
enablers and results are presented. Each of the parts of
the enabler criteria is evaluated according to approach and
deployment. Approach is concerned with the methods the
organization uses to address the criterion parts.
The score given takes account of the achievement of these
items:
• the appropriateness of the methods, tools and tech-
niques used;
• the degree to which the approach is systematic and
prevention based;
• the use of review cycles;
• the implementation of improvements resulting from
review cycles;
• the degree to which the approach has been integrated
into normal operations.
Deployment is concerned with the extent to which the
approach has been implemented to its full potential. The
score given will take account of appropriate and effective
application of the approach:
• vertically through all relevant levels;
• horizontally through all relevant areas and activities;
• in all relevant processes to all relevant products and
services.
Each of the parts of the results criteria is evaluated
according to the degree of excellence and the scope of the
results presented.
235
L. Bohigas and C. Heaton
The degree of excellence of results takes account of:
• positive trends and/or sustained good performance;
• comparison with own targets;
• comparison with external organizations (including
competitors and ‘best in class’ organizations wherever
possible);
• evidence that results are caused by appropriate ap-
proaches.
The scope of results takes account of:
• the extent to which the results cover all relevant areas
of the organization;
• the extent to which a full range of results, relevant to
the criterion part, is presented;
• the extent to which the relative importance of the
results is understood and presented.
An overall percentage score is then derived and converted
into points according to the values shown in the EFQM
model.
In visitatie consensus within the team is the first criterion
for each formal conclusion or recommendation made. The
second criterion is that judgement must be based on more
that one argument mentioned somewhere in the report.
Thirdly, a balance in positive conclusions and re-
commendations for improvement is necessary.
Report writing. Most accreditation programmes require each
surveyor to write a section of the report. Surveyors may be
given a time limit within which to do this, ranging from
completion during or on the day of the survey to completion
2 weeks after the visit. Several accrediting bodies use client
managers or analysts employed by the accreditor to write the
report, based on the surveyors’ notes and observations.
In ISO, the audit or assessment report is prepared by the
team under the direction of the lead auditor.
The findings of the visitatie team are written up in a report,
which is completed within 2 or 3 weeks of the visit by
someone from CBO.
Evaluation
The evaluation of reports is an area of significant divergence
among the four types of external health service review
programmes. The report may be examined by an individual or
a committee that applies explicit rules and implicit judgement
when making the decision of whether or not to accredit/
certify the institution.
The following elements may be involved:
• an evaluation committee;
• rules for making decisions;
• a certificate or award;
• an appeal;
• publication of results.
236
The evaluation committee
The evaluation committee is the group of people within or
affiliated to the evaluating body that evaluates the report and
makes the decision whether or not to award accreditation/
certification to the institution evaluated.
Accrediting bodies call this committee the assessment
panel, accreditation committee, or independent, professional
board. The status and composition of this group also varies:
the committee may include any combination of the following:
senior, independent health care professionals; surveyors; client
managers; representatives of the accrediting body; the chief
executive or other representatives of the institution itself. In
cases of non-agreement or no accreditation, the decision may
be referred to a higher body. The relationship of the surveyors
to the board is a distinction between many accreditation
programmes, as some keep the survey and final evaluation
entirely separate, whereas others link the two activities closely.
The length of the evaluation process may indicate the
quality of the accreditor’s performance. Most accrediting
bodies designate a maximum period of time between survey
and accreditation decision.
ISO guidelines state that the decision should not be made
by any person participating in the audit, but that it should
be taken by the personnel within the certification body
responsible for auditing the facility, rather than delegated to
any outside person or body. There is therefore no independent
committee involved in the decision. ISO does not stipulate
whether an individual or group should take the final decision,
nor does it give guidelines on the composition of the decision
making team. Exactly who within the certification body is
responsible for the decision varies from organization to
organization. Some have appointed assessment committees
elected from the governing board of the organization, others
allocate the responsibility for all decisions to a chief auditor
(not involved in the audit in question) employed by the
certificating body.
Rules for making decisions
In three of the four models (accreditation, ISO and EFQM),
the report with graded compliance is converted into a final
decision, whether that be accreditation, certification or an
award. This process can be divided into two distinct activities:
the first is to aggregate the many scorings into one single
score or a limited list of scores, based on the aggregation
rules; the second activity follows this and involves a decision
on whether this score merits the award or not, depending
on the decision rules. These rules may be implicit in the
minds of the members of the evaluation committee or explicit,
documented in a series of rules or algorithms forming part
of the policy of the evaluating body.
In the accreditation field the aggregation and decision rules
may be published as part of the standards manual, surveyor
handbook or award recognition guidelines, and used when
reviewing reports. These may include additional elements
such as key safety/business risk criterion. The Hospital
Accreditation Programme Board uses a numerical scoring
mechanism to assess the level of accreditation through ag-
gregated scoring by the surveyors. This is monitored closely
against the surveyors’ recommendation for accreditation.

The rules for deciding whether to award ISO certification
are developed individually by each certification body and
audited by an external organization such as United Kingdom
Accreditation Service (UKAS). Most base their decision cri-
teria on the level of compliance with the standards and on
the severity of non-compliant aspects of the quality system.
HKZ in The Netherlands states that the audittee must comply
with all norms within the certification scheme in order to
gain certification. The auditors’ recommendations may be
graded and the decision made depends on the grades given.
The certificate
A crucial aspect of accreditation and ISO that differentiates
them from other external evaluations is the certificate. This
certificate is a public announcement by the evaluating body
that the institution has been reviewed against a set of standards
and is significantly compliant with them.
The accreditation certificate has two main features, the
time span that it covers (usually up to 3 years) and the degree
of compliance. The 3-year accreditation award is common
among accreditors, although some use a range of time periods
between 1 and 5 years.
The levels of accreditation awarded by the Hospital Ac-
creditation Programme (UK) are as follows:
• accreditation for 3 years;
• accreditation for 2 years;
• accreditation for 1 year;
• focus survey (pending award);
• non-accreditation.
When the ISO certifier reaches a positive decision, the
audittee is provided with a certificate for 3 years, stating the
name, address, dates, terms and ISO standards against which
the health care facility’s quality system have been assessed.
Detail on the scope of certification is included and the status
of the certification body, as well as the expiry date of the
certificate. If the facility has undertaken the audit process
through an accredited certification body, their certificate
should recognize this. In the UK, accredited certification is
recognized by the tick and crown symbol, in the Netherlands
by the HKZ and RvA logos.
For most scientific associations the visitatie process has a
5-year cycle. After this time, the facility is reviewed by
another team of visitors to establish the degree to which
recommendations have been followed. The return visit must
demonstrate compliance with previous suggestions for im-
provement. Some scientific societies have introduced return
visits after 1 or 2 years, particularly in cases of widespread
non-compliance with the quality norms.
Appeal
In a case where the accreditation committee recommends
non-accreditation, the hospital may choose to follow an
appeals procedure, in order to have its case reviewed again.
The appeal is usually dealt with by the board or committee,
who ask the hospital to submit data or information in favour
External evaluation of health care
of accreditation. The board examines the survey report, the
rules applied by the accreditation committee and the evidence
submitted by the hospital in order to reach the final decision.
Publication of results
Most accreditors publicize a list of accredited institutions
(not including the grade of award or the names of those
institutions that have failed to achieve accreditation). In
response to public demand, the JCAHO has a policy of
publicising, under request, the main findings of the ac-
creditation survey by comparing the institution with a range
of similar ones.
The results of visitatie are strictly confidential. The report
is sent only to the physicians visited, who decide whether to
discuss the survey results with their employers, the hospital
management and other parties involved in the visitation
report.
Between surveys
Most accrediting bodies survey each participating institution
every 3 years. To avoid institutions dropping their standard
of performance between surveys, accreditors have devised
several mechanisms. An agreed quality action plan on how
the institution will implement recommendations may be
followed, involving an interim visit to review progress. Un-
announced surveys may be made, to a sample of institutions
in the years in between surveys, and facilities may be required
to submit standard performance data. The Hospital Ac-
creditation Programme requires facilities to indicate the action
taken on the previous recommendations in advance of further
survey visits.
Most organizations using the ISO model conduct annual
or biannual audits on previously agreed aspects of the quality
system. If the audittee has not achieved certification, a number
of types of re-audit may be offered. Facilities may receive a
follow up visit 3 months after the initial audit, to ensure that
the one or two serious problems identified have been rectified.
There may also be a partial re-assessment 6 months after the
original audit, or for a number of serious non-compliances,
another full audit.
Conclusions
Though the terminology of the four main models of external
review of health services may differ (for example evaluators
have different names according to the model: accreditation-
based models use the title surveyors; ISO calls its reviewers
auditors and lead assessors; EFQM uses the term assessors;
and visitatie the terms visitors) the evaluation processes of
the four models share a similar evaluation methodology.
Common features include:
• voluntary initiation by the institution;
• self-assessment;
• agenda or audit plan;
237
L. Bohigas and C. Heaton
• evaluation visit;
• trained reviewer or evaluation team;
• written or verbal report;
• evaluation of findings.
This suggests that these elements are essential to a struc-
tured approach to the external evaluation of health care
facilities. It does not necessarily follow that the differences
between the programmes are superfluous, rather, the areas
where the approaches diverge result from the different em-
phases or purposes of the four models.
One of the common purposes of accreditation and ISO
is to provide evidence that the quality of health service
provision reaches a specified standard. This evidence is in
the form of a certificate awarded to the evaluated institution
on successful completion of the evaluation process. The
corresponding purpose of EFQM is to select and give an
award to institutions achieving excellence in their business
or service category. Only the winners receive an award
confirming high quality health care provision. Visitatie offers
no corresponding award or certificate in its medical peer
review process.
Other distinctions may be drawn between the different
focal points of each model. Accreditation looks at the entire
organization, either department by department, or with a
patient/client focus following the functions surrounding the
patient across all relevant departments. ISO 9000 examines
designated quality systems, focusing on how the institution’s
stated objectives are achieved, rather than how the institution
as a whole meets the needs of its patients. Visitatie directs
its attention to the appropriateness of service delivery pro-
vided by the medical practitioner.
These different purposes make some phases of the method-
ology more necessary than others. Where accreditation uses
focused surveys, workshops and training events between
surveys to facilitate continual improvement in the provision
238
care, EFQM has no need of activity between visits as its
purpose is not continuous evaluation of an individual in-
stitution. The emphasis on self-assessment is important in
determining whether an institution may be eligible to compete
for an award. EFQM may promote improvement, but through
competition rather than collaborative organizational de-
velopment. In contrast, visitatie has no need of an appeal
process, as its main purpose is to offer advice and support
for personal and organizational development between peers.
This paper covers some of the common criteria and
distinctive features of the four models’ evaluation processes.
In order to reach a generic set of core criteria for objective,
consistent, external evaluation programmes across Europe,
these elements should be discussed both in association with
the other components of evaluation programmes and within
the context of each of the four models.
References
1. Shaw CD. External quality mechanisms for health care: summary
of the ExPeRT project on visitatie, accreditation, EFQM and
ISO assessment in European Union countries. Int J Qual Health
Care 2000; 12: 169–175.
2. Bohigas L, Smith D, Brooks T et al. Accreditation programs
for hospitals: funding and operation. Int J Qual Healthcare 1996;
8: 583–589.
3. Bohigas L, Brooks T, Donahue T et al. A comparative analysis
of surveyors from six hospital accreditation programs and a
consideration of the related management issues. Int J Qual
Healthcare 1998; 10: 7–13.
4. Bohigas L. Accreditation across borders: the introduction of
Joint Commission accreditation in Spain. J Qual Improv 1998;
24: 226–231.
Accepted for publication 3 February 2000