STANDING COMMITTEE FOR QUALITY AND COMPETENCE (QCC)

GUIDANCE ON THE CONDUCT OF PROFICIENCY TESTS AND COLLABORATIVE EXERCISES WITHIN ENFSI
DOCUMENT TYPE : GUIDANCE REF. CODE: QCC-PT-001 ISSUE NO: 003 ISSUE DATE: 18-02-2005

1 2 3 4 5 6 7 8 9 10 11 12 13

Introduction .................................................................................................................. 2 International standards ............................................................................................... 2 Scope .............................................................................................................................. 3 Definitions ..................................................................................................................... 3 Types of proficiency tests and collaborative exercises .............................................. 4 Design of proficiency tests or collaborative exercises................................................ 4 Preparation of materials .............................................................................................. 6 Distribution ................................................................................................................... 6 Return of results ........................................................................................................... 7 Assessment of performance ......................................................................................... 7 Reporting of results ...................................................................................................... 8 Review of report ........................................................................................................... 8 Implementation of improvements ............................................................................... 9

Ref code: QC-PT-001

Issue No. 002

Page: 1/9

1
1.1

Introduction
The ENFSI Memorandum of Understanding encourages co-operation between its members, their laboratories and other international organisations in the development of new scientific methods and procedures, standards of practice, training and quality assurance. The ENFSI Board wishes to promote consistent and reliable scientific evidence through the whole forensic process from scene of incident to court. As one part of this aim it is the policy of the Board that all member laboratories should have achieved or should be taking steps towards EN ISO/IEC 17025 compliant accreditation for their laboratory testing activities. In determining this policy the Board accepts that progress will be slower in some countries than others for a number of reasons including differences in national accreditation systems and differences in the operation of legal systems. Where EN ISO/IEC 17025 compliant accreditation cannot be achieved, the Board encourages the use of other Quality Management standards with broadly equivalent objectives. The vision of ENFSI is to ensure that the quality of development and delivery of forensic science throughout Europe is at the forefront of the world. One of its key objectives is to identify and collate the appropriate data on competencies, working practices, instrumentation, training, standards of accreditation, performance indicators, etc. in order to establish best practice and promote its use to consolidate the work of ENFSI as both high quality and trans-European. In order to develop best practice, the ENFSI Expert Working Groups are carrying out inter-laboratory tests or exercises through proficiency tests and collaborative exercises. Therefore when there is an appropriate test offered within ENFSI, the laboratory members should participate in.

1.2

1.3

International standards

The following reference documents provide information on the conduct of proficiency tests and collaborative exercises. The present guidance is based on them. 2.1 2.2 EN ISO/IEC 17025 - General requirements for the competence of testing and calibration laboratories - 1999 ISO/IEC guide 43-1 Proficiency test by interlaboratory comparison Part 1 : Development and operation of proficiency testing schemes - 1997 ISO/IEC guide 43-2 - Proficiency test by interlaboratory comparison - Part 2 : Selection and use of proficiency testing schemes by laboratory accreditation bodies 1997 ISO/IEC Guides 43 list the relevant ISO standards and other publications, and provide an appropriate vocabulary and framework for all inter-laboratory comparisons. Compliance with this Guide will improve communication in this area and will also be essential if recognition is to be sought from an accrediting body. 2.4 ISO 5725-2 - Accuracy (trueness and precision) of measurement methods and results Part 2: Basic method for determination of repeatability and reproducibility of a standard measurement method 1994
Issue No. 002 Page: 2/9

2.3

Ref code: QC-PT-001

2.5

ISO/DIS 13528 Statistical methods for use in proficiency testing by interlaboratory comparisons February 2002 ILAC G13 Guidelines for the requirements for the competence of providers of proficiency testing schemes 2000. This provides interpretation and amplification of the requirements in ISO/IEC 43-1 EA-3/04 Use of proficiency testing as a tool for accreditation in testing August 2001 EURACHEM Selection, use and interpretation of proficiency testing (PT) schemes by laboratories 2000

2.6

2.7 2.8

Scope

This document provides guidelines for the conduct of proficiency tests and collaborative exercises by the Expert Working Groups and identifies a role : (a) for the Expert Working Group in applying this guidance document (b) for the Quality and Competence Committee in providing advice and support (c) for the Expert Working Group Committee in providing monitoring.

Definitions

For the purpose of this guidance the following definitions apply in addition to those described in ISO/IEC Guide 43-1 : 1997. 4.1 Inter-laboratory test or exercise Inter-laboratory test or exercise can be carried out for a number of purposes : (a) to determine the performance of individual laboratories for specific tests or measurements and to monitor laboratories continuing performance ; (b) to identify problems in laboratories and initiate remedial actions which may be related to, for example, individual staff performance or calibration of instrumentation; (c) to determine the performance characteristics of a method and to establish the effectiveness and comparability of new tests or measurement methods ; (d) to assign values to reference materials and assess their suitability for use in specific test or measurement procedures. 4.2 Proficiency testing Proficiency testing is the use of inter-laboratory tests or exercises for purpose (a) and requires that there is a known expected result. 4.3 Collaborative exercises Collaborative exercises do not require known expected outcomes and are the most appropriate approach for trouble-shooting, validation work or the characterisation of reference materials, although information for these purposes can also be generated incidentally from proficiency tests.

Ref code: QC-PT-001

Issue No. 002

Page: 3/9

5
5.1 (a) (b) (c) (d) 5.2

Types of proficiency tests and collaborative exercises

Proficiency tests and collaborative exercises can involve various types of activity: Qualitative identification/comparison; Quantitative measurement; Data transformation; Interpretation. They can be carried out using material supplied to all individual participants for concurrent examination or with the material being provided for sequential examination by the participants on a round-robin basis. Sequential examinations can result in very drawn out exercises, problems with the stability or integrity of the material involved and delays in the overall assessment and reporting. Therefore, they should only be used when there is no other choice. Proficiency tests and collaborative exercises can be conducted overtly (declared tests) or covertly (blind tests). It is unlikely that covert tests or exercises will be practicable within ENFSI for comparisons between laboratories, although individual laboratories may find benefit for themselves in introducing some overt tests or exercises covertly as normal case work.

5.3

Design of proficiency tests or collaborative exercises

The aim of all inter-laboratory tests or exercises is to establish, monitor and/or improve performance in case work. Proficiency tests and collaborative exercises should therefore be realistically designed to reflect the case work position as closely as possible. Where appropriate, they should take proper account of the different legal frameworks in which participants may be working and the different requirements for their services. 6.1 Inter-laboratory tests or exercises can also be time consuming and expensive, and they can be overly disruptive to case work if they are too demanding or too frequent. It is therefore essential that they are designed so that: the tests or exercises require no more than 1-2 man days of effort from individual participants ; they reflect the work carried out routinely by laboratories in that area of work ; they are focused on the priority issues (agreement within the Expert Working Group) ; they have a clear stated purpose ; they are properly designed (according to these guidelines) to achieve that purpose with minimum effort; and the information derived from them is maximised and used to best effect ; they are neither too simple nor over-ambitious.

Ref code: QC-PT-001

Issue No. 002

Page: 4/9

6.2

The responsibility for running proficiency tests and collaborative exercises remains with individual Expert Working Group. The Proficiency Tests and Collaborative Exercises Project (PTCEP) team is charged by the ENFSI Board with taking an overview of these tests and exercises (list on the QCC webpage) and providing advice and support to other Expert Working Groups on their design and conduct with a view to promoting best practice. The Expert Working Group Committee is charged with providing an annual report to the ENFSI Board on the content and effectiveness (expected outcome(s) reached, improvement actions carried out, etc.) of the programme (see EWGC framework). For each test or exercise a Co-ordinator should be appointed by the relevant Expert Working Group. The Co-ordinator should have, or should collaborate closely with those holding adequate qualifications and experience in the design, implementation and reporting of inter-laboratory comparisons. Such support is available through the Proficiency Tests and Collaborative Exercises Project (PTCEP) team. When the test or exercise involves highly specialised knowledge or when statistical treatment of the results is appropriate, it is also essential to involve relevant experts or statisticians at the design stage. The Co-ordinator should be responsible for : establishing the purpose of the test or exercise ; establishing the criteria for the participants and whether prospective participants meet the criteria ; setting the timing and return date for the test or exercise ; indicating who will assess the returns and how this will be done ; and providing the protocol for communication of the results and assessment. The criteria for participation are particularly relevant given the diversity in expertise, experience and equipment available to ENFSI laboratories. The Co-ordinator should also be responsible for : planning, documentation and communication ; designing the test or exercise ; establishing, when appropriate, the expected or desired outcome ; obtaining and providing the test materials; producing clear, unambiguous instructions for participants on what they are required to do and how they are to report their results and be aware of the difficulties arising from participants who may not be highly proficient in the language ; packaging and transportation arrangements ; ensuring compliance with any legal, health and safety requirements in the design and distribution of the test or exercise ; establishing (if necessary) a team of qualified assistants to help monitor the progress of a particular test or exercise. provide the necessary information (EWG organizer, PT or CE code, examination/analysis type, materials tested) to the PTCEP project leader in order to update the overview on the QCC webpage. In order to facilitate communication, all instructions and reports should be in English.

6.3

6.4

6.5

6.6

Ref code: QC-PT-001

Issue No. 002

Page: 5/9

7
7.1

Preparation of materials
Every effort should be made to ensure that all test materials provided are uniform and, when appropriate, homogeneous, and that they will not deteriorate. They should be prepared by a person or institution with the appropriate competence so as to guarantee their integrity. The same standards should apply to the preparation of test materials as would apply to case work. The test materials should be properly characterised before being issued. The characterisation should be sufficient for the purposes of the proficiency test or collaborative exercise, using a reference laboratory acceptable to all the participants and any available primary or reference standards. When it is necessary to carry out a specific calibration to ensure the compatibility of the test or exercise results, such standards should also be available to all participants. Consideration should be given to the preparation of spare test materials, to be made available to participants in the event of their original test materials being lost or compromised in any way, and to assist in resolving any issues which may arise concerning the integrity of the test materials. Spare test materials, when not needed for other purposes, could also provide useful material for training purposes. Details of the test materials and their preparation and characterisation should be fully documented. Where there are any possible health or safety considerations associated with the test materials or their examination, it is the responsibility of the Co-ordinator to identify these and to bring them to the attention of participants.

7.2

7.3

7.4

7.5

8
8.1

Distribution
Test materials should be packaged in such a way as to ensure their integrity, stability and security whilst in transit. Any specific requirements for their handling or storage should be made explicit, particularly if this could affect the health or safety of anyone involved. The distribution of some materials, such as drugs, firearms and explosives substances, is controlled by legislation. If there is a need for import and export licences to be obtained then this must be complied with. Attention will need to be given to any time limits associated with such licenses. The Co-ordinator should ensure that all the necessary requirements are in place. Details for the packaging and distribution should be fully documented. Distribute the test materials to the Laboratory Working Group member, quality assurance manager and/or Laboratory director. Ensure that is labelled as an official rather than a personal communication so that it will receive attention in the event of an individual being absent from the Laboratory.

8.2

8.3 8.4

Ref code: QC-PT-001

Issue No. 002

Page: 6/9

9
9.1

Return of results
It is particularly important to identify how the results should be returned before the test or exercise is prepared. The use of standard forms greatly facilitates the analysis of the returns but can constrain valuable comment. When assessment of the strength of evidence is required, there is advantage in providing a recommended verbal scale. Ideally, the same verbal scale should be used for all areas of case work and for all tests and exercises. Where measurement units are involved, there may be different national practices, so it is helpful to indicate the units that should be used. The Co-ordinator should specify the latest date for return of results. The policy for dealing with late returns can be left to individual Expert Working Groups, but consideration should be given to discouraging them. Consideration should be given by the Co-ordinator to the need for the test materials themselves or working documents/materials to be returned with the results. All returns should be retained by the Co-ordinator until the test or exercise has been completed.

9.2

9.3

9.4

9.5

9.6

10

Assessment of performance

10.1 If the purpose of the test or exercise and its expected outcome are clear, then assessment of performance should be straightforward. The method of assessment should have been agreed upon at the trial design stage. 10.2 Assessments are in the responsibility of the relevant Expert Working Group. They can be carried out by individuals or panels, but consideration should be given to the value of including in the assessment process other recognised experts in the particular field, or statisticians, if such expertise is not available within the Expert Working Group. 10.3 Many proficiency tests and collaborative exercises yield useful information which is incidental to the main aim. This should be collected and disseminated, but in such a way that it does not affect the primary issues.

Ref code: QC-PT-001

Issue No. 002

Page: 7/9

11

Reporting of results

11.1 The protocol for reporting the results should be established at the design stage. The report should be checked for accuracy before it is released and should be provided according to a deadline previously determined. It is best if this is sufficiently in advance of any meeting of the Expert Working Group to allow them to digest the results and consider what actions might be recommended. 11.2 An important consideration in reporting results is whether they should be anonymous and the participants referred to by codes. Such an approach is more common in proficiency testing, or when there are commercial or other sensitivities. For benchmarking purposes there is obviously advantage in individual participants being able to be identified, either openly or via the Co-ordinator. 11.3 The report should provide digests and analyses of the results returned and should also address performance against the declared aims and any expected or desired outcomes. It should be objective and report the facts (e.g. when possible, the information concerning the various techniques or methods used), although it is also appropriate to include any issues of concern and to identify matters requiring further consideration by the Expert Working Group. 11.4 Together with any significant issues identified during the review of the report, a copy should be provided to the Expert Working Group Committee in charge of the monitoring of the proficiency tests and collaborative exercises carried out by the Expert Working Groups (see EWGC framework). 11.5 The full report contains sensitive information, so for the PT and CE conducted within the ENFSI list available on the QCC web page, ENFSI members can request information just through the EWG webpage link. In this way, the responsibility to deliver information (report, etc.) will be given to the Expert Working Groups. However, in all cases anonymity must be guaranteed.

12

Review of report

12.1 All proficiency tests and collaborative exercises should be reviewed at the earliest opportunity at a meeting of the relevant Expert Working Group. This review can be preceded by an exchange of information and views in writing, but this should not replace group discussion. 12.2 The Expert Working Group should consider: how far the aims of the test or exercise have been met; recommended improvement actions; the timetable for improvement actions to be implemented; the provision of support to effect the improvements; learning points for the future design of similar tests or exercises; the timing of any similar tests or exercises after the improvement actions arising from the current exercise or test have been implemented.

Ref code: QC-PT-001

Issue No. 002

Page: 8/9

13

Implementation of improvements

13.1 ENFSI exists to identify and collate data in order to establish best practice and promote its use through co-operation between its members, their laboratories and other international organisations. This can only be achieved if there is a commitment from members to make improvements when the need is identified and to provide whatever support they can to assist others in this. Both should be a pre-requisite for all participants in inter-laboratory proficiency tests and collaborative exercises. 13.2 The annual report from the Expert Working Group Committee should be written to preserve the anonymity of all participants.

Ref code: QC-PT-001

Issue No. 002

Page: 9/9