CAEAL Policy on the Estimation of Uncertainty of Measurement in Environmental Testing CAEAL Uncertainty Policy

1. Laboratories accredited under the joint SCC-CAEAL Accreditation Program for Environmental Laboratories shall fulfil the requirements of CAN-P-4D (ISO/IEC 17025) with respect to the estimation of uncertainty of measurement associated with environmental testing for those tests which produce numerical results. This applies whether the test methods are rational or empirical.. 2. They shall report the expanded uncertainty estimate as part of the reported result when the reporting of the estimate of measurement uncertainty is Required by the client, or Required to establish that the data is 'fit-for-purpose', or Required because the data is being used to establish compliance (of the body being represented by the analysed sample) with a requirement. 3. The requirement which underlies this policy is that given in CAN-P-4D, Clause 5.4.6. Other documents and Guides may be used by laboratories to develop methods in meeting this requirement.

Implementing the CAEAL Uncertainty Policy

4. All laboratories affected by this policy shall submit to CAEAL, on or before 31 December 2002, documentary evidence of their commitment to this policy within their quality system and a plan for its implementation within the laboratory. 5. Beginning with the 2003 assessment year, laboratories shall demonstrate their implemented use of adequate procedures for their estimation of the uncertainty of measurement associated with their accredited tests and shall have begun reporting the estimates (as expanded standard uncertainties) in accordance with the requirements of CAN-P-4D Clause 5.10.3.1 c).

Requirement from CAN-P-4D 6. The following excerpt from CAN-P-4D articulates the requirement to estimate the uncertainty of measurement associated with testing. 5.4.6 Estimation of uncertainty of measurement

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 1 of 39

5.4.6.1 A calibration laboratory, or a testing laboratory performing its own calibrations, shall have and shall apply a procedure to estimate the uncertainty of measurement for all calibrations and types of calibrations. 5.4.6.2 Testing laboratories shall have and shall apply procedures for estimating uncertainty of measurement. In certain cases the nature of the test method may preclude rigorous, metrologically and statistically valid, calculation of uncertainty of measurement. In these cases the laboratory shall at least attempt to identify all the components of uncertainty and make a reasonable estimation, and shall ensure that the form of reporting of the result does not give a wrong impression of the uncertainty. Reasonable estimation shall be based on knowledge of the performance of the method and on the measurement scope and shall make use of, for example, previous experience and validation data. NOTE 1 The degree of rigor needed in an estimation of uncertainty of measurement depends on factors such as: - the requirements of the test method; - the requirements of the client; - the existence of narrow limits on which decisions on conformance to a specification are based. NOTE 2 In those cases where a well-recognized test method specifies limits to the values of the major sources of uncertainty of measurement and specifies the form of presentation of calculated results, the laboratory is considered to have satisfied this clause by following the test method and reporting instructions (see 5.10). 5.4.6.3 When estimating the uncertainty of measurement, all uncertainty components which are of importance in the given situation shall be taken into account using appropriate methods of analysis. NOTE 1 Sources contributing to the uncertainty include, but are not necessarily limited to, the reference standards and reference materials used, methods and equipment used, environmental conditions, properties and condition of the item being tested or calibrated, and the operator. NOTE 2 The predicted long-term behaviour of the tested and/or calibrated item is not normally taken into account when estimating the measurement uncertainty. NOTE 3 For further information, see ISO 5725 and the Guide to the Expression of Uncertainty in Measurement (see bibliography) [1]. Guidance and Interpretations adopted by PALCAN for use by laboratories 7. SCC Document D92.5 contains all of the interpretive notes and guidance used throughout the PALCAN program and is also for those laboratories accredited under the joint SCC-CAEAL Environmental Laboratory Accreditation Program. This document is available from the SCC. Relevant citations are from Clause 5.4.6 of D92.5 [2].

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 2 of 39

ILAC guidance to clause 5.4.6 (G.5.4.6.) G.5.4.6 Guidance on this aspect is to be found in the ILAC document Expression of Uncertainty of Measurement in Calibration, and in the EURACHEM document Quantifying the Uncertainty in Analytical Measurement. SCC Interpretive Note: Measurement uncertainty as defined by the GUM, is the only pertinent product of calibration activities. It is important to address both calibration done in-house by the laboratory itself and conducted by outside suppliers. Testing laboratories that are providing their own calibrations will have to provide measurement uncertainties for these calibrations. Testing laboratories are to ensure that they also receive appropriate uncertainties of measurement from outside sources of calibration. ILAC guidance to clause 5.4.6.2 (G.5.4.6.2) G.5.4.6.2 The complexity involved in estimation of uncertainty of measurement in the case of testing varies considerably from one test field to another and also within one field itself. It is also often achieved by a less metrologically rigorous process than that which can be followed for calibration. Clause 5.4.6.2 of ISO/IEC 17025 allows for these factors and accreditation bodies should take them into account during assessments. (ILAC Laboratory Liaison Committee is developing a strategy for implementation of measurement uncertainty in testing). SCC Interpretive Note: this requirement cannot be applied unequivocally at the present time because of the diversity of the fields of testing that are covered by the SCC accreditation process. Also, there is a great deal of work that still needs to be conducted on the international scene before this requirement can be applied in a consistent manner for specific fields and specific product/services classifications. SCC must remain in line with the interpretations that will be published by ILAC. SCC can have input into the development of the ILAC document and will wait for more guidance from ILAC before applying this requirement rigorously. Assessors are reminded to proceed as they have done in the past and to continue to require this where it is common practice. For the other areas where it is not common practice to express uncertainty of a test result, assessors are requested to remind the laboratory of this requirement and to ask them to consider what they feel would be feasible in their operation and that when guidelines are available, they will be distributed to the laboratories and considered for future assessments. SCC Interpretive Note: The Horwitz Trumpet can be used for an initial estimation of uncertainty in chemical and microbiology/biology testing. (APLAC Common Assessor Training Course 2000-04-10 to 14)

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 3 of 39

Estimating the Uncertainty of Measurement in Environmental Laboratories 8. Several organisations and groups have published advice on the estimation of measurement uncertainty to the analytical laboratory community. These include APLAC and the AOAC, ILAC, NMKL, EURACHEM/CITAC, the SCC TG Labs Mineral Analysis Working Group and the Ontario Ministry of the Environment. CAEAL has adopted a single approach which is deemed to have a minimum impact on the costs incurred by laboratories to meet the requirements of CAN-P-4D. This policy considers the expected costs to laboratories, is based on consensus from within the Canadian environmental laboratory community and provides a generic approach from among the organisations listed above. 9. There are two approaches that can be taken in estimating the uncertainty of measurement associated with testing. The first method, termed Type A, estimates uncertainties through the use of experimental data such as that from routine laboratory QA/QC work (duplicates, reference material usage, method validation studies, and proficiency testing (PT) and other interlaboratory programs, for example). 10. The second method, Type B, involves the cause and effect-based metrological estimation of specific uncertainties from each identified source of uncertainty. It is similar to the approach used by calibration laboratories. 11. The first approach (Type A) is the one used by most specifier agencies when requiring estimations of the uncertainty of measurement for analytical laboratories. It is also the basis of the approach used in this Policy.

Why is the CAEAL Uncertainty Policy based on Type A?

The EURACHEM CITAC Guide Quantifying Uncertainty in Analytical Measurements states Where a factor has been representatively varied during the course of a precision experiment that factor needs no additional study. Virtually all of the data required is already present in laboratory files. Little time is required to estimate the uncertainty for individual methods using the laboratory historical data. Little training is required to enable laboratory staff to do the necessary calculations. The resulting estimate is robust and defendable to clients and specifiers. The resulting estimate is relatively easy to assess during assessments.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 4 of 39

CAEAL Uncertainty Protocol

12. Measurement uncertainty shall be expressed as a combined Standard Deviation (SD) with the same units as those of the measurand. 13. The final result shall be reported with an expanded uncertainty that 'produces an interval (the expanded uncertainty) about the measurement result that may be expected to encompass a large, specified fraction (e.g. 95%) of the distribution of values that could reasonably be attributed to the measurand.' Guide to the Expression of Uncertainty in Measurement, ISO, 1 ed. 1993, ISBN 92-67-10188-9. 14. The expanded standard uncertainty shall be calculated to give a confidence level of 95% using an expansion factor 'k' of: k = 2 when n is 30 or more (n = number of observations from which the SD is calculated). k = the appropriate (95% confidence level) Student distribution 't' (two tailed) factor for n<30.

15. In cases where test methods generate multi-analyte data of 10 species or more (as from ICP/OES, GC/MS or ICP/MS methods), laboratories shall select 3 analytes that represent each of three levels of uncertainty of the results small, medium and large levels of uncertainty - for which to estimate the measurement uncertainty. In cases where the analyte is expected to occur over a wide concentration range (more than a factor of 10), the estimation of uncertainty should be done at low, medium and high concentrations within that range. This will reflect the increase in the uncertainty with concentration. The other analytes will be classified within these three categories. NOTE: The objective of this approach is to minimise the amount of work required in the estimation of uncertainty for multi-component tests. It is intended that the uncertainty for all constituents in a category would be assumed to have the uncertainty of the surrogate constituents, expressed as the pooled SD of the 3 measured SDs at each concentration range. 16. Laboratories will be required to re-estimate measurement uncertainty only when changes to their operations are made that may affect sources of uncertainty and these sources have not been shown to be unaffected through method validation or other studies. If validation of the method has shown that the uncertainty of a test is not significantly affected by a change of analysts for example, then a change of analysts will not require a new

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 5 of 39

estimate to be made. Proof that such a change has not affected the measurement process can be found in control charts monitoring results from spikes or reference materials. If, according to the definition of the laboratory quality program, there is no change in the spread of data about the mean (no loss of precision), or no evidence of the introduction of a bias, when such a change has been implemented (a different analyst in this example) then that factor does not have to be included as a possible source of uncertainty. Annotated control charts created when such changes occur are considered sufficient documentary evidence of such a conclusion. 17. Laboratories shall make independent estimates of measurement uncertainty for tests performed on samples with significantly different matrices (organic constituents in ground waters versus waste waters for example) only when such differences have been shown to have an effect on the estimate. 18. Laboratories shall make independent estimates of measurement uncertainty for analyte concentrations that vary over orders of magnitude (at low, medium and high concentration levels for example). If the relationship between SD and concentration is shown to be linear, the laboratory can estimate an expanded RSD (see Reporting the Result in the Appendix). 19. When laboratories select an analytical portion from a sample that may not be homogeneous, the laboratory shall include sub-sample uncertainty as part of the combined standard uncertainty calculation (see Sample duplicate insertion, under Laboratory Repeat Data Sets in the Appendix) when that uncertainty is sufficiently large (see 'Tabulate Uncertainty Estimates' in the Appendix). 20. As stipulated in Note 2 to Clause 5.4.6.2 of CAN-P-4D (ISO/IEC 17025): 'In those cases where a well-recognized test method specifies limits to the values of the major sources of uncertainty of measurement and specifies the form of presentation of calculated results, the laboratory is considered to have satisfied this clause by following the reporting instructions.' Using the Type A Approach 21. The following steps involve the use of experimental data to estimate the uncertainty of measurement for environmental laboratories. (See the Appendix for more detail). Using the method SOP and the final result-calculation equation, identify and list all potential sources of uncertainty. Identify and compile recent laboratory repeat analysis and PT data that is available. Match each repeat data set with those sources of uncertainty that are likely to have varied during the collection of the repeat data and identify double counted sources of uncertainty. Estimate the magnitude of any source of uncertainty that is not varied during the collection of any of the repeat data sets.
Rev 1.4 Page 6 of 39

P19 - CAEAL Policy on Uncertainty

Tabulate each source of uncertainty and its associated SD, and/or relative SD (RSD) derived from the repeat data set(s) matched to it, or from the estimate made. Eliminate double counted sources. Using only those SDs that are 1/3 or more the size of the largest individual SD, calculate the combined standard uncertainty using standard propagation of error rules (the square root of the sums of squares of SDs known as the root sum of squares - RSS). Apply the appropriate coverage factor 'k'. (see paragraph 14 above) Report the result with the expanded uncertainty and with a description of how the uncertainty was calculated.

Reporting the Uncertainty Associated with Environmental Measurement 22. CAN-P-4D, Clause 5.10.3.1 c) details the requirement to report uncertainty of measurement associated with testing and includes the following circumstances [1]: i. When it is relevant to the validity or application of the test result, ii. When so instructed by a client, or iii. When the uncertainty affects compliance to a specification limit. 23. Estimates of measurement uncertainty quoted in reports shall reflect conservative worst case scenarios of variability incorporating long term effects on significant sources of uncertainty such as different analysts, instrument drift and other factors that reflect routine laboratory operations. A short description of how the estimate of uncertainty was determined should also be included. This description would include information on the source(s) of the data used to estimate the SDs included in the calculation of the combined standard uncertainty.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 7 of 39

APPENDIX 1 Definitions of Terms Used in this Policy (reprinted from A2LA Guide[8]) And References accuracy (of measurement) (VIM 3.5): closeness of the agreement between the result of a measurement and a true value of the measureand NOTES: Accuracy is a qualitative concept The term precision should not be used for accuracy. an accepted reference value may be used in place of a true value in this definition.} bias (ISO 3534-1): the difference between the expectation of the test results from a particular laboratory and an accepted reference value NOTE: Bias is the total systematic error as contrasted to random error. There may be one or more systematic error components contributing to the bias. A larger systematic difference from the accepted reference value is reflected by a larger bias value. combined standard uncertainty (GUM 2.3.4): standard uncertainty of the result of a measurement when that result is obtained from the values of a number of other quantities, equal to the positive square root of a sum of terms, the terms being the variances or covariances of these other quantities weighted according to how the measurement result varies with changes in these quantities correlation (ISO 3534-1): the relationship between two or several random variables within a distribution of two or more random variables NOTE: Most statistical measures of correlation measure only the degree of linear relationship. coverage factor (GUM 2.3.6): numerical factor used as a multiplier of the combined standard uncertainty in order to obtain an expanded uncertainty NOTE: A coverage factor, k, is typically in the range of 2 to 3. error (of measurement) (VIM 3.10): result of a measurement minus a true value of the measureand NOTES: Since a true value cannot be determined, in practice a conventional true value is used. When it is necessary to distinguish error from relative error, the former is sometimes called absolute error of measurement. This should not be confused with absolute value of error, which is the modulus of the error. an accepted reference value may be used in place of a true value in this definition.}

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 8 of 39

expanded uncertainty (GUM 2.3.5): quantity defining an interval about the result of a measurement that may be expected to encompass a large fraction of the distribution of values that could reasonably be attributed to the measureand. NOTES: The fraction may be viewed as the coverage probability or level of confidence of the interval. To associate a specific level of confidence with the interval defined by the expanded uncertainty requires explicit or implicit assumptions regarding the probability distribution characterized by the measurement result and its combined standard uncertainty. The level of confidence that may be attributed to this interval can be known only to the extent to which such assumptions may be justified. influence quantity (VIM 2.7): quantity that is not the measureand but that affects the result of the measurement EXAMPLES: temperature of a micrometer used to measure length; frequency in the measurement of the amplitude of an alternating electric potential difference; bilirubin concentration in the measurement of hemoglobin concentration in a sample of human blood plasma. level of confidence (GUM C.2.29): The value of the probability associated with a confidence interval or a statistical coverage interval NOTE: The value is often expressed as a percentage. measureand (VIM 2.6): particular quantity subject to measurement EXAMPLE: Vapor pressure of a given sample of water at 20C. NOTE: The specification of a measureand may require statements about quantities such as time, temperature, and pressure. measurement (VIM 2.1): set of operations having the object of determining a value of a quantity precision (ISO3534-1): the closeness of agreement between independent test results obtained under stipulated conditions NOTES: Precision depends only on the distribution of random errors and does not relate to the true value or the specified value. The measure of precision is usually expressed in terms of imprecision and computed as a standard deviation of the test results. Less precision is reflected by a larger standard deviation. Independent test results means results obtained in a manner not influenced by any previous result on the same or similar test object. Quantitative measures of precision depend critically on the stipulated conditions. Repeatability and reproducibility conditions are particular sets of extreme conditions.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 9 of 39

repeatability (VIM 3.6): closeness of the agreement between the results of successive measurements of the same measureand carried out under the same conditions of measurement NOTES: The conditions are called repeatability conditions. Repeatability conditions include: the same measurement procedure; the same observer; the same measuring instrument used under the same conditions; the same location; repetition over a short period of time. Repeatability may be expressed quantitatively in terms of the dispersion characteristics of the results. reproducibility (VIM 3.7): closeness of the agreement between the results of measurements of the same measureand carried out under changed conditions of measurement NOTES: A valid statement of reproducibility requires specification of the conditions changed. The changed conditions may include but are not limited to: principle of measurement; method of measurement; observer; measuring instrument; reference standard; location; conditions of use; time. Reproducibility may be expressed quantitatively in terms of the dispersion characteristics of the results. Results are here usually understood to be corrected results. standard uncertainty (GUM 2.3.1): uncertainty of the result of a measurement expressed as a standard deviation trueness (ISO 3534-1): the closeness of agreement between the average value obtained from a large series of test results and an accepted reference value NOTES: The measure of trueness is usually expressed in terms of bias. Trueness has been referred to as accuracy of the mean. This usage is not recommended. type A evaluation of uncertainty (GUM 2.3.2): method of evaluation of uncertainty by the statistical analysis of observations type B evaluation of uncertainty (GUM 2.3.3): method of evaluation of uncertainty by means other than the statistical analysis of a series of observations

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 10 of 39

uncertainty of measurement (VIM 3.9): parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measureand NOTES: The parameter may be, for example, a standard deviation (or a given multiple of it), or the half-width of an interval having a stated level of confidence. Uncertainty of measurement comprises, in general, many components. Some of these components may be evaluated from the statistical distribution of the results of series of measurements and can be characterised by experimental standard deviations. The other components, which can also be characterised by standard deviations, are evaluated from assumed probability distributions based on experience or other information. It is understood that the result of the measurement is the best estimate of the value of the measureand, and that all components of uncertainty, including those arising from systematic effects, such as components associated with corrections and reference standards, contribute to the dispersion. This definition is that of the Guide to the expression of uncertainty in measurement in which its rationale is detailed (see in particular 2.2.4 and Annex D to VIM). References 1. CAN-P-4D: General Requirements for the Competence of Testing and Calibration Laboratories (Verbatim Canadian adoption of ISO/IEC 17025 same title). 2000, Standards Council of Canada: Ottawa, ON. 2. SCC: D92.5: PALCAN Interpretations for Conducting Assessments of Testing and Calibration Laboratories. 2000, SCC: Ottawa. 3. Ellison, S.L.R., M. Rosslein, and A. Williams, Editors, Quantifying Uncertainty in Analytical Measurement, 2nd Edition, Eurachem/CITAC, available on internet at http://www.measurementuncertainty.org/mu/quam2.pdf, 2000. 4. http://www.measurementuncertainty.org/mu/quam2.pdf, 2000. 5. ILAC Guide 17: Introducing the Concept of Uncertainty of Measurement in Testing in Association with the Application of the Standard ISO/IEC 17025. 2002, ILAC: Rhodes, NSW, Australia, http://www.ilac.org, 2002. 6. Albert, R. and W. Horwitz, A Heuristic Derivation of the Horwitz Curve. Anal. Chem., 1997. 69(4): p. 789-790. 7. Uncertainties in qualitative testing and analysis. Accreditation and Quality Assurance, 2000. 5( 8.): p. 346-348. 8. APLAC Policy, Interpretation and Guidance on the Estimation of Uncertainty of Measurement in Testing, Asia-Pacific Laboratory Cooperation, (APLAC) 2002. 9. Adams, T.M., A2LA Guide for the Estimation of Measurement Uncertainty In Testing. 2002, American Association for Laboratory Accreditation (A2LA): Frederick, MD. p. 42.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 11 of 39

10. Barwick, V.J. and S.L.R. Ellison, VAM Project 3.2.1. Development and harmonisation of measurement uncertainty principles. 2000, LGC, UK, www.vam.org.uk, http://www.caeal.ca/VAM_uncertainty.pdf, 2000. 11. Estimation and Expression of Measurement Uncertainty in Chemical Analysis. 1997, NMKL. p. 15. 12. McQuaker, N., Quality Control for Environmental Laboratories. Revision 4.5 October 2001, CAEAL: Ottawa, ON. 13. Taylor., J.K., Quality Assurance of Chemical Measurements. 1987, Boca Raton, FL: Lewis Publishers Inc. 14. A2LA Interim Policy on Measurement Uncertainty for Testing Laboratories. 2000, American Association for Laboratory Accreditation (A2LA): Frederick, MD. 15. Excel spreadsheet used on the Course in Measurement of uncertainty in microbiological examination of food. 2002, NMKL, http://www.nmkl.org/Engelsk/publications.htm,2002. 16. Measurement of uncertainty in microbiological examination of foods. 2nd. Ed. 2002, NMKL: Norway, http://www.nmkl.org/Engelsk/reports.htm,2002. 17. NMKL, Measurement of Uncertainty in Microbiological Examination of Foods. 1999, NKML (Nordic Committee on Food Analysis. p. 22, www.nmkl.org,1999. 18. Accreditation in Microbiological Laboratories. 2002, European Cooperation for Accreditation (EA), http://www.europeanaccreditation.org/,2002. 19. Mills, W.J. Uncertainty in Microbiological Analysis of Environmental Samples. in CAEAL Uncertainty Workshop. 2001. Edmonton, AB: CAEAL. 20. Niemela, S.I., A semi-empirical precision control criterion for duplicate microbiology colony counts. Letters in Applied Microbiology. 22(4): p. 315319.1996. 21. Voysey, P.A. and K. Jewell, Uncertainty Associated with Microbiological Measurement. 1999, Campden & Chorleyword Food Research Association. p. 271999. 22. Niemi, R.M. and S.I. Niemela, Measurement Uncertainty in Microbiological Cultivation Methods. Accred. Qual. Assur. 6: p. 372-375.2001. 23. Niemela, S.I., Uncertainty of Quantitative Determinations Derived by Cultivation of Microorganisms. 2002, Centre for Metrology and Accreditation: Helsinki, Finland. p. 75, http://www.mikes.fi/documents/upload/Publication%20J3%202002_1.pdf, 2002. 24. Norli, H.S. NMKL Procedure no 8, 2nd Ed., 2002: Measurement of uncertainty in microbiological examination of foodsProf. Eystein Skjerve. in AOAC Annual Meeting. 2002. Los Angeles, CA: AOAC. 25. Schop, R., Personal Communication, D.W.J. Mills, Editor. 2002: Toronto, ON2002. 26. USEPA, Membrane Filter Method for the Simultaneous Detection of Total Coliforms and Escherichia coli in Drinking Water. 2000, USEPA, Office of Research Environmental Protection and Development Cincinnati OH 45268: Washington, DC. p. 21, http://www.epa.gov/nerlcwww/MI_emmc.pdf,2000.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 12 of 39

27. USEPA, Improved Enumeration Methods for the Recreational Water Quality Indicators: Enterococci and Escherichia coli. 2000, United States Environmental Protection Agency, Office of Science and Technology , Washington DC 20460, http://www.epa.gov/ost/beaches/rvsdman.pdf,2000. 28. McQuaker, N.R., Measurement Uncertainty for Environmental Laboratories. 2000, CAEAL: Ottawa, ON2000. 29. Mills, W.J., Uncertainty Estimate for a Microbiological Dataset. 2002, Unpublished Data 2002. 30. Tholen, D., Telephone Conversation, W.J. Mills, Editor. 2002: Chicago, IL2002.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 13 of 39

APPENDIX 2 Measurement Uncertainty for Analytical Chemistry Aim 1. This appendix considers and expands each of the CAEAL Protocol steps, as they apply to analytical chemistry, in more detail. It also explains what is meant and how to perform each task required. Sources of uncertainty: 2. The possible sources of uncertainty for an analytical method are tabulated in many of the sources listed in paragraph 5 of the Policy. Close examination of the steps in the laboratory method SOP, and of the parameters found in the final concentration calculation, will usually help to identify the likely sources of uncertainty in the method. ILAC Guide 17 lists these as[4]: a. b. c. d. e. f. g. h. i. j. k. definition of the measurand sampling transportation, storage and handling of samples preparation of samples environmental and measurement conditions the personnel carrying out the tests variations in the test procedure the measuring instruments calibration standards or reference materials software and/or, in general, methods associated with the measurement uncertainty arising from correction of the measurement results for systematic effects. source of bias and examples reference

3. Spike or reference material recovery efficiency is an example of a variation in the test procedure category above. Inter-laboratory standard deviations derived from proficiency testing programs are of uncertainty in the correction for systematic effects. Data from material analyses can also be used for this purpose. Laboratory Repeat Data Sets

4. These are sources of repeated measurements from which SDs and RSDs can be calculated. The laboratory can vary one or more of the above sources of uncertainty during the collection of the repeat data and the SD calculated will include uncertainty attributed to the varied source(s). The various repeat data sets include: Proficiency testing programs These are a source of reproducibility SD (SDR) that includes both intra- and inter-laboratory sources of uncertainty. It is larger than the intra-laboratory uncertainty (SDr) known as repeatability - of a laboratory whose methods are in statistical control. In

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 14 of 39

the absence of any other source of repeated data, reproducibility from proficiency testing and other round robin studies can be used as an estimate of measurement uncertainty provided the laboratory can demonstrate that their bias is within certain bounds, consistent with the collaborative study estimate of between-laboratory SD. It is, however, very likely to be an over estimate of what the intra-laboratory uncertainty actually is (by a factor of 2 according to conventional wisdom). Note the possibility of using PT results for bias detection and correction. Reference sample insertion both certified reference samples and inhouse reference samples inserted into routine runs for control charting applications are a source of long term uncertainty data. Sources of uncertainty that can vary during repeated insertion of these samples over time are analysts, calibration sets, calibration solution sources, environmental conditions, instrument drift and many more. As a consequence, the standard deviation calculated from this data will reflect the uncertainty contributions from these sources. Sources of variability that are not included however, are factors that can change from sample to sample such as matrix effects and sample non-homogeneity (or heterogeneity). If a reference sample is to be used to estimate a bias, the uncertainty in the bias estimate must include the uncertainty in the certified value of the reference material. The combined standard uncertainty must include the bias uncertainty if the result is corrected for bias. Spike recovery data Can give the same information as reference sample insertion and in some cases can reflect variability due to different sample matrices. This type of interpretation should be made with caution however since the spike portion may be 100% recovered but the analyte portion may not be (due to speciation differences for example). Method validation replicate data This is a source of data from repeat analyses run to establish precision estimates at different analyte concentration levels. The results from those run at low concentrations for the calculation of detection and quantitation limits can also be used to assess uncertainty at low analyte concentration ranges. The validation data can also serve as a source of information on the uncertainty contributed by other sources (such as analyst, instrument, temperature, time etc.) depending on how the validation work was planned and executed to include such variables. This is especially the case if ruggedness studies were incorporated as an integral part of the validation program to assess the effect of varying parameters likely to be significant sources of uncertainty. A thorough discussion of the use of method validation data in the estimation of uncertainty is VAM Project 3.2.1 Development and Harmonization of Measurement Uncertainty Principles; Part (d): Protocol for uncertainty evaluation from validation data, by V.J. Barwick and S.L.R. Ellison, January 2000, Version 5.1. This can be downloaded as a pdf file from the VAM web site.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 15 of 39

Sample duplicate insertion This can be a valuable source of uncertainty data known as replicability SDdupl - that reflects the variability due to differences between analytical portions (non-homogeneity) and other factors that can vary between replicates (weighing, volumetric manipulations, and short term instrument drift are examples). Note: If the duplicates are measured in the same analytical run, as is usually the case, any uncertainty associated with the instrument set up and calibration is not accounted for. More than 20 duplicate pairs should be run of samples of a similar concentration. The SDdupl = (R2/2N) where R is the difference between duplicate pairs and N is the number of duplicate pairs. This should be calculated for low, medium and high concentration ranges to reflect the concentration dependence of the SD. Alternatively, the RSD can be calculated (at low, medium and high concentration ranges as well) as RSDdupl = {[(ai - bi)/xi]2/2N} where (ai - bi)/xi is the relative difference between duplicates for sample i and N is the number of samples for which duplicates have been run. This value makes allowances for the concentration dependence of the SD for concentrations between those at which the calculation was made (see paragraph 9 below).

Match Repeat Data with Uncertainty Sources 5. The objective of this step is to select laboratory Quality Control and validation data that includes as many sources of variability as possible so that these do not have to be estimated using the more difficult (and time consuming) Type B approach. The most effective means of achieving this is to design the analytical method to ensure spikes, reference samples and duplicates are inserted as early as possible into the analytical run. In addition, from the "Method Validation Replicate Data" section in paragraph 4 above, the method validation program should include the variation of as many potentially significant sources of uncertainty as possible. Estimate the Uncertainty for any Sources not Accommodated by Repeated Data 6. In the unusual cases where it is necessary to estimate uncertainties for any sources not accommodated by repeated data, the estimation of the uncertainty from these sources is based on information from manufacturer specifications that accompany instruments and equipment (such as volumetric ware), tabulated data from handbooks, experience from other methods and/or laboratories and other sources. Examples of these calculations are found in the EURACHEM/CITAC guide Quantifying Uncertainty in Analytical Measurement available as a pdf file from their web page. Tabulate Uncertainty Estimates 7. Compile the values estimated from the repeated experimental data with that for each of the potential sources of uncertainty identified as not being reflected in the repeated data variability (if any) and rank them in decreasing
P19 - CAEAL Policy on Uncertainty Rev 1.4 Page 16 of 39

numerical order. Those sources that have a SD less than 1/3 of the largest SD can be ignored in the subsequent calculation of the combined uncertainty since their contribution to the combined uncertainty will be negligible. Calculation of the Combined Uncertainty 8. SDs cannot be manipulated to calculate the combined standard uncertainty. Instead, the SDs are converted to variances by squaring them and the variances are used for the calculation of the combined standard uncertainty. The combined standard uncertainty is the square root of the sum of the squares of the SDs (known as the Root Sum of Squares). 9. If RSDs have been calculated, the SD at a specific concentration C should be calculated by SD = RSDC. This allows for taking the concentration dependence of the SD into account. .(NMKL Procedure No. 5 (1997) Estimation and expression of measurement uncertainty in chemical analysis). 10. If no actual data is available, a first approximation of the inter-laboratory reproducibility RSD is given by RSDR = 2C(-0.15). The intra-laboratory RSD is one half of that (Official Journal of the European Communities L 77, 16.3.2001, p. 14). The formula is matrix and analyte independent but is unreliable at low and high concentration extremes. 11. Precautions must be taken to not count the contribution of a source of uncertainty more than once in the calculation of the combined standard uncertainty. The between run SD calculated from daily spike recoveries for example, will include the variability found in the entire analytical process provided the spike was inserted at the very beginning. This is also true however, of the SD calculated from the routine inclusion of any reference sample that is inserted at the very beginning of the analytical process. Calculating the combined standard uncertainty by using the SDs from both of these sets of data would double count all of the contributing sources and result in an estimate of the measurement uncertainty that is too large. The established procedure in such an instance is to use the larger of the two SDs in order to give a worst case estimate. As an example, if we have established the between run SD from historical spike recovery data to be SDspike, the bias uncertainty from a Proficiency Testing Program to be SDPT and the sample non-homogeneity SD from sample duplicate insertions to be SDhom and that no other sources of uncertainty have an SD larger than 1/3 of the largest of these three, the combined standard uncertainty SDC is given as: SDC = (SDspike2 + SDPT2 + SDhom2). Applying the Coverage Factor k 12. The Expanded Uncertainty is derived by multiplying the Combined Standard Uncertainty by a coverage factor k. The value of k for 95% coverage is
P19 - CAEAL Policy on Uncertainty Rev 1.4 Page 17 of 39

selected on the basis of the number of values n that are used for the calculation for the SDs. If n 30, k = 2. If n < 30, k is the appropriate Students t factor for n-1 degrees of freedom and a 95% confidence level. Reporting the Result 13. The final concentration result C is then reported as C kSDC with a description of how the measurement uncertainty was calculated. Uncertainty at the Limit of Detection and at the Limit of Quantitation 14. Only when a measured value is larger than the uncertainty with which it can be measured does it have any credibility. This point is known as the Limit of Detection (LOD). The lowest concentration at which a result can have a meaningful uncertainty assigned to it is the Limit of Quantitation (LOQ). The LOD has been most commonly set at a concentration that gives a signal that is 3 times the standard deviation of the measurement process at zero concentration or 3so. Similarly, the LOQ has been set at 10so. 15. The value for s o is the Method Detection Limit determined as described in the CAEAL document Quality Control for Environmental Laboratories. This gives a relative uncertainty at the LOD and LOQ of 100% and 30% respectively, both with a 95% confidence level. (J.K. Taylor, Quality Assurance of Chemical Measurements Lewis Publishers Inc., pages 79-82 (1987). Hierarchy of Data Selection for Estimation of Uncertainty 16. The following hierarchy is presented to provide laboratories with guidance on which types of data they might use to estimate uncertainty within the laboratory. This list is given in order of priority from (I) Most Suitable, to (IV) Least Suitable. I. Uncertainty Specified within the Method. In those cases where a wellrecognized test method (such as a peer-reviewed AOAC method or one published by agencies such as the Ontario MOE, the US EPA or ASTM) specifies limits to the values of the major sources of uncertainty of measurement and specifies the form of presentation of calculated results, the laboratory should follow the reporting instructions (see Note 2 to Clause 5.4.6.2 of CAN-P-4D (ISO/IEC 17025). NOTE: The laboratory would be expected to demonstrate that their results obtained when using this method have the reliability specified in the method in order for this clause to apply. II. Laboratory Control Samples (LCS) and Matrix Spikes. In cases where matrix specific LCS and/or matrix spike data are available, include uncertainty estimated from the standard deviation of the LCS or matrix spikes of more than 50 points collected from their insertion into routine analytical runs. See paragraph 4 of this Appendix above.
P19 - CAEAL Policy on Uncertainty Rev 1.4 Page 18 of 39

III. Pooled Sample Replicate Data. In cases where sample replicates are analysed and there is sufficient data above the limit of quantitation, include pooled sample replicate data to estimate uncertainty that incorporates sub-sample uncertainty as a source. See paragraph 4 of this Appendix above. IV. Proficiency Testing Sample Data. In cases where the previous options are not available and where Proficiency Testing samples are analysed with sufficient data above the limit of quantitation, pooled Proficiency Testing sample data can be used to estimate uncertainty. See paragraph 4 of this Appendix above.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 19 of 39

Description of Uncertainty Source

Example Table to compile MU information Value x Uncertainty u(x) as u(x)/x measured or Standard found Deviation

Source of u(x) information

Steps to using this Table: 4. Define the measureand(s) the analyte, the measurement objectives required for
data to be fit-for-purpose (includes LOD, precision, accuracy, analytical range, selectivity etc.) 5. List the anticipated sources of uncertainty (including parameters found in the equation used to calculate the final result to be reported). 6. List the repeated data sources (spikes, certified reference materials, in-house reference materials. duplicates, method validation files) both short term (one day or one run for example) and long term (over several months or longer). 7. Match the sources of uncertainty with repeat data that was collected while the sources of uncertainty may have varied. Long term spike recovery data may include changes in analysts, calibration sets, and laboratory environment. 8. Identify those sources of uncertainty that are included in more than one repeat data set. Both long-term spike and reference material standard deviation values will include uncertainty due to different analysts, calibration sets etc.; if these were varied while the spike and reference material data were being collected in routine runs. Use only one of these two standard deviation values to estimate the contribution to measurement uncertainty from the sources identified as being varied usually the larger to be conservative. Alternatively, the two standard deviations can be pooled and the pooled value included for compilation into the overall estimate of measurement uncertainty. 9. Estimate the uncertainty due to those sources that have not varied during the collection of repeat data either during method validation or routine analysis. This may involve using certificates for balances and masses or some other source of uncertainty information. 10. Compile the information into the table above and check to ensure that a source of uncertainty has not been counted more than once. 11. Remove those sources of uncertainty that have a standard deviation less than 1/3 the largest standard deviation. 12. Combine the remaining standard deviations using root sum of squares (RSS) technique. (See paragraph 8 of this Appendix) 13. Multiply this combined standard deviation by the appropriate expansion factor to determine the expanded uncertainty. 14. Ensure the data meets the fit-for-purpose criteria. 15. If applicable, report the result with the expanded uncertainty. Indicate the expansion factor (k) and the confidence interval (usually 95%).

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 20 of 39

Aim

APPENDIX 3 Measurement Uncertainty for Microbiological Testing

1. This Appendix applies to environmental microbiological testing methods which: a. Are quantitative but do not have normally (or Gaussian) distributed data. For example, the results from many microbiological tests, such as total coliforms in surface water, are considered to be generally described by the Poisson Distribution [14-20]; and/or b. Are qualitative or semi-quantitative in nature (for example presence/absence types of tests, most probable number (MPN), strains of bacteria, DNA testing). Note 1: Data which is normally (Gaussian) distributed can be handled in the manner outlined in Appendix 2, wherever the required information does exist. Data that are log transformed may be amenable to data analysis for normally distributed data. Note 2: The Poisson distribution is not completely symmetrical, but can be considered to be so for mean values greater than 25 to 30 [16]. 2. Four main types of Microbiological testing have been identified by APLAC[7] and are presented below. Note that Italicized text has been added for CAEAL purposes. a. General quantitative procedures e.g. total coliforms in Surface waters. Depending upon the range of the colony counts, these general quantitative procedures may be considered semi-quantitative (for example in the region of 1-10 or 20 counts). b. MPN procedures c. Qualitative procedures e.g. presence/absence type testing, The microbiological analysis for E.Coli. in drinking water can be considered a Qualitative procedure, since in Ontario at least, the limit is 0. d. Specialist tests, e.g. pharmaceutical testing, or DNA testing The applicability of measurement uncertainty (MU) to each of these types of microbiological tests may differ. 3. This appendix considers and expands each of the CAEAL Protocol steps in more detail. It also explains what is meant and how to perform each task required.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 21 of 39

Sources of uncertainty: 4. The possible sources of uncertainty for an analytical method are tabulated in many of the sources listed in Paragraph 8 of this Policy. Close examination of the steps in the laboratory method SOP, and of the parameters found in the final concentration calculation, will usually help to identify the likely sources of uncertainty in the method. ILAC [4] lists these as: a. b. c. d. e. f. g. h. i. j. definition of the measureand sampling transportation, storage and handling of samples preparation of samples environmental and measurement conditions the personnel carrying out the tests variations in the test procedure the measuring instruments calibration standards or reference materials software and/or, in general, methods associated with the measurement k. uncertainty arising from correction of the measurement results for systematic effects. 5. For microbiological methods additional sources of uncertainty exist, including [16, 20, 21-23] a. Colony count reliability b. microbial distribution i. single plate ii. multiple plate count c. Agglomeration and clustering of microbes d. Microbial Growth (Poisson Scattering) i. difference in incubation time ii. difference in incubation temperature e. Lack of Reference Materials, especially for quantitative measurements f. Confirmation of identification and counts g. Volumetric issues i. Dilution factors ii. Volumetric uncertainties h. Procedural differences. Niemela [22] and [19] has provided a good discussion of these sources of uncertainty for microbiological methods. This document is highly recommended and can be downloaded from the internet at http://www.mikes.fi/documents/upload/Publication%20J3%202002_1.pdf (currently it is free).

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 22 of 39

Laboratory Repeat Data Sets 6. As discussed in Paragraph 4 of the Appendix 2 of this Policy, Proficiency Testing (PT) programs, Reference Samples, Spike Recovery, Method Validation Replicate and Sample Duplicates are typical sources of Laboratory Repeat Data Sets and are sources of repeated measurements from which SDs and RSDs can be calculated. The laboratory can vary one or more of the above sources of uncertainty during the collection of the repeat data and the SD calculated will include uncertainty attributed to the varied source(s). Note: There are limitations with the use of any these for microbiology excepting sample duplicates: Proficiency testing programs These are a source of reproducibility SD (SDR) that includes both intra- and inter-laboratory sources of uncertainty. It is larger than the intra-laboratory uncertainty (SDr) known as repeatability - of a laboratory whose methods are in statistical control. Unlike the case for typical chemical analysis, the data from proficiency test programs alone is often inadequate to establish the quality of a microbiological analysis. Reference sample insertion both certified reference samples and inhouse reference samples inserted into routine runs for control charting applications are a source of long term uncertainty data in chemical analysis. There is a general lack of reference materials available for routine use in microbiological methods. However there are exceptions, such as a reference material with a specified minimum number of organisms, or the preparation of materials in house [16, 24] At least one vendor claims to be able to provide freeze dried pellets which will provide a guaranteed range of colony forming units (CFU). Spike recovery data Can give the same information as reference sample insertion and in some cases can reflect variability due to different sample matrices. There is a general lack of materials available for routine use in quantitative spiking for microbiological methods. However there are exceptions, the Ontario Ministry of the Environment has prepared spiking solutions for certain bacteria. While the procedure can be time consuming and is not performed on a routine basis, properly applied, it can provide useful information [24] Method validation replicate data This is a source of data from repeat analyses run to establish precision estimates at different analyte concentration (or measurement i.e. colony forming units) levels. For the same reasons as the reference materials and spikes, this information is difficult to obtain and is less useful for microbiological tests. A thorough discussion of the use of method validation data in the estimation of uncertainty in chemical analysis is VAM Protocol [9] which was available from the VAM web site (www.vam.org.uk).

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 23 of 39

Sample duplicate insertion known as replicability SDdupl . The SDdupl reflects the variability due to differences between analytical portions (non-homogeneity) and other factors that can vary between replicates and the distribution of counts. In general for microbiological tests this will be the most readily available source of uncertainty data

Match Repeat Data with Uncertainty Sources 7. The objective of this step is to select laboratory Quality Control and validation data that includes as many sources of variability as possible so that these do not have to be estimated using the more difficult (and time consuming) Type B approach. The most effective means of achieving this is to design the analytical method to ensure spikes, reference samples and duplicates are inserted as early as possible into the analytical run. In the case of microbiological testing, sample duplicates will represent the primary source of data for uncertainty. Microbiological laboratories are encouraged to consider the use of ruggedness testing.[9] It should be noted here that while this policy recommends the use of the Type "A approach (use of statistical data to estimate uncertainty), uncertainties estimated using the Type B approach (estimation of the uncertainty through the mathematical manipulation of all relevant contributions to the overall estimate), are acceptable provided they are complete. Estimate the Uncertainty for any Sources not Accommodated by Repeated Data 8. In the unusual cases where it is necessary to estimate uncertainties for any sources not accommodated by repeated data, the estimation of the uncertainty from these sources is based on information from manufacturer specifications that accompany instruments and equipment (such as volumetric ware), tabulated data from handbooks, experience from other methods and/or laboratories and other sources. Examples of these calculations are found in the EURACHEM/CITAC guide [3] which is available for downloading from www.measurementuncertainty.org. Unfortunately none of these references cover microbiological tests. However Niemela [22] has provided some information on observed and computed values of components of uncertainty which may be useful to laboratories in the absence of other information. This type of information will be accepted by CAEAL provided it is clearly referenced, copies of the cited reference are maintained, and the information is applied properly to the uncertainty determination. Calculation of the Combined Uncertainty 9. Precautions must be taken to not count the contribution of a source of uncertainty more than once in the calculation of the combined standard uncertainty. The combined uncertainty can be determined from the standard uncertainties for the different components. The combined uncertainty is calculated base upon the fact that variances are additive. In practice for
P19 - CAEAL Policy on Uncertainty Rev 1.4 Page 24 of 39

microbiological testing it may be found that the uncertainty associated with one of the components overwhelms the contribution of all the other sources (i.e. the RSD or standard uncertainty is 3 times larger than any other components), in which case the others may all be considered negligible. Applying the Coverage Factor k 10. The Expanded Uncertainty is derived by multiplying the Combined Standard Uncertainty by a coverage factor k. The value of k for 95% coverage is selected on the basis of the number of values n that are used for the calculation for the SDs. If n 30, k = 2. If n < 30, k is the appropriate Students t factor for n-1 degrees of freedom and a 95% confidence level. Note: Although it is possible to calculate the Expanded Uncertainty for microbiological tests in this manner, it will not typically be required to be reported in this manner by CAEAL for these parameters. If the expanded uncertainty is reported at the 95% confidence interval based upon the results from proficiency testing then there is no need to apply a coverage factor. However if a laboratory is making use of a RSD reported for a using any of the other Laboratory Repeat Data Sets described in 22 above, then a coverage factor of 2 should typically be used. Uncertainty at the Limit of Detection and at the Limit of Quantitation 11. The concept of LOD and LOQ are much less straightforward in microbiology. The principal application would be to presence absence type of tests (such as for E.Coli in drinking water). In this case the laboratory should at a minimum provide information on the false positive and false negative rates. If these rates cannot be determined an explanation as to why not should be maintained in the method records and reviewed on a regular basis. Reporting the Result 12. For microbiological tests the final result C (in counts per mass or volume) is then reported as a. C kSDC with a description of how the measurement uncertainty was calculated. For microbiological tests measurement uncertainty will typically only be reported for Quantitative measurements (with numerical uncertainty values) b. for semi-quantitative the results (in the range of 1-20) in the absence of laboratory QA data, the results should be noted as semi quantitative in nature c. for qualitative test the negative result should be reported as zero with an upper value for false negative rates( or similar value Hierarchy of Data Selection for Estimation of Uncertainty 13. The following hierarchy is presented to provide laboratories with guidance on which types of data they might use to estimate uncertainty for
P19 - CAEAL Policy on Uncertainty Rev 1.4 Page 25 of 39

microbiological measurements within the laboratory. This list is given in order of priority from (I) Most Suitable, to (IV) Least Suitable. I. Uncertainty Specified within the Method. In those cases where a wellrecognized test method (such as a peer-reviewed AOAC method or one published by agencies such as the Ontario MOE, the US EPA or ASTM) specifies limits to the values of the major sources of uncertainty of measurement and specifies the form of presentation of calculated results, the laboratory should follow the reporting instructions (see Note 2 to Clause 5.4.6.2 of CAN-P-4D (ISO/IEC 17025). NOTE 1: The laboratory would be expected to demonstrate that their results obtained when using this method have the reliability specified in the method in order for this clause to apply. NOTE 2: The laboratory must review the reported uncertainty to determine if it is a standard uncertainty, combined uncertainty or expanded uncertainty. For example the USEPA has reported a single laboratory precision of 3.3 to 27.3 % (depending upon the range) for E.Coli in drinking water [25] this would need to be multiplied by 2 for the Expanded Uncertainty. Another recent USEPA document [26] indicates that a particular E. Coli. method has as 4% false negative rate. II. Pooled Sample Replicate Data. In cases where sample replicates are analyzed and there is sufficient data for each of the cases in II (i), (ii) and (iii) below, include pooled sample replicate data to estimate uncertainty that incorporates sub-sample uncertainty as a source. Sufficient data are considered to be a minimum of 30 sets (i.e. minimum of 60 results). i. Samples above 10 Counts: express MU as an uncertainty interval. ii. Samples with 1 to 10 Counts: semi-quantitative, express MU as an uncertainty interval. iii. Samples with < 1 Counts: qualitative, express MU as a false negative rate. III. Laboratory Control Samples (LCS) and Matrix Spikes. In microbiological tests, matrix specific LCS and/or matrix spike data are typically unavailable. However if available, this type of data can be used to provide an estimate of MU provided a sufficient number of data points exist. Typically data from a minimum of 30 analyses will be required. IV. Proficiency Testing Sample Data. In cases where the previous options are not available and where Proficiency Testing samples are analysed with sufficient data for each of II (i), (ii) and (iii) above, pooled Proficiency Testing sample data can be used to estimate uncertainty. Use of PT sample data will often result in a much larger MU than a single lab will have. For example the results from the analysis of CAEAL PT samples for Microbiological and Toxicity test parameters [27] are shown in the table 1 below. The MU of 0.30 (or 30%) is a 3X larger than the MU calculated for one of the participating laboratories [28]. It is important for a lab to
P19 - CAEAL Policy on Uncertainty Rev 1.4 Page 26 of 39

understand the sources of uncertainty, which were included in the system that produced the PT result(s). Since only four samples are analysed per round, and there are two rounds per year, it could take a substantial length of time to accumulate the 30 samples for each range of analyses.
Table 1: CAEAL Proficiency Testing 1991-1999 Measurement Uncertainty Results [27]

Parameter

Fecal Coliform 0.35 4.5 Total Coliform 0.30 3.6 Average microbiology 0.324 Trout LC50 (96 h) 2.077 8.491 0.248 0 Daphnia LC50 (96 h) 2.185 38.357 0.159 1.89 Microtox (15 min.) 4.390 10.784 0.301 0 Average Toxicity 0.236 Testing Note: slope is calculated based upon the 95% confidence intervals. Some Suggested Procedures for Obtaining Measurement Uncertainty Data for Microbiology Testing Quantitative Testing 14. Log Transforming Data: A number of authors have recommended the use of log transformed data [16, 7, 21, 22, 29]for determining statistics, including standard uncertainties. The CAEAL policy does not advocate or restrict this treatment of data. However it is recommended that at least the first set of data (minimum 30 sets) be analysed both with and without log transforming. 15. The NMKL procedure[16] provides a simple way for evaluating the uncertainty of quantitative microbiological testing on a day to day basis using data from duplicates and/or parallel plating. These include a. b. c. d. Poisson variance : for plating methods Possion Index of Dispersion (D2) Extra Variance (u) Calculation of Routine Parameter (Q) and Steering Diagram

Concentration or Counts Low High 14 82 5 220

Slope (MU)

Intercept

Note that for ease of reference, the original Formula number from NMKL has been retained. 16. Poisson Variance: During the initial method validation, the laboratory should collect data for determining data distribution. Formula 1 provides a simple method for determining an approximate of the 95% Confidence Interval for the case where more than 25 to 30 colony counts are found and the Poisson distribution is applicable.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 27 of 39

Where: Y=estimate of probable number of micro organisms per unit (in this case, Volume (V)) Var=variance SD=standard deviation CI=confidence interval 17. Poisson Index of Dispersion: A test knows as the Dispersion test has been suggested by Niemela [19] to evaluate if variance is larger in the case of two parallel results (A and B) than would be expected from the Poisson distribution assumption. The test parameter (D2) is calculated using Formula 5.

Where, in this case: A= reading from filter 1 B=reading from filter 2 D=dispersion parameter If the D2 value is less than 4, it is approximately 95% probable that the Poisson distribution assumption has been met. It can be seen that as the sum of A and B increases, the deviation that can be tolerated for the Poisson distribution increases also.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 28 of 39

Formula 6 can be used for comparing dilutions for the Poisson distribution, where V1 and V2 represent results from the same sample with different dilutions.

18. Calculation of extra variance, u: In the case where repeated tests show a greater variance than predicted by the Poisson distribution, the extra variance is calculated using the factor u from Formula 7 from the NMKL procedure. Note that the factor u is matrix and test method specific.

the factor u can then be used to calculate the 95% CI using Formula 10.

A different u value will result if several analysts are pooling their results, if different dilutions are being used etc.. The value of u may be found to be count level dependent (e.g. for 1-10, 10-100, 100-1000) in which case different u values should be used. Note that if u values change significantly over time, with the same procedures, the method must be reviewed. 19. Steering Diagram: The use of the routine parameter Q, suggested by Niemela [19] and NMKL is a precision measure for parallels A and B that is similar in concept to repeatability and reproducibility. The parameter Q is calculated using Formula 8.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 29 of 39

The value of Q is expected to be less than 2 and similar for all analyses. It can be plotted over time using a so-called steering diagram, in a manner similar to the typical Shewart control charts used for many chemical analysis methods. Note that the use of logarithms serves to minimise the effects of results which may diverge. NMKL has indicated this Q variable is probably the one most appropriate for use in routine laboratory operations as it is designed to provide results using the same unit and limit value for all analyses. 20. Calculation of Combined Uncertainty: In addition to the uncertainty associated with the microbiological colony distribution, microbiology laboratories must also consider the uncertainty contributions from other identified sources. These may include contributions in measuring volume (initial filtration volume, dilutions, parallels volumes, etc.), media, different sample preparations, and the inter- and intra-analyst variation in colony counting. The exact factors or sources, which must be examined, will depend on the laboratory. The combined uncertainty is then calculated from the standard uncertainties for each of the components using Formula 11

As indicated in the main body of this Policy, and in Appendix 2, any standard uncertainties which are 1/3 or less than the largest uncertainty may be ignored. 21. Calculation of Expanded Uncertainty: The Expanded Uncertainty can be calculated in several ways. It can be calculated directly from RSD (or SDR) information by multiplying by a coverage factor (k=2) to give the Expanded Uncertainty. In the case where a combined uncertainty has been calculated the Expanded Uncertainty is determined using Formula 12 below

22. NMKL has recently updated their 1999 document [23] and has prepared an electronic calculation aid (in MS-Exceltm) that can be downloaded from their website at www.nmkl.org and which laboratories may find useful. Laboratories are still responsible for validating the use of the software.
P19 - CAEAL Policy on Uncertainty Rev 1.4 Page 30 of 39

23. In addition to the procedures and Formula provided above, the recent document by Niemela [22] is available for download at http://www.mikes.fi/documents/upload/PublicationJ3202002_1.pdf and provides a detailed discussion on combining uncertainty components from the Poisson distribution, inter-analyst colony counting, volumetric and dilution factors etc.. While the steps outlined in the Niemela document are not compulsory for microbiology testing laboratories, they are highly recommended. Most Probable Number Methods 24. MPN Methods: The Draft APLAC MU Guidance [7] accepts the data in the McCradys tables as reasonable estimates of MU. For the purposes of this Policy, they can be used as estimates of MU for a test, provided the laboratory reviews the results of unusual combinations and flags or rejects such data. As knowledge advances in this aspect of MPN testing, this aspect of the Policy will be reviewed. A procedure has been suggested by NMKL and Niemela [16, 22] which allows laboratories to calculate a standard deviation for MPN results. For samples in which positive results are found, the relationship in Formula 2 is considered useful.

Where: Y=most probable number V=volume n=total no. of tubes q=no. of negative tubes Note that the CI is skewed. In many cases this uncertainty will usually overwhelm all other sources (such as volumetric etc.). Qualitative Tests

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 31 of 39

25. Qualitative Examinations: For some microbiology tests the result is more semi-quantitative such as E. Coli in drinking water, where there must be zero colony forming units; or qualitative such as presence/absence testing. In these instances, the worst case involves false negatives due to some error. The following Formula 3 can be used to express an upper limit on the number of positives that might be expected in the case where all samples were negative. Sample homogeneity information would need to identify if all samples were from the same location, the same time, etc.

Where: P=confidence level required p=maximum possible positive rate n= no. of samples that test negative In order to increase statistical validity using this approach, more tests must be run compared to other approaches. Laboratories are encouraged to utilise a number of low-level samples to determine false negative rates. In addition to the false negative rates, the measurement uncertainty should include an evaluation of the positive and negative controls. A Shortcut to Uncertainty of Multiple Plate Instruments 26. Maximum log-likelihood ratio statistic G2: The differences between the colony numbers observed are partly due to design (volumes and dilutions) and partly due to random variation (Poisson scatter, volume uncertainties, uncertainty of reading counts). The uncertainty due to design can be removed by using the log-likelihood ratio statistic, G2. The uncertainty of the microbial count can then be estimated directly without using separate uncertainty components. The microbial concentration of the final suspension can be calculated from the weighted mean using formula 63 [22]:

where: zi=the observed colony count of the ith plate vi=the volume of the final suspension inoculated into the ith plate combined relative uncertainty for the test result (wy=ucombined/c) can be calculated using Formula 64 [22]

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 32 of 39

27. To obtain the relative uncertainty estimate that includes all the random components affecting counts within the instrument, the log-likelihood ratio statistic G2 is computed using Formula 66 [22].

28. The relative standard uncertainty squared (or variance) of the colony count in Formula 64 is calculated using Formula 67 [22].

Comparisons to Norms, Guidelines or Regulations 29. Comparison to a norm or guideline: In the case where a test result is to be compared to a regulatory limit, norm or guideline, the observed value should

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 33 of 39

include the confidence limits as calculated in Formula 1. However, an alternative method to Formula 1, using the chi-test can be used. The chi-test parameters are shown in Formula 4 below

For a 95% confidence level the critical value is 4 (approximated from 3.84=1.962). Where: x2=chi squared value C=no. of colonies L= limit value Note that this Policy is based on statistically valid sampling. Regulatory agencies may specify other ways of determining compliance with a guideline or standard. Where a regulatory guideline or standard requires an approach different that this Policy, the regulatory guideline or standard shall govern. Limitation on Uncertainty 30. The boundaries for the system for which the uncertainty is being determined must be recognised and specified. If the laboratory only determines the sources of uncertainty after the sample has been received, the uncertainty estimate represents only part (possibly only a small part) of the uncertainty associated with the final result. 31. The following Table 2 contains a listing of all equations/formula used in this appendix, along with the original reference location.

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 34 of 39

Table 2: Summary of Formula Presented in Appendix 3 Formula No. 1 Formula Equation Comments C= counts V= volume Y=counts per volume Vary=variance SD= Standard Deviation Reference and Page NMKL[16] page 9

N=no. of tubes q= no. of negative tubes V= volume of sample in each tube

NMKL[16] page 11

[22] page 40

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 35 of 39

Formu la No. 3

Formula Equation

Comments

P= desired probability n= no. of negative samples p= max. positive rate C= counts L= limit value _2= chi squared test statistic= approximately 4 for 95% probability D2=dispersion test parameter A= parallel plate reading B= second parallel plate reading V1=initial volume V2= dilution volume C1=undiluted counts C2=diluted counts

Original Reference and Page NMKL[16] page 12

NMKL[24] page 13

NMKL[16] page 15

NMKL[24] page 16

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 36 of 39

Formu la No. 7

Formula Equation

Comments u= extra variance measure C= counts Q=test value or routine parameter

Reference and Page NMKL[16] page 18 NMKL[16] page 18

10

NMKL[16] page 2

11 12

uc= combined uncertainty k=coverage factor (generally 2-3) uc=combined standard uncertainty

Adapted from CITAC [2] Adapted from CITAC [2]

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 37 of 39

Formu la No. 63

Formula Equation

Comments
c=mean concentration zi=the observed colony count of the ith plate vi=the volume of the final suspension inoculated into the ith plate

Original Reference and Page Niemela [22] Page 42

64

wc=relative standard uncertainty of colony counting wF=relative standard uncertainty of dilution factor

Niemela [22] Page 42

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 38 of 39

Formula No. 66

Formula Equation

Comments

zi=colony count on ith plate vi=test portion volume (final suspension) ith plate N=number of test plates(filters) Z=sum of all colony counts V=sum of all test portion volumes

Original Reference and Page Niemela [22] Page 42

67

Niemela [22] Page 43

P19 - CAEAL Policy on Uncertainty

Rev 1.4

Page 39 of 39