Clinica Chimica Acta 279 (1999) 97105

Indirect reference limits estimated from patients

results by three mathematical procedures
a

Montse Ferre-Masferrer
, Xavier Fuentes-Arderiu a , *, Rafael
b
Puchal-Ane
a

`
`
i Universitaria
de Bellvitge, 08907 L Hospitalet de
Servei de Bioqumica
Clinica, Ciutat Sanitaria
Llobregat, Barcelona, Spain
b
`
`
Servei de Medicina Nuclear, Ciutat Sanitaria
i Universitaria
de Bellvitge, 08907 L Hospitalet de
Llobregat, Barcelona, Spain
Received 1 July 1998; received in revised form 28 September 1998; accepted 4 October 1998

Abstract
Presently, only a few clinical laboratories produce their own reference values, while the great
majority use reference intervals reported in the literature. An alternative to this unsatisfactory
situation is to estimate indirect reference limits by means of mathematical / statistical procedures
from patients results obtained routinely in the laboratory. The procedures of Bhattacharya (A
simple method of resolution of a distribution into Gaussian components. Biometrics 1967;23:115

135) Martin et al. (Reference values based on populations accessible to hospitals. In: Grasbeck
R,
T, editors. Reference Values in Laboratory Medicine. Chischester: Wiley, 1981:233262)
Alstrom
and Kairisto et al. (Generation of reference values for cardiac enzymes from hospital admission
laboratory data. Eur J Clin Chem Clin Biochem 1994;32:789796) were applied to 14 biochemical
quantities. In order to verify these procedures, the indirect reference limits obtained from patients
results were validated by statistical comparison with reference limits estimated from a reference
sample according to recommendations of the International Federation of Clinical Chemistry
(IFCC). Calculated indirect reference limits for most quantities studied were reliable, but indirect
reference limits for bilirubins and potassium ion substance concentrations, alanine aminotransferase, and aspartate aminotransferase catalytic concentrations in serum were not suitable. We
conclude that indirect reference limits can be obtained from patients results by all procedures
studied when skewness and kurtosis of mixed distribution are not too large, but other factors also
seem to have an influence on the reliability of these procedures. 1999 Elsevier Science B.V.
All rights reserved.
Keywords: Reference values; Statistical methods; Patients results
*Corresponding author.
0009-8981 / 99 / $ see front matter 1999 Elsevier Science B.V. All rights reserved.
PII: S0009-8981( 98 )00164-8

98

et al. / Clinica Chimica Acta 279 (1999) 97 105

M. Ferre-Masferrer

1. Introduction
Reference values are necessary to compare patients results. The International
Federation of Clinical Chemistry (IFCC) has published recommendations on the
theory of reference values, the selection of individuals for obtaining reference
values and the statistical treatment of data in order to estimate reference limits
[1]. Several European societies of clinical chemistry have published documents
on this topic, concluding that every clinical laboratory should produce its own
reference values [2]; however, only a few laboratories do so. The selection of
reference individuals is difficult, time-consuming and costly. Furthermore, the
continued evolution of measurement procedures requires this process to be
repeated too often.
Thus, it is not surprising that, in many laboratories, reference values are taken
from the scientific or commercial literature. However, reference limits included
in text books or in manufacturers package inserts should not be used for
diagnostic purposes; these texts usually contain no information about the
metrological quality (systematic error and imprecision) with which the reference
values have been produced. In fact, the reliability of reference limits shown in a
text book is usually inversely proportional to the between-laboratory coefficient
of variation observed in an external quality assessment scheme [3].
An alternative to IFCC recommendations is the estimation of indirect
reference limits 1 by means of mathematicalstatistical procedures applied to the
patients results obtained daily in the laboratory, after differentiating them into
pathological and non-pathological (health-related) results [412]. The aim of
this study is to apply, compare and validate three different procedures for this
alternative [8,10,12].

2. Materials and methods

2.1. Reference individuals

For validation purposes, 120 healthy reference individuals (the reference
sample) were selected according to IFCC recommendations. For practical
reasons, we enrolled only women aged 2045 years because they were easily
available in our hospital. These women were not pregnant and had no history of
either severe diseases or surgery in the previous 3 months. They were advised to
avoid physical exercise and cigarette smoking the night before testing and any
alcohol consumption during the prior day. In the last 8 h they had neither eaten
solid food nor drank any liquid except water, and they had remained seated for
1

This term has been proposed by Kairisto et al. [11].


et al. / Clinica Chimica Acta 279 (1999) 97 105
M. Ferre-Masferrer

99

15 min before blood sampling. Venous blood was drawn between 8:00 and 10:00
h, using evacuated blood-collection tubes with gel (Vacutainer, Becton Dickinson, ref. 368510). After 30 min, the tubes were centrifuged at 1200 g for 15 min.
Serum samples were tested within 4 h of blood collection.

2.2. Patients
The hospital where the work was conducted has 1000 beds comprising all
main clinical medical specialities, except paediatrics and obstetrics. During
1995, the clinical biochemistry laboratory of the hospital received 119 583
requests corresponding to 85 618 different patients, 67 565 of which were
outpatients (79%), and 18 053 (21%) were inpatients.
All patients results have been stored during 1995 in a laboratory information
system (Omega, Boehringer Mannheim, Barcelona, Spain); among them, results
from 5578 women aged 2045 years were selected and filed using a personal
computer connected on-line to the laboratory information system. Only the first
result for each patient was included (Table 1).

2.3. Measurement procedures

Biochemical quantities selected for this study were:
Mass concentration of albumin and protein in serum
Substance concentration of bilirubins, calcium(II), creatininium, phosphate,
potassium ion, sodium ion, urate and urea in serum
Catalytic concentration of aspartate aminotransferase, alkaline phosphatase,
alanine aminotransferase and g-glutamyltransferase in serum
All measurements were made with a Hitachi 747 analyser (Boehringer
Mannheim, Germany); details of the measurement procedures are irrelevant for
this study and have been omitted. A human serum freeze-dried calibrator
(Boehringer Mannheim, Germany, ref. 759350) was used once a day for
calibration. During the study, internal quality control was made using Moni-Trol
I-X and Moni-Trol II-X (Dade International Inc. Miami, FL).

3. Procedures for dividing mixed distributions

3.1. Bhattacharya procedure

The Bhattacharya procedure, as modified by Naus et al. [6], assumes that
health-related distributions can be described by a gamma function. Health-

100

et al. / Clinica Chimica Acta 279 (1999) 97 105

M. Ferre-Masferrer

Table 1
Distributions of patients results used to estimate reference limits
Quantity

Refererence limit rex

sults (%)
Lower

Upper

limit

limit

G1

G2

SAlbumin (g/l) r

5067

23

45

21.7

SBilirubins (mmol/l)c

3391

NC

10

13

SCalcium(II) (mmol/l)c

4993

SCreatininium (mmol/l)c

5711

NC

SPhosphate (mmol/l)c

2177

1.17

0.3

1.8

SPotassium ion (mmol/l)c

5578

4.37

0.4

0.8

5.2

SProtein (g/l) r

2947

10

70

20.3

4.4

SSodium ion (mmol/l)c

5578

141

20.2

6.9

SUrate (mmol/l)c

5104

NC

236

79

1.4

5.9

2.30
86

0.1
65

20.4
12

4.8
240
4.8
159
11

SUrea (mmol/l)c

3128

NC

11

5.7

3.6

SAlanine aminotransferase

5722

NC

15

0.50

1.2

3297

NC

18

0.65

1.2

1.3

5098

NC

1.1

0.7

85

5098

NC

24

0.53

1.0

109

5.8
25

49
925

(mkat/l)a
SAspartate aminotransferase

1.5

(mkat/l)a
SAlkaline phosphatase
(mkat/l)a
Sg-Glutamyltransferase
(mkat/l)a
Lower limit, lower reference limit obtained in reference sample according to IFCC recommendation; upper limit, upper reference limit obtained in
reference sample according to IFCC recommendation.
mean value of mixed population; s, standard deviation of mixed population; G1 , skewness coefficient of mixed
n, number of patients used; x,
population; G2 , kurtosis coefficient of mixed population; NC, not calculated, because they have no diagnostic utility.
a

Catalytic concentration.

Substance concentration.

Mass concentration.

related and disease-related distributions from patients results can be separated

mathematically if the two subpopulations do not overlap too much and if the
sample size is large enough. The total frequency distribution is divided into a
number of classes with the same width. When the logarithm of the ratio of the
frequencies in two sequential classes is plotted against the centre of the first
class, a gamma distribution will display a straight line, their statistics (m and a )
can be calculated, and the fractiles 0.025 and 0.975 can be estimated from a
chi-squared distribution.
To estimate indirect reference limits based on the Bhattacharya procedure, a
computer program developed by H.F.M. Borgonjen (ZCA, Juliana Hospital,
7300DV Apeldoorn, Holland) was used.


et al. / Clinica Chimica Acta 279 (1999) 97 105
M. Ferre-Masferrer

101

3.2. Martin et al. procedure

The procedure described by Martin et al. [10] is based on the assumption that
the best function to describe the distribution of the health-related population is a
GramCharlier series. This series includes such terms as the distribution height
at the mean or mode, the mean, the standard deviation, and an added parameter
for the skewness, which makes the function applicable to distributions of many
shapes. When the coefficient of skewness is equal to zero, the formula reduces to
that of a Gaussian curve; when the coefficient of skewness term is not zero, it is
introduced into the function.
The initial estimates for mean and standard deviation of the healthy
population are required; these can be obtained from a reduced group of reference
individuals but, for this purpose, any published reference limits may be used.
The program readjusts all the quantities and generates a solution when the sum
of the squared differences is minimum. From a health-related subpopulation,
0.025 and 0.975 fractiles are calculated by a non-parametric procedure.
To obtain the indirect reference limits using the Martin et al. [10] procedure,
one of the authors (R.P-A.) developed a personal computer program written in
FORTRAN language.

3.3. Kairisto et al. procedure

This procedure was developed using some of the principles first described by
Pryce and Hoffmann [13,14]. The main modifications are the splitting of the
distribution and forcing the mode (rather than the mean) of the hypothesised
health-related distribution to be the same as the mode in the original distribution
[11].
In this procedure, each side of the health-related distribution was estimated
separately, so that the resulting health-related distribution consisted of two
halves of two different Gaussian distributions, with the same mode and mode
frequency but different standard deviations. To estimate indirect reference limits
according to Kairisto et al. [10], a procedure included in the program
GraphRoc TM for Windows [12] was used.

3.4. Validation of procedures for dividing mixed distributions

To validate these procedures for obtaining indirect reference limits from
patients results, we have used a reference sample according to IFCC recommendations [1]. Since fractiles 0.025 and 0.975 are usually selected as reference
limits, it was established that any lower or upper reference limit obtained by a
non-IFCC procedure will be valid if it is not statistically different (P . 0.05)
from fractiles 0.025 or 0.975, respectively, in the reference sample. Thus, a

102

et al. / Clinica Chimica Acta 279 (1999) 97 105

M. Ferre-Masferrer

statistical test comparing an observed fractile with a theoretical one [15] was
used.

4. Results
The distribution of patients results for each quantity has been reported by
some statistics (Table 1). In order to calculate the proportion of pathological
results in the mixed distribution, the reference limits obtained following IFCC
recommendations with the reference sample have been used. The results outside
the reference intervals defined by these reference limits have been considered as
disease-related.
The indirect reference limits estimated from patients results using the three
procedures and the reference limits estimated following to IFCC recommendations are given in Table 2.
In this work, the three procedures studied gave indirect reference limits
similar to those obtained in a reference sample according to IFCC recommendations. However, after validating the indirect reference limits, statistical
differences have been found for some quantities.
As regards serum concentration of bilirubins, creatinium, urate and urea, we
have only validated the upper indirect reference limit, because this is the only
one having diagnostic utility in our hospital.
The upper indirect reference limits for substance concentration of bilirubins
and catalytic concentration of alanine aminotransferase in serum obtained by the
three procedures are significantly different (P , 0.05) from fractile 0.975 of the
reference distribution.
The upper indirect reference limit for catalytic concentration of aspartate
aminotransferase in serum obtained using the Kairisto et al. procedure [11] is
statistically different (P , 0.05) from fractile 0.975 of the reference distribution.
The lower indirect reference limit for substance concentration of potassium
ion in serum obtained using Martin et al. [10] and Kairisto et al. procedures [11]
is statistically different (P , 0.05) from fractile 0.025 of the reference distribution.

5. Discussion
The establishment of reference values is difficult, time-consuming and costly;
consequently, it is not surprising that only a few laboratories do it. Probably, one
of the main problems is to obtain a sufficient number of adequate samples for
obtaining reference values, particularly for quantities that are not distributed
according to LaplaceGauss law and for which partitioning criteria should be


et al. / Clinica Chimica Acta 279 (1999) 97 105
M. Ferre-Masferrer

103

Table 2
Indirect reference limits obtained from patients results using Bhattacharya [4], Martin et al. [10],
and Kairisto et al. [11] procedures
Quantity (units)

SAlbumin (g/l) r
SBilirubins (mmol/l)c
SCalcium(II) (mmol/l)c
SCreatininium (mmol/l)c
SPhosphate (mmol/l)c
SPotassium ion (mmol/l)c
SProtein (g/l) r
SSodium ion (mmol/l)c
SUrate (mmol/l)c
SUrea (mmol/l)c
SAlanine aminotransferase
(mkat/l)b
SAspartate aminotransferase
(mkat/l)b
SAlkaline phosphatase
(mkat/l)b
Sg-Glutamyltransferase
(mkat/l)b

Procedure
Bhattacharya
[4]

Martin at al.
[10]

Kainsto at al.
[11]

IFCC [1] a

4152
314*
2.132.48
6496
0.841.55
3.744.95
6381
138146
129366
2.98.2
,0.47*

3951
316*
2.122.48
6293
0.821.52
3.78*5.04
6381
138146
120345
2.67.8
,0.52*

3952
213*
2.112.47
6295
0.841.56
3.80*5.05
6180
136146
120337
2.37.5
,0.51*

4153
418
2.152.46
6596
0.871.45
3.664.89
6479
138146
134325
3.07.3
,0.60

,0.45

,0.49

,0.41*

,0.46

,1.5

,1.7

,1.5

,1.4

,0.41

,0.48

,0.44

,0.43

Reference limits obtained from reference sample were produced according to the FCC recommendations.
Catalytic concentration.
c
Substance concentration.
r
Mass concentration.
* Significantly different (P#0.05) than fractile 0.975 of the reference distribution.
b

used. For this reason, there are three alternatives to solve this problem: (i)
adoption of reference limits produced by other laboratories after verifying their
transferability [1], (ii) the multicentric production of reference values [16], and
(iii) the use of patients results to estimate reference limits by direct or indirect
methods [414].
In any case, the ideal situation is to have true interchangeability of reference
limits, and consequently, common interpretation of results coming from different
laboratories.
Among these alternatives, the estimation of reference limits from patients
results has many advantages:
It is the easiest procedure
It is the cheapest procedure
Reference limits are derived from values obtained with the same pre-

104

et al. / Clinica Chimica Acta 279 (1999) 97 105

M. Ferre-Masferrer

metrological conditions as the patients results which will be compared with

reference limits
Patients results can be easily selected according to biological variation
factors when it is necessary.
The upper indirect reference limits for substance concentration of bilirubins
and catalytic concentration of alanine aminotransferase in serum obtained by the
three procedures have not been used for diagnostic purposes. This is probably
due to the shapes of the mixed distributions; these distributions have coefficients
of skewness and kurtosis greater than the other quantities studied (Table 1). The
shape of mixed distributions seems to have an influence on the reliability of
these procedures. We observed that the three procedures are reliable when
distribution of patients result has a coefficient of skewness of 12 or lower and a
coefficient of kurtosis of 159 or lower, but they are not reliable when these
values are 13 and 240 or higher, respectively.
The upper indirect reference limit for catalytic concentration of aspartate
aminotransferase in serum obtained by the Kairisto et al. procedure [11] and the
lower indirect reference limit for substance concentration of potassium ion in
serum obtained by the Martin et al. [10] and Kairisto et al. [11] procedures are
not valid, but we do not have an explanation for these differences.
We conclude that the three procedures used to estimate reference limits from
patients results are good enough for the following quantities: substance
concentrations of calcium(II), creatininium, phosphate, sodium ion, urate and
urea in serum; mass concentration of albumin and protein in serum; and catalytic
concentration of alkaline phosphatase and g-glutamyltransferase in serum.
The most effective procedure studied is the one proposed by Bhattacharya [4]
modified for gamma distributions.

References
[1] International Federation of clinical chemistry. Approved Recommendation on the Theory of
Reference Values. Part I to VI. J Clin Chem Clin Biochem 1987;25:337342; J Clin Chem
Clin Biochem 1987;25:639644; J Clin Chem Clin Biochem 1987;25:645656; J Clin Chem
Clin Biochem 1987;25:657662; J Clin Chem Clin Biochem 1988;26:593598; J Clin Chem
Clin Biochem 1991;29:531535.
J.M., dir. Teora
de los valores de referencia. Documentos de la Comision

[2] Queralto Compano

de Valores de Referencia. Barcelona: SEQC, 1993.

[3] Fuentes-Arderiu X, Ferre-Masferrer

M. On the reliability of published reference limits. Eur J
Clin Chem Clin Biochem 1996;34:143.
[4] Bhattacharya CG. A simple method of resolution of a distribution into Gaussian components.
Biometrics 1967;23:11535.
[5] Naus AJ, Borst A, Kuppens PS. The use of patient data for the calculation of reference
values for some haematological parameters. J Clin Chem Clin Biochem 1980;18:6215.


et al. / Clinica Chimica Acta 279 (1999) 97 105
M. Ferre-Masferrer

105

[6] Naus AJ, Borst A, Kuppens PS. Determination of n-dimensional reference ellipsoids using
patient data. J Clin Chem Clin Biochem 1982;20:7580.
[7] Baadenhuijsen H, Smit JC. Indirect estimation of clinical chemical reference intervals from
total hospital patient data: application of a modified Bhattacharya procedure. J Clin Chem
Clin Biochem 1985;23:82939.
[8] Hemel JB. Critical discussion on a method for derivation of reference limits in clinical
chemistry from a patient population. J Autom Chem 1985;7:2030.
[9] Oosterhuis WP, Modderman TA, Pronk C. Reference values: Bhattacharya or the method
proposed by the IFCC?. Ann Clin Biochem 1990;27:35965.
[10] Martin HF, Hologgitas JV, Driscoll J, Fanger H, Gudzinowicz BJ, Reference values based on

T, editors. Reference Values in

populations accessible to hospitals. In: Grasbeck
R, Alstrom
Laboratory Medicine. Chischester: Wiley, 1981:233262.

V, Voipio-Pulkki LM, Iriala K.

[11] Kairisto V, Hanninen
KP, Leino A, Pulkki K, Peltola O, Nanto
Generation of reference values for cardiac enzymes from hospital admission laboratory data.
Eur J Clin Chem Clin Biochem 1994;32:78996.
[12] Kairisto V, Poola A. Software for illustrative presentation of basic clinical characteristics of
laboratory tests: GraphROC for Windows. Scand J Clin Lab Invest 1995;55(222):4360.
[13] Pryce SD. Level of haemoglobin in whole blood and red blood-cells, and proposed
convention for defining normality. Lancet 1960;2:3336.
[14] Hoffmann RG. Statistics in the practice of medicine. J Am Med Assoc 1963;185:86473.
[15] Conover WJ, Practical Non-parametric Statistics. New York: Wiley, 1980:105111.

R, Castineiras-Lacam[16] Ferre-Masferrer
M, Fuentes-Arderiu X, Alvarez-Funes V, Guell-Miro
bra MJ. Multicentric Reference Values, Shared Reference Limits. Eur J Clin Chem Clin
Biochem 1997;35:7158.