Non-sampling errors

(Session 20)
SADC Course in Statistics

Learning Objectives
By the end of this session, you will be able to
describe the types of non-sampling errors
that arise in survey work
explain actions that may be taken to
minimise commonly occurring non-sampling
errors
have a greater appreciation that sampling
errors is only a small component of all
errors that may arise and that close
attention to reducing non-sampling errors is
equally or more important in survey work.
To put your footer here go to View > Header and Footer

Non-sampling errors: 1
Non-sampling errors cover all errors other
than those due to sampling a subset of the
population.
In both surveys and censuses, it is quite usual
to find non-sampling errors because their
absence implies that the data collection
process has been:
(a)Implemented and enumerated perfectly, &
(a)Completely free of measurement errors,
i.e. inaccuracies in the recording of
information from selected units.
To put your footer here go to View > Header and Footer

Non-sampling errors: 2
Non-sampling errors are not all due to
avoidable mistakes and/or deficiencies.
They can often occur because of decisions
by the researchers to balance the need for
good quality data with the need to obtain
timely data at acceptable cost.
The problem then reduces to one of defining
and minimising errors associated with the
data collection and data processing
procedures.
To put your footer here go to View > Header and Footer

Types of non-sampling
errors

Non-sampling errors can be of various types

Coverage (or Frame) errors

Non-response errors

Measurement errors

Data handling errors

Note that the first more often applies to

sample surveys, while the last three apply
to both surveys and censuses.

To put your footer here go to View > Header and Footer

Coverage (frame) errors

In surveys, the sample is selected from a list,
i.e. a sampling frame, of all population
members.
An inadequate frame leads to coverage errors.
Often can have either
under-coverage (missing elements), or
over-coverage (duplicates)
Both lead to biased results. See below for an
example.
To put your footer here go to View > Header and Footer

Minimising frame errors

For under-coverage, consider re-defining the
population, i.e. the target population is
simply considered as the population which
can be accessed by the frame.
For duplicates, develop a system to identify
the duplicates, e.g. by using additional
information on the recording unit.
Both under & over-coverage are minimised by
using up-to-date frames, e.g. in UK the
Postcode Address File is updated every 3
months, and is hence often used by the Office
of National Statistics (ONS).
To put your footer here go to View > Header and Footer

Non-response errors
Non-response errors are all errors arising
from:
Unit non-response, i.e. failure to obtain
information from a pre-chosen sampling
unit or population unit
Item non-response, i.e. failure to get a
response to a specific question or item in
the data recording form.

To put your footer here go to View > Header and Footer

Types of non-response errors

Discussion:
What are the typical forms of non-response
(both unit and item non-response) you
encounter in your work?
What are the reasons for non-response?
How can such non-response errors be
minimised?

To put your footer here go to View > Header and Footer

Measurement Errors
Measurement errors arise when the recorded
response differs from the true value.
They can occur for a variety of reasons, e.g.
by respondent (e.g. heads of households)
giving an incorrect answer
because of instrument or question error
by interviewer error.
Further, errors may be greater for some subgroups of the population, e.g. those less
literate, or those unwilling to co-operate.
To put your footer here go to View > Header and Footer

10

Reasons for respondent errors

Respondent errors arise for many reasons e.g.
respondent gives an incorrect answer, e.g. due to
prestige or competence implications, or due to
sensitivity or social undesirability of question
respondent misunderstands the requirements
lack of motivation to give an accurate answer
lazy respondent gives an average answer
question requires memory/recall
proxy respondents are used, i.e. taking answers
from someone other than the respondent.

How can such errors be minimised?

To put your footer here go to View > Header and Footer

11

Instrument Errors
Instrument or question errors arise when
The question is unclear, ambiguous or difficult
to answer
the list of possible answers suggested in the
recording instrument is incomplete
requested information assumes a framework
unfamiliar to the respondent
the definitions used by the survey are different
from those used by the respondent (e.g. how
many part-time employees do you have? See
next slide for an example)

How can such errors be minimised?

To put your footer here go to View > Header and Footer

12

An example of instrument error

The following example is from Ruddock (1998) see slide 18

In the Short Term Employment Survey (STES)

conducted by Office of National Statistics in UK,
data are collected on numbers of full-time and
part-time employees on a given reference date.
Some firms ignored the reference date and gave
figures for employees paid at the end of the
month, thus including those who joined and those
who left in that month leading to an overestimate.
Firms found it difficult to give details of part-time
employees as their definition of part-time did
not agree with that used by ONS.
To put your footer here go to View > Header and Footer

13

Interviewer errors
Interviewer errors arise when
different interviewers administer a survey in
different ways
differences occur in reactions of respondents
to different interviewers, e.g. to interviewers
of their own sex or own ethnic group
inadequate training of interviewers
inadequate attention to the selection of
interviewers
there is too high a workload for the
interviewer

How can such errors be minimised?

To put your footer here go to View > Header and Footer

14

Data handling errors

Data handling errors can occur from the stage of data
collection up to the final stages of data analysis.
Types of errors that can arise include:

errors in transmission of data from the field to

the office
errors in preparing the data in a suitable format
for computerisation, e.g. during coding of
qualitative answers
errors in computerisation of the data
errors during data analysis, e.g. imputation and
weighting.

Do any of these types of error occur in your

work. If so, what can you do to minimise them?
To put your footer here go to View > Header and Footer

15

Measuring non-sampling errors

Measuring non-sampling errors is difficult and
often impossible. Attempts have often been
through specific additional studies, e.g.
characteristics of non-respondents in the 1996
British Crime Survey were investigated by a
mini-questionnaire to those living in 25% of
non-responding addresses.
Several studies to assess non-sampling errors
can be found in Ruddock (1998) (see slide 18
for full ref.) & in Lessler, J.T. and Kalsbeek,
W.D. (1992) Non-sampling error in surveys;
Wiley.
To put your footer here go to View > Header and Footer

16

Non-sampling errors: Key Points

Non-sampling errors are inevitable in production
of national statistics. Important that:

At planning stage, all potential non-sampling

errors are listed and steps taken to minimise them
are considered.

If data are collected from other sources, question

procedures adopted for data collection, and data
verification at each step of the data chain.

Critically view the data collected and attempt to

resolve queries immediately they arise.

Document sources of non-sampling errors so that

results presented can be interpreted meaningfully.

To put your footer here go to View > Header and Footer

17

References
Ruddock, V. (1998) Measuring and Improving Data
Quality UK Govt. Statistical Service Methodology
Series No. 14, for a very comprehensive coverage of
non-sampling errors. This document may be
downloaded from

http://www.statistics.gov.uk/methods_quality/publications.asp

Lepkowski, J. (2004) Non-observation error in

household surveys in developing countries. Chapter
VIII, pp 149-169 of the UN Publication An Analysis of
Operating Characteristics of Household Surveys in
Developing and Transition Countries: Survey Costs,
Design Effects and Non-Sampling Errors. Available at
http://unstats.un.org/unsd/hhsurveys/index.htm

To put your footer here go to View > Header and Footer

18

Practical work follows

To put your footer here go to View > Header and Footer

19