Академический Документы
Профессиональный Документы
Культура Документы
Correspondence:
Edwin van Teijlingen,
Department of Public Health
and Dugald Baird Centre for Research
on Women's Health,
Medical School,
University of Aberdeen,
Aberdeen AB25 2ZD,
UK.
E-mail: van.teijlingen@abdn.ac.uk
V A N T E I J L I N G E N E .R ., R E N N I E A -M ., H U N D L E Y V . & G R A H A M , W .
(2001)
The importance of conducting and reporting pilot studies: the example of the
Scottish Births Survey
Background. In many research papers, pilot studies are only reported as a means of
justifying the methods. This justication might refer to the overall research design,
or simply to the validity and reliability of the research tools. It is unusual for reports
of pilot studies to include practical problems faced by the researcher(s). Pilot studies
are relevant to best practice in research, but their potential for other researchers
appears to be ignored.
Objective. The primary aim of this study was to identify the most appropriate
method for conducting a national survey of maternity care.
Methods. Pilot studies were conducted in ve hospitals to establish the best of four
possible methods of approaching women, distributing questionnaires and encouraging the return of these questionnaires. Variations in the pilot studies included (a)
whether or not the questionnaires were anonymous, (b) the staff involved in
distributing the questionnaires and (c) whether questionnaires were distributed via
central or local processes. For this purpose, ve maternity hospitals of different sizes
in Scotland were included.
Results. Problems in contacting women as a result of changes in the Data Protection
Act (1998) required us to rely heavily on service providers. However, this resulted in
a number of difculties. These included poor distribution rates in areas where
distribution relied upon service providers, unauthorized changes to the study
protocol and limited or inaccurate information regarding the numbers of questionnaires distributed.
Conclusions. The pilot raised a number of fundamental issues related to the process
of conducting a large-scale survey, including the method of distributing the
289
Introduction
This paper reports the lessons learnt from a number of small
pilot studies conducted in preparation for the Scottish Births
Survey, a national survey of maternity services in Scotland
(Hundley et al. 2000). This paper argues for more and better
reporting of the process and outcomes of pilot studies.
The Scottish Births Survey aimed to include all women who
had delivered a baby in Scotland during a given 10-day
period in the autumn of 1998. Considering the scale of the
sample, the geographical size of Scotland and the limited
resources, the obvious (that is realistically viable) research
method was either a postal questionnaire or a telephone
survey. The latter option was not considered viable as some
women did not have a telephone and these women were likely
to come from a particular socio-economic background
(Bowling 1997). Thus, using telephone interviews would
not give opinions from women across the whole socioeconomic spectrum. Furthermore, identifying women and
gaining access to their telephone numbers would have been a
problem. Hence the decision was made to conduct a postal
questionnaire survey.
Ethical issues
Deciding to conduct a postal survey immediately led to a
number of ethical considerations. Ethical approval was
sought from the Multi-Research Ethics Committee, which
considers applications for research conducted in more than
one region of Scotland. However, the committee felt this to
be unnecessary because it regarded the main study to be an
audit and not as research (Wilson et al. 1999). However,
classifying a study as audit does not `solve' ethical considerations such as: `How could we make sure that the survey was
not going to upset women who had had a delivery in the
designated period, but whose baby was seriously ill and/or
had subsequently died?' One could argue that such women
should be included in the survey to cover the range of
experiences of maternity care in the population. At the same
time it is clear that women with these experiences should be
290
Methods
A self-completion questionnaire was developed by the survey
team, drawing on the criteria agreed by the multidisciplinary
Standard Setting Group for Scotland, and using existing,
validated questionnaires (Mason 1989, Audit Commission
1996, Lamping & Rowe 1996, Hundley et al. 1997, Rennie
et al. 1998). The majority of the questions were closed,
although a section for open comments was included. The
questionnaire was pretested on 10 postnatal women to assess
its acceptability and reviewed by the research team and the
funders for validity. Minor amendments were made to
the text and the questionnaire was then piloted to assess
the acceptability of the revised questionnaire as well as the
feasibility of the proposed research process.
The pilot studies were restricted to births within a
limited period in July 1998 in order to replicate the full
survey as much as possible. The ve maternity units
involved in the pilot studies, and four different methods of
distributing and returning questionnaires are shown in
Table 1. The units were selected by size (i.e. number of
deliveries per year). As the main study covered all maternity units in Scotland, a survey process was needed which
worked in both small and large units. Hence, two large
hospitals were selected for the pilot studies, a medium-sized
one and two smaller ones.
In hospital `Large 1', the questionnaire was distributed
anonymously by midwives working in the community, thus,
making it impossible for the research team to send out
reminders to nonrespondents. The original plan for hospital
`Large 2' was to distribute the questionnaires with a study
number attached, which allowed reminders to be sent to
those women who did not return their questionnaires.
However, the local head of midwifery voiced concerns about
the ethics of this strategy, partly as a result of one previous
Identication? (Named/
numbered or anonymous)
Distribution of questionnaire
Method of reminder
Large 1
Large 2
Small 1
Small 2
Medium 1
Anonymous
CMW*
N/A
Named/numbered
CMW
Head of midwifery
Named/numbered
CMW
Research team
Named/numbered
CMW
Research team
Anonymous
Hospital and CMW (leaet)
Hospital records department
local incident, where the parents of an ill baby had been sent
a questionnaire that was felt to be inappropriate, and partly
because of the changes in the United Kingdom Data Protection Act (1998). Consequently, reminders were sent out via
the head of midwifery in case there were any problems with
the new-born baby.
In units `Small 1' and `2' the midwifery managers did not
raise any objections to providing us with the names of women
to whom the questionnaire was administered. Consequently
numbered questionnaires were distributed by midwives in the
community, who also returned to us a list with the women's
names (after gaining consent) matched to the study number
on the questionnaire. This allowed us to send out reminders
directly to the women if required. Finally, in unit `Medium 1'
the plan was to distribute numbered questionnaires to the
women on discharge from hospital. At the same time, staff in
the hospital record department completed an additional form
for each woman containing the woman's name, address and a
unique study number. The names and numbers were held
locally to allow the hospital to send out of reminders. In
addition, the midwives in the hospital were asked to hand out
a leaet to the women to inform them about the survey.
All ve hospitals were asked to inform us about the
number of questionnaires handed out to women to allow us
to calculate the response rates (see Table 2).
Results
The ndings of the pilot studies related to both process and
outcome. As the outcomes were more straightforward, these
will be presented rst. Table2 shows the actual number of
births that took place during the period of the pilot studies.
The number of births in hospital `Large 2' (n 27) did not
differ greatly from the number in `Medium 1' (n 22) and
this may be because of seasonal uctuations in births. It
became clear during the pilot studies that midwives were not
informing the research team of all questionnaires handed out
to new mothers in their area. This was particularly obvious in
units `Large 1' and `Medium 1'. The proportion of women
that the midwives reported receiving the questionnaire varied
from 0% to 96%. Moreover, response rates, when calculated
on the basis of information provided by the midwives, were
often unrealistically high. In `Large 1' midwives reported that
they had handed out 12 questionnaires, representing only
24% of all births registered in the pilot period, yet 24
questionnaires were returned. Similarly, in `Medium 1',
where 22 births took place, no questionnaires were reported
by the midwives to have been handed out, however, 10 were
returned. In the two small hospitals, the number given
equalled those returned, although in `Small 1', the midwives
included only 20% of all registered births in the pilot.
Large 2
Small 1
Small 2
Medium 1
50
27
22
12
26
200
24
58
15
100
1
100
2
10
100 (2/2)
45 (10/22)
48 (24/50)
55 (15/27)
20 (1/1)
291
Discussion
Pilot studies are a crucial step in the research process. The
results of the pilot studies made the research team reconsider
aspects of the main survey. We opted for a system whereby a
midwife in the community handed out a numbered questionnaire on the tenth postnatal day. After gaining consent, the
midwife also lled in a form with the woman's name and
address and the study number on the questionnaire. The
latter form was then forwarded to the research team.
Lessons learned
The politics of the research process is sometimes equally or
more important than other considerations in deciding which
research design to use. In our case involving midwives was
perceived to be benecial for two reasons: (1) increased
recruitment rates as a result of face-to-face recruitment and
(2) a greater sense of participation in and ownership of the
national audit amongst the midwifery profession. However,
using intermediaries to distribute the questionnaire added an
extra link in the chain of distribution and communication.
The inuence of a `gatekeeper' (Holloway 1997) probably
weakened the research process. The midwives could also have
had an effect on our ndings by inuencing the women's
responses either directly (consciously or unconsciously), or
indirectly with women giving responses that they thought
were more acceptable to the midwives (Riley 1977).
The fact that managers in some units chose to change the
method of distribution during our pilot, indicates the chal292
Conclusion
Our message is that everybody needs pilot studies, not just
their own, but also those of other researchers in the eld.
293
Acknowledgements
These pilot studies would not have been possible without the
participation of the Scottish community midwives, heads of
midwifery and link supervisors. Thanks are also due to the
staff at the Dugald Baird Centre for Research on Women's
Health for their help in conducting the pilot studies. The
survey was commissioned and funded by the Scottish
Programme for Clinical Effectiveness in Reproductive
Health, Edinburgh. Finally, we like to thank the anonymous
reviewers for their positive and helpful comments on our rst
submission.
References
Audit Commission (1996) Maternity Care Survey. Audit Commission/Institute of Child Health, Bristol, UK.
Baird C. (2000) Taking the mystery out of research. The pilot study.
Orthopaedic Nursing 19, 4243.
Baker R. (1990) Development of a questionnaire to assess patients'
satisfaction with consultations in general practice. British Journal
of General Practice 40, 487490.
Barriball K.L. & While A.E. (1999) Non-response in survey research:
a methodological discussion and development of an explanatory
model. Journal of Advanced Nursing 30, 677686.
Beurskens A.J.H.M., De Vet H.C.W. & Kant I.J. (1998) Dwalingen
in de methodologie. VIII. Pilotonderzoeken: zin en onzin. Neder2 lands Tijdschrift Voor de Geneeskunde 142, 21422145.
Bourne I. (2000) The Data Protection Act 1998 selected highlights
of the new legislation. The British Journal of Healthcare Computing & Information Management 17, 2729.
Bowling A. (1997) Research Methods in Health: Investigating Health
& Health Services. Open University Press, Buckingham, UK, p. 232.
Burns N. & Grove S.K. (1999) Understanding Nursing Research, 2nd
edn. W.B. Saunders Company, Philadelphia, PA, p. 40.
Clarke R., Breeze E., Sherliker P., Shipley M., Youngman L., Fletcher
A., Fuhrer R., Leon D., Parish S., Collins R. & Marmot M. (1998)
Design, objectives, and lessons from a pilot 25 year follow-up
294
295