The Delphi Technique

Brian Garrod
Institute of Rural Sciences
University of Wales Aberystwyth
What is the Delphi Technique?
• Qualitative research technique …
• … but with quantitative elements
• Relies on the judgement of a panel of experts
• Iterative process, taking place over a number of
‘rounds’
• Much admired:
¤ Very flexible
¤ Good at getting beneath the surface of issues
¤ Good at addressing difficult or ‘expert’ questions
¤ More structured than conventional interviewing
• But also much criticised:
¤ Often dismissed as being ‘discredited’
¤ Accused of being anti-democratic/anti-participatory
¤ Reputation tarnished by sloppy execution in early days
The Delphi Technique
• Name eludes to the Oracle of Ancient Greece
• First used in 1950s by RAND Corporation in the USA
• Cold war application
• Panel of experts formed
• Question related to likely
number of USSR nuclear
missiles
• Mean and standard deviation
• Feed back between rounds
• Repeated until ‘consensus’ approached
The Delphi Technique
• What is the Delphi Technique?
‘a systematic method of collecting opinions from a group
of experts through a series of questionnaires, in which
feedback on the group’s opinion distribution is provided
between question rounds while preserving the anonymity
of the respondent’s responses.’ (Helmer, 1972, cited in
Masser and Foley, 1987: 217-218)
• Key features
¤ Systematic
¤ Questionnaire
¤ Expert opinions
¤ Iterative process, i.e. ‘rounds’
¤ Feedback - individual opinions mediated by group
¤ Yet anonymity of individuals
The Delphi Technique
Example applications
• Technological forecasting, e.g. medicines in the 1990s
• Demand forecasting, e.g. Hawaii tourism in the Year 2000
• Forecasting changes in hotel management in Hong Kong
post-1997
• Marketing of the island of Gozo separately from Malta
• Environmental Impact Assessment of development of
Salt’s Mill
• The setting of priorities in managing heritage attractions
• Definitions of ‘ecotourism’
The Delphi Technique
Example: “Managing Heritage Tourism” (Garrod and
Fyall, 2000)
• Follow-up study to a mail-based survey of 300 managers
of heritage attractions
• Delphi used to explore some intriguing issues in greater
depth
• Survey asked respondents if they
were willing to be involved in a
follow-up study
• Panel members recruited from
among those who said yes
The Delphi Technique
• 17 panel members
Heritage management consultant 2
Local authority officer 2
Heritage organisation officer 2
Historic property manager 6
Academic 3
Museum officer 2
• Three topic areas
? What should be the major priorities in the mission of heritage
attractions?
? What factors are most likely to influence your admission pricing
policy over the next decade
? What should be the funding priorities of major heritage
organisations such as the National Trust and English Heritage?
The Delphi Technique
• The rounds
> Round 1 – establish themes
> Round 2 – assess themes and place in rank order
> Round 3 – assess ranking and re-rank
• Example – Heritage attraction mission
Second Round Third Round Change
Conservation 1 1 -
Accessibility 2 2 -
Finance 3 4 ↓
Education 4 3 ↑
Quality 5 5 -
Relevance 6 6 -
Recreation 7 7 -
Local Community 8 8 -

Rs = 0.9462
The Delphi Technique
Panel membership

Round: 1 2 3
Heritage management consultant 2 2 1
Local authority officer 2 2 2
Heritage organisation officer 2 2 2
Historic property manager 6 5 5
Academic 3 2 2
Museum officer 2 2 2
Total: 17 15 14
Strengths of the Delphi Technique
The technique is flexible enough to be applied in a
variety of situations and to a wide range of complex
problems, for which there is often no other suitable
means of analysis
The iterative approach allows experts to reconsider their
judgements in the light of feedback from peers
The process also gives participants more time to think
through their ideas before committing themselves to
them, leading to a better quality of response
The anonymity of the approach enables experts to
express their opinions freely, without institutional
loyalties or peer group pressures getting in the way
Strenghts of the Delphi Technique
The potential influence of personality is also removed in
this way
Redundant ‘noise’ (issues that tend to side-track the
debate) can be controlled by the project manager
The process generates a record of the group’s thoughts,
which can be reviewed as required
The method can be used to evaluate the spread of
opinion as well as consensus points.
Weaknesses of the Delphi Technique
Delphi can be extremely sensitive to:
 the level of panellists’ expertise
 the composition of the panel
 the clarity of the questions
 the way in which the project manager reports reasons for outliers
 the administration of the questionnaire
It assumes that experts are willing to allow their
judgements to be re-formed by the opinions of others
Expert panel is vulnerable to high rates of attrition due to:
 boredom with the subject matter
 disillusionment with the process, and/or
 lack of time to complete the questionnaire before the following
round commences
Weaknesses of the Delphi Technique
Some Delphi practitioners use monetary payments or
moral persuasion to encourage panellists to stay the
course; however, this may bias the results of the study
There is also a risk of ‘specious consensus’ being
formed, whereby panellists acquiesce and conform to
the median judgement (‘group think’)
Where consensus is being sought there is a problem in
determining what actually constitutes consensus
The technique often requires a substantial period of time
to complete and can be costly in terms of the
researcher’s time
Proposed Best-Practice Guidelines
1. The Delphi technique should not be seen as a main tool
of investigation but a means of supporting/extending
studies which better established and more reliable
methods of investigation.
2. The topic must be appropriate, for example there must
be no widely-perceived ‘correct answers’ to the
questions posed.
3. Questions must be pilot-tested to avoid ambiguity
4. Panellists should be recognised experts in their field (a
self-assessment selection procedure may be useful in
this respect).
5. The panel should comprise a good balance of different
disciplines and areas of expertise.
6. Adequate time must be given to experts to think deeply
about the questions at hand.
Proposed Best-Practice Guidelines
1. Once a subsequent round has commenced, those
completing the previous round late should nevertheless
be excluded from continuing.
2. Criteria for panel balance should be set in advance.
Should these no longer be met the study should be
terminated.
3. Attrition of the panel may be minimised by selecting
experts who already have a strong interest in the
outcome of the project.
4. This is preferable both to using monetary payment and
moral persuasion as a means of ensuring that experts
remain committed to the project.
5. Experts must also believe that the Delphi technique is a
valid way of going about the task at hand.
Proposed Best-Practice Guidelines
1. Full anonymity must be preserved at all times between
the panellists (but not necessarily between the panellists
and the coordinating researchers).
2. The coordination group should make themselves
available as a resource for locating further information
on specific subjects or clarifying the questions.
3. The coordination group should intervene in the process
as little as possible.
4. The panellists must do the initial scoping themselves,
the coordination group should not set the agenda for
discussion (although they will have to determine the
research questions that will need to be answered
through this process).
5. Where consensus is being sought, the coordination
group should determine the criteria for bringing the
consensus rounds to a close before the project begins.
Conclusions
• ‘Quick and dirty’?
• Delphi – a warning from history!
• Need to establish best practice