SURVEY

DEVELOPMENT

PRESENTATION
OVERVIEW
What are questionnaires and what do they

measure?
Objectives in questionnaire research
Advantages/Disadvantages
Main parts and formats
Dos/Donts of questionnaire writing
Questionnaire administration
Issues in questionnaire research
Ethical principles of questionnaire research

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

WHAT ARE
Questionnaires are any written instruments
QUESTIONNAIRES?
that present respondents with a series of
questions or statements to which they are
to react either by writing out their answers
or selecting them from among existing
answers. (Brown, 2001:6)
Referred to by different names
Inventories, forms, opinnionaires, tests, batteries,
checklists, scales, surveys, profiles, indexes
(Aiken, 1997)

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

WHAT DO
QUESTIONNAIRES
Questionnaires (Qs) can yield different
types of information (Dornyei, 2003)
MEASURE?
Factual questions
Find out information about who respondents are;
demographics

Behavioral questions
Find out what respondents are doing or have done in
the past

Attitudinal questions
Find out what people think
Attitudes, opinions, beliefs, interests & values

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

OBJECTIVES IN
QUESTIONNAIRE RESEARCH
To obtain accurate and relevant information/data
To maximise the response rate (the proportion of
subjects answering the Q)

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

DECIDING WHAT TO ASK

Information we are primarily interested in (dependent
variables)
Information which might explain the dependent
variables (independent variables)
Other factors related to DV and IV factors which may
distort the results and have to be adjusted for
(confounding variables)

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

ADVANTAGES
Unprecedented efficiency in terms of researcher time,
effort and financial resources (Dornyei, 2003)
Versatility can be used successfully with a variety of
people in a variety of contexts targeting a variety of
topics
Collect a large amount of data in a short time (Brown,
1988)
Easier and less expensive than other forms of data
collection (Seliger & Shohamy, 1989)
Can be used to research any aspect of teaching and
learning (Nunan, 1989)
Can be easily used in field settings such as classrooms
(Nunan, 1992)

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

DISADVANTAGES
Simplicity and superficiality of answers
Qs must be understood by all; written in
simplistic way; cant probe deeply into an
issue; results in superficial data (Moser &
Kalton, 1971)

Unreliable and unmotivated respondents

Results vary greatly from person to person

Respondent literacy problems

Assumption that all can read and write well

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

DISADVANTAGES
Social desirability or prestige bias
People do not always provide true answers
about themselves
Over reporting good behavior or under
reporting bad or undesirable behavior
Happens because Qs are often transparent
(i.e. Rs have a fairly good idea of what the
desirable/acceptable answer is)
Presenting ourselves in a good light is a
natural human tendency

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

DISADVANTAGES
Acquiescence bias

Tendency for people to agree with sentences when

they are unsure or ambivalent
Go along with anything that sounds good

Halo effect

One's judgments of a persons character can be

influenced by one's overall impression of him or her
Concerns the human tendency to overgeneralize
If our overall opinion about someone/thing is positive
we are disinclined to say anything negative

Fatigue effects

Kicks in if Q is overly long

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

MAIN PARTS OF
QUESTIONNAIRES
Title

To provide Rs with initial orientation & activate content

schemata (Dornyei, 2003)
Qs with titles are generally perceived as more
serious/credible than those without

Instructions: Two Types

General: opening greeting at beginning of Q
Should contain
What the study is about and why its important
Organization responsible for conducting study
Emphasis that there are no right or wrong answers
Promise of confidentiality
A thank you

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

MAIN PARTS OF
QUESTIONNAIRES
Instructions

Specific
Explains and demonstrates how Rs should answer
questions
Each new task type requires instructions

Questionnaire Items
Additional Information

Contact details of researcher; how Qs should be

returned; note promising a copy of results;
invitation for follow-up interview

Final thank you

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

OPEN OR CLOSED
FORMATS?
Responses can be in open or closed formats
Open-ended formats
Rs formulate their own answers

Closed formats
Rs are forced to choose between several given options

It is possible to use a mixture of these formats

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

OPEN OR CLOSED FORMATS:

ADVANTAGES
Open-ended formats

Allows exploration of
the range of possible
themes arising from
an issue

Closed (or forced)

choice format

Easy and quick to fill in

Minimise
discrimination against
the less literate or the
Can be used if a
comprehensive range less articulate
of alternative choices Easy to code, record,
cannot be compiled
analyse results
quantitatively
Easy to report results

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

CLOSED OR FORCED CHOICE

FORMATS
Choice of Categories

What is your marital status?

Circle/tick: single, married, divorced, separated

Likert Scale

LSS 2533 Research Methods is an interesting

course.
Circle/tick SD D NA or D A SA

Differential Scale

How would you rate this presentation?

Very Boring 1 2 3 4 5 6 7 8 9 10 Very interesting

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

CLOSED OR FORCED CHOICE

FORMATS
Checklists
Circle the leisure time activities you are particularly interested in
Video games
Watching TV
Playing football
Reading
Going to movies
Going to the beach

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

CLOSED OR FORCED CHOICE

FORMATS
Ranking
Please rank your abilities in English. (1=best;
5=worst)
Listening
Reading
Writing
Speaking
Grammar
Vocabulary

Probably the least used format because of

difficulty with analysis

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

OPEN-ENDED QUESTIONS
Specific open questions
Ask about concrete pieces of information
(i.e. facts, past activities, preferences)
How many languages have you studied?

Clarification questions
Questions that are so important you need
follow-up information
If you rated the textbook you are using as
poor or very poor, please briefly explain
why

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

OPEN-ENDED
QUESTIONS
Sentence completion questions
One thing I liked about this class
was
I found this activity
Short-answer questions
Exploratory inquiry about an issue
What was it that you found most
useful about this class?

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

LENGTH?
General temptation is always to cover
too much ground by asking everything
that might turn out to be interesting
Experts agree that anything over 4-6
pages and requires more than 30 min
to complete is too long
As a general rule, long questionnaires
get fewer responses than short Qs

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

QUESTION
ARRANGEMENT
Go from general to particular
Go from easy to difficult
Go from factual to abstract
Start with closed format questions
Start with questions relevant to the
main subject
Do not start with demographic and
personal questions

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

HOW TO WRITE GOOD

ITEMS
Most problems can be traced back to
the design phase

Ask only Qs that address the study goals

Use short and simple sentences.

less confusing & ambiguous
most sentences should contain one/two clauses & not exceed 20
words.

Ask for only one piece of information

at a time.
Please rate the lecture in terms of its content and presentation
asks two questions.

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

HOW TO WRITE GOOD

ITEMS
Ask respondents to rate both positive and

negative statements
tendency to mark every item at the same
end of the scale
by offering positive/negative responses Rs
must evaluate every statement
Begin with non-threatening and interesting
items
Leave space for comments
Place most important Qs in the first half of Q
Ss often send back partially completed Qs

Hold the Ss interest

provide a variety of item types; this will prevent

response sets

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

WHAT TO AVOID
Avoid scale point proliferation
Never/rarely/occasionally/fairly
often/often/very often/almost always/always
Research has shown that most Ss cannot
reliably distinguish between more than six or
seven levels of response
four to five scale points are sufficient
Distinguish between a neutral and no
response
when you dont want to have a neutral
option you can sometimes have a prefer
not to answer, not applicable, or no
basis for judgement

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

QUALITIES OF GOOD
QUESTIONNAIRES
Brings out the truth

Qs must be non-threatening
if your Q has sensitive items, be sure to
clearly state your policy on confidentiality

Ask for an answer on only one

dimension

ex. Where you satisfied with the time and

location of your class?

Can accommodate all possible answers

Has mutually exclusive options

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

QUALITIES OF GOOD
QUESTIONNAIRES
Produces a variety of responses
Follows comfortably from the previous
question

transitions between Qs should be smooth

grouping Qs that are similar will make the Q
easier to complete

Do not presuppose a certain knowledge

Do not imply a desired answer
Dont you think that the textbook should be
changed?

Do not use emotionally loaded or

vaguely defined words; unfamiliar words
or abbreviations
ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI
MEN'S COLLEGE

QUESTIONNAIRE
ADMINISTRATION
Sample

Difference between sample and population

Different types of samples

Convenience/opportunity: most common

Snowball: where researcher asks a group to
recommend others who have similar qualities
Quota: defines certain subgroups within a
population and samples according to population
(i.e. male vs females in UAE tertiary education)
Random: selection by random basis; said to be
the most powerful sampling technique

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

QUESTIONNAIRE
ADMINISTRATION
How large should the sample size be?
No hard/fast rules
10% of population is generally ideal
recommendation
Basic requirement is normal distribution
which equates to at least 30 Ss (Hatch &
Lazarton, 1991)
For statistical significance at least 50
For multivariate statistical procedures like
factor analysis at least 100

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

ISSUES IN QUESTIONNAIRE
ADMINISTRATION
Most common way to administer is by
mail or online

The cover letter/cover email

Necessary to sell the Q & create rapport
with Rs
What they should include?

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

ETHICAL PRINCIPLES OF
QUESTIONNAIRE RESEARCH
Five Ethical Principles (Oppenheim,
1992; Sudman & Bradburn, 1983)

No harm should come to Rs as a result of

their participation
Rs right to privacy should always be
respected
Rs should be provided with sufficient info to
complete the Q
In the case of children, permission should be
sought from caretakers
Researcher should not promise a higher
degree of privacy and confidentiality than
he/she can deliver

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

ISSUES IN QUESTIONNAIRE
RESEARCH
Rs that tend to sit the fence

relevant for Likert scales with odd numbered response options

Sensitive issues
Difficult to obtain truthful answers to sensitive questions.

Controversial topics
Various biases in research

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

CONCLUSION
Qs must be carefully designed to yield
valid information.
Attention must be paid to ensure that
individual questions are relevant,
appropriate, intelligible, precise and
unbiased.
The order of Qs must be carefully
arranged and the layout of the Q must
be clear.

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE

FINAL THOUGHTS
In writing questionnaire items
no amount of textbook admonition
can take the place of common
sense.
(Moser & Kalton, 1971:310)

ADAPTED FROM DR. CHRISTINE COOMBE'S PRESENTION AT DUBAI

MEN'S COLLEGE