Questionnaire Design &

Survey Methodology for

Medical Education Research
Proposals
ADE Medical Education Research
Faculty Development Program
February 14, 2006
Faculty Introductions

Wendy Cohn, Ph.D.

Jim Martindale, Ph.D.
Natalie May, Ph. D.
Lisa Rollins, Ph.D.
Goals of Session I

How to write a questionnaire

Best ways to administer a survey
Measurement and Analysis
Q&A
Prepare for Session II
What You Must Consider
What is the question?
Has the question been addressed previously? If so,
how?
What is the appropriate methodology?
Whom do you need to sample and how?
What do you need to ask?
How should you ask it?
Is there an existing survey/validation?
How should the survey data be analyzed?
How should the results be reported?
Finding Your Question

Begin with a purpose!

For example: The purpose of this study is to
test (the theory) by relating (the independent
variable) to (the dependent variable) for
(participant group) at (the research site).
Finding Your Question
(contd)
Pose a question
Begin with how, what, or why
Specify your variables
Use words to describe the connection
between/among variables (describe,
compare, relate, etc.)
Indicate the participants and setting
Conceptual Survey Outline

A conceptual map of your questionnaire.

Helps guide the development of your
questionnaire so that you include relevant
questions and exclude irrelevant questions.
Key elements:
Concept; term; operational definition;
measurement; priority
Types of Questions

Dichotomous Questions
Nominal Questions
Ordinal Questions
Interval/Ratio Questions
Filter or Contingency Questions
Question Formats

Structured
Fill-in the blank
Rating
Likert Scale
Check all that apply
Unstructured
Open ended question
I oppose electronic prescribing and
electronic medical records for
patients.

SA A N D SD
Which of these five statements best
describes your chairman?
Innovative but lacking in leadership
qualities
About the same on innovation and
leadership qualities
Stronger on leadership than innovation
A born leader
A real innovator
 How do you feel about
healthcare in the United
States?
About how many books have you
read for leisure during the past
year?

___________ Number of books

 Dont you agree that the new
resident work hour regulations are
limiting to your education?

SA A N D SD
 How supportive is your spouse about
your efforts to quit smoking?

Very Supportive Somewhat Not

at all
supportive supportive
supportive
If you fixed dinner at home last
night, did you eat meat as part of
that meal?
Yes
No
Please identify your insurance
carrier:

Aetna
Blue Cross
Health South
Self Pay
Do you incorporate obstetrics/
gynecology into your practice?

SA A N D SD
Do you favor or oppose not
allowing the state to raise taxes
without approval of 60% of the
voters?
Favor
Oppose
Summary of Typical
Problems
Double-barreled questions
Vague questions
Leading questions
Premature assumptions/bias
Insufficient alternatives with forced-choice options
Answer options dont fit with the question
Double negatives
Issues to Consider Regarding
Content
Is the question necessary/useful?
Are several questions needed?
Do respondents have the needed info?
Does the question need to be more specific?
Is the question biased or loaded?
Is the question asking about sensitive
information?
Layout Issues

Self-explanatory
Visually clear and uncluttered
Group tasks/types of questions
Consistent and clear response options
Layout Issues (contd)

Consider question placement

Skip patterns kept to a minimum (if needed,
use arrows and boxes)
Beware of using double-sided pages
There is elegance in simplicity!
Tips to Reduce Non-
Response
Professional, personalized, attractive, easy to
complete
Tasks/directions should be clear
Easy to read, uncluttered, visually appealing
Response task should be easy
Length of form
Survey Implementation

A well designed questionnaire is critical

however:
Implementation procedures have a greater
influence on response rates.

Experimental research has identified the

factors that influence survey response.
Choices for implementation

Telephone
Costly; need expertise
Web
Efficient; cheap; limited range of question types;
hard to apply full range of implementation
procedures
Mail
Very well studied; moderately costly; can do well
Multi-method
Can be done but complicated; get help
Implementation procedures that
affect response
Multiple contacts
Contents of letters
Appearance of envelopes
Incentives
Personalization
Sponsorship & its explanation
The five most important
elements for achieving high
response
rates questionnaire
Respondent-friendly
Four contacts by first class mail
Pre-notice letter; questionnaire; thank you
postcard; replacement questionnaire; final
Return envelopes with real first class stamps
Personalization of correspondence
Token prepaid financial incentives
First Contact:
Pre-Notice Letter
Provides positive notice that the recipient will
be receiving a questionnaire
Important characteristics:
Brief
Personalized
Positively worded
Aimed at building anticipation vs. providing too
many details
Sent about 1 week in advance
Second Contact:
The Questionnaire Mail Out
Cover letter (1 page; date; purpose of letter; why
request is important; confidentiality; voluntary
participation; enclosures of stamped envelop and
incentives; who to contact with questions
Questionnaire
Return envelope with stamp
Assembling the packet
Third Contact:
The Postcard Thank
You/Reminder
Written to remind participants that a

questionnaire was sent to them

Elements:
Reminder that questionnaire was sent
Thank you to those who have returned; request
for others to do so.
Invitation to ask for a replacement questionnaire.
Fourth Contact:
The First Replacement
Questionnaire
Elements:

Havent heard from you

Others have responded and answers are
important
Eligibility
Confidentiality
Voluntary
The Fifth Contact:
Special Procedures
Less frequently done in practice
Last attempt; most intense
Certified mail
Telephone
Sampling

The survey population consists of all units

(e.g. households, individuals) to which one
desires to generalize survey results.
The sampling frame is the list from which a
sample is to be drawn to represent the
population.
A sample are all units of the population that
are drawn for inclusion in the survey.
Probability Sampling

Get help!
How large of a sample do you need?
How precise do you need your estimates?
Size of the population
How varied the population is with respect to your
characteristic of interest
Amount of confidence you wish to have in the
estimates of the entire population.
What Am I Measuring?
A Look At Reliability
Reliability has to do with the quality of
measurement. Practically speaking, reliability
is the "consistency" or "repeatability" of your
measures.
Internal Consistency is one type of reliability
measure. Cronbach's alpha measures
internal consistency by how well a set of
items (or variables) measures a single uni-
dimensional latent construct.
What Am I Measuring?
A Look at Validity
Construct-the degree to which inferences can
legitimately be made from the
operationalizations in your study to the
theoretical constructs on which those
operationalizations were based.
External-the degree to which the conclusions
in your study would hold for other persons in
other places and at other times.
Summary Points

There are a lot of things to consider when

developing surveys
Leave enough time for the process
See if it has been done before
Get help if you need it
No matter what, another set of eyes can be
helpful
Length and simplicity
Next Session (Feb. 21)

You bring:
Questionnaire drafts
Research questions
Ideas
Questions

Well bring:
List of resources
IRB contacts & information
Other Ideas for Final
Session?
Contact Information

Wendy Cohn wfc2r@virginia.edu

Jim Martindale jrm7e@virginia.edu
Natalie May nlb7r@virginia.edu
Lisa Rollins lkr2h@virginia.edu