So Now You Want to Do a

Survey...
Words of Advice, Words of Caution

Jay P. Paul, PhD

UCSF Center for AIDS Prevention Studies
Key Questions To Consider
When Asking Questions
 Is this item understandable to the
participant?
 Is this item something the participant can
answer?
 Is this item something the participant will
be willing to answer honestly?
 How important is this item compared to
others I have to avoid respondent
burden?
 Areas to Consider in
Designing Your Survey
What type of survey methods are
out there and what can they do for
me?
What are the things to consider in
maximizing the quality of data with
respect to questions I ask?
How do I consider using existing
scales and putting it all together?
Modes of Survey Delivery
 Survey Method Options
IAQ
SAQ
Including Diary Methods, Mail Surveys
CASI/CAPI (and A-CASI)
CATI
T-ACASI
Internet
 Interviewer-Administered
Questionnaire (IAQ)
 Most flexible and responsive to respondent
re: comprehension of question, selection of
response options.
 Able to handle skip patterns that could
confuse the respondent in a self-
administered questionnaire.
 Human element, rapport & responsiveness
also can introduce problems in lack of
standardization of delivery, self-
presentation.
 Self-Administered
Questionnaire (SAQ)
 Allows for greater privacy than face-to-
face interviewing
 Tends to be associated with higher rates
of reporting of sensitive behaviors
 Literacy requirements for respondent
 Cannot be adjusted to be responsive to
needs of individual participant
 Cannot handle complex “skip patterns”
 Low-cost survey delivery option
 Diary Methods
 Provides optimal potential for recording of accurate rates
of target behaviors
 Recall
 Exploration of variations over time
 Allows for question design to examine co-occurrence of key
behaviors (e.g., sexual behavior and drug use)
 Burden of consistent completion of information means that
data may be missing in possibly non-random pattern, or
completed at a later time than designed.
 NOTE: Can be useful to design for Web-based
implementation.
 Mail Surveys
 Many of considerations of SAQ are applicable -- with
format even more crucial.
 Many respondents report liking the flexibility of
completing surveys at their own pace.
 Need for clear and easy format to maintain
respondent interest and ease of completion.
 Critical to have contact information and clear
procedures to follow up on respondents to ensure
adequate response rates.
 Computer-Assisted Self-
Interview /Personal Interview
(CASI/CAPI)
 Survey is delivered via computer (can
be laptops for maximal flexibility), with
respondent keying in responses.
 Usual mode is “Audio-CASI” or A-CASI,
which involves respondent listening to
audio recording of survey as they see
items on-screen.
 For special PCs with touch screens, can
avoid keyboard and use fingers to select
choice by touching on-screen.
 CASI/A-CASI Advantages
 Privacy
 Branching in complex questionnaires
 Automated consistency checks
 Automated range checks
 Automated adaptation of question
wording based upon prior responses
 Audio-CASI allows for standardized
verbal delivery of all questions and does
not require respondent literacy
 Computer-Assisted
Telephone Interview (CATI)
 List samples of “live” residential phone lines are
called to screen households and determine
eligibility of members.
Once consent is obtained, interviewer uses a
computer to guide them through asking survey
questions, providing prompts, entering data, etc.
as in A-CASI.
Interviewer also has some flexibility in dealing
with respondent comprehension/literacy level.
 CATI Advantages
 Opportunity to access representative
sample (costly with eligibility
requirements that are less common).
 Some advantages of live interviewer,
but also anonymity of phone call.
 Has capacities of computer-assisted
technology with respect to branching,
adaptation of wording,
consistency/range checks, etc.
 Telephone Audio Computer-
Assisted Self-Interview (T-
ACASI)
 First a telephone interviewer contacts a
household, screens for an eligible
respondent, and secures consent.
 Phone call then transferred to automated
system where computer-controlled, pre-
recorded questions are read aloud.
 Respondent provides answers by
pressing keys on a touch-tone phone.
 At end, respondent can be returned to
live person to close out interview.
 Internet Surveys
 Access to large sample at relatively low cost in relatively short
time frame.
 May provide access to difficult-to-reach populations.
 As with other computer-assisted methods, may result in higher
rates of reporting of sensitive information, and provides
consistency and validity checks.
 Rates of internet access are growing (if still disparities exist in
different populations).
 Participation rates of target population hard to gauge
 Dropout rates
 Direct data entry into database facilitates cleaning and analysis.
Summary: Survey Delivery
Considerations
Mode of survey delivery clearly influence:
Respondent comfort and likelihood of self-disclosure
(due to level of privacy)
Respondent ease of navigating through the survey if
there are any skip patterns
Time and effort expended per respondent
The audience you can reach with your survey
Quality of data, data entry and analysis
While important, how you actually ask the
questions is just as critical -- our next focus.
Survey Construction
 Developing the Survey:
Question Organization
 Topics should be generally organized
from least to most intimate
 Initial questions are crucial -- respondent
must get “hooked” by at least some
questions which are felt to be personally
salient
 Initial questions must also require low
effort (cognitive demands) to answer
 Thus, if possible, avoid demographic
questions to start (especially income!)
 Developing the Survey:
Question Organization
 A survey -- no matter how delivered -- is a type of
conversation.
 Topics should follow an orderly sequence, with
common questions grouped together.
 This flow allows transitions to more intimate matters
without jarring or acute awareness.
 Transitional phrases can help in shifting set.
 All this also provides respondent with notion that
there is a rationale to questions asked.
 Question Ordering
 Be careful with filter questions: if not
worded carefully, respondents will learn
that a “no” response means avoiding
subsequent questions.
 Be careful about summary attitudinal
questions: more specific queries to
begin will influence summary responses
by forcing respondent to think about
their overall beliefs more carefully.
 Question Ordering
Having similar response choices reduces
cognitive burden on respondent; grouping items
with such options together is thus useful. It can
also be useful to have the same number of
options to reduce complexity (e.g., always using
a four-point or five-point scale).
However, arrange types of questions to provide
some variety and reduce response set.
Avoid asking unnecessary questions!!
 Question
Wording/Construction
Comprehension/Interpretat
ion
Not just question of literacy, but influenced by
cognitive burden and attention required
Wording choices
Length and complexity of survey items
Ordering of clauses
Consistency of phrasing
Avoid “not’s” or items whose meaning will be
lost by inattention to a single word
Phrasing also influences response distribution
along a Likert-type agree-disagree scale
 Cultural Considerations
 Individuals can come up with very different
understandings of survey items, based upon their
personal experiences.
 The meanings that respondents bring to the survey
experience can be strongly influenced by culture,
acculturation and other background variables -- not just
education and economic status.
 These differences are highlighted in the vast literature
that exists on different response patterns of different
ethnic groups to the same standardized scales. (For a
review of some issues, see Pasick, Stewart, Bird &
D’Onofrio, 2001.)
 Key Considerations
 Examine all survey items with eye for their
appropriateness for target population.
 Be aware of pre-existing measures so as to not
have to reinvent the wheel.
 When using pre-existing scales, review in same
way for clarity & ease of response.
 Get feedback on item wording and construction
from as many colleagues as possible.
 Pre-test survey, using methods first described by
Cannell et al. (1989).

Cannell C, Oksenberg I, Kalton G, Bischoping K & Fowler F. (1989). New Techniques in Pretesting
Survey Questions (NCHSR #HS 05616). Survey Research Center, University of Michigan.
 Cognitive Burden & Recall
Demand
 Recall accuracy is influenced by a variety
of factors, including:
salience/vividness of behavior,
time frame,
complexity of response information necessary
(ever, counts, close-ended categories, etc.),
frequency and consistency of behavior,
effort respondent is willing to exert,
cueing techniques and prompts
Time Frame - Sexual/Drug
Use Behaviors
 3 month time frame generally considered the
outside range of accurate reporting; some
suggest limiting to 1 month.
 Longer time frame may be possible if asking
if a behavior ever happened.
 One important timeframe consideration is
dealing with infrequent behaviors -- what do
you want to capture with respect to either
behavior events or possible behavior change
(if longitudinal study)?
 Sensitivity
 Item sensitivity not consistent across populations
(e.g., sexual behavior among MSM vs. general
population, drug use behavior among heavy
drug abusers/IDUs, discussing sexuality among
different ethnic groups).
 Rationale for sensitive questions helps response
rates.
 Asking if “ever” occurred first, prior to asking
about (for example) the last month, may be less
charged for respondent.
 Importance of neutral wording of question and (if
IAQ) careful training of interviewer
 In Closing ...
 Selecting your methods of data collection must be
carefully considered based upon target population,
study questions and design, and costs.
 No matter what the method of data collection, your
instrument is critical.
 Survey design is a combination of art and science,
with the “art” strongly reliant upon interviewing skills.
 The science of survey design currently has a wealth of
research literature to examine and guide you.