______________________________________________ Research

2.02 Survey Research Advantages and disadvantages of different survey methods

Dr. Domantay & Batch 2019 / October 13 & 27, 2016 ***study this table very well
Midterm Exams Self Self- Phone Interview in
Administered administered interview Person
Mail/Email in Person
Scenario: You want to conduct a survey as part of a research study. Cost ++ + - -
Time ++ + - -
1. What important steps should be undertaken BEFORE Standardi- + + +/- +/-
deciding on the survey method to use? zation
Formulation of a clear research question and specific Depth/ - - + ++
objectives. Detail
Guide the design of a survey questionnaire. Questions Response - ++ + ++
must be based on the research question and it should rate
follow a logical structure, it should not be chopsuey. Missing - ++ ++ ++
responses
It should be clear what part of the questionnaire
(+)ADVANTAGES; (-) DISADVANTAGES; (+/-) NEUTRAL
answers what research problem.
The need to know the answer.
3. How should questions be developed in terms of:
A crucial step is to review the literature.
A. FORMAT
Investigation of survey topics with focus groups
Open - ended questions vs. Close - ended questions
Survey topics may need to be investigated with focus
Open-ended questions are exploratory in
groups (composed of 6-10 members) in order to know
nature and provide rich qualitative data.
the specific issues that are important to study.
NO options
Done in person or over the telephone. Interview with
Permit the subject to respond in his or her
resource people / experts can also be helpful especially
when conducting a research from scratch. own words (primary advantage:
Conduct a focus group discussion before a survey, if the reporting using subjects words)
questions to be asked are still unclear Disadvantage: Lack the statistical
Done in order to generate the specific questions or the significance needed for conclusive research.
specific statements that the respondents will react to Open ended is kept at minimum
Closed-ended questions are conclusive in
2. What survey methods may be used? What should be taken nature as they are designed to create data that is
into consideration when deciding on the survey method to easily quantifiable.
use? More difficult to write, but payoff is the ease
with which the answers can be analyzed
A. Questionnaires
and reported.
A paper-and-pencil instrument
The usual questions found in questionnaires are Close ended is mostly utilized for survey, to
closed-ended questions, which are followed by minimize the writing of the respondent
response options. However, there are
questionnaires that ask open-ended questions to
explore the answers of the respondents.
Survey methods: according to how they are given
Self-administered (mail survey method)
However, since the response rates related to
mail surveys had gone low, questionnaires
are now commonly administered online, as in
the form of web surveys.
Group-administered
Household drop-off

B. Interviews Examples:
Are more personal and probing (provides the Open- ended questions
freedom to ask follow-up questions to explore the What habits do you believe increase a
answers of the respondents) persons chance of having a stroke?
Includes two persons: the researcher and ** Lack statistical significance
the respondent Closed- ended questions
Several survey methods Which of the following do you believe
Personal or face-to-face interview increase the chance of having a stroke?
Phone interview (Check all that apply.)
Online interview Smoking
Being overweight
Stress
Drinking alcohol
_________________________________________________________________________________________________
TRANSCRIBERS: ASBC, YES, Darth Vader Page 1 of 7
______________________________________________ Research
B. SCALE OF MEASUREMENT Interval
The precision with which a characteristic is show the order of things, but with equal
measured. intervals between the points on the scale
Determines the statistical methods for analysing the
data.
If open ended questions are used, it is recommended
that researchers collect data in as much detail as is
reasonable
If closed ended questions are used, the researcher
needs to decide the level of detail required in the
answers Ratio
Different forms of Measurement: The intervals between numbers are equal
1. NOMINAL Scales have an absolute zero
categories are in words and there is no
particular order
used to classify things into categories
weakest form of measurement
Examples: Gender, color, etc.
*** Form of Measurement is important because it
What brand of stethoscope do you trust determines the statistical test to be used in analyzing a
most? data from a survey.
1 = Littmann 3 = Omron
2 = Baxtel 4 = Welch Allyn C. BALANCE OF CATEGORIES
Questions using an ordinal response should provide
2. ORDINAL as many positive as negative options
categories are in words with particular A neutral option is also included to serve as a balance
order between the two positive and negative options.
Allow classification/ranking. However, the trend is to do away with the middle
Ordinal scales also imply rank ordering option so that you will force your respondent to choose
There is no difference between the either + or side.
importance of the choices Example:
Example: Severity / Stage of disease, etc.
How did you find the service of SLU-HSH
The information desk staff of SLU-HSH is: information desk?
1 - Extremely Helpful
2 - Very Helpful
3 - Moderately Helpful
4 - Not Very Helpful
5 - Not Helpful At All

**Usually mistreated as numerical (when

weighted mean is used) ; should be treated as
ordinal by just counting number or percentage of
respondents said helpful / not helpful, etc.
3. NUMERICAL
responses are in numbers
Traditionally divided into interval & ratio.
For practicality, we do not need to consider
them anymore because they use the same
statistical test anyway
Example: Blood glucose levels in mg/dL,
blood pressure, cholesterol levels, etc.
Analyzed using weighted means
D. AVOIDANCE OF VAGUE QUALIFIERS
Adverbs like usually that can mean different things
to different people. Using such words makes it
difficult to interpret the participants responses
because we cannot be sure how they interpreted the
qualifier.
Response categories must be quantified rather
than using always often never. We can
categorize the responses by using range. Instead of
using always we can use 6-10 times.

_________________________________________________________________________________________________
TRANSCRIBERS: ASBC, YES, Darth Vader Page 2 of 7
______________________________________________ Research
E. USE OF MUTUALLY EXCLUSIVE CATEGORIES Often used in health research (people, as they are
Response categories should be mutually exclusive, or complicated human beings, usually dont just have 1
independent. This means that none of the response choice, but have several choices.)
categories in a question should overlap or
categories must be distinct enough that no
observations will fall into more than one category.
It should be clear to respondents where a response
fits and which response category should be selected.
Respondents should not have to puzzle over
which category their response fits in to. This also
applies to questions where multiple responses are
allowed so that respondents' answers can be
accurately interpreted as a discrete response.
Example:
These questions can be tricky to analyze. The best
approach is to treat each option as a yes/no variable
and calculate the percentage of time each was selected.
Might be tedious, but this will make data analysis easier.

F. POTENTIALLY OBJECTIONABLE QUESTIONS

Some questions are viewed as too personal and
people hesitate to divulge information.
Income
Personal habits
Sexual activity
Two ways to deal with them: H. USE OF DONT KNOW
Softening the manner in which the questions are Dont know or Undecided category
asked. Dont ask straightforward Qs; Be creative No established consensus
Placing the question near the end of the Opportunity for the respondent not to commit an
questionnaire. Ask from the least sensitive to the answer (which some researchers do not like to provide
most sensitive Qs or hide them in the middle. while other point out that not providing the dont
Ask for less details. Instead of asking for a specific know option forces respondents to indicate opinions
amount, give the participant multiple options. It they do not really hold)
is generally better to obtain approximate answers
than none at all.

Which of the following broad

What was your total categories best describes your full
income from all sources income from all sources in 2003?
in 2003? a. P 10,000 or less
b. P 10,001 - 20,000
c. P 20,001 - 35,000
P _______________ total
d. P 35,001 - 50,000
income for 2003 e. P 50,001 100,000
f. More than P 100,000
Placed at the end of the question rather than the
G. USE OF CHECK ALL-THAT-APPLY ITEMS middle (increase in completion rates by 9%)
Often used when survey questions asks about the It might be better to use not applicable and omit a
qualities of a product or about the bases of selection middle anchor such as dont know or neutral
i.e. why consumers selected a specific This option is more applicable for factual questions
model/school/service (requires fact-based answers)
Items in which subjects can choose as many options
as they wish.
They do not force the subject to single out the best
feature.
_________________________________________________________________________________________________
TRANSCRIBERS: ASBC, YES, Darth Vader Page 3 of 7
______________________________________________
I. RANK VS. RATING
Ranking Questions
Asks survey respondents to compare a list of
different objects to one another
The researcher wants to know which choice is
viewed as the best by most subjects
Relative judgments against other similar objects
Compare two or more objects and make choices
among them
Selecting the top 3 choices also works well;
However, beyond that, many people begin to have
difficulty. For instance, some people omit one
number and/or rank two options with the same
number. Furthermore, although most people can
discriminate among a limited number of choices,
they have trouble when the list becomes long.
Example:
Please indicate the order of importance of these
subjects to you by placing 1 beside the most
important, 2 beside the next most important and
so forth.
__ Genesis of Medicine
__ Family and Community Medicine
__ Physiology
__ Biochemistry
__ Neuroscience
Rating Questions
Asks survey respondents to compare compare
different items using a common scale
Qualitative description of a limited number of
aspects of a thing or of traits of a person
Judge properties of objects without reference to
other similar objects
Three to seven points scales are generally used
More points on a scale provides greter sensitivity
of measurement
Disadvantage: responses often to cluster at one
end of the scale
Asks to indicate their personal levels of
agreement, satisfaction or frequency, etc.
Graphic Rating Scale
the rater indicates his rating by simply
making a mark at the appropriate point on
a line that runs from one extreme to the
other.
Itemized Rating Scale
Also known as numerical scale
Presents a series of statements from which
a respondent selects one as best reflecting
his evaluation. Ex. Likert scale - allow
answers that range from strongly
disagree to strongly agree, and so on.
_________________________________________________________________________________________________
TRANSCRIBERS: ASBC, YES, Darth Vader Page 4 of 7
Research
*** Issue on whether Rank or Rating is not yet resolved. As
much as possible, AVOID RANKING because it is more
difficult to analyze and might get a lot of responses, while
RATING/SCALE is easier to analyze.
J. USE OF SCALES
Precision with which a characteristic is measured
Decide the level of detail required in the answers
Helps determine which methods can be used to
analyze results
3 TYPES:
NOMINAL
No order is implied, often referred to as a
categorical scale
Simply a count of the frequency of the cases
assigned to the various categories.
No arithmetic properties and act only as
labels. This is simply a set of frequency
counts.
Knowing whether a subject agrees with the
question
ORDINAL
Used if knowing gradations of importance is
desired
Allow classification / ranking. It is the
order that matters
In a table where subjects are ranked base on
preference, the researcher knows the order
of preference but nothing about how much
more one subject is preferred to another,
there is no information about the interval
between any two subjects.
NUMERICAL
Open-ended question in which the subject
provides a number
ORDINAL and LIKERT SCALE
Ordinal
Likert
Ordinal psychometric measurement of
attitudes beliefs and opinions
In each question, a statement is presented
in which a respondent must indicate a
degree of agreement or disagreement in a
multiple choice type format.
Advantages:
- Most universal method for survey
collection
- Easily quantifiable
______________________________________________ Research
- Easily understood 4. What should be observed regarding the layout of the
- Accommodates neutral answer questionnaire?
- Quick, inexpensive, efficient A. Length
Shorter surveys are generally preferable than longer
Disadvantages: surveys
- Uni-dimensional (5-7 options of choice) B. Instructions
STANDARD is 5 Well-designed questions
- Each choice cannot possibly be equidistant, Place instructions where needed even if it
means repeating the instructions at the top of a
fails to measure the true attitudes of
continuation page.
respondents
C. Avoid branching questions
- Avoidance of extreme options although
Avoid skipping or branching questions (i.e. Question
they are most accurate
# 1 : Yes or No, if YES, proceed to number 23)
- ORIGINAL Likert uses agree and
Use directional arrows and other visual guides to
disagree assist the respondents
D. Order
There is an option to list the categories horizontally if Easier questions first
each question similar options: **More efficient in terms List questions in logical order so that questions about
of space, especially if the options are the same the same topic are grouped together.
E. Scales
Scale direction is listed in a consistent manner (i.e.
very important to least important)
F. Placement of demographic items
There is an option to list the categories vertically if (usually at the beginning)
each question utilizes different options: Age
Education
Ethnicity
Employment Status

5. How can the reliability of the questionnaire be

established?
RELIABILITY
Is the degree to which an assessment tool produces
stable and consistent results

TYPES OF RELIABILITY
A. Test- retest reliability
Obtained by administering the same test twice
over a period of time to a group of individuals
B. Internal consistency reliability
A.k.a. CRONBACHS ALPHA
Used to evaluate the degree to which different
test items that probe the same construct
produce similar results
Testing agencies sometimes use alternative
forms of a test in which the items on the tests
*** No consensus exists in the survey literature about the number
differ, but they measure the same thing and have
of categories to use. An EVEN number of categories, forces the
the same level of difficulty.
respondent to choose from two opposite options, even if he or she
Most Efficient form of reliability
is totally neutral (4 options strongly agree, agree, disagree, and
Advantage: Administer questionnaire once
strongly disagree). The preferred number of categories is usually
It is INTERNAL because there is a comparison of
ODD numbers, usually with either 5 or7 categories which has a
the odd and even responses (responses from nos.
middle or neutral option.
2,4,6,8,10 and from nos. 1,3,5,7,9)
C. Interobserver Reliability
*** There is an issue in removing the middle category, since tendency
Used to assess the degree to which different
measures/central tendency must be avoided (ODD.) Whereas the
raters/observers give consistent estimates of
EVEN category will force the respondent more to make a choice.
the same phenomenon
Used to assess the degree of agreement among
*** The choice of number of categories can have a major effect on repeated administrations of a diagnostic test
the conclusion made with the survey. performed by a single rater

_________________________________________________________________________________________________
TRANSCRIBERS: ASBC, YES, Darth Vader Page 5 of 7
______________________________________________ Research
D. Intraobserver Reliabilty More abstract and difficult to define - is
referred to nominal measurement (positive vs. generally established by using several
negative) instruments or tests on the same group of
E. Alternate forms Reliability individuals and investigating the pattern of
Measure of "equivalence" in reliability testing relationships among the measurements
Involves the development of a second equivalent
form of a tool to estimate consistency Measure of Validity
Disadvantage: 2 questionnaires are needed,
which measure the same thing Characteristics How Measured
Content Appropriateness of
Subjective
Types of reliability Validity content by experts
Type Characteristics How Measured Associated with
Criterion- Correlation between
Responses are stable Administer twice similar skills at
Testretest concurrent scores on instruments
over time (generally estimated) present time

Agreement among Criterion- Predict similar skills in

Internal Cronbach's alpha
items (measuring the predicative future
consistency (average correlation)
same thing)
It looks on point when
Face Validity Subjective
Alternative Different items Correlation between item seen by non-experts
form measure same topic scores Construct Correlation with other
Theoretical measure measures
Intraclass correlation or Validity
Intraobserver Observer is consistent
McNemar statistic
7. How should the survey be administered, in terms of:
Different observers Intraclass correlation or A. Conduct of a Pilot Test
Interobserver
agree kappa A pilot test is carried out after the questionnaire is
designed but before it is printed or prepared for
*** If the measurement is numerical, such as the score on a test or a administration
section of a questionnaire, intra- and interobserver agreement is May reveal
measured by the intraclass correlation coefficient which is sensitive reading level is not appropriate for the
to both random error and systematic error, also called statistical bias. intended subjects,
Statistical bias occurs when one observer always scores candidates some questions are unclear or are
higher by the same amount than another observer. objectionable and need to be modified,
instructions are unclear
6. How can the validity of the questionnaire be established? A large sample is not required to pilot test an
Validity is defined as how well an instrument measure instrument; it is more important to choose people
what it purports to measure who will provide feedback after completing the
4 Commonly used Measures: questionnaire.
A. Content Validity If a large number of changes are required, it may
Degree to which the items on the instrument are be necessary to repeat the pilot test with another
representative of the knowledge being tested or group of subjects.
the characteristic being investigated
B. Criterion Validity B. Attempts to increase response rates?
Instruments capacity to predict a characteristic increases confidence in the validity of the results and
that is associated with the characteristic the likelihood that the results will be used
Ideally, criterion validity is established by Send out additional mailings
comparing the measurement to a gold standard, If demographic information on the sampled group
if one exists is available, compare the sex, age, etc.
C. Face Validity 3-4 follow-ups at approximately 2-week intervals;
Degree to which a questionnaire or test appears each subsequent follow-up has a smaller yield (may
to be measuring what it is supposed to measure increase responses by 34%
In other words, a questionnaire about domestic Telephone follow-up with a shortened questionnaire
violence training should have questions related to With effort, it should be
that issue. possible to obtain responses
D. Construct Validity of 50% or more
Consists of demonstrating that the instrument is
If people are very interested in the topic, response
related to other instruments that assess the rates often approach 70%
same characteristic or not related to assess
85% or more in a highly selected sample or a
other characteristics.
captive population

_________________________________________________________________________________________________
TRANSCRIBERS: ASBC, YES, Darth Vader Page 6 of 7
______________________________________________ Research
C. Advance notification? she returns the questionnaire. Only the postcard and
Lets people know they will have a survey not the questionnaire itself contains any information
Methods: Letter, Telephone, Email that can be used to identify the respondent. This
Increase response rate by 7-8% practice permits the researcher to remove the
Content: responders name from the follow-up list, thereby
Who is doing the survey saving administrative costs and potentially annoying
Purpose the responder.
Why subject was selected
How results will be used Confidentiality
Whether responses will be anonymous, and Subjects always need to be assured that their
When the questionnaire will be mailed (or responses will be kept confidential
emailed) or the interview scheduled No response can be identified in any information
reported or otherwise communicated
D. Inclusion of a cover letter and return envelopes? Required by institutional review boards (IRBs)
Cover Letter If questionnaires can be identified with a number
Short or code, it is easier to know who has returned the
Relevant questionnaire and streamlines the follow-up
With letterhead process. Regardless, subjects always need to be
Signed assured that their responses will be kept
Content confidential. No individuals response can be
Purpose of the survey identified in any information reported or
Importance of response otherwise communicated. With the increasing
Use of gathered data protection for human subjects, confidentiality is
Option to share the results with the almost always required by institutional review
recipient (To maintain recipient anonymity, boards (IRBs) as a prerequisite to approving the
a separate postcard can be included for the survey.
recipient to return indicating a desire for a
copy of the results.)
Return Envelopes
Self addressed stamped envelopes
increase response rates by 69%
Business Reply envelopes
May seem less personal
Might suggest plain advertising to some
people
E. Provision of incentives?
Controversial
Higher response rates
MONETARY INCENTIVES
Modest incentive: Few dollars
- 16-30% increase in response rates
NON-MONETARY INCENTIVES
- 8% increase in response rates
MATERIAL INCENTIVES
Increases response rates: half as much of
the other incentives
LOTTERY TICKETS
Little effect on increasing response rate

F. Maintenance of anonymity and confidentiality?

Cover letter should contain information on
anonymity or confidentiality
It is advisable to make the returns completely
anonymous
Separate postcard containing responders
information: for identification purposes
Depending on the purpose and sensitivity of the
questionnaire, it may be advisable to make the returns
completely anonymous. In order to keep track of who
returns the questionnaire, the researcher may ask the
responder to mail a separate postcard at the time he or

_________________________________________________________________________________________________
TRANSCRIBERS: ASBC, YES, Darth Vader Page 7 of 7