Surveys

A. Some background information

– The basic component of surveys consists of QUESTIONS THAT PEOPLE ANSWER however
sampling and analysis are also fundamental parts of survey research
– Arguably the most popular data collective strategy in the social sciences
– Victim Surveys and Offender surveys represent major data collection efforts in criminology
and criminal justice

BASIC MODEL: **test , describe these steps, provide answers

ASKING QUESTION -> UNDERSTANDING QUESTION -> RETRIEVING INFORMATION -> REPORTING
ANSWER

Answering survey questions

1. Understanding of questions

A. Interpretation and meaning

Example : Are the 2010 Winter Olympics good for Vancouver?

– “good” in what sense?

– “Olympics” does this include the para-olympics?
– “Vancouver” Vancouver people? Greater Vancouver? British Columbia? Canada?
– Is the meaning/interpretation affected or dependent upon previous questions? (i.e. if all
the questions before this question dealt with the homelessness problem in Vancouver
would they affect the answer?

A. Information Processing
– Is respondent being asked to think about actual experience or hypothetical situations?

1. Retrieving/constructing responses
– Memory
– Mental capacity
– Qualified to answer **

Memory

– If the time frame is from jan.1 –dec.31 (what crime have you been a victim of)
– Forward telescoping – talking about events that occurred before Jan. 1 acting like they did
happen after jan1
– Keep time period as small as possible for accuracy or relating it to other events (it
occurred before my birthday , after my anniversary

Mental capacity

– Wouldn’t ask a kindergartener about the economic status in Canada

 – Drug users who are high won’t have

Qualified to answer**

– How old were you when you started to walk? How old were you when you didn’t need
diapers? Wouldn’t ask the person who experienced them

1. Reporting answer
– Social and conversational norms (e.g. social desirability- many will not disclose that they
might think that would make them look bad)
– Confidentially and anonymity (helps get a more accurate answer without worrying about
social desirability, and insurance like a consent form)

TYPES OF SURVEYS

Distinction based on the Mode of Surveying/how it is carried out.

Self administered – given survey and done by yourself

Interviews - surveyor asks you questions and they write down your answer

Half – interviewer asks questions but you write down answers

Self administered |-----------------------|-------------------------|Interviews

– Mailed Half - Face to Face

– Emailed - Telephone
– Drop off

QUESTIONNAIRES

Mailed surveys

– At one time a common type of self-administered survey especially consumer/marketing

research
– Can be effectively used with targeted groups (e.g. membership lists for a golf club)

Some basic considerations:

a. Cover letter
– Describes the study
– Has to get people interested in participating in the study

Enlists respondent’s cooperation/participation

– Stresses the importance of the study

– Stresses the importance of the respondent

Instructions

– How to complete questionnaire

 – How/where/when to return completed questionnaire
a. Typically LOW response rates

Non-respondents could be distinct in some way

Factors affecting participation/response rates:

– Who’s doing it
– Inducements to participate (awards, token of appreciation, being entered into draws)
– Characteristic of respondents (group member/non member, age, gender, ethnicity, social
class)
– Mailing date
– Follow-up procedures (reminder letter, additional letter and fresh questionnaire, telephone
call!!!) got to do a lot of work to get the most and best responses possible
– Confidentiality/anonymity

INTERVIEW SCHEDULES

1. Role of interview (can include)

a. Locate and enlist cooperation of respondents
b. Conduct “good” interview
i. Motivate respondent
ii. Answer questions/clarify confusion
iii. Using PROBES to elicit address information
iv. Accurately recording information (not easy)

1. Training of interviewers (must include)

a. Knowing key aspects of study (objectives, sponsors, sampling strategy etc.)
b. Interviewing basics (how to ask questions and record answers)

Telephone Surveys

two reasons behind widespread use:

– Almost all households had at least ONE phone

– Use of computers (CATI – computer assisted telephone interviews)
a. Random digit dialling
b. Standardized script (present interviewer with everything he or she will say and
they’ll read the script) allow you to build in an enormous amount of complexity
in the script that not even the interviewer can screw up. For example, if you’re
asking if someone has been a victim of a crime or not, depending on the answer,
it’ll send the questions to a different part of the script
c. Direct data entry

NOTE: the methodology used by Statistics Canada to conduct victimization surveys

Face to Face interview

– A direct meeting between interviewer and interviewee

– Some see this as the GOLD standard for survey research because of the high quality data
that can be generated
 – But it has disadvantages

Comparing Face to face interviews, phone surveys, and self-administered questionnaires

Face to Face Phone Quest.

Cost High Low/med Low/med

Response rates High Low Low/med

Time to answer Med/high Low/med High

Complex questions High High Low/med

Educational Bias Low Low High

Safety (teacher) High Low Low

Interviewer bias High High Low

****

Cost

– Includes travelling , experience, etc.

Response rates

– higher in face to face generally, easier to get out an answer out of

Time to answer

– Phone interview have to ask questions that really shouldn’t take a lot of time, tied to
response rates . Very concise questions (because it’s easier to terminate interview
– face to face/questionnaire more lengthy amount of time to answer questions, different
questions

Complex questions

– higher in face to face and phone because of the way the questions are formatted as
opposed to questionnaire where it’s very strict

Educational bias

– Quality and completeness of answers is closely tied to the level of education the person
has.
– In questionnaire the people who can read and write and understand correctly will give
much better responses
– In face to face/phone, you can get help whereas in questionnaires you really can’t

Safety

– If the interviewer is at risk of harm

*** Ways in which the interviewer affects the data

a. During the interview

○ Changing the word order of questions (HAVE YOU EVER COMMITED SUICIDE)
 ○ Adding words to questions (you haven’t committed suicide, have you?)
○ Forcing answers (calling someone a liar)
○ Recording errors (e.g. making up answers by the interviewer)
a. Interviewer characteristics
○ Race/ethnicity (same race as the interviewee or different)
○ Gender ( male interviewing females on sexual assault)
○ Age (young person asking old person about some questions)
○ Physical appearance (clothing, tattoos)

Types of surveys based on level of standardization/uniformity

|----------------------------------------- |----------------------------------------- |

Unstructured semi-structured structured

Question Structured

1. Closed-ended questions ask respondents to choose among a set of fixed or predetermined

set of answers

Have you used marijuana in the past year?

a) No b) Yes

How often have you used marijuana in the past year?

Never 1

1 or 2 times 2

3 or 4 times 3

5 or more times 4

Might not be enough precision saying 5, (some might use it 50-100 times a year)

1. Open ended questions ask respondents to supply their own responses and in some cases
to record his/her responses

e.g. how often have you used marijuana in the past year?

____________________________________________________________

Comparing closed ended questions and open ended questions

Closed Open

Facilitates completion Yes No

Known responses must be Yes No

known
Complex and subtle No Yes
responses

Response set or bias Yes No

Add new information No Yes

(exploration)

Education bias No Yes

Classifying/coding No Yes
responses

Known responses must be known

– Don’t get responses you don’t anticipate

Complex and subtle responses

– A B C D on whether you smoked pot vs. Open ended where you can talk about it

Response set or bias

– Based on responses you can see in their answers that the answers don’t make sense since
they’re getting lazy creating a pattern

Add new information (exploration)

– Ask question and let them answer themselves opposed to closed ended abcd

Education bias

– Persons level of education affects answers

Classifying/coding bias

– If you ask someone to describe in an open ended question what work you do for pay, got
to be required to code all the responses to numerical values to analyze the numbers. The
responses can vary greatly by job and could make it really hard to code everything

Contingency questions

– Questions that apply to a subgroup of respondents based on answers to “filter or

screener” questions
– Filter/screener questions direct respondents to next relevant

Have you used marijuana in the past year?

1. No- skip to Q 5
2. Yes- go to next question
3. How many times?
1 or 2 times 1
3 or 4 times 2
5 or more times 3
Matrix questions

– A set of closed ended questions that have the same response categories

How afraid are you that someone will.... 1 not at all, 2 moderately, 3 extremely

Break into your house

Break into car

Steal your car

Etc.

Problem – response sets

General guidelines to writing questions

1. Keep the wording as simple as possible (if necessary provide definitions)

2. Keep the questions short
3. Avoid double barrelled questions
– Asking two questions as one question (i.e. do you think the police do a good job in
informing the public and enforcing the law?)
1. Avoid hypothetical questions (poor predictors of actual behaviour/experiences)
2. Don’t push the limits of what people can remember (how did you feel over the past year?
Happy, sad etc)
3. Avoid negative questions
– Ie. The police should not carry guns? Agree/disagree
1. Response categories must be mutually exclusive
2. Response categories must be exhaustive
3. Avoid leading questions (wording that encourages the respondent to answer in a particular
way)
– i.e. you don’t agree that the police should carry a gun, do you?

Question placement (depends on mode of data collection)

1. begin with non sensitive demographic questions

2. questions of major research interest next
3. “sensitive” items should be last (if what you’re studying is highly sensitive, it should be
moved up)

Get as much information as you can in the major research section before people start bailing

Question patterning

a. Funnelling sequence- moving from general to narrow

b. Inverted funnelling sequence – moving from narrow to general
Example:

Would you say that the police in Canada do a good job, average job, or a poor job? (scope is all
of Canada)

Would you say that police in your city do a good job? (scope is the city)

Would you say that police in your neighbourhood do a good job? (scope is neighbourhood)

SELF REPORT SURVEYS (offenders)

– One of the three (3) major ways of measuring delinquent and criminal involvement
– Basic approach is to ask individuals about disreputable behaviour
– First published findings of self report surveys in mind 1940’s
– James Short and Ivan Nye (1957) carry out first methodologically sophisticated survey
○ Used scale construction, assessed measurement reliability and validity, probability
sampling
○ Multi level scales

Early studies focused on [two distinct but related]

○ Measuring the ‘Dark figure’

○ Relationship between SES and offending

***Early studies show that almost everyone as an adult has at some time committed a very
serious act

*** the early studies also assumed that the class/crime relationship doesn’t exist

Limitations

– The “domain problem” (narrows scope)

– Inability to measure ‘chronic and career’ offending (scale problems)

Studies in 1970’s and onwards have measured other areas of an individual’s life (etiological
factors, the study of causal connections) which permit theory testing/development

Assessing reliability and validity of self reports

A. Measurement reliability : measurement strategy yielding the same results on repeated

trials
a. Internal consistency of items (more complex version of splitting techniques)
i. But should we expect items to correlate?
a. Test-retest method involves collecting panel data but re-asking the same questions at
some later time

Test results indicate major self report measures have acceptable reliability

Q1 drink alcohol, Q2 smoke Cig ,Q3 drop acid (lsd), Q4 use inhalants

Split them into 2 questions in order to see if they’re consistent, split half technique (if they’re
high here, they should be high there too)
Internal consistency- To see how each question relates to every other question

Expect high correlation but not always doesn’t

Q1 Q2 Q3 Q4

Q1 ---------------------
--------

Q2 -------------------------
----
Q3 -------------------------
----
Q4 -------------------------
----

A. Measurement validity : a measurement strategy that accurately measures what it is

intended to measure

Recall you need a nominal definition (delinquency/crime/deviance)

a. Content Validity – less of an issue with more recent measures

b. Construct validity – measures shown to correlate with socio-demographic variables in
theoretically expected ways
A. Criterion validity – comparison (delinquency of a respondent)
i. self report measures – official records
ii. self report measures – report by friends/classmates
iii. self report substance use – blood/urine/saliva tests

***there appears to be substantial underreporting , people don’t always tell the truth

Test results indicate major self report measure have acceptable validity/accuracy

Major concerns with self report surveys

PEOPLE UNWILLING TO REPORT CRIMINAL BEHAVIOUR

Are there method effects?

– Self administered/face to face/ telephone

– Anonymous/non-anonymous

Generally the available research findings and unclear

Spatial methods for collecting sensitive (illegal and embarrassing) information

– Randomized response technique

Computer assisted self-administered interview (CASI)

– Essentially the respondent reads and questions and enters responses into computer
– Convince them that it’ll be completely confidential

Victimization surveys
 – First major victim surveys undertaken in the late 1960’s in the US- now carried out in
many countries, including Canada
– 1966-1967 large scale victim surveys started taking place in US
○ Canadian urban victimization survey 1981 (URBAN survey, all participants in 1 of 8
countries in Canada)
○ General social survey 1988, 1993, 1999, 2004 (5 year increments)
○ Violence against women survey 1993
** All done by statistics Canada, done by telephone (CATI), excludes territories
–
– Like self report offender surveys, victim surveys attempt to measure crime events
reported and not reported to the police (dark figure)
– Historically, have provided more detailed info (info about victim, offender, circumstances
surrounding event) than UCR data – but this has changed since late 1980s
– UCR data criticized for giving very limited data about crime ( # of homicides, charges,
males/females etc) and are monthly totals (aggregate data) , don’t know if people are
done more than one offences. Very little information relating to the offender
– Now asked more (age of offender, were the people involved drinking, circumstances of
situation, etc)

Victim surveys have led to creation of new victim-centered theories (routine activities, lifestyle,
opportunity, rational choice, repeat victimization)