Measuring and Monitoring

with Surveys
Erich Stiefvater, NWREL
NW Network Conference 2009
March 2-4, 2009

1
Guiding Questions

Why should I use surveys?

What survey tool is right for me?
How do I design an effective survey?

2
Session Roadmap

Take workshop survey

General discussion
Sign up for free account
Survey-building practice
Report out
Wrap-up

3
Why should I use surveys?

4
A Variety of Uses

Surveys can help you:

Investigate community needs
Design new programs and services
Evaluate impacts and outcomes
Assess constituent satisfaction
Plan events
Test communication media and
messages

5
Survey Pros

Surveys:
Are ideal for gathering data and feedback
that can be quantified
Collect information in a standard,
systematized way
Yield datasets that can be sorted, filtered,
cross-tabbed, and graphed to provide
deeper analysis

6
Survey Cons

Surveys:
Take time to plan for, design, and collect
and analyze responses
Can yield unreliable or unusable
information if poorly planned or designed
Are often ignored by targeted users

7
Questions to Ask Yourself

What question do I need answered,

or what decision do I need to make?
What data or feedback will help me
answer that question or make that
decision?
Is a survey the best way to get that
data or feedback?

8
What survey tool is right for me?

9
Paper or Online?

Paper-survey tools
Desktop-publishing software (e.g., Word)
Optical-scan readers (e.g., Scantron)
Online-survey tools
Survey software (e.g., Zoomerang)
Web-design tools (e.g., Dreamweaver)
Widgets and templates in social-
networking software (e.g., Google Docs)

10
A Myriad of Options

11
How to Pick

Price
Maximum numbers of:
Surveys
Questions
Responses
Features
Storage
Support
12
Basic Tool Features

Battery of common question types

Save/copy/edit surveys
Launch via email or URL
View survey results online
Knowledge base/FAQs/tutorials

13
Advanced Tool Features
Survey templates you can adapt
Skip/branch logic
Customizable appearance and messages
Upload and manage email lists
Filter and cross-tabulate results
Summary reports in several formats
Export survey data
Phone and email tech support

14
Questions to Ask Yourself

How often will I create surveys?

How many people will I typically be
surveying?
How complex will my surveys be?
How long will I need to save my data?
What combination of pricing and
features would best meet my needs?

15
 How do I design
an effective survey?

16
Survey-Design Process

17
18
Determine Your Goal

What do I want to know?

A clear goal:
Tells you what information to collect
Helps you identify participants
Helps you write questions
Begins with analysis in mind

19
Sample Goal Statements-1
Investigate community needs
What gaps exist in the continuum of
homeless services in our town?
Design new programs and services
What new recreation classes would local
seniors be most interested in?
Evaluate impacts and outcomes
How many of our graduates find a job or
go on to college within a year?

20
Sample Goal Statements-2
Assess constituent satisfaction
What do our AmeriCorps members think
of our new staff mentoring program?
Plan events
What topics do our partners want included
on the agenda for the gang summit?
Test communication media and
messages
Does our website meet visitors’ needs?
21
Determine Your Audience

Who can give me the data or feedback

I need?
Audience analysis:
Identifies potential participants
Helps you figure out how to reach them
Determines the number of responses you
need

22
Determine Your Methodology

Key questions to ask

Number of responses needed
Print or online survey
How will be marketed
Offer incentive or not
Format raw data will need to be in
Format of finished analysis

23
24
General Design Tips

Introduce the survey

Keep it short
Move from broad to narrow questions
Put the most important questions first
Limit use of open-ended questions
Keep answer schemes consistent
End with profile questions

25
Introduction Example

26
Question Types
Multiple Choice-Single Answer

27
Question Types (cont’d)
Multiple Choice-Multiple Answer

28
Question Types (cont’d)
Rating Scale-Single Answer

Note consistent scaling

29
Question Types (cont’d)
Rating Scale-Table/Matrix

30
Choose Words Carefully

Make sure your survey questions are:

Brief, clear, and direct
Free of jargon or confusing language
Asking only one thing each
Free of bias

31
Question Example 1

What do you think?

32
Question Example 1 (cont’d)
biased biased jargon/unclear

assumption/leading
“double-barreled”
question

33
Question Example 1 (cont’d)

Try this

34
Question Example 2

What do you think?

35
Question Example 2 (cont’d)

slightly
unclear

36
Question Example 2 (cont’d)

Try this

37
Skip/Branch Example

38
Skip/Branch Plan

39
40
Deployment Options

Send link via email

Embed URL in webpage or domain

41
Pre-Test Your Survey

Have colleagues test/review:

Link (make sure it works)
Email invitation subject line and text
Questions/instructions
Appearance and navigation

42
43
Monitor Your Survey

Review real-time collection stats

Send reminders if needed

44
45
View Quick Reports . . .

46
. . . Or Download Full Ones

47
Some Can be Customized

48
49
Making Sense of the Data

Tie survey results back to the initial

research question or goal statement
Summarize key findings from survey
and write recommendations
Put actionable data in the hands of
decision-makers

50
Happy Surveying!

Thank you, and please contact me with

any questions

Erich Stiefvater
(503) 275-0761
stiefvae@nwrel.org

51
 Resources and References
Dillman, D.A. (2007). Mail and internet surveys: The tailored design
method. Hoboken, NJ: John Wiley & Sons, Inc.

Griffiths, K. & Toledano, Y. (August 26, 2008). Creating effective online

surveys. (Online). Retrieved February 23, 2008 from
https://cc.readytalk.com/cc/schedule/display.do?rfe=wviry6spmzb3&udc
=pi7os9o8bpc7.
Kasunic, M. (September 2005). Designing an effective survey. (Online).
Retrieved February 24, 2008 from
http://www.sei.cmu.edu/pub/documents/05.reports/pdf/05hb004.pdf.
Nielsen, J. (February 2, 2004). Alertbox: Keep online surveys short.
(Online). Retrieved February 24, 2008 from
http://www.useit.com/alertbox/20040202.html.
Zoomerang. (n.d.). 10 Tips to Improve Your Surveys. (Online).
Retrieved February 22, 2009 from http://www.zoomerang.com.

52