1/25/12

Is It Right To Test For Significant Differences in Convenience Samples? |

Online Survey Software - SurveyGizmo Login Search Search for: Search Go Free Trial Pricing Software Features Full Service Survey Guides Support Blog About Us Have Questions? (800) 609-6480 Start your 14 day Enterprise trial Or try
Professional

Is It Right To Test For Significant Differences in Convenience Samples?

Posted 06.29.10 by Michaela Mora - A Survey Expert Filed in: Market Research, Survey Best Practices, Survey Expert

You'll have access to all Enterprise Plan features. If you get stumped you can call or email us any time during your trial. Name: Company: Phone: Email: Password: Confirm Password:

Share Tweet

I agree to the terms and conditions and privacy policy. Trials limited to 25 responses, upgrade for more. Create My Account 2005-2012 Widgix, LLC dba SurveyGizmo (S48)

I often get the question about what sample size is needed to get a large enough sample so that statistically significant differences can be found and inferences to a larger population can be made. But something that is often ignored is that these statistical tests were meant to work within the probabilit sampling theor framework. Since the advent of online panels and the increase of online surveys using
surveygizmo.com/ /significant-differences-and-convenience-samples/ 1/4

1/25/12

Is It Right To Test For Significant Differences in Convenience Samples? |

panel-provided samples, the issue of testing for significant differences using standard parametric tests has become a moot point in many research studies. Nowadays many of the surveys conducted online use samples provided by online panels, but these are mostly convenience samples (nonprobability). The populations of online panels include respondents who are willing to participate in studies, excluding those unwilling to be part of the panel who may be members of the target population we are after. In probability sampling, each possible respondent from the target population has a known probability to be chosen. Probability sampling helps us avoid some of the selection biases that can make a sample not representative of the target population. For more on this read Representative Samples Does Sample Size Really Matter? Unfortunately, taking a probability sample is hard and costly. For most consumer research studies and social behavior studies, we really don t know the size of the actual population of consumers behaving in certain ways or consuming certain product. Trying to find this out would make the research prohibitively expensive. This is why we often have to settle for convenience samples like the ones offered by online panels. They still can offer valuable insights if designed with care, but again doing statistical testing in a convenience sample is pointless since the assumptions about probability sampling are violated. Online panels are here to stay, and they will continue to be a source for affordable sample for market research. Research using convenience sample is often better than no research at all if the survey is well designed and screening criteria are used to define the target population. A more appropriate case for testing statistically significant differences are random samples taken from a customer database, since this is essentially the population frame where we can count all members and estimate their probability to be chosen. However, if you don t have a customer database or are interested in surveying non-customers, then use a convenience sample. You may feel more confident in your sample if you are able to replicate the results in repeated surveys, but always be cautious about inferences made from convenience samples since there could be a hidden systematic bias in the data. It is always important that whenever you use convenience samples you consider the following when analyzing the results: 1. Who is systematically excluded from the sample? 2. What groups are over or under represented in the sample? 3. Have the results been replicated with different samples and data collection methods? If testing for significant difference gives you peace of mind, even when using convenience samples, do it to confirm the direction of the data, but restrain yourself from doing inferences to a larger population. For help on sample size calculation check out Survey Sample Size What Should It Be? To calculate sample size and margins of error, use the Sample Size and Margin of Error Calculators from Relevant Insights.
surveygizmo.com/ /significant-differences-and-convenience-samples/ 2/4

1/25/12

Is It Right To Test For Significant Differences in Convenience Samples? |

Photo courtesy of lrargerich

Flickr, Creative Commons (Attribution)

Tags: convenience sample, Michaela Mora, Representative Sample, statistical significance, Survey Expert Share This: Have you tried SurveyGizmo yet? Try our 14 day free trial

About the Author

Michaela Mora - A Surve E pert Michaela is the president of Relevant Insights, LLC, a Texasbased full-service market research agency serving B2B and B2C clients and offering research services for the general and Hispanic market. With 20 years of hands-on market research experience, Michaela has been involved in the design and implementation of many studies ranging from market segmentation, to pricing optimization, customer satisfaction tracking, new product testing, advertising testing, and brand tracking studies among others. Michaela holds a MS in Marketing Research from The University of Texas at Arlington, a MS in Marketing, Advertising and PR from Stockholm University, and a BS in Psychology from Havana University. She also holds a Professional Researcher Certification (PRC) at the Expert Level, issued by the Marketing Research Association (MRA). If you need help with survey design or other phases of a research project, Michaela can be an invaluable resource. She can be reached at michaela@relevantinsights.com

The Surve E pert Blog

Get a Project Quote from Professional Services

When you need survey building expertise or have a project to complete right away, we can help.

By Kipp Chambers

surveygizmo.com/

/significant-differences-and-convenience-samples/

3/4

1/25/12

Is It Right To Test For Significant Differences in Convenience Samples? |

New Surve Gi mo File Librar

Get help with our Surve Gi mo January 24 2012 - We ve upgraded our File Library with a project
cleaner interface and bulk upload/bulk delete features. Check out the changes! How-To Tutorials and Webinars or search by topic. Ask a Question - Our team of survey experts can answer your question and get you back on track! Ja . 20 By Marni Zapin

Contact us
Questions about SurveyGizmo plans or upgrading your account Reseller and Affiliate Programs Inquiries

Using an iPad Surve to Conduct a Qualitative Stud

Marni Zapin shares an example of how to conduct a qualitative study using an iPad survey at The Market Research Event 2011. Ja . 18 By Kipp Chambers

New Surve Templates & Surve Guides

Our new Survey Guides section offers several survey examples you can use to get ideas of how to use our software. You can also check out our new survey templates! More from our Survey Experts

surveygizmo.com/

/significant-differences-and-convenience-samples/

4/4