A Survey of Crowdsourcing as a means of Collaboration and the Implications

of Crowdsourcing for Interaction Design

Yue Pan & Eli Blevis
School of Informatics & Computing
Indiana University at Bloomington
{panyue,eblevis}@indiana.edu
ABSTRACT
Crowdsourcing has emerged as a new learning and
online collaborative paradigm, in which crowds of
people can collaborate and complete a specific task. In
this paper, we provide a survey of crowdsourcing as a
means of collaboration in three different contexts,
namely academic, enterprise, and social. The survey
also includes a short inventory of how different factors
contribute to the development of crowdsourcing. Finally,
we develop and outline open questions, insights, as well
as a research agenda for enhancing crowdsourcing
development in the perspective of HCI.
KEYWORDS: Crowdsourcing, HCI, interactive design,
collaboration.

1. INTRODUCTION
Crowdsourcing is a new paradigm that relies on the
intelligence of crowds of people to solve a specific
problem or complete a tasksometimes with a
monetary reward [10, 14, 16]. Jeff Howe invented the
term crowdsourcing in 2006. The most recent
definition
of
crowdsourcing
he
offers
is:
Crowdsourcing is the act of taking a job traditionally
performed by a designated agent (usually an employee)
and outsourcing it to an undefined, generally large
group of people in the form of an open call. (Howe
2011)
Crowdsourcing has prompted models and applications
for problem solving and knowledge building in a wide
range of domains, from science to enterprise to social
networks. The development of crowdsourcing has been
leveraged in a way that largely satisfies social demands
for working with people in a larger geographic scope, as
well as task demands for accomplishing human
intelligence
tasks
(HIT).
[12]
Successful
crowdsourcing applications, such as Amazons

978-1-61284-639-2/11/$26.00 2011 IEEE

Mechanical Turk (AMT), Everything2, Wikipedia,

Innocentive etc., serve as innovative learning and
network collaboration paradigms.
The role of interaction design and research in facilitating
the development of online collaborative practices has
not been fully explored so far. Of interest is the question
of how can interaction design play the role of motivating
and sustaining participation and collaboration in
different contexts (academic, enterprise, social)? This
paper provides a survey of crowdsourcing as a means of
collaboration, especially related to interaction design,
and proposes a research agenda of interaction design
insights and methods of promoting applicability of
crowdsourcing in different domains.
This paper is divided into three parts. First, we review
the literature about crowdsourcing in different contexts,
namely academic, enterprise, and social. The second
part of the paper describes crowdsourcing in more
detailed dimensions, including role, behavior and media,
and with different variables, including leadership,
ownership, coordination and conflict, integration and
attribution, mobile crowdsourcing, and ubiquitous
crowdsourcing. In the last part, based on the literature
review in the first and second sections, this paper
concludes by developing several open questions about
crowdsourcing and puts forward several design insights
and considerations, as well as a research agenda for
further investigation and research in crowdsourcing
development.

2. CROWDSOURCING & INTERACTION

DESIGN IN DIFFERENT CONTEXTS
Crowdsourcing has become a very essential tool and is
quite popular among various areas. In order to better
understand various motivations behind different groups
of contributors, this paper provides a general review of
crowdsourcing apropos of the HCI literature in three
different contexts, namely academic, enterprise and
social.

397

2.1. Crowdsourcing for Enterprise

Stewart et al. in [14] state that design principles can be
used to influence two domains in crowdsourcing:
enterprise (inside a firewall) and the public
domain. They raise the notion of extrinsic motivation
monetary rewards that can draw a crowd without any
number limitation. Stewart et al. employ a large
crowdsourcing community across a wide range of
influence at IBM with about 400,000 employees located
in 160 countries. The value of Stewart et al.s work lies
in three hypotheses and tests about motivating
crowdsourcing, as well as two design principlestask
decomposition and incentive model to motivate and
sustain participation in enterprise level.
In [10] Maja Vukovic talks about the usage of
crowdsourcing in the software development domain at
the enterprise level. She categorizes crowdsourcing
platforms along two dimensions: crowdsourced
function and crowdsourcing mode. She also proposes
22 required features for crowdsourcing and administers
an assessment of the four most famous crowdsourcing
systems illustrative of each feature, namely MTurk
which illustrates features, such as Comprehensive
model of the crowdsourcing request, and Automated
provisioning of crowdsourced request deliverable etc.;
Innocentive, which illustrates features, such as
Crowdsourcing platforms, and Collaboration
services etc.; iStockphoto, which illustrates features,
such as Comprehensive model of the crowdsourcing
request, and Registration facility etc. ; Threadless,
which illustrates features, such as Registration
facility, and Collaboration services etc. Finally, she
proposes 7 research challenges which can be a good
guidance for system developers and interaction
designers to improve crowdsourcing systems. For
instance, one of the research challenges is that
Designing a mechanism for virtual team formation,
incorporating not only skill-set, but also discovered
social networks. [10]

2.2. Crowdsourcing for Academic

Regarding the application and impact of crowdsourcing
in academic and science field, Buecheler et al. in their
2010 paper [15] describe how crowdsourcing can
benefit the Research Value Chain and they provide a
research agenda about the applicability of
crowdsourcing in fundamental science based on
Artificial Intelligent research. The research brings
insights and methods for facilitating the processes of
crowdsourcing in scientific and academic fields. It also
supports the idea that non-scientists can support
scientific projects with high quality contributions. The

same idea is also held by [3], which is targeted

specifically at the academic purpose. He holds that data
collected by volunteers (non-scientists) can be of great
educational value in a larger geographic scale and over a
longer period of time. These two papers hold the belief
that the collaboration between professionals (scientists)
and non-professionals (general public) deserves more
attention from academic and scientific field.
In [13], Nov et al. introduce a term to represent internetbased crowdsourcing in the field of science, namely
SciSource and they explore the motivations behind
what drives people to make contributions to share
resources voluntarily. Based on findings from two
preliminary studies (SETI@home and CWOP), they list
several prominent factors that serve as motivations for
continuous crowdsourcing participation in academic and
scientific related fields, including Enjoyment,
Reputation,
Values,
Enhancement,
and
Affiliation to a team and tenure. Understanding
different aspects of motivations that contribute to
scientific crowdsourcing can be a first-class
consideration for designing scientific projects.

2.3. Crowdsourcing for Social Network

Chung et al. [6] introduce the Stress OutSourced
(SOS) system which is designed to enable people to
relieve stress by sending each other therapeutic
massages as an instance of the concept of
crowdsourcing. They state: The SOS system consists of
members wearing customizable signal modules for
sending and responding to an anonymous SOS call, and
wearable massage actuation modules to receive the
massage strokes sent by others. Chung et al. discuss
three innovative design concepts of the SOS system.
Based on their prototype of the massage module, they
describe a design framework which can be useful to help
better understand and develop scalable social media and
haptic networks with crowdsourcing. The prototype of
the SOS system is achieved by wearable garments (such
as sweater, shirt etc.) that can send and receive signals
through an embedded or attachable signal module and
a massage module.
Huberman et al. [4] describe social networking as a
construction of the interaction that people have with
their friends, families and acquaintances. From data
gathered from Twitter, they found that the degree of
peoples contributions to crowdsourcing depends on the
attention people receive from others. Based on this
notion, they describe two types of networks, a very
dense one made up of followers and followees and a
sparser and simpler network of actual friends. This
distinction can be part of the design and research agenda
for design online collaboration network based on social
aspects that we describe in the last section of this paper.

398

3. CROWDSOURCING IN DIFFERENT
DIMENSIONS
In this part, we examine crowdsourcing in more detailed
dimensions but from a different angle. We list a set of
factors that contribute to motivations and participation
of volunteers at different levels. The set of factors can
be classified into following categories: (i) role-oriented
crowdsourcing with leadership and ownership, (ii)
behavior-oriented crowdsourcing with attribution,
coordination and conflict, and (iii) media-oriented
crowdsourcing with mobile computing and ubiquitous
computing.

3.1. Role-oriented Crowdsourcing

Leadership & Crowdsourcing

Crowdsourcing
requires
communication
and
cooperation from people who are geographicallydistributed [7] and who have diverse backgrounds. At
the same time, Kittur and Kraut [2] argue that it can be
very ineffective for peer-to-peer communication if
there are too many people involved in a free and open
environment. They also conducted research which
suggests that it is highly beneficial to have people who
can set direction and provide a structure to which others
can contribute. Under this circumstance, high-quality
leadership is quite essential and necessary to guarantee
successful crowdsourcing participation and online
collaboration at different levels. Based on empirical
qualitative studies of three online communities, Luther
and Bruckman introduce two major challenges that
online collaboration leaders face in the entertainment
domain. [7] They identify four themes, namely
originality, completion, subjectivity and
ownership, which can be used to guide us on
improving design to deal with these challenges that are
faced by online collaboration leaders.
Based on [8], Luther also explores the role of leadership
in online collaboration in the context of movie-making.
In [8], he offers concrete design and research principles
and insights about the potential and limitations of online
creative collaboration.

3.2. Ownership & Crowdsourcing

Regarding existing features of Wikipedia, ThomSantelli et al. [5] raises a notion of territoriality, namely
the expression of ownership towards an object. This
can be valuable regarding a successful hierarchical style
of collaboration, and present design implications for
supporting expressive territory behaviors such as
marking, control, and defense. These design
insights and implications can be used for later research

on how to maintain online collaborative system in a

healthy and productive way, as well as providing
support and encouragement to further user experience
design of collaborative systems.

3.3. Behavior-oriented Crowdsourcing

Integration
&
Crowdsourcing

Attribution

within

In [9], Luther et al. hold based on their research that

recognizing integration and attribution of online
reputations in a proper way can contribute to motivating
creative online collaborations, as an instance of
collaborations that are possibly in the sense of
crowdsourcing. Based on a qualitative study (involving
17 experienced animators) on Newgroundsthe largest
Flash animation portal on the Web, this paper also
provides novel insights on promoting online creative
collaboration by identifying frustrations with existing
practices and systems, as well as proposing design
considerations for alleviating them. Especially, three
design insights, namely cr-editing, collaborative
authoring, and community value can be crucial
guidelines for better understanding and designing for
online creative practices that prompt collaboration.

3.4.
Coordination
Crowdsourcing

&

Conflict

within

Kittur and Kraut in their [2] emphasize the importance

of coordination by distinguishing (i) explicit
coordination,
which
is
based
on
direct
communication and verbal planning and (ii) implicit
coordination, which is workshop structure, unspoken
expectations and shared mental models of the task to be
accomplished. They adopt different research
methodologies (such as cross-sectional analysis and
longitudinal approach) to examine different factors that
contribute to the quality of crowdsourcing, including the
number of contributors and the differing
coordination techniques. Their findings can be helpful
in providing a better understanding on what kind of role
that coordination plays in improving crowdsourcing
quality.
Recognizing a phenomenon with Wikipedia that the
rate of creation of new articles and content is
decreasing, while levels of maintenance and indirect
work are increasing, Kittur et al. [1] build a
characterization model and a user conflict model to
investigate motivations and conflict at the local article
level. Their models have been testified by adopting
several scientific methods, such as human-labeled
controversy tags and history flow visualization etc.
as well as by conducting an online survey of 28 articles.
The scientific methods, conflict models, visualization

399

tools and techniques that they used to investigate and

predict conflict and coordination costs are very intuitive
and can be applied to analyze and generalize
crowdsourcing systems in a larger scale.

3.5. Media-oriented Crowdsourcing

Mobile Crowdsourcing

Mobile computing has potential implications for

crowdsourcing. Mobile computing can facilitate the
development and expansion of crowdsourcing by
enabling mobile users to work on crowdsourcing tasks
with their mobile devices. In [16], Yan and his
colleagues investigate existing crowdsourcing services
and develop a mobile application mCrowd based on
sensor-rich mobile devices which allow users to post
sensing tasks, such as posting text or image tagging
queries. The idea and design of the mCrowd
application can serve as a good demonstration of the
ways in which mobile technology can facilitate the
development of crowdsourcing in a sustainable way.
Considering crowdsourcing applications using mobile
technology, another good example is Txteagle by
Nathan Eagle [12]. Txteagle is a very typical mobile
application using mobile computing technology in the
crowdsourcing domain that enables people from
developing countries to conduct real-time, peer-topeer financial transactions. In their paper, they
describe four interaction scenarios that demonstrate how
Txteagle facilitates financial earnings in rural settings.
For example, in one user scenario a home-maker in
Butare Rwanda was unable to afford a phone herself.
But now, she gets paid by completing txteagle
transaction tasks for a village phone operator, and
eventually she will be able to buy her own cell phone.
Txteagle can be a good design insight and implication
on applying mobile technology and crowdsourcing in
providing financial sources to people at remote and
impoverished areas of the world.

3.6. Ubiquitous Crowdsourcing

Vukovic et al. in their Ubicomp 10 workshop paper [11]
describe several important aspects of ubiquitous
crowdsourcing, including the interaction model,
crowdsourcing applications enabled by ubiquitous
computing systems, and protocol between ubiquitous
computing and crowds. They raise three critical
challenges about crowdsourcing and ubiquitous
computing, from the perspective of (i) creating
interaction model and protocol, (ii) facing challenges of
ensuring crowdsourcing quality, and (iii) developing
crowdsourcing application within ubiquitous computing
domain. All of these three challenges serve as novel
insights for HCI and interaction design and they also

contribute to part of our research agenda described in

the last section of the paper.

4. RESEARCH AGENDA
In this section of the paper, we propose several open
questions that interaction designers may face apropos of
HCI and interaction design in crowdsourcing, and
develop insights that own directly to our secondary
research in the previous sections. More specifically, the
design insights include a summary of crowdsourcing
values in different context (academic, enterprise, and
social), the assessment of different variables on
crowdsourcing, as well as the motivation model, all of
which can lead to concepts and principles for interaction
design.

4.1. Design Problems

We provide the following inventory of open research
questions based on our literature review of
crowdsourcing within HCI and interaction domain.
1. What are the factors spurring creative online
collaboration in different contexts (academic,
enterprise, social) as well as targeted at different
dimensions (role, behavior, and media)?
2. What are the challenges that interaction design is
facing according to improving and maintaining
development of crowdsourcing?
3. What are human-computer interaction designer and
researcher roles in motivating, enhancing and
sustaining motivations and participation in
crowdsourcing?
4. Are there ways in which HCI design can improve
creative collaborative crowdsourcing?
5. What values are advanced by crowdsourcing systems
in promoting social values, such as diversity, social
awareness, respect, sustainability etc.?

4.2. Insight Based on Surveys

This part of the paper describes crowdsourcing values in
different contexts, roles of different variables in
crowdsourcing, as well as the motivation model that are
derived from the survey of crowdsourcing as a means of
collaboration in the previous part.
4.2.1. Crowdsourcing Values in Different Contexts
Generally speaking, the role of crowdsourcing is to
collect intelligence from the crowd to complete a
specific task. With the purpose of better understanding
the contribution of crowdsourcing, we derive

400

crowdsourcing values separately in the three contexts of

academic, enterprise, and social that we identified in the
previous part, which is displayed in Table 1.

crowdsourcing. Keeping these factors in mind can shape

the design of crowdsourcing applications with more
specific targets and objectives.

4.2.2. Crowdsourcing in Different Dimensions

In the previous section, we identify a list of factors that
contribute to crowdsourcing from three different
dimensions, including role, behavior and media. In
Table 2, we describe the role that each factor may play
in crowdsourcing. Each variable according to different
dimension can contribute to crowdsourcing from
different perspectives. More specifically, some variables
may act as intrinsic factors to motivate and direct
collaborators, such as to motivate ones personal values,
enhance ones reputations, etc.; some variables may
influence the quality and development of crowdsourcing
in a more direct way, such as affecting the cost and
development process of crowdsourcing; some variables
may contribute from the perspective of technology and
communication media.

4.2.3. Motivation Model

Considering the values people have that prompt them to
participate in crowdsourcing, we created the Incentive
Model shown as Table 3 that presents an inventory of
motivating factors from different aspects.

It is useful to be familiar with each factor and its

respective roles as part of any design process involving

4.3. Research Agenda Based on Insights

Crowdsourcing as a means of collaboration is a very
promising model of interactivity that can be applied in
different disciplines. At the same time, it also prompts
wide-ranging research challenges along the contexts of
academic, enterprise, and social. What follows is the set
of challenges that interaction design and HCI may face
when designing and developing crowdsourcing
applications:
1. Design interfaces that can attract and sustain a large

Table 1. Crowdsourcing Value in Different Contexts

Table 2: The Roles of Different Variables on Crowdsourcing

Table 2. The Roles of Different Variables on Crowdsourcing

401

Table 3. Motivation Model

number of people to join in crowdsourcing.

2. Design
mechanisms
that
support
efficient
crowdsourcing formation as well as avoid conflict
among collaborators.
3. Design experience models that evaluate the quality
of experience in crowdsourcing process. [10]
4. Design a mechanism for balancing communication
and independency within crowdsourcing. [15]
5. Design in a way to increase the robustness of
crowdsourcing in dealing with collaborations between
unknown people within unknown situations and/or
environments. [15]

[1] Aniket Kittur, Bongwon Suh, Bryan A. Pendleton, and

Ed H. Chi. 2007. He says, she says: conflict and
coordination in Wikipedia, In Proceedings of the
SIGCHI conference on Human factors in computing
systems (CHI '07). ACM, New York, NY, USA, 453462.
[2] Aniket Kittur and Robert E. Kraut. 2008. Harnessing
the wisdom of crowds in wikipedia: quality through
coordination, In Proceedings of the 2008 ACM
conference on Computer supported cooperative work
(CSCW '08). ACM, New York, NY, USA, 37-46.
[3] Cohn, J.P. (2008) Citizen science: Can volunteers do
real research? Bioscience 58(3): 192197.
[4] Huberman, B. A., Romero, D. M., and Wu, F.. Social
networks that matter: Twitter under the microscope,
arXiv: 0812.1045v1, Dec 2008.

5. CONCLUSION
We have surveyed several key articles about
crowdsourcing as a means of collaboration and created
three tables that serve as a classification scheme for the
content of this survey.

[5] Jennifer Thom-Santelli, Dan R. Cosley, and Geri Gay.

2009. What's mine is mine: territoriality in
collaborative authoring, In Proceedings of the 27th
international conference on Human factors in computing
systems (CHI '09). ACM, New York, NY, USA, 14811484.

No matter if it is in the context of academic, enterprise,

or social, the benefits of crowdsourcing merit research.
The survey of crowdsourcing in this paper can be a first
step in exploring crowdsourcing values and listing
various factors that determine the development and
expansion of crowdsourcing. How to address the
research challenges enumerated above for developing
further crowdsourcing services from HCI and design
perspectives remains as future related work.

[6] Keywon Chung, Carnaven Chiu, Xiao Xiao, and Pei-Yu

(Peggy) Chi. 2009. Stress outsourced: a haptic social
network via crowdsourcing, In Proceedings of the 27th
international conference extended abstracts on Human
factors in computing systems (CHI '09). ACM, New
York, NY, USA,
[7] Kurt Luther and Amy Bruckman. 2008. Leadership in
online creative collaboration, In Proceedings of the
2008 ACM conference on Computer supported

REFERENCE

402

cooperative work (CSCW '08). ACM, New York, NY,

USA, 343-352.
[8] Kurt Luther. 2010. Supporting and transforming
leadership in online creative collaboration, In
Proceedings of the 28th of the international conference
extended abstracts on Human factors in computing
systems (CHI EA '10). ACM, New York, NY, USA,
2931-2934.
[9] Kurt Luther, Nicholas Diakopoulos, and Amy Bruckman.
2010. Edits & credits: exploring integration and
attribution in online creative collaboration, In
Proceedings of the 28th of the international conference
extended abstracts on Human factors in computing
systems (CHI EA '10). ACM, New York, NY, USA,
2823-2832.
[10] Maja Vukovic. 2009. Crowdsourcing for Enterprises,
in Proceedings of the 2009 Congress on Services - I
(SERVICES '09). IEEE Computer Society, Washington,
DC, USA, 686-692.
[11] Maja Vukovic, Soundar Kumara, and Ohad Greenshpan.
2010. Ubiquitous crowdsourcing, in Proceedings of
the 12th ACM international conference adjunct papers
on Ubiquitous computing (Ubicomp '10). ACM, New
York, NY, USA, 523-526.
[12] Nathan Eagle. 2009. txteagle: Mobile Crowdsourcing,
In Proceedings of the 3rd International Conference on
Internationalization, Design and Global Development:
Held as Part of HCI International 2009 (IDGD '09),
Nuray Aykin (Ed.). Springer-Verlag, Berlin, Heidelberg,
447-456.
[13] Oded Nov, Ofer Arazy, & David Anderson. (2010).
Crowdsourcing for science: understanding and
enhancing SciSourcing contribution, ACM CSCW
2010 Workshop on the Changing Dynamics of Scientific
Collaborations.
[14] Osamuyimen Stewart, Juan M. Huerta, and Melissa
Sader. 2009. Designing crowdsourcing community for
the enterprise, In Proceedings of the ACM SIGKDD
Workshop on Human Computation (HCOMP '09), Paul
Bennett, Raman Chandrasekar, Max Chickering, Panos
Ipeirotis, Edith Law, Anton Mityagin, Foster Provost,
and Luis von Ahn (Eds.). ACM, New York, NY, USA,
50-53.
[15] Thierry Bcheler, Rudolf M. Fchslin, Rolf Pfeifer, Jan
Henrik Sieg. (2010). In Proceedings of Artificial Life
XII -- Proceedings of the Twelfth International
Conference on the Synthesis and Simulation of Living
Systems. Pages 679686. MIT Press.
[16] Tingxin Yan, Matt Marzilli, Ryan Holmes, Deepak
Ganesan, and Mark Corner. 2009. mCrowd: a platform
for mobile crowdsourcing, In Proceedings of the 7th
ACM Conference on Embedded Networked Sensor
Systems (SenSys '09). ACM, New York, NY, USA,
347-348.

403