Asynchronous Discussion Forum: Integration of Content

Analysis and Social Network Analysis

Erlin

Norazah Yusof

Azizah Abdul Rahman

Faculty of Computer Science and

Information System

Faculty of Computer Science and

Information System

Faculty of Computer Science and

Information System

Universiti Teknologi Malaysia, 81310 Universiti Teknologi Malaysia, 81310 Universiti Teknologi Malaysia, 81310
Skudai, Malaysia
Skudai, Malaysia
Skudai, Malaysia

erlin_by@yahoo.com

norazah@utm.my

ABSTRACT
This paper is to propose a methodology to integrate content
analysis and social network for analyzing asynchronous
discussion forum (ADF) within collaborative learning
environment. Integrating these methods may offer promising
advantages in which the transcript and the structural network of
interaction are able to be analyzed during discussion. Presently,
the application of these methods is used in separate way and lack
established methodologies to analyze it. This approach filters
unrelated messages from the discussion content, before being
analyzed by SNA. A more reliable and meaningful result is
expected.

Categories and Subject Descriptors

K.3.1 [Computers and Education]: Computer Uses in Education
collaborative learning, distance learning
I.2.7 [Artificial Intelligency]: Natural Language Processing text
analysis, discourse. E.1 [Data Structures]: graphs and networks

General Terms
Design, Measurement, Performance

Keywords
asynchronous discussion forum, content analysis, social network
analysis, communication transcripts, communication structures

1. I#TRODUCTIO#
The use of information and communications technology (ICT) in
educational settings is increasing rapidly, mainly due to the many
assumed benefits of the use of CMC. Similarly, the increasing
popularity of the internet and its ability to provide seemingly
transparent communication between different computing
platforms has simplified the processes of providing learning
opportunities to remotely located learners.
Asynchronous discussion forum are now regularly used as a

azizahar@utm.my

component of distance education courses in tertiary education as a

means of promoting interaction between course participants [1].
Discussion forums create an environment similar to the face-toface classroom environment where knowledge can be critically
constructed validated and shared. In traditional face to face
classrooms, observation will tell instructors how the class is
doing, whether the students know what to do, and if they are
enjoying what they are doing [2].
Since asynchronous discussion forum take a central place in
CSCL or CMC, assessing methodologies to analyze it is
important. An increasing number of researchers have attempted to
produce technique that measure and analyze quality of
asynchronous discussion [1], having reviewed current literature,
suggest that the majority of studies can be loosely categorized into
one of four categories, according to the construct being measured;
levels of disagreement, argument structure analysis, interaction
based and content analysis. However, there has not been much
exploration and evaluation to integrate the method of content
analysis and social network analysis.
This paper proposes the use of content analysis with an
incorporation of social network analysis as a method to analyze
asynchronous discussion forum. These methods analyze not only
the transcript but also the structures of interaction during
discussion. Objective of this paper is to propose a method that can
analyze asynchronous discussion forum that is more reliable and
meaningful.
The rest of the paper is organized as follows. In Section 2, discuss
the content analysis and social network analysis approach. Next in
section 3, present the background leading to the statement
problem. Then, in section 4, discuss the evaluation on content
analysis and social network analysis research in ADF. Hence, in
section 5 discuss on research methodology to illustrate research
design and procedure and conceptual framework. Finally, sections
6 are conclusion and future work.

2.

CO#TE#T A#ALYSIS A#D SOCIAL

#ETWORK A#ALYSIS APPROACH
2.1 Content Analysis Approach

The content analysis approach codes messages according to the

message type. A review of literature suggests that content analysis
is the most popular approach used by researchers to evaluate
quality in discussion forum postings. A central idea in content
analysis that the many words of the text are classified into much
fewer content categories. Each categories may consist of one,
several, or many word. Words, phrases or other unit of text

classified in the same category are presumed to have similar

meaning. Several models are developed and they all have their
advantages and disadvantages. Most researchers have based their
model on theoretical foundations, but often the link between these
theoretical foundations and the models used in practice is not very
explicit.
In general, the aim of content analysis is to reveal information that
is not situated at the surface of the transcript. To be able to
provide convincing evidence about the learning and the
knowledge construction that is taking place, in-depth
understanding of the asynchronous discussion is needed [3].
This paper accepts a broad-based definition in a recent content
analysis [3]. For the purpose of this paper, content analysis is a
technique often used to analyze transcripts of asynchronous,
computer mediated discussion groups in formal educational
settings.

2.2 Social #etwork Analysis Approach

SNA techniques have been applied to a variety of problems and
they have been successful in uncovering relationships not seen
with any other traditional method [2]. Also, visualization
techniques are important aids in helping researchers understand
social and conversational patterns in online interactions. SNA
techniques paired with recent developments in software for
visualization could help provide a clearer picture of what is
happening in the online environment.
SNA can also be used to visualize the network connections by
creating a graphical representation called a sociogram. A
sociogram is a representation of all participant connections in a
social network. Visualizations of social networks can show
whether interactions are occurring between all members of a
group or whether some group members are communicating more
(or less) with other specific individuals [4].
Using SNA methods for analyzing online networks, in particular
learner networks suggested by many researchers. Several authors
have demonstrated the applicability of SNA to specific learning
situations. In these research, the collaborating persons, (student,
tutors, experts, and so) are the actors.
Moreover, Social Network Analysis (SNA) may help in
identifying patterns of relationship between people who are part
of a social network [5]. It may assist us in the analysis of these
patterns by illuminating the flow of information and/or other
resources that are exchanged among participants.

3.

BACKGROU#D LEADI#G TO THE

STATEME#T PROBLEM

Collaborative
learning
can
be
either
asynchronous
(communications are sent and received at the different times) or
synchronous (communication are sent and received at virtually the
same times). Although, many researchers have argued that content
analysis and social network are the popular method to analyze
asynchronous discussion [2][6][7], there has not been much
exploration of methodologies to integrate both of them as shown
in table 2.
Many researchers in the field argue for the use of content analysis
as a vehicle for classifying, analyzing and measuring
communication transcripts with respect to different aspects of
communication and learning [8]. Unfortunately, analysis in CSCL

or CMC research focused on surface level characteristic of the

communication [9].
Moreover, researchers should not heavily depend on the
information recorded in the log file for analyzing collaborative
process, but to combine this with an analysis of its content,
especially the content of collaborative dialogue [10]. Furthermore,
the reliability of content analysis of computer conference
communication is not, or only briefly, and not solved in a
satisfactory way-in much of the literature arguing for and using
content analysis [11].
Synthesis of communication transcripts in virtual learning
normally involves analysis of social interaction or using
communication tools to find out what participants are talking
about, and the motivation driving them to socially connect with
others in virtual spaces [12]. SNA techniques paired with recent
developments in software for visualizations could help provide a
clearer picture of what is happening in the online discussion [2].
On the other hand, SNA is usually used to study the way people
participated and interacted with each other, especially investigates
the relationship between participants rather than the discussion
content [13]. Furthermore, SNA calculate all message in ADF for
measuring quality of network interaction without consider
unrelated message. The summary leading to the statement problem
in this paper is shown in figure 1.
The important to analyze ADF
Discussion forums are a powerful way to conduct conversations
online and a significant component of online learning
In spite of the importance of these forums, the method and
technique to analyze the quality of ADF is needed.
However, it is apparent that the field of CMC or CSCL lacks
established methodologies to analyze it.

Issues in CA and SNA

Analysis on CSCL or CMC research focused on surface
level characteristic of the communication
The Reliability of CA is not, or only briefly & not solved in
a satisfactory way
SNA focused on interaction rather than discussion content
SNA calculate all message in ADF for measuring quality of
interaction without consider unrelated message

Communication
transcripts is evaluated
through Content
Analysis

integrating

Network structures
are analyze using
Social Network
Analysis

Problem Statement
How to analyze ADF in collaborative learning that can give
more reliable and meaningful analysis by integrating content
analysis and social network analysis?

Figure 1. Background that leads to the problem statement

4. EVALUATIO# O# CA A#D S#A I# ADF

4.1 Evaluation on CA Research in ADF
The applied methods reflect a wide variety of approaches and
differ in their level of detail and the type of analysis categories
used. Further differences are related to diversity in their
theoretical base, the amount of information about reliability, and
the choice for the unit of analysis.
Table 1. Studies on content analysis research in ADF
Author
Hendri
(1992)

Newmann
(1995)

Gunawardena

Theoretical
background

Unit of
Analysis

Cognitive &
Metacognitive

Thematic
unit

Critical
thinking

Thematic
unit

construction
of knowledge

Message

(1997)

Garrison,
Anderson
& Archer
(2001)

Community
of inquiry cognitive
presence

Dimension/Category/
Phase
5 variables:

Participation
Social
Interaction
Cognitive
Metacognitive
10 categories:
relevance, importance,
novelty, outside
knowledge,
ambiguities, linking
ideas, justification,
critical assessment,
practical utility, and
width of understanding
Uses 5 phases:

Sharing andcomparing
information

Discovery and
exploration

Negotiation of meaning

Testing and
Modification

Agreement and
application
Message
5 phases:

An initiation
An exploration
An integration
A resolution
Not reported
3 category:

Interrater
reliability
Not
reported

Not
reported

Not
reported

Holtis
coefficient
Cohens
kappa

Using
S#A

Cohens
kappa

de Laat
(2002)

To study interaction pattern

and knowledge construction


(unfilter)

Aviv
(2003)

To study network analysis of

knowledge construction


(unfilter)

Poon &
Daniel
(2006)

To employ SNA and CA to

understand
interaction
patterns in one videomediated virtual community


(unfilter)

Suh &
Lee
(2006)

To design and develop a

prototype for an agent that
support online collaborative
learning

de Laat
(2007)

To investigate patterns of
interaction
in networked
learning and CSCL

5 variable as Henrys
model

Percent
agreement

Soller
(2004)

Knowledge
sharing
Knowledge
construction

Sentence

2 categories:

Cohens
Kappa

Weinberger &
Fischer

Argumentativ
e-knowledge
construction

micro4 different process:

level and

Participation
macro
Epistemic
level

Argumentative

Social mode
Message
5 categories:

Oriogun
(2006)

Cognitive
Engagement

Suggestion
Question
Unclassified
Answer
Delivery

Table 2. Studies on CA and S#A research in ADF

Using
CA

paragraph

(2005)

A number of studies using both content analysis and social

network were selected based on the objective criteria. Although,
this does not represent an exhaustive collection of literature
regarding both content analysis and SNA, but reflects a balanced
sample of what is currently used in the research field.

Objective

Cognitive and
Metacognitif

Effective

Ineffective

Most technique used to measure inter-rater reliability are Cohens

Kappa and Percent Agreement because it is intuitively and simple
to calculate.

Author

Hara,
Bonk &
Angeli
(2000)

Interaction

Participation

Critical thinking

The table also shows that researchers can consider each individual
sentence as a single unit of analysis. A second option is to identify
a consistent "theme" or "idea" (unit of meaning) in a message and
to approach this as a unit of analysis. A third option is to use the
complete message as the unit of analysis. The fourth option is to
use a paragraph as the unit of analysis. Every researcher has his or
her reasons to choose for one of these possibilities, and there is
not really an agreement. Researchers also have their own view of
theoretical background as a basis of their work.

Percent
agreement

Sentence

(2000)

Table 1 shows several models of content analysis. This helps to

qualify the current state-of-the-art of CSCL-research based on
content analysis model. Although many researchers have argued
that without the establishment of reliability, content analysis
measures are less meaningful, but these literature shows,
reliability on content analysis data is not considered.

4.2 Evaluation on CA and S#A Research in

ADF

Structural &
interactional
exchange
pattern

Fahy, et al

In order to present an overview of the current state-of-the-art, a

number of studies were selected, based on the following criteria:
methods applied, cited, or reflected upon in CMC journals and
CSCL-conferences; since these are the most important for a where
scientific discussions about the development, use, and study of
such model take place. Further, this selection was extended with
recently developed methods with a unique approach or a
noticeable theoretical background.

Percent
agreement
Cohens
kappa
Cohens
Kappa
TRCP

(unfilter)


(unfilter)

This list give illustration that there has not been much exploration
of methodologies to integrate both content analysis and social

network, whereas these incorporation method has more

advantages in analysis asynchronous discussion. As mentioned in
section 1 that these incorporation methods can analyze not only
the transcripts but also the structure of network. Hence
exploration to use these incorporation methods is needed.

research area of online asynchronous discussion have been

reviewed. This reviewed in order for both major areas
complement each other. In addition to the mentioned areas of
research, the data analysis methods will be reviewed.

From the reviews it is cleared that the current approaches have

variety of capability to analyze transcript and interaction, but little
of them support aspects from both content analysis and social
network. Furthermore, the table also shows that related or
unrelated message will be calculated for SNA. Most approaches
typically provide only limited supports to one method to analyze
online discussion.

Based on the analysis method selected during theory building, the

data will be analyzed using certain criteria. This research also
coded the message for content to determine emerging categories
to guide in the identification of relevant topic. The unit of analysis
to perform a content analysis will be elected. In this stage also
develop a method to analyze communication transcripts based on
knowledge sharing model [15]. This stage also involves
examining the social network to indicate the flow of interactions
which subsequently determine community structure as well as
patterns of discussion. Moreover, design auto-coding categorizing
text and develop prototype tool for converting text based data to
SNA will be proposed.

5. RESEARCH METHODOLOGY
5.1 Research Design and Procedure
To ensure the appropriate research perspective, the multimethodological research approach [14] as shown in figure 2 will
be used to guide this research. This approach has been selected as
it allows multiple perspective and flexible choices of methods to
be considered in various stages of the research process. This
methodology research approach integrating theory building,
systems development, experimentation, and observation will be
used to address the research questions associated with this
research. This methodology consists of three stages
(experimentation and observation will be combined in one stage)
which are described in the following sections.
Theory
Building
1

System Development

Experimentation & Observation

Finally, findings from theory building and system development
will be integrated to test the content analysis and social network
analysis to case study. Data of students asynchronous discussion
forum for one subject will be observed and test uses the propose
method. The data will be collected from weekly discussion forum
held on Moodle. The test will be conducted by comparative result
and will be validated by two case studies, using filter data and unfilter data. This comparative result will be presented to consider
that using filter data will give the result to be more reliable and
meaningful. Furthermore, testing and review result obtained by
the tool to get feedback to improve the propose method. This
integrate is necessary to ensure that the methods functionalities
satisfy the known constraints, objectives and requirements.
This research involves three stages activities. These activities
corresponds the research strategies discussed in the previous
section depicted in figure 2. Correlation the objectives, activities
and expected outcome are shown in table 3.

System
Development - 2

Table 3. Correlation the objectives, activities and expected

outcome

Observation
4

Experimentation
3

Figure 2. Multi-methodological research (#unamaker, 1991)

Theory Building
During theory building, review of the literature is done. The
literature review started with overview on different approach to
identify quality of online discussion, next, the general view of
content analysis (CA) and social network analysis (SNA). Issues
of content analysis and social network in online asynchronous
discussion have been reviewed. This stage also includes the
investigation of existing methods and research finding related to
asynchronous discussion. The particularly here in establishing a
conceptual framework based on evaluation of existing CA and
SNA method. In order to look at communication transcripts
processes and interaction pattern in collaborative learning, the

Objective

Activity

Outcome

Stage 1 : Theory Building

To propose a
methodology that
can analyze
asynchronous
discussion forum
that is more reliable
and meaningful

Literature study
Evaluation on
existing methods
(previous work)
Propose conceptual
framework of the
research
Conduct reliability
test of content
analysis
Stage 2 : System Development
To develop
transcripts analysis
tool that can be
used to evaluate
communication
transcripts of ADF

Comparative
evaluation methods
Conceptual
Framework
The appropriate
reliability to this
study

Design
Design of proposed
documentation
auto-coding
categorizing message Prototype tool
Design and develop Execute the tool
prototype tool for
converting text based
data for SNA

Stage 3 & 4 : Experimentation and Observation

To analyze
interaction using
Social Network
Analysis (SNA) by
integrating Content
Analysis for
collaborative
learning interaction

Experiment of CA
Analysis result
and SNA to case
Final Result
study
Testing and review
the result obtained
by the tool
Improve proposed
methods and finalize
the result

5.2 Conceptual Framework

From literature review and evaluation on existing methods for
analyzing asynchronous discussion forum leading to propose a
conceptual framework is as below.
Interaction
through the
computer

text
based

CA

Tool to
examine
the KS

Coding
Transcripts

Out comes

ADF

Information
regarding the
communication
structures

CA
module

filter
text
based

Information
regarding the
transcript
analysis

Tool for
converting
SNA
module

Out comes

SNA
Analyzing and
investigating

Participant

Figure 3. Conceptual Framework

Figure 3 describes the conceptual framework of the research. As
depicted in the figure, there are three elements involved in the
methods, i.e. the content analysis environment which
communication transcripts will be analyzed, the social network
analysis environment which contains tool to convert text-based
into data for SNA and SNA module and the outcomes to shows
the result from integrate these methods.
This research employs content analysis and social network
technique to analyze asynchronous discussion for students
participating in a course. Hence, there are three elements involved
in this methods, i.e. the content analysis environment which
communication transcripts will be analyzed, the social network
analysis environment which network structure of interaction will
be identified and the expected outcomes to shows the result from
integration these methods.
The content analysis environment contains three sub-elements for
the process, i.e. the coding transcripts procedure; tool to examine

the knowledge sharing and content analysis module. The

procedure for coding transcripts should comprise five steps:
determination of the unit of analysis, development of
segmentation procedure, determination of the reliability of the
segmentation procedure, development of coding categories and
determination of the reliability of the coding categories.
The social network analysis contains two sub-elements: tool for
converting text based data for SNA and SNA module. This
research will develop a conversation program that scan the SQL
database of the messages of a given ADF, Next, it constructs the
response matrix of ADF. Rows and columns of a response matrix
are labeled with numbers representing the participants. The
content of the response matrices was then written by conversation
program into data file in the ntf format which is acceptable for
analysis by SNA.
These two main elements that compose the integration methods in
order to analyze ADF that is more reliable and meaningful. The
content analysis will use tabulation technique to represent result
that collecting same or similar recording units in categories and
representing counts of how many instances are found in each.
Two categories, not related and specific, will be added for coding
the messages which means this message not related to the topic or
theme and for message that combine time and activity.
Furthermore, not related message will be filtered or ignore it as
data input for SNA. This technique will be used that can support
the result of SNA to be more reliable. The SNA provide much
information regarding the communication structure of the group,
level of participant, group cohesion, role of each participant,
identifying central participant, bridge and isolate (able to
determine who are not engaged in the discussion). This technique
also will give comparative result with using filter data and unfilter data and will be represented to consider that using filter data
will give more meaningful and reliable result.

6. CO#CLUSIO# A#D FUTURE WORK

We believe that the incorporation of content analysis and social
network provides a scientific and systemic way to analyze the
quality of asynchronous discussion forum. It filters unrelated
messages from the discussion content, before being analyzed by
SNA. It is as a potential method to analyze not only the transcripts
but also the structures of interaction during discussion.
In future work, we integrate the tool to examine content analysis
based on knowledge sharing to content analysis module. Also
integrate tool for converting data to SNA to perform the
experiment in this research.

7. REFERE#CES
[1]

Spatariu, A., Hartley, K. and Bendixen, L.D. 2004. Defining

and Measuring Quality in On-line Discussion. Journal of
Interactive Online Learning. Vol 2(4).

[2]

Willging, P. A. 2005. Using Social Network Analysis

Techniques to Examine Online Interactions. Journal of USChina Education Review, Vol 2, No.9 (Serial No.10).

[3]

De Wever, B., Schellens, T., Valcke, M. and Van Keer, H.

2005. Content analysis schemes to analyze transcripts of
online asynchronous discussion groups: A review,
Computers & Education 46 (2005) (1), pp. 628.

[4]

[5]

Haythornthwaite, C. 2002. Building social networks via

computer networks: Creating and sustaining distributed
learning communities. In K. A. Renninger & W. Shumar
(Eds.), Building virtual communities: Learning and
change in cyberspace (pp. 159190). Cambridge:
Cambridge University Press.
de Laat, M., Lally, V., Lipponen, L. and Simon, R., J.
2007. Investigating patterns of interaction in networked
learning and computer-supported collaborative learning: A
role for Social Network Analysis. Journal of Computer
Supported Collaborative Learning.

[6]

Neuendorf, K. A. 2002. The content analysis guidebook.

Thousand Oaks, CA: Sage Publications.

[7]

de Laat, M. 2002. Network and Content in an Online

Community Discourse, from:
http://www.uu.nl/uupublish/content/2002%20Networked
%20Learning%201.pdf

[8]

[9]

Henri, F. 1992. Computer conferencing and content

analysis. In A. R. Kaye (Ed.), Collaborative learning
through computer conferencing. London: Springer.
Strijbos, J. W., Martens, R., Prins, J., and Jochems, W.
2006. Content Analysis: What are they talking about?
Journal of Computer & Education, Elsevier. 46. 29-48.

[10] Nurmela, K., Palonen, T., Lehtinen, E. and Hakkarinen, K.

2003. Developing tools for analyzing CSCL process,
proceeding of the international conference on computer
support for collaborative Learning, Bergen, Norway. pp
333-342.
[11] Ratfell. 2007. The Reliability of Content Analysis of
Computer Conference Communication. Journal of
Computer & Education, Elsevier. 49.230-242.
[12] Poon, N. and Daniel, B. K. 2006. Social Network and
Content Analysis of Interactions in a Video-Mediated
Virtual Community. Proceedings of the Sixth International
Conference on Advanced Learning Technologies
(ICALT'06).
[13] Li, Y., Wang, J., Liao, J., Zhao, D and Huang, R. 2007.
Assessing Collaborative Process in CSCL with an
Intelligent Content Analysis Toolkit. Journal of Seventh
IEEE International Conference on Advance Learning
Technologies (ICALT), 2007.
[14] Nunamaker, J.F., Jr.; Chen, M. and Purdin, T.D.M. 1991.
System development in information systems research.
Journal of Management Information Systems, 7 (3), 89106.

[15] Soller A. 2004. Computational Modeling and Analysis of

Knowledge Sharing in Collaborative Distance Learning.
User Modeling and User-Adapted Interaction 14: 351
381.
[16] Newman, G., Webb, B. and Cochrane, C. 1995. A content
analysis method to measure critical thinking in face-toface and computer supported group learning. Interpersonal
Computing and Technology, 3(2), 56-77. from
http://www.helsinki.fi/science/optek/1995/n2/newman.txt
[17] Gunawardena, C. N., Lowe, C. A and Anderson, T. 1997.
Analysis of global online debate and the development of
an interaction analysis model for examining social
construction of knowledge in computer conferencing.
Journal of Educational Computing Research, 17(4), 397
431.
[18] Garrison, D. R., Anderson, T., and Archer, W. 2000.
Critical thinking in a text-based environment. Computer
conferencing in higher education. Internet in Higher
Education, 2(2), 87-105.
[19] Fahy, P. J., Crawford, G., Ally, M., Cookson, P., Keller,
V. and Prosser, F. 2000. The development and testing of a
tool for analysis of computer mediated conferencing
transcripts. Alberta Journal of Educational Research,
46(1), Spring, 85-88.
[20] Hara, N., Bonk, C. J. and Angeli, C. 2000. Content
analyses of on-line discussion in an applied educational
psychology course.Instructional Science,28(2),115152.
[21] Weinberger, A and Fischer, F. 2005. A framework to
analyze argumentative knowledge construction in
computer-supported collaborative learning, Journal of
Computers & Education 46(1), pp. 7195.
[22] Oriogun, P. K. 2006. Content Analysis of Online
Transcripts:
Measuring
Quality
of
Interaction,
Participation and Cognitive Engagement within CMC
Groups
by
Cleaning
of
Transcripts,from:
http://www.itdl.org/Journal/Mar_06/article03.htm
[23] Aviv, R., Erlich, Z., Ravid, G., and Geva, A. 2003.
Network analysis of knowledge construction in
asynchronous learning networks. Journal of Asynchronous
Learning Networks, 7 (3).
[24] Suh, H. J., and Lee, S. W. 2006. Collaborative Learning
Agent for Promoting Group Interaction, ETRI Journal,
Vol 28, No.4, 461-474, 2006.