Facilitating Critical Thinking Skills through Content Creation  1 

W. Ian O’Byrne

University of Connecticut

Neag School of Education

249 Glenbrook Rd., Unit 2033

Storrs, CT 06269

Facilitating Critical Thinking Skills through Content Creation: Results from a Pilot Study

 
 Facilitating Critical Thinking Skills through Content Creation  2 

Facilitating Critical Thinking Skills through Content Creation:

Results from a Pilot Study

No technology has impacted literacy and learning with the scope and speed of the

Internet (Coiro, Knobel, Lankshear & Leu, 2007). The Internet has quickly become this

generation’s defining technology for reading. This reliance on the Internet as a dominant

text has also been shown in the habits of adolescents; students now report spending 48

minutes a day reading online compared to 43 minutes offline (Kaiser Family Foundation,

2005). Similarly adolescents read online at a rate far greater than other segments of the

world’s population, indicating an important shift in the generational privileging of texts

(Lenhart, Madden, & Hitlin, 2005).

This rate of online reading applies to a variety of academic tasks. More than 90%

of adolescent students in the U.S with home access to the Internet, report using the

Internet for homework (Lenhart, Simon, & Graziano, 2001). Over 70% of these students

used the Internet as the primary source for information on their most recent school report

or project (Lenhart, Simon, & Graziano, 2001). Only 24% of these students reported

using the library for the same task (Lenhart, Simon, & Graziano, 2001). More recent

research paints a clearer picture of teens and their relationship to information searching.

Individuals aged 18-30 stated that they valued the resources, especially a connection to

the internet that libraries provided (Lenhart, Simon, & Graziano, 2001). If they need to

find solutions to a problem, they would use the Internet first, or the library in conjunction

with the Internet to find the answers they needed (Estabrook, Witt, & Rainie, 2007). The

respondents, which included adults, stated that they used and trusted the Internet as a

means to find answers and advice, far more than they would consult a friend or an expert
 Facilitating Critical Thinking Skills through Content Creation  3 

(Estabrook, Witt, & Rainie, 2007). Our youth are increasingly online and consulting the

Internet as an expert in their daily lives. However, they frequently do not have the skills

and strategies necessary to succeed in this environment (Livingstone, 2004; Bennet,

Maton, & Kervin, 2008; Jewitt, 2008).

Increasingly, students are using the Internet to obtain information about both

general and academic topics (Lubans, 1999; Jones & Madden, 2002; Shackleford,

Thompson & James, 1999). Along with this trend there is a growing concern about the

dubious nature of online information, and users’ ability to validate or evaluate this

information (Alexander & Tate, 1999; Flanagin & Metzger, 2000; Browne, Freeman &

Williamson, 2000). Research shows that students are frequently deceived when viewing

online content (Leu et al., 2007; Johnson & Kaye, 1998; Rieh & Belkin, 1998).

Particularly, students are not able to judge the validity of a website, even when given

procedures to do so (Lubans, 1998, 1999). There is still little known about building the

healthy skepticism needed by students while reading online information. Because of the

reliance of students to use the Internet to find information it is even more paramount to

their success as online readers to be able to evaluate the validity and reliability of

websites (Leu et al., 2008). This paper reports on the results of a pilot study that

investigates if critical evaluation skills could be improved by first analyzing the

techniques authors use to make websites more credible (Britt & Gabrys, 2002; Fogg,

Marshall, Laraki, Osipovich, Varma, Fang, et al., 2001) and then having students build

their own hoax websites by using these techniques. Hoax websites are defined as website

“fabrications” that have been created for entertainment purposes, usually invoking the
 Facilitating Critical Thinking Skills through Content Creation  4 

ridiculous, but maintaining a “superficial appearance of scientific professionalism”

(Brem, Russell, & Weems, 2001, p. 198).

Theoretical Perspective

To effectively address the complex nature of the Internet and other

communication technologies (ICTs) as they affect the way in which individuals learn this

study needed to be framed by using multiple theoretical perspectives (Labbo & Reinking,

1999). Theoretical perspectives from critical literacy, new literacies, critical evaluation,

and multimodal design were used to frame the work presented in this paper.

Critical Literacy

Deeply rooted in socio-cultural perspectives of reading, critical literacy seeks to

use literacy and learning to “build access to literate practices and discourse resources”

(Luke, 2000, p. 449) for use as social capital in the community (Freebody & Luke, 1990;

Lankshear & Knobel, 1998). Critical literacy moves away from the “self” in critical

reading to understand how texts interact in different contexts (Luke, 2000). Texts are

analyzed to determine the purpose, which ideologies are represented, and how students

can “reject them or reconstruct them” (Cervetti, Pardales, Damico, 2001, p. 8) to support

their own experiences (Luke, 2000). Analysis of the lexical and mechanical operations of

text can be refined to the ideological representations, social relations, and textual

formations of the information (Halliday, 1994). Viewed within a classroom context this is

seen as examining text by analyzing: the field, or contexts; the tenor, or social functions;

and the mode, or technical characteristics (Luke, 2000). The four social practices

identified by Freebody & Luke (2000) within critical literacy impact the design and
 Facilitating Critical Thinking Skills through Content Creation  5 

revisions of this study. These practices take place at while students are at work critiquing

texts: code-breaker, meaning maker, text user, and text critic (Freebody & Luke, 2000).

New Literacies

The study is framed within both the larger definition of New Literacies, emerging

from a variety of disciplines, as well as the more specific definition of the new literacies

of online reading comprehension (Leu, O’Byrne, Zawilinski, McVerry, Everett-

Cacopardo, 2009). In an analysis of new literacy theory, Coiro, Knobel, Lankshear, &

Leu (2008) note four principles, common to the broader definition of New Literacies that

is beginning to inform work in fields as distinct as enthographies of cyberspace (e.g.

Wittel, 2000), semiotics (e.g. Kress, 2003; Lemke, 2002), media literacy (e.g.

Livingstone & Bober), or sociolinguistics (e.g. Street, 2003):

1. new technologies for information and communication require us to bring new

potentials to their effective use;

2. new literacies are central to full civic, economic, and personal participation in a

globalized community;

3. new literacies are deictic and regularly change;

4. new literacies are multiple, multimodal, and multifaceted.

Within this broader definition of New Literacies, this study is framed within the more

specific theory of the new literacies of online reading comprehension. This perspective

defines online reading comprehension as a process of problem-based inquiry, which

requires new skills, strategies and dispositions (Leu, Kinzer, Cammack, & Coiro, 2004;

Leu, et. al, 2008). One important skill identified within online reading comprehension is

the ability to evaluate critically the information that an individual encounters online.
 Facilitating Critical Thinking Skills through Content Creation  6 

Because students cannot be assured of the reliable or reputable nature of online texts as

they would traditional print texts, this skill is essential (Brandt, 1997; Henry, 2006).

Work conducted in the new literacies of online reading comprehension informed and

generated questions about the manner in which students critically evaluate online

information in this study.

Critical Evaluation

There are multiple definitions of critical evaluation informed by varying theoretical

perspectives (Coiro, 2007). In order to represent the varying viewpoints and research

involved, a review of the literature indicated several pieces of research that helped inform

the specific task of critically evaluating websites. A value added model of judging

information quality identified five values founding information systems: accuracy,

comprehensiveness, currency, reliability, and validity (Taylor, 1986). Tate & Alexander

(1996) identified five criteria in an analysis of print texts as applied to web pages:

accuracy, authority, objectivity, currency, and coverage. As a counterpart to identifying

specific elements of a text to determine authority and credibility, research determined that

students needed to be able to negotiate multiple points of view, consider conflicting

information, and construct their own meaning (Hannafin & Land, 1997; Hannafin &

Land, 2000). Extending from this, models called for readers to construct a two level

model while evaluating sources: integration (the text based, or internal model) and

situational (text mixed with prior knowledge and inter-text with other sources) (Britt,

Perfetti, Sandak & Rouet, 1999). These criteria began to boil down into two main factors

with credibility (expertise and trustworthiness) and relevance (importance and currency)

being used by undergraduates to evaluate information on web pages (Judd, Farrow &
 Facilitating Critical Thinking Skills through Content Creation  7 

Tims, 2006). Graesser et al. (2007) maintained that readers are to evaluate truth and

relevance of information, think about quality of sources, trace implications of evidence

and claims made, and ask how information is linked to learner’s goals. Metzger’s (2007)

review of the literature on elements of credibility and identified five values: believability,

accuracy, trustworthiness, bias, and completeness of information. Finally, critical

evaluation was defined as critical thinking abilities used to: 1) question, analyze and

compare resources; 2) judge quality of information on various characteristics; 3) defend

opinion with evidence form multiple sources and prior knowledge (Coiro, 2008). Kiili,

Laurinen & Marttunen (2008) identified critical thinking skills used by students when

evaluating information on the Internet into: credibility (distinguishing reliable from

unreliable information) and relevance (judgment about essential or non-essential nature

of information).

Multimodal Design

When working with or comparing two digital texts, the “argumentative” nature of

the work is often times in the visual elements and temporal arrangements, more than the

verbal (Wysocki, 2001). Originating as an extension from multimodalities (Kress & van

Leeuwen, 2001; Jewitt, 2008), design specifies the interchange between the linguistic,

visual, audio, gestural, spatial and multimodal (New London Group, 2000). When

integrated with a New Literacies framework, students operate as “designers” and “apply

critiqued knowledge of the subject/topic synthesized from multimodal sources” (Kimber

& Wyatt-Smith, 2006, p. 26). In multimediating (Doneman, 1997; O’Brien, 2003), the

student constructs a “representation of new knowledge” (Kimber & Wyatt-Smith, 2006,

p. 26) and communicates this with the intention of engaging their audience. As a
 Facilitating Critical Thinking Skills through Content Creation  8 

pedagogical tool, design holds together “process and product” (New London Group,

2000), and allows students to consider how literacy practices shape truth (Street, 1984;

Alvermann & Hagood, 2000).

Method

The population consisted of a convenience sample of forty seventh-grade students

from a middle school in the Northeast. Instruction was delivered by the researcher, a

teacher with experience working with IRT in a one-to-one laptop classroom (Leu,

Reinking et al., 2008), and a student teacher. The intervention took place two times a

week during the students’ regular English language arts ninety-minute class time.

Students remained in their class groups throughout the intervention. The intervention

lasted twelve weeks and consisted of three phases. The data collected during the pilot

study consisted of results from focus groups and content analysis coding of student

storyboards and hoax websites. The study was directed at answering three research

questions: 1) What are the themes and patterns present as students construct online

content? 2) What are the themes and patterns that emerge as students critically evaluate

online information? 3) What are the changes that take place as the content construction

process moves from planning to storyboards to the final product?

Phase One

Students were instructed in basic skills of online reading comprehension (ORC),

specifically evaluation of online information and initial use of the online content

construction (OCC) tools. Instruction focusing on the critical evaluation component of

ORC had students focus on seven websites of varying credibility and relevance. The

topic for all seven websites was Asthma due to the fact that some prior knowledge
 Facilitating Critical Thinking Skills through Content Creation  9 

because the students had covered it in their health class. The seven websites were

selected because they fell into two of the categories typical of web environments: weaker

sincere sites & stronger sincere sites (Brem, Russel, & Weems, 2003). Weaker sincere

sites are identified as more “balanced between reputability and disreputability” (Brem,

Russel, & Weems, 2003, p. 198) than hoax websites or stronger sincere sites. The claims

made are believable, and backed up by supporting data found online, but do not stand up

to close examination. Stronger sincere sites present: “professional markers” of

organization, more credible experts, and an “air of precision and authority. (Brem,

Russel, & Weems, 2003).

________________

Insert Table 1 Here

_________________

Students were asked to view the webpages via hyperlinks from a website set up

by the researcher (http://newliteracies.typepad.com/asthma_quiz/). The students were

asked to select websites that would help a peer learn more about asthma. The students

were to view the seven websites and rank order them in terms of credibility and

relevance. Students completed a section defining the two criteria (credibility &

relevance), and the seven websites. Students were given 45 minutes to look through and

rank order the websites. The students were allowed to view any of the pages within each

of the sites, and could use any ORC strategy they felt would help them in reviewing the

websites.

The Brem, Russel & Weems study included hoax websites in their investigation

of the varying levels of credibility and relevance of online information. To attend to the
 Facilitating Critical Thinking Skills through Content Creation  10 

presence of hoax websites, as part of the instructional model in phase one the students

were presented with five websites and asked to provide support for the credibility and

relevance of each of the sites. The sites are listed in Table 2.

________________

Insert Table 2 Here

_________________

The instructors did not tell students that the five websites were hoaxes. Students

presented their findings about the hoax websites as part of a whole-group discussion with

the class. At the conclusion of the discussion, the instructors revealed to the class that all

five websites were hoaxes. This activity expanded students’ understanding of the

presence of sites with varying levels of sincerity, but also websites whose purpose is to

deceive the reader.

Focus Groups. After the pretest, the instructor and researcher delivered four days

of instruction in critical evaluation of websites. Focus groups (Krueger & Casey, 2008)

were conducted to determine criteria students used while evaluating online content. The

main research question of the focus groups was: What features of a website make it more

credible and relevant? The focus groups consisted of four meetings of 12-15 students.

The classroom teacher or student teacher served as the moderator, while the researcher

took notes. The role of the moderator was that of a “seeker of knowledge” (Krueger &

Casey, 2008). The focus group met for a half-hour, three times over the course of two

weeks. The focus groups constructed student-derived criteria using exemplars from the

seven asthma websites and the five hoax websites that would be used to guide OCC
 Facilitating Critical Thinking Skills through Content Creation  11 

(Orsmond, Merry, Reiling, 2000; 2002). The students used these criteria as guides in the

editing and revision of their websites.

OCC Tools. The final element of phase one was the initial instruction in the use of

the OCC tools. Online content was constructed using several tools that came as part of

the Mac OS suite, and as noted in Table three.

________________

Insert Table 3 Here

_________________

The instructors provided an initial demonstration of the affordances of iWeb. The

additional tools were demonstrated to each group that expressed a need for creation of

that type of content as identified by their storyboards.

Phase Two

Phase two included the planning, construction and revision of the student created

spoof websites. The students were allowed to select their own groups to work in while

constructing their websites. The groups consisted of from two to four students, with the

classroom teacher giving final approval for each group. The classroom teacher and the

researcher, following the IRT instructional model (Leu, Coiro, Castek, Hartman, Henry,

& Reinking, 2008; Leu et al., 2008), delivered classroom instruction with modifications

noted for future iterations of the model. To scaffold student and group progress from

beginning to end of the process, hardcopy worksheets of student work were kept in-group

portfolios. Some of these worksheets required teacher signature to signify that the student

or group had completed a stage sufficiently and could move on to the next step of the

content construction process.

 Facilitating Critical Thinking Skills through Content Creation  12 

The OCC process initiated with groups brainstorming to determine topics that

could be used for each hoax website. Students were to select an initial pool of five

potential topics for their website, and then meet with an instructor to discuss the rationale

for selecting each of the topics. Discussions between the instructors and groups ensured

that each group selected a topic that the students were interested in, and could effectively

create a hoax website for.

After agreeing upon the topic for the hoax website, each group worked together

using paper graphic organizers and colored pencils to storyboard (Bailey & Blythe, 1998)

plans for their website. Upon teacher approval noted by a signature, students were

allowed to move on to use of OCC tools and website construction.

The groups created online content two to three times per week for six weeks. The

researcher and classroom teacher met with each group several times during each class to

ensure on task behavior, high rigor of work, and collaboration by all group members.

Extended use of the content construction tools was taught to students on an “on demand”

basis (Gee, 2003). After the initial instruction of usage of a tool, these students became

experts of the tool, and were expected to share this expertise with other students.

Three times during the OCC process the instructors conducted a whole class

discussion aimed at reminding students as to the criteria constructed as a class which

outlined elements of credibility and relevance in a website. The student derived marking

criteria were used to direct students, as well as exemplars selected by the instructors

(Orsmond, Merry, & Reiling, 2000; 2002). In addition to these whole-group discussions,

instructors conducted meetings with each group focused on formative feedback using the

exemplar websites shared as a group (Orsmond, Merry, & Reiling, 2002). The exemplar
 Facilitating Critical Thinking Skills through Content Creation  13 

websites were often viewed as “competition” for their site to model future design choices

and changes (e.g., the gaming site group consulted other websites selling video games).

The groups were asked to review the competition site, and their site and agree upon

strengths, weaknesses and opportunities to add features to their website.

Phase Three

Phase three consisted of a final review of each website with the instructors, as

well as a showcase of the websites with members of the students’ class. Several of the

best websites constructed within a class were also showcased to students in the other

class. In class discussions, students were asked to present their site, along with major

decision choices and changes made during the online content construction process.

Students were asked to critique the work from other groups according to the class rubrics.

Finally, students were to take the critical evaluation instrument again as a posttest.

Analysis

Qualitative analysis for the pilot study focused on results from the two sources of

data: focus group interviews; and student-constructed storyboards and hoax websites. The

results from the focus group interviews were used to triangulate (Jick, 1979) the results

from the coding of the storyboards and hoax websites to establish more consistent themes

and patterns.

Focus Groups

The researcher analyzed the notes from the focus groups and the criteria

developed by the students to identify themes. The classic approach (Krueger & Casey,

2008) to the analysis of focus group data was employed to find themes. An initial read of

the researcher notes and criteria identified essential themes. The themes were then
 Facilitating Critical Thinking Skills through Content Creation  14 

compared to the main research question of the focus groups to ensure the themes

answered the question of: What features of a website make it more credible and relevant?

During a second examination of the materials, the notes and criteria were then assigned a

code according to the theme they most closely suited. At the conclusion of coding, the

results were categorized according to the frequency, specificity and extensiveness of

comments (Krueger & Casey, 2008). Frequency refers to how often an element of a

website was commented on by students. Specificity refers to comments that

comprehensively detail the elements of a feature of a website. Extensiveness is related to

the frequency of comments, but identifies the many different ways in which numerous

students identify a theme. (Krueger & Casey, 2008).

Storyboards and Hoax Websites

At the conclusion of the intervention the storyboards and hoax websites from

groups that completed all work of the intervention were collected. Six groups of students

completed all aspects of work on their storyboards and hoax websites. The decision was

made to focus the analysis on the work from these groups that had completed all phases

of the work. The analysis focused intensively on a smaller number of groups to develop

an understanding of the themes and patterns shown by students that successfully finish all

aspects of the OCC process. A rigorous content analysis (Mayring, 2000) was conducted

of the student-constructed storyboards and hoax websites in order to inductively analyze

(Patton, 2002) and find common themes. Preliminary inductive categories were created

as the documents were read through. After half of the documents of the groups had been

read, the researcher checked to see if the categories remained consistent with the

identified constructs of credibility and relevance. The check to address concerns of the
 Facilitating Critical Thinking Skills through Content Creation  15 

inductive categories was conducted again after all of the studies had been reviewed. After

the final reliability check, the categories and their subcategories were: credibility and

relevance. Credibility refers to text, images or video that builds the expertise,

trustworthiness or reliability of a website. This includes content that describes the source

of the site, products or services offered, or testimonials from individuals that support the

website. Relevance refers to text, images or video that builds the importance, currency, or

essential nature of the website. This includes content that addresses or persuades the

audience; or tries to negotiate the value of a product or service with the audience. Once

the categories had been created through analysis of all the student work, codes were

assigned to all markers of either credibility or relevance in student work.

Results

Focus Groups

The purpose of the focus groups was to identify elements of web page design that

students examine while judging the credibility or relevance of a website. These results

were coded and organized into criteria that students also used while engaged in the OCC

process. The results were collected as students reviewed the seven asthma websites, and

the five hoax websites. The results identify the major elements of web design that most

effectively adjust the audience’s view of the materials presented. The most popular

response (23 responses) was that students were influenced by the web address (e.g., .com,

.org, .edu) when viewing a website. A second important element in the evaluation of a

web page was the belief that the web site contained “enough information” (17 responses),

or information that seemed “believable” (15 responses). Multimedia elements

significantly affected the viewing of a web site by adolescents as students valued the
 Facilitating Critical Thinking Skills through Content Creation  16 

presence of images (12 responses) or videos (11 responses). Surprisingly, a larger number

of students valued the aesthetics of a webpage, but did not know the terminology of how

to express this. Students commented that it was “well-organized” (5 responses),

“professional looking” (6 responses), “was nice to look at” (4 responses). An example of

this is illustrated by the differences in color, organization and style shown by WebMD

and KidsHealth.org. A relatively small portion of the students identified the inclusion of a

“about us”, “contact us”, or “author” page (5 responses) as being a necessary element of a

website. Finally a small portion of students (4 responses) valued links to other sites

online as affecting the credibility of a web site.

Storyboards and Hoax Websites

The coding of the storyboards and hoax websites created by the students included

a content analysis (Mayring, 2000) that identified markers of credibility and relevance

across the two student works. The analysis sought to identify the themes and patterns

identified in student created storyboards and hoax websites, and then triangulate this data

with results from the focus group interviews to ensure reliability. Finally the analysis

sought to measure the increased presence of markers of credibility and relevance across

OCC process, as documented by student work. The results of the coding will be

organized with a brief description of the hoax website constructed.

Pulchritude. (http://newliteracies.uconn.edu/projects/hoaxsites/pulchritude 

/Site/Welcome.html) This website was created by two students and the audience was

identified by the students as “middle aged women”. The purpose of the site was to sell

jewelry that offered supposed health benefits to the individual that wore it. The

storyboard constructed by the students contained 19 markers of credibility and 15

 Facilitating Critical Thinking Skills through Content Creation  17 

markers of relevance. The final website constructed by this group contained 21 markers

of credibility and 20 markers of relevance.

Tikistar Island. (http://newliteracies.uconn.edu/projects/hoaxsites/tikistar 

/Site/Welcome.html) This website was constructed by two students and the targeted

audience was identified as “anyone that wanted to purchase an exotic animal”. The

purpose of the site was to sell exotic fish and pets from an island identified as Tikistar

Island. The storyboard constructed contained 33 markers of credibility and 2 markers of

relevance. The final version of the hoax website contained 75 markers of relevance.

Toething. (http://newliteracies.uconn.edu/projects/hoaxsites/toething 

/Site/_welcome.html) This website was constructed by three students and the targeted

audience was any one that wanted to purchase their product. The purpose of the site was

to market and sell toething, or “clothes for your toes”. The storyboard contained 20

markers of credibility and 1 marker of relevance. The hoax website constructed contained

52 markers of credibility and 4 markers of relevance.

DAT-a-Way. (http://newliteracies.uconn.edu/projects/hoaxsites/dat%20a%20way 

/Site/Welcome.html) This site was constructed by two students and the identified

audience was described as “teenagers with acne problems”. The purpose of the site was

to market their product, an acne treatment formula that was made from the cleanest

substance on the face of the earth, dog slobber. The storyboard contained 25 markers of

credibility and 9 markers of relevance. The final draft of the hoax website contained 65

markers of credibility and 33 markers of relevance.

Pillow3. (http://newliteracies.uconn.edu/projects/hoaxsites/Pillow3/Site 
 Facilitating Critical Thinking Skills through Content Creation  18 

/Home.html) This site was constructed by three students and the targeted audience was

identified as “teenage boys”. The purpose of the site was to market the newest installment

of the Pillow game series, a videogame in which not only the game characters went to

sleep, but many times the game players slept as well. The major advance of Pillow3 was

that is was now a massively multiplayer online game. The storyboard contained 41

markers of credibility and 8 markers of relevance. The hoax website constructed

contained 51 markers of credibility and 10 markers of relevance.

Fruitilicious. (http://newliteracies.uconn.edu/projects/hoaxsites/fruitiilicious 

/Site/VVellcome.html) This site was constructed by two students. The targeted audience

was identified as “women of all ages”. The purpose was to market a new hair color

product that loosely was marketed as “scratch and sniff hair”. The storyboard contained 4

markers of credibility and no markers of relevance. The final draft of the hoax website

contained 70 markers of credibility and 5 markers of relevance.

Discussion

The purpose of this pilot study was to identify patterns and themes that exist as

students evaluate online information in conjunction with creation of online content. The

results inform future iterations of not only the intervention design, but also the feasibility

of facilitating the critical evaluation skills of adolescents. The results suggest that with

the aid of instructors, and providing time to examine the methods with which credibility

and relevance are created through online content, adolescents can build awareness of the

questionable nature of online content. The coding of storyboards and hoax websites

within the OCC process showed that increased construction and presence of credibility

and relevance markers increased.

 Facilitating Critical Thinking Skills through Content Creation  19 

The patterns and themes recognized by students as they construct online content

revealed interesting details about not only the manner in which students work, but how

teachers work with them. The instructors negotiated a subtle balance between getting the

students excited about the construction process, but also reminding students to focus on

the content and final draft of their work. Attempts were made early in the intervention to

instruct the entire class as to the affordances and operation of each OCC tool, but students

would rather receive “just-in-time” (Gee, 2003) lessons. Additionally, student strategy

exchange was difficult to foster within the classroom. When a student was struggling and

wanted to learn a strategy, they would instantly call upon the instructors without seeking

help from members of the group, or classmates. Many times a student sitting right next to

them had the answer to their quandary.

The patterns and themes identify as students critically evaluate online information

suggest that as students spend increased time with direct instruction on the presence of

online information with varying levels of sincerity their healthy skepticism grows. What

is not known at this point is whether these skills and behaviors follow the students after

the intervention concludes. Additionally, it is not known whether all students understand

and employ the strategies, or merely the more vocal students. Finally, the extent to which

students employ these skills is unknown. It is not known whether these students knew

and used these skills, or simply correctly used the terminology.

The use of IRT as an instructional model proved effective in working with

students multimediating (Doneman, 1997; O’Brien, 2003) in a one-to-one laptop

classroom. Additional skills and strategies were needed to scaffold students as they work

on construction of online texts. The six groups whose work was coded outlined the
 Facilitating Critical Thinking Skills through Content Creation  20 

benchmark for success in constructing online content. Scaffolding measures (e. g.,

storyboards, criteria) were extremely successful in keeping students cognizant of the path

from beginning to completion of websites. It is hypothesized that in future iterations of

this study, more scaffolding, and time working with OCC tools ahead of the intervention

is needed.

Their work presents results that an intervention designed after this model would

be successful in building OCC skills, and critical evaluation skills in these students. What

is probably the more interesting question is the other half of the sample that did not finish

all aspects of the work involved, and what steps are necessary to bring all students up to

the benchmark of completion. In future iterations of this study more scaffolding of skills

and strategies may be necessary to assist these learners. In addition, the other factors that

influence the classroom environment, such as classroom management, previous

technological tool use, and academic ability need to be investigated in relation to this

work.

The student-derived criteria identify opportunities for future research in

differences in collective construction of criterion as a group of adolescents view

multimodal content. Research suggests that in many cases students do not evaluate online

content at all, and if they do, the evaluation of content is cursory at best. The focus

groups identified a multitude of elements of web design that students use while

evaluating online content. What is not known is whether student knowledge and

application of these skills extends beyond the length of the intervention. Additionally,

one of the most used strategies for evaluating the credibility or relevance of a website

was to look at the URL of the website, or search for more information of the creator of, or
 Facilitating Critical Thinking Skills through Content Creation  21 

the subject of the website (Bråten, Strømsø, & Britt, 2009). Because the websites created

by students were HTML files, or contained a web address that included university

markers, it was difficult for them to test the veracity of peers’ constructions.

The results from the coding of the storyboards and websites present an interesting

conclusion. With extended time working with OCC tools, the markers of credibility rose

in the websites as opposed to the storyboards. What is not known, is whether or not the

added markers is a result of affordances of the OCC tools, or prowess of the students in

recognizing the need for and the ability to construct these markers. In addition there was

a relatively low use of markers of relevance in the storyboards and hoax websites created

by students. It is not known if this is a reflection of the students not having studied

persuasive writing up until this point in their schooling, or that the instructors did not

address it effectively as part of the instructional model. More research is needed to

explore the lack of placement of markers of relevance, and the justification for the high

levels of markers of credibility in student work.

Given the work conducted and limited results obtained it is highly important that

continued work investigates ways to build the healthy skepticism of online learners. The

skilled critical evaluation skills of students as they sift through online information are

paramount as the Internet becomes more of a dominant text in their lives. Online content

creation proves to be a popular habit in the lives of students outside of the classroom;

research like this suggests that it may be feasible to conduct this construction in a one-to-

one laptop classroom. Finally, the attitudes and aptitudes of adolescents skilled in online

environments need to be studied and given a place in the traditional classroom context.
 Facilitating Critical Thinking Skills through Content Creation  22 

References

Alexander, J. E., & Tate, M. A. (1999). Web wisdom: how to evaluate and create

information quality on the Web. Hillsdale, NJ: Lawrence Erlbaum.

Alvermann, D. & Hagood, M. (2000) Fandom and critical media literacy. Journal of

Adolescent & Adult Literacy, 43, 436-446.

Bailey, G. & Blythe, M. (1998). Outlining, Diagramming and Storyboarding –

Three Essential Strategies for Planning and Creating Effective Educational

Websites. Learning & Leading With Technology, 6-11.

Bennet, S., Maton, K., & Kervin, L. (2008). The digital natives debate: A critical review

of the evidence. British Journal of Educational Technology, 19, 775–786.

Brandt, S. (1997). Constructivism: Teaching for understanding on the Internet.

Communications of the ACM, 40, 112-116.

Bråten, I., Strømsø, H.I., & Britt, M.A. (2009). Trust matters: Examining the role of

source evaluation in students' construction of meaning within and across multiple

texts. Reading Research Quarterly, 44, 6-28.

Brem, S. K., Russell, J., & Weems, L. (2001). Science on the Web: Student evaluations

of scientific arguments. Discourse Processes, 32, 191–213.

Britt, M., Perfetti, C., Sandak, R., & Rouet, J. (1999). Content integration and source

separation in learning from multiple texts. In S. R. Goldman, A. C. Graesser, & P.

van den Broek (Eds.), Narrative, comprehension, causality, and coherence:

Essays in honor of Tom Trabasco (pp. 209-233). Mahwah, NJ: Erlbaum.

Britt, M. & Gabrys, G. (2002). Implications of document-level literacy skills for Web site

design. Behavior Research Methods, Instruments & Computers, 34, 170-176.

 Facilitating Critical Thinking Skills through Content Creation  23 

Browne, M. N., Freeman, K. E., & Williamson, C. L. (2000). The importance of critical

thinking for student use of the Internet. College Student Journal, 34(3), 391–398.

Cervetti, G., Pardales, M., & Damico, J. (2001, April). A tale of differences: Comparing

the traditions, perspectives and educational goals of critical reading and critical

literacy. Reading Online 4(9). Available:

http://readingonline.org/articles/art_index.asp?HREF/articles/cervetti/index.html

Coiro, J. (2003). Rethinking comprehension strategies to better prepare students for

critically evaluating content on the Internet. [Electronic Version]. New England

Reading Association Journal.

Coiro, J. (2007). Exploring changes to reading comprehension on the Internet:

Paradoxes and possibilities for diverse adolescent readers. Unpublished

dissertation: University of Connecticut.

Coiro, J. (2008). Exploring the relationship between online reading

comprehension, frequency of Internet use, and adolescents’ dispositions toward

reading online. Paper presented at the annual meeting of the National Reading

Conference, Orlando, FL.

Coiro, J., Knobel, M., Lankshear, C. & Leu, D. (Eds.) (2008). Handbook of research on

new literacies. Mahwah, NJ: Lawrence Erlbaum Associates.

Doneman, M. (1997). Multimediating. In C. Lankshear, C. Bigum, & C. Durant (Eds.),

Digital rhetorics: Literacies and technologies in education–current practices and

future directions, Vol. 3 (pp. 131–148). Brisbane, Australia: QUT/DEETYA.

Eagleton, M. B., Guinee, K. & Langlais, K. (2003). Teaching Internet literacy strategies:

the hero inquiry project. Voices from the Middle, 10, 3, 28–35
 Facilitating Critical Thinking Skills through Content Creation  24 

Eagleton, M., & Dobler, E. (2007). Reading the Web: Strategies for Internet inquiry.

New York: Guilford.

Estabrook, L., Witt, E., Rainie, L. (2007). Information Searches That Solve Problems.

Pew Internet & American Life Project: Washington, DC. Retrieved December 20,

2007, from: http://www.pewinternet.org/pdfs/Pew_UI_LibrariesReport.pdf

Flanagin, A. J., & Metzger, M. J. (2000). Perceptions of Internet information credibility.

Journalism and Mass Communication Quarterly, 77, 515–540.

Fogg, B.J., J. Marshall, O. Laraki, A. Osipovich, C. Varma, N. Fang, et al. (2001). What

makes web sites credible? A report on a large quantitative study. Presented to the

Computer-Human Interaction Conference, Seattle, Washington.

Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R.

(2003). How do users evaluate the credibility of Web sites? A study with over

2,500 participants. Proceedings of the 2003 Conference on Designing for User

Experiences (DUX’03). Retrieved April 15, 2008, from

http://portal.acm.org/citation.cfm?id=997097&coll=ACM&dl=ACM&CFID=362

36037&CFTOKEN=18606069

Freebody, P. & Luke, A. (1990) Literacies programs: Debates and demands in cultural

context. Prospecct: Australian Journal of TESOL 5(7), 7-16.

Gee, J. (2003). What video games have to teach us about learning and literacy.

Macmillan. New York.

Graesser, A.C., Wiley, J., Goldman, S.R., O’Reilly, T., Jeon, M., & McDaniel, B. (2007).
 Facilitating Critical Thinking Skills through Content Creation  25 

SEEK Web Tutor: Fostering a critical stance while exploring the causes of

volcanic eruption. Metacognition and Learning, 2(2-3), 89-105.

Doi:10.1007/s11409-007-9013-x

Halliday, M.A.K. (1994). An Introduction to Functional Grammar. 2nd ed. London:

Edward Arnold.

Hannafin, M. J., & Land, S. M. (1997). The foundations and assumptions of technology-

enhanced student-centered learning environment. Instructional Science, 25, 167-

202.

Hannafin, M., & Land, S. (2000). Technology and student-centered learning in higher

education. Journal of Computing in Higher Education, 12(1), 3-30.

Henry, L. (2006). SEARCHing for an answer: The critical role of new literacies while

reading on the Internet. Reading Teacher, 59, 614–627.

Jewitt, C. (2008). Multimodality and Literacy in School Classrooms. Review of Research

in Education, 32(1), 241-267.

Jick, T. (1979). Mixing qualitative and quantitative methods: Triangulation in action.

Administrative Science Quarterly, 24, 602-61 1.

Johnson, T., & Kaye, B. (1998). Cruising is believing? Comparing Internet and

traditional sources on media credibility measures. Journalism & Mass

Communication Quarterly, 75, 325–340.

Jones, S. & Madden, M. (2002). The Internet goes to college: How students are living in

the future with today's technology. Washington DC: Pew Internet & American

Life Project. Retrieved January 20, 2007 from

http://www.pewinternet.org/pdfs/PIP_College_Report.pdf.
 Facilitating Critical Thinking Skills through Content Creation  26 

Judd, V. C., L. I. Farrow and B. J. Tims. 2006. 'Evaluating public web site information: a

process and an instrument'. Reference Services Review, 34 (1): 12-32.

Kaiser Family Foundation. (2005). Generation M: Media in the lives of 8-18 year olds.

Retrieved March 9, 2005, from

http://www.kff.org/entmedia/entmedia030905pkg.cfm

Kiili, C., Laurinen, L., & Marttunen, M. (2008). Students evaluating Internet sources:

From versatile evaluators to uncritical readers. Journal of Educational Computing

Research, 39(1): 75-95.

Kimber, K. & Wyatt-Smith, C. (2006). Using and creating knowledge with new

technologies: a case for students-as-designers. Learning, Media and Technology,

31(1), 19-34.

Kress, G. (2003). Literacy in the new media age. London: Routledge.

Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of

contemporary communication. London: Arnold.

Krueger, R. & Casey, M. (2008). Focus Groups: A practical guide for applied research.

Thousand Oaks, CA: Sage.

Labbo, L. & Reinking, D. (1999). Negotiating the multiple realities of technology in

literacy research and instruction. Reading Research Quarterly, 34, 478-492.

Lankshear, C. & Knobel, M. (1998). Critical literacies and new technologies. Paper

presented at the American Education Research Association convention in San

Diego, CA.

Lemke, J (2002) Travels in hypermodality. Visual Communication 1(3), 299-325.

Lenhart, A., Simon, M., & Graziano, M. (2001). The Internet and education:
 Facilitating Critical Thinking Skills through Content Creation  27 

Findings of the Pew Internet & American Life Project. Washington, DC: Pew

Internet & American Life Project.

Lenhart, A., Madden, M. & Hitlin, P. (2005). Teens and technology: Youth are leading

the transition to a fully wired and mobile nation. Washington DC: Pew Internet &

American Life Project.

Leu, D. J. (2000). Literacy and technology: Deictic consequences for literacy education

in an information age. In M.L. Kamil, P. Mosenthal, R. Barr, & P.D. Pearson

(Eds.), Handbook of reading research: Volume III (pp.743-770). Mahwah, NJ:

Erlbaum.

Leu, D.J., Jr., Kinzer, C.K., Coiro, J., Cammack, D. (2004). Toward a theory of new

literacies emerging from the Internet and other information and communication

technologies. In R.B. Ruddell & N. Unrau (Eds.), Theoretical Models and

Processes of Reading, Fifth Edition (1568-1611). International Reading

Association: Newark, DE. Retrieved October 15, 2008 from

http://www.readingonline.org/newliteracies/lit_index.asp?HREF=/newliteracies

/leu

Leu, D. J., Zawilinski, L., Castek, J., Banerjee, M., Housand, B., Liu, Y., et al. (2007).

What is new about the new literacies of online reading comprehension? In L.

Rush, J. Eakle, & A. Berger, (Eds.). Secondary school literacy: What research

reveals for classroom practices. (37-68). Urbana, IL: National Council of

Teachers of English.

Leu, D., Coiro, J., Castek, J., Hartman, D., Henry, L., & Reinking, D. (2008). Research
 Facilitating Critical Thinking Skills through Content Creation  28 

on instruction and assessment in the new literacies of online reading

comprehension. In Cathy Collins Block, Sherri Parris (Eds.). Comprehension

instruction: Research-based best practices (pp. 321-346). New York: Guilford

Press.

Leu, D. J., Reinking, D., Hutchinson, A., McVerry, J. G., Robbins, K., Rogers, A.,

Malloy, J., O’Byrne, W. I., Zawilinski, L. (2008). The TICA Project: Teaching the

New Literacies of Online Reading Comprehension to Adolescents. An alternative

symposium presented at the National Reading Conference, Orlando, FL.

Leu, D. J., O’Byrne, W. I., Zawilinski, L. McVerry, J. G., & Everett-Cacopardo, H.

(2009). Expanding the New Literacies Conversation. Educational Researcher,

38(4), 264-269.

Livingstone, S. (2004). Media literacy and the challenge of new information and

communication technologies. The Communication Review , 7 (1), 3-14.

Livingstone, S., & Bober, M. (2005). UK children go online: Final report of key

project findings. Project Report. London: London School of Economics

and Political Science.

Lubans, J. (1998, April). How first-year university students use and regard Internet

resources. Available from Duke University Libraries Web site:

www.lib.duke.edu/staff/orgnztn/lubans/docs/1styear/firstyear.htm (accessed may

2000).

Lubans, J. (1999, May). Students and the Internet. Available from Duke University

Libraries Web site: www.lib.duke.edu/staff/orgnztn/lubans/docs/study3.htm

(accessed may 2000).

 Facilitating Critical Thinking Skills through Content Creation  29 

Luke, A. (2000). Critical literacy in Australia: A matter of context and standpoint.

Journal of Adolescent & Adult Literacy, 43, 448-461.

Metzger, M. (2007). Making sense of credibility on the web: Models for evaluating

online information and recommendations for future research. Journal of the

American Society for Information Science and Technology, 58(13):2078–2091.

The New London Group. (1996). A pedagogy of multiliteracies: Designing social futures.

Harvard Educational Review, 66, 60–92.

O’Brien, D. (2006). “Struggling” adolescents’ engagement in multimediating: Countering

the institutional construction of incompetence. In D. E. Alvermann, K. A.

Hinchman, D. W. Moore, S. F. Phelps, & D. R. Waff (Eds.), Reconceptualizing

the literacies in adolescents’ lives (pp. 29-46). Mahwah, NJ: Erlbaum.

Orsmond, P., Merry, S. & Reiling, K. (2000) The use of student driven marking criteria

in peer and self assessment, Assessment & Evaluation in Higher Education, 25(1),

23–38.

Orsmond, P., Merry, S. & Reiling, K. (2002) The use of exemplars and formative

feedback when using student derived marking criteria in peer and self-assessment,

Assessment & Evaluation in Higher Education, 27(4), 309–323.

Rieh, S., & Belkin, N. (1998). Understanding judgment of information quality and

cognitive authority in the WWW. Proceedings of the ASIS Annual Meeting, 35,

279–289.

Shackleford, J., Thompson, D., & James, M. (1999). Teaching strategy and assignment

design: assessing the quality and validity of information via the Web. Social

Science Computer Review, 17(2), 196–208.

 Facilitating Critical Thinking Skills through Content Creation  30 

Street, B. (1984). Literacy in theory and practice. Cambridge: Cambridge University

Press.

Street, B. (2003). What’s “new” in New Literacy Studies? Critical approaches to

literacy in theory and practice. Current Issues in Comparative Education, 5(2), 1–

15.

Tate, M., & Alexander, J. (1996). Teaching critical evaluation skills for World Wide Web

resources, Computers in Libraries, 16(10), 49-55.

Taylor, R. S. (1986). Value-added processes in information systems. Norwood, NJ: Ablex

Publishing.

Wittel, A. (2000). Ethnography on the move: From field to net to Internet. Forum:

Qualitative Social Research, 1(1). Retrieved on Dec. 24, 2006, from

http://www.qualitative-research.net/fqs-texte/1-00/1-00wittel-e.htm

Wysocki, A. (2001). Impossibly distinct: On form/content and word/image in

two pieces of computer-based interactive multimedia. Computers and

Composition, 18, 209–234.

 Facilitating Critical Thinking Skills through Content Creation  31 

Table 1 Stronger and Weaker Sincere Sites

Stronger Sincere Site Website Address

Asthma page for Wikipedia.com http://en.wikipedia.org/wiki/Asthma

Asthma page for WebMD.com http://www.webmd.com/asthma/

Asthma & allergies page for http://kidshealth.org/kid/health_problems/

KidsHealth.org allergy/asthma.html

Weaker Sincere Sites Website Address

Angry Asthma Mama blog http://angryasthmamama.blogspot.com/

Mike Reed’s memoir about http://usads.ms11.net/reed.html

Asthma the dog

Asthma page from Dr. Greene.com http://www.drgreene.org/body.cfm?id=21& 

action=detail&ref=1031 

Asthma Girl blog http://www.asthmagirl.com/asthma-on-parade/

 Facilitating Critical Thinking Skills through Content Creation  32 

Table 2 Hoax Websites

Hoax Website Website Address

Computer Tan http://www.museumofhoaxes.com/hoax/weblog

/permalink/computer_tan/

Pacific Northwest Tree Octopus http://zapatopi.net/treeoctopus/

Buy Dehydrated Water http://buydehydratedwater.com/

CLONES – R - US http://www.d-b.net/dti/

Dog Island http://www.thedogisland.com/

Table 3 OCC Tools

Tool Purpose Tool Name

Website Construction iWeb

Photo Editing Aviary (add-on for Firefox)

Audio Editing Audible

Video capture & editing iMovie

Text Editing Microsoft Word

 Facilitating Critical Thinking Skills through Content Creation  33 

Appendix 1 Storyboard Example