Академический Документы
Профессиональный Документы
Культура Документы
a r t i c l e i n f o
a b s t r a c t
Article history:
Received 23 May 2014
Received in revised form
3 September 2014
Accepted 7 October 2014
Available online 16 October 2014
The requirements of a contemporary workplace include the ability to think critically and creatively in
order to solve problems and respond to changes in economic and social conditions. Unfortunately,
vocational education often fails to prepare graduates for this environment due to limited resources, low
student motivation, or the reliance upon outdated instructional strategies. The use of digital game-based
learning (DGBL) for vocational education has been proposed, but has yet to be effectively implemented,
particularly in terms of the promotion of higher order thinking skills (HOTS). Data from 68 eleventh
grade vocational high school students were evaluated after a quasi-experimental, 27 week intervention.
Pretest and posttest results were evaluated by MANCOVA and demonstrated that the experimental group
(blended DGBL incorporating integrative HOTS activities) outperformed the comparison group (technology enhanced learning) in terms of creative thinking, critical thinking, problem solving, and academic
achievement, with signicant improvements on all four measures. While technology-enhanced learning
was effective in promoting academic achievement and creative thinking, the DGBL condition was
deemed most effective in providing an authentic context for developing employment-related skills and
knowledge. Based on these results, a blended approach for DGBL, which incorporates instructor
orchestration and scaffolding, provision of learning aids, and the use of collaborative learning, is recommended, particularly for vocational learners. This paper provides examples of a concrete model of
DGBL instruction that was veried empirically as successful in signicantly improving all three higher
order thinking skills, including creative thinking, critical thinking, and problem solving.
2014 Published by Elsevier Ltd.
Keywords:
Applications in subject areas
Cooperative/collaborative learning
Improving classroom teaching
Secondary education
Simulations
1. Introduction
Current workplace environments, information-intensive and continually evolving, require employees to continually acquire and transfer
skills and competencies to changing conditions and economic demands (Boahin & Hofman, 2013). The demand for general employability
skills and the ability to develop professional competence highlights the importance of creative thinking, critical thinking, and problemsolving, as well as communication skills, information and communication technology (ICT) skills, and collaboration (Partnership for 21st
Century Skills, 2009). Research has shown that fostering these employability skills and competences benets from a student-centered
approach to problem-based learning which adopts authentic employment-based learning and involves teamwork (Boahin & Hofman, 2013).
The exible and relevant skills and knowledge necessary for preparing students for employment in real world businesses cannot be
developed through traditional instruction that lacks authenticity and ignores the complex realities of the marketplace (Lainema & Nurmi,
2006). On the other hand, simulations or games can provide a) hands-on and relevant experiences with virtual business or employment
settings (Siewiorek, Saarinen, Lainema, & Lehtinen, 2012; Williams, 2008), and b) develop higher order thinking skills (HOTS) when
adopting social constructivist principles, such as active knowledge construction, the provision of support tools, and interaction among
participants (Kiili, 2005; Li, Cheng, & Liu, 2013).
282
While DGBL has been used for decades, the adoption of simulation games for developing work-related knowledge while promoting HOTS
for vocational students has yet to be explored. In fact, many scholars suggest that rigorous empirical research demonstrating the effectiveness of digital games in learning is lacking (Bourgonjon et al., 2013; Erhel & Jamet, 2013). As Papastergiou (2009) noted, four important
gaps in the literature concerning DGBL should be explored: the lack of emphasis on learning outcomes within authentic classroom environments; the tendency of studies to emphasize traditional subjects, such as mathematics; a lack of studies with adolescent students; and
the problematic use of digital games as an alternative rather than a complement to classroom teaching. In order to address these
outstanding research gaps, this study adopts an authentic blended learning context, evaluating learning outcomes from adolescents in a
vocational education context, and exploring the impact of DBGL on higher order thinking by adopting a research design that integrates game
play with classroom teaching rather than viewing game play as an alternative to classroom instruction.
1.1. Vocational education in Taiwan
Vocational Education and Training, concerned with the development and training of technical and work-related skills, is critical for
increasing employment and ensuring economic stabilization (Inayat, ul Amin, Inayat, & Salim, 2013). As such, most countries rely upon some
form of vocational education to develop a competent workforce and provide an alternative to higher education for learners who seek
employment immediately following graduation. However, two main challenges face vocational educators and students: students' low
engagement and motivation and outdated approaches to instruction that fail to meet the demands of the marketplace.
In Taiwan, students often enroll in vocational schools because of low academic achievement and have, as a result, lower motivation,
condence, and attitudes toward learning (Lee, Shen, & Tsai, 2010). Taiwan's vocational schools also face the difculties of low enrollment,
little public investment, and students who may be distracted by part-time work and lack engagement in the classroom (Lee, Shen, & Tsai,
2008). Additionally, vocational schools are often private and must compete for students, accepting applicants with low levels of academic
achievement (Tsai, Lee, & Shen, 2013). Along with the resulting lack of achievement, many students also lack the problem-solving and
critical thinking skills which are highly valued by employers.
In terms of educational approaches to vocational education, a current report on the state of technical and vocational education in Taiwan
(Taiwan Ministry of Education, 2013) noted that vocational schools often lack an emphasis on job training and are increasingly adopting
exam-oriented approaches to both instruction and evaluation, prompting students to pursue vocational college after high school rather than
enter the marketplace, since vocational high school graduates are unprepared and, as a result, under-paid. The report recommends an
employment-oriented approach to vocational education, facilitating the transition from vocational school to the workplace through the
provision of diverse and exible curricula, hands-on experiences, experimentation, and training in skills relevant to specic industries in
which graduates might seek employment.
Solutions to these challenges may be provided by technology-enhanced learning (TEL). In fact, some studies (i.e., Lee, et al., 2010) have
suggested emphasizing authentic learning experiences with the assistance of e-learning in order to promote problem-solving and selfregulation among Taiwanese vocational students. However, the design of Lee et al.'s experiment relied upon applications of technology
(such as spreadsheet and word processing programs) which did not adequately simulate the complexities and realities of the workplace. As
well, student involvement, rather than actual problem-solving skills, was used as a dependent measure. Therefore, in this study, the
integration of digital games with employment-oriented instruction will be empirically evaluated.
1.2. Digital Game-Based Learning (DGBL)
As a form of TEL, digital games are believed to offer several advantages to learners, some of which may be particularly useful for
vocational learners. Given the changing demands of marketplaces within knowledge-based economies, scholars have argued that contextbased learning with realistic or simulated learning activities, and augmented by information and communications technology, will provide
learners with much needed vocational skills and experiences.
Game-based learning is grounded in the notion of play as an intrinsically motivating experience and, as an extension of Problem Based
Learning (PBL), as a constructivist principle guiding learning through active learning (Wu, Chiou, Kao, Hu, & Huang, 2012). In terms of game
play, Malone (1982), in an early analysis of learner' intrinsic interest in computer games, concluded that three main guidelines inuenced
learner enjoyment: challenge, including the provision of clear goals along with a range of uncertain and adaptive outcomes; fantasy, which
involves an appeal to players' emotions; and curiosity, which is elicited through humor and visual effects and which balances players'
background knowledge with opportunities for acquiring new information. The adoption of a PBL approach to DGBL was explored in detail by
Warren, Dondlinger, McLeod, and Bigenho (2012) who proposed that PBL allows for an authentic learning environment which engages
learners in story-like problem-based tasks.
While digital games have been used over the past few decades, games are increasingly focused on learner-centered approaches and
many have begun including instructional supports, such as clues, feedback, or explanations, to assist learning (Kebritchi & Hirumi, 2008). As
a result, opportunities for collaboration and problem-solving are plentiful in digital games, particularly for simulation games which allow
learners to explore and test hypotheses. Kiili's (2005) model of experiential gaming emphasizes learner engagement, requiring the provision
of immediate feedback, and the use of clear and appropriate challenges which take into consideration players' skills and prior knowledge.
Additionally, cognitive load is a critical factor in successful digital game-based learning. As such, the use of a blended learning environment
was adapted for this study in order to reduce the cognitive load which might arise from the presence of multiple, simultaneous challenges
during game play.
While few studies in the area of digital game-based learning emphasize the area of business management, business simulation games
(BSGs) can provide more authentic and practical opportunities for exploring realistic work environments and engaging in the decisionmaking necessary for success in the marketplace (Chang, Chen, Yang & Chao, 2009; Lin & Tu, 2012). Simulations have even been adopted for developing leadership skills in business education at the graduate school level (Siewiorek et al., 2012) and for professional development with construction professionals (Wall & Ahmed, 2008). The appropriateness of digital games for vocational learners at the high
283
school level will be evaluated by this study. In particular, we will focus on two principle learning outcomes: the acquisition of HOTS and
academic achievement in a course focused on employment-oriented knowledge and skills.
1.3. DGBL and higher order thinking skills
Since employers demand workers who have the ability to think critically and solve problems, classes which rely upon lower-level
thinking and fail to provide opportunities for critical thinking will jeopardize the employability of graduates (Kiener, Ahuna, & Tinnesz,
2014). Thus, HOTS, including creativity, critical thinking, and problem solving, are crucial for vocational students whose training is oriented toward future employability and the development of transferable skills which will lead to a professional career.
Recent papers in the eld of secondary education have highlighted the potential of digital games for fostering 21st century skills,
including HOTS as well as soft skills such as collaboration and global awareness (Ray, Faure, & Kelle, 2013). Well-designed digital games
cannot only assist learners in knowledge construction, but develop HOTS which extend beyond the game to students' personal lives (Kiili,
2005). However, many such studies do not empirically evaluate the impact of digital games on HOTS. For example, in a recent study, Kiener,
et al. (2014) adopted an action research approach to evaluate the role of critical thinking in developing a professional disposition and the
higher order thinking necessary for employability, offering classes which combined lecture components with discussion and case study
analysis. Their ndings indicated that an emphasis on critical thinking skills was an important contributor to students' development of a
professional perspective.
In the context of business or vocational education, simulation offers several advantages, including linking learning with authentic duties,
offering opportunities for collaboration and sharing of experiences, and introducing students to the complexities of the real world, allowing
them to experiment and take risks in a safe context (Siewiorek et al., 2012). Problem-based approaches to games or simulations provide
learners with opportunities to explore and experience realistic scenarios in which all learning tasks are linked to a larger, authentic, and
complex problem in which players can test alternative ideas, take ownership for the consequences, and receive feedback and opportunities
for reection in order to develop HOTS (Warren et al., 2012).
This type of problem-based gaming ts with what Kiili (2007) describes as the three fundamental DGBL characteristics for effectively
developing problem-solving skills: authentic and realistic simulations which leads to learning that can be transferred to the real world;
cooperative team play and knowledge sharing; and learning through active, hands-on participation. One important factor in Kiili's (2007)
problem-based gaming study was an emphasis on critical reection, necessary for problem-solving, planning, and adaption during game
play. Based on these ndings, the present study adopts an authentic simulation game with collaborative small groups for game play.
Additionally, support is provided through integrative learning activities which foster student-centered, active learning, engagement with
the learning topic, support of game play, and sufcient time and scaffolding for critical reection.
1.4. DGBL and academic achievement
Digital games can be successful in addressing the real-world needs of vocational students, creating opportunities for learning trans m
ferable skills and rehearsing workplace-specic knowledge (Ha
al
ainen, Oksanen, & H
akkinen, 2008). It is expected that digital games will
improve student attention and motivation while providing a problem-based learning environment which promotes the rehearsal and
application of curricular content, resulting in improved academic performance. In fact, according to Ebner and Holzinger (2007), digital
games provide opportunities for in-depth learning of subject matter by simulating authentic experiences through realistic settings. The
advantage of simulation games in terms of decision-making and problem-solving was explored by Pasin and Giroux (2011) through the
examination of student errors over the course of simulation play, nding that students benetted from the simulation's realistic context,
emphasis on the need for continual adjustment, and monitoring and responding to ongoing changes and consequences. Thus, it appears that
learners can develop the knowledge and skills necessary to succeed academically, when provided with appropriate and relevant digital
gaming experiences.
However, due to mixed empirical results, Erhel and Jamet (2013) suggest that it is difcult to conclude decisively that digital games and
simulations have a positive impact on learning and motivational outcomes. As such, the authors advocate a value-added approach, which
explores the features of digital games which can enhance the quality of learning, nding that the nature of game instructions and feedback
provided can inuence the development of deep learning. In extending the evaluation of digital game based learning to a blended learning
environment, the current study designs supplemental and scaffolding activities for integrating HOTS with curricular objectives and game
play.
1.5. Suggestions for the effective integration of DGBL
Based on a review of the literature, certain key recommendations and suggestions were discovered which relate to the more effective use
of DGBL. These recommendations concern the use of collaboration, provision of learning aids, scaffolding of learning and instructor
orchestration, and a blended learning environment.
1.5.1. Collaboration
First, collaboration is essential for online learning environments to be successful in promoting learning and higher order thinking, such as
critical reection, questioning, reasoning, and problem-solving. In order to design a computer-supported collaborative learning environ m
ment (CSCL) for vocational students, Ha
al
ainen, et al. (2008) adopted collaboration scripts within the game to prompt learners to interact,
thereby enhancing their learning of authentic, transferable skills which would have been difcult to achieve in traditional classroom environments. Student collaboration in online vocational learning environments was also explored by Inayat et al. (2013) in four vocational
courses, in which it was found that group interaction (among students and between students and instructors), instructor feedback, and the
provision of support materials were signicant contributors to learning.
284
285
Will participants who learn through DGBL improve more signicantly in terms of critical thinking, as compared to those who learn
through other forms of TEL (comparison group)?
Will participants who learn through DGBL improve more signicantly in terms of problem solving, as compared to those who learn
through other forms of TEL (comparison group)?
Will participants who learn through DGBL improve more signicantly in terms of academic achievement, as compared to those who
learn through other forms of TEL (comparison group)?
3. Methods
In order to evaluate the above research questions, a 27 week quasi-experimental design was adopted. Fig. 1 illustrates the overall design
of the experiment, including both independent and dependent variables.
3.1. Participants
Participants included seventy-seven students (16e17 years old) who were recruited from and completed two sections of an 11th grade
Store Planning and Management class at a vocational senior high school in Taiwan. Both sections were taught by the same instructor, using
the same textbook, schedule, and evaluations but were taught using two different instructional strategies. One class, with 41 students (12
males and 29 females), was taught with TEL activities and served as the comparison group, while the other class, with 36 students (7 males
and 29 females), was taught using DGBL and served as the experimental group.
3.2. Independent variable
The independent variable was instructional strategy: TEL vs. DGBL. The comparison group learned through classroom activities that
featured the integration of technology for enhancing learning (such as the use of an interactive whiteboard, multimedia presentations, and
software for spreadsheets, word processing, and desktop publishing), and included class discussions, debates, and student-centered
learning activities which were designed to enhance both student's higher order thinking, as well as academic achievement. Meanwhile,
students in the experimental group engaged in DGBL activities, including both tutorials and game challenges, supplemented by activities
aimed at cohesively integrating higher order thinking with digital game play. The class activities and allocation of time for the two groups
are shown in Table 1.
3.3. Dependent variables
The four dependent variables evaluated in this study were students' HOTS (including creative thinking, critical thinking, and problem
solving) and academic achievement in Store Planning and Management. These variables were measured by the following instruments.
3.3.1. Creative thinking skills
Creative thinking was measured by the Torrance Test of Creative Thinking with Pictures (Torrance, 1998) which evaluates creativity in
ve dimensions: uency, originality, elaboration, resistance to premature closure, and abstractness of titles. The test form consists of three
parts: construction and naming of a large abstract picture; completion and labeling a set of incomplete gures; and using a series of
repeating shapes (such as circles or parallel lines) in order to create as many gures as possible. Ten minutes is provided for each part. In
addition, bonus points are awarded for thirteen creative strengths, such as expressiveness, storytelling, and movement and action. The
reliability of the measure has been reported as Cronbach's a .81 (Torrance, 1981).
In the case of uency, one point is allotted for each unique and meaningful gure which is drawn or completed. To evaluate originality,
the scoring manual provides a list of responses which, based on many thousands of tests being evaluated, are considered too common to be
original. Figures which are drawn that are not listed in the scoring manual are statistically uncommon and, therefore, awarded points for
286
Table 1
Summary of the two research groups including time allocated for learning tasks.
Instructional activities
Content delivery
Student-centered learning tasks (targeting HOTS
and academic achievement)
originality. Elaboration is evaluated by the degree to which of meaningful elements added which go beyond the minimum required, such as
color, shading, or additional details. Resistance to premature closure relates to Activity Two, and evaluates the test-taker's resistance to
making the shortest or easiest possible completion of an incomplete gure. Abstractness of title is also evaluated by a rubric, which considers the amount of description, particularly non-concrete, which is used to describe or name gures. Other creative strengths, such as
humor add bonus points to the overall score.
3.3.2. Critical thinking skills
Critical thinking skills were measured by the Critical Thinking Test-Level I (Yeh, 2003), which includes a total of ve subscales: recognition of assumptions (identifying implicit claims), induction (inferences based on a set of known facts), deduction (drawing a rational
conclusion on the basis of given premises), interpretation (identifying causal relationships implied by statements), and evaluation of arguments (assessing the strength of an argument). Each subscale includes 5 multiple choice questions worth one point each, with a
maximum score for the test of 25 points. The overall Cronbach's a of the CTT-I is .76.
Two examples will be provided to illustrate the subscales of the CTT-I. For example, one question from the recognition of assumptions
subscale asks students to consider the assumptions behind the statement I have to check if there is a warm coat in my suitcase, since I am
going on a trip to Yangmingshan National Park tomorrow. Three choices are provided: a) Yangmingshan National Park will be cold
tomorrow; b) There is already a warm coat in the suitcase; and c) People should bring a coat when going to Yangmingshan. Of these options,
a) is the correct answer. For the subscale of deduction, one question provides students with two premises and asks them to identify the
correct conclusion: The amusement park is crowded on holidays; tourists will be in a bad mood when it is crowded. The three options are:
a) Tourists who go to the amusement park on holidays will be in a bad mood; b) The reason why people have a bad temper is because of
crowded spaces; and c) People have a bad temper on holidays. Of these options, a) is the correct answer.
3.3.3. Problem solving skills
Problem solving was measured by the New Test of Problem Solving (Chan & Wu, 2007) which includes three subscales: nding causes,
nding solutions, and avoiding problems. This test proposes several scenarios, to which students can provide a maximum of three responses
in terms of the cause, solution, and future avoidance of problems. This measure evaluates both the exibility (i.e., the variety and
uniqueness) and effectiveness (i.e., the practicality and usefulness) of the solution in computing a nal score. The test-retest reliability of the
test is .86 with a Cronbachs a of .91. A sample question for the nding solutions subscale of the NTPS is provided as follows: On your way
to school, the tire of your bicycle went at. You worry that you cannot arrive at school on time. What will you do now?
Students' answers are rst identied according to certain categories of response, which are provided in the NTPS scoring manual. For this
question, answers are organized into four categories: A (xing the bicycle by yourself), B (asking somebody to give you a ride), C (going to
school on your own), and D (informing your teacher of the situation). A rubric in the NTPS manual provides sample items for evaluating
effectiveness scores for each response, ranging from a score of 0 (non-effective solution) to 2 (full points). For exibility, one point is awarded
for each answer that is effective (i.e., received a raw score greater than zero) with a maximum of one point awarded for each category
answer. Effectiveness is calculated as the sum of the raw scores for each response to the question. Using effectiveness and exibility scores,
an overall problem solving score can be computed. As noted above, this score is equal to (exibility 2) effectiveness.
3.3.4. Academic achievement
To evaluate students' academic achievement in Store Planning and Management, a posttest was developed based on the national
vocational education curriculum. The pretest scores of both comparison and experimental groups were obtained from students' nal exams
of a business class from their previous semester. Content/expert validity was achieved through the cooperation of two experienced
vocational high school teachers with approximately 10 years of teaching experience. The academic achievement test consisted of four
categories: principles of store planning and management (four questions), business processes (four questions), inventory management (four
questions), and marketing strategies (eight questions). Each of the 20 multiple choice questions was worth 5 points for a maximum score of
100 points.
3.4. Research procedures
The duration of the quasi-experiment was 27 weeks, including two weeks for pretests and two weeks for posttests. The remaining 23
weeks of the experiment was divided into two semesters, as shown in Table 2.
An elective Store Planning and Management class was selected as the setting for this study, since it was deemed most appropriate for
the integration of digital games or simulations targeting employment-related skills and knowledge. During the rst semester, following two
weeks of pretests, one 50 min class per week was taught for each section (Weeks 3 through 12). However, based on a design-based research
orientation, teacher input was welcomed for the modication of the instructional design. Thus, during the course of the intervention, the
287
Table 2
Experimental procedure.
Research group
Pretests
Semester 1
Semester 2
Posttests
Week
Comparison
Experimental
1e2
O1
O1
3e12
X1
X2
13e24
X1
X2
25e27
O2
O2
Notes. X1: Technology-enhanced learning (comparison group). X2: Digital game-based learning (experimental group).
classroom teacher determined that the class time was too short for both groups and used exible study hours to provide an additional
50 min per week for both groups during the second semester of the course (Weeks 13 through 24), in order to ensure that the effectiveness
of the two approaches could be reasonably compared. This coincides with Taiwan Ministry of Education's (2013) suggestion that diverse and
exible curricula be provided for learners. The study concluded with two weeks of posttests.
For both groups, the rst semester of both courses included the topics of business processes and inventory management, while the
second semester focused on marketing, market segmentation, and market strategies. For both groups, reviews and reports were administered at the end of the rst semester, while debates and group presentations were used at the end of the second semester. A summary of
the two instructional strategies is provided in the following sub-sections, while Tables 3 and 4 provide details on the instructional content
for both groups.
3.4.1. Experimental group: DGBL
The experimental group played the Mandarin version of two commercially available digital games, Chief Executive Ofcer (CEO) produced
by T-Time Technology, and Capitalism 2 by Enlight. As shown in Table 1, approximately 15% of class time was allocated to the presentation of
textbook content, with another 15% used for previewing of games and game tutorials. Once students were familiar with the games, around
35% of the class time was used for game play and reection on the game tasks. Furthermore, in order to integrate course content with game
play, integrated learning tasks were provided, requiring around 20% of the class time. These activities were designed by the researcher and
Table 3
Weekly activities for comparison and experimental groups (semester 1).
Week Instructional content
(for both groups)
1e3
4
Pretest
Introduction
Multimedia presentations
- Introduce and compare Tabulate and compare
types of processes
processes
Business Processes
Internet search
- selection factors
Downloading of related
- product life cycle
content
Inventory management
Multimedia
- dene stock and
Group brainstorming:
inventory types
importance of inventory
- the function of
management
inventory
Inventory management
Internet search
- performance evaluation Downloading of related
- classication
content
Case study 1:
Case study 2:
11
12
Introduction to the
course and grouping
Business processes
10
Group reports
Review week
Introduction to the
course and grouping
4: Produce aluminum
activity: select
team name
Creative thinking Design three
potential team
logos
Creative thinking Create an original
team chant
2: Produce lumber
Tutorials: processing 3: Produce paper
materials to
manufacture
desks.
6: Retail the desks for prot
Game play: CEO
Critical thinking
Critical thinking
Problem solving
Problem solving
Problem solving
critical thinking:
sample questions
based on Store
Management
Induction and
deduction: evaluate
statements about
management
Interpretation,
Evaluation,
Recognition of
Assumptions
activities
Dening problems
and causes
associated with
failing to earn
100 million
Preventing future
problems: solutions
for improving future
performance
SWOT analysis of
production strategies
288
Table 4
Weekly activities for comparison and experimental groups (Semester 2).
Week
Instructional content
(for both groups)
13
Introduction
14
Marketing I:
- producer-oriented
marketing
management
15
Marketing II:
- producer and
consumer oriented
marketing management
16
Marketing III:
- social marketing
- evolution of
marketing
17
Market Segmentation I:
- geographic and
demographic
variables
18
- requirements
19
Marketing Strategies I:
- denition and
types of marketing
strategies
20
- functions
21
- international and
internet marketing
22
Debate
23e24
Group presentations
25e27
Posttests
Introduction to the
course, grouping,
vote for group leader
Introduction to the
course, grouping, and
voting for group leader
(mayor)
Creative
thinking Revise team
Students go online to
read a blog
Class discussion with
SMART board
Design a table to
contrast producer
and consumer
oriented approaches
Online search
Tutorials: marketing in
capitalism 2 1: Basics of retail
Critical
thinking Induction:
Tutorials: marketing in
capitalism 2 2: Retailing: part 2
Problem
solving Dening
3: Manufacturing
causes
Preventing problems
Tutorials: marketing in
capitalism 2 4: Advertising
Creative
thinking Evaluate
million NTD
and Brand
Game challenge II: advertising
Debate which is
more important for
marketing: corporate
branding or product
quality
Group presentations
on marketing
strategies (topics
approved by the
instructor) and
student assessment
patterns of
consumption
Evaluate and
explain prot goal
progress
Suggest reasons for
the results of game
play
the meaning
and creativity
of an award logo
Logo redesign
Problem
solving Evaluating
attributes of top
Taiwan brands
Game challenge III: emerging
markets Boomtown (single-player):
Desktop publishing
to creating an
advertisement
Multimedia of
international and
internet strategies
Review quiz for
Strategies unit
name, logo,
and chant
Critical
thinking Evaluate a
mobile
marketing
strategy (4Ps)
Evaluate brand loyalty
claims
Creative
thinking Design a
mobile phone
(creativity criteria)
Game challenge IV: management
team contest Boomtown
(multi-player):
Groups compete online
to earn a prot
Problem
Solving SWOT analysis
of marketing
strategies to
compete in the
Team Contest
Critical
thinking Assessment
rubric
classroom teacher in order to link textbook content with game play while fostering higher order thinking. It should be noted that while
game play itself, as well as reporting on game play and other class activities, were designed to promote higher order thinking and academic
achievement, the highlight of the HOTS-integrated learning tasks was to synthesize game play elements with course content in order to
provide scaffolding for game play as well as hands-on, blended, learning activities. This was deemed important in order to allow DGBL
learners with equal opportunities to engage with course-relevant material outside of the context of game play, in order to create a truly
blended learning environment that balanced the strengths of both digital and face-to-face environments.
289
In keeping with the principles of digital game play covered in the introduction, the experimental group included the following
characteristics.
3.4.1.1. Collaboration. Students formed collaborative teams of six groups of six students each. While a smaller group number was desired,
limits on available computers necessitated larger groups. At the beginning of both semesters, activities were aimed at fostering collaboration
and a sense of group identity, including creating group names, logos, and chants.
3.4.1.2. Learning aids. Following the recommendation of previous studies, clear learning aids were provided. Game tutorials, based on
examples provided by the game publisher or created by the instructor, were assigned in order to ensure learner's familiarity with the games'
functions and tasks and to avoid frustration. Furthermore, the HOTS integrative activities were designed to lead students to reect on their
game play experience in order to effectively integrate textbook knowledge with the games' simulation activities.
3.4.1.3. Scaffolding and teacher orchestration. Given the blended nature of the learning environment, discussed below, the teacher's presence during game play was ensured, and explicit orchestration and regulation of game play was provided in cases where students were
unsure how to proceed independently. Instructional scaffolding was provided both through the aforementioned learning aids, as well as
through modeling of game play, the provision of advice and suggestions during game play, and the weekly integrative HOTS tasks.
3.4.1.4. Blended learning environment. While a certain amount of time was required for face-to-face instruction (including the introduction
of course content and class discussion), a signicant portion of the class time was allocated for autonomous learning through computermediated gaming activities. As such, the advantages of online multi-player gaming and self-paced game play were balanced with access
to teacher support and periods of group reection and discussion.
3.4.2. Comparison group: TEL
While the experimental group used digital games to apply course content to simulated employment-oriented activities, learners in the
comparison group relied upon other forms of TEL in addition to direct instruction, which constituted approximately 20% of class time.
This design was adopted in order to provide a fair comparison between DGBL and an alternative, state-of-the-art instructional strategy.
Activities which utilized the benet of instructional technology for fostering higher order thinking and academic achievement
(approximately 60% of the class time) included inquiry-based learning tasks using internet searches and blog reviews, the adoption of an
interactive whiteboard with videoconferencing capabilities for centralizing classroom discussions by constructing and documenting
knowledge collaboratively, and the integration of interactive multimedia for illustrating principles related to the course content. As Lee,
et al. (2010) have suggested, the use of TEL is expected to bring about improvements to higher order thinking skills, such as problem
solving. Thus, the activities for the comparison group were designed to foster higher order thinking, based more directly on the content of
the course textbook, such as debates and discussions (emphasizing critical thinking skills), case studies and blog reviews (focusing on
problem solving), and students' use of multimedia and design of presentation materials (highlighting creative thinking skills). While
these traditional TEL activities were designed to foster higher order thinking, they did not integrate elements of digital game play and,
thus, were focused on the content of the textbook and supplemental materials. Details of these instructional methods are included in
Tables 3 and 4
Principles of TEL for the comparison group also mirror many of the improvements and innovations adopted for DGBL in this study. For
example, collaboration was emphasized as students also engaged in group work for each of the TEL elements of the course. Furthermore,
scaffolding and teacher orchestration is typical in technology-enhanced classes, and learning aids (in the form of multimedia presentations,
templates for organizing online research, and prociency-leveled blog readings) were provided for comparison group learners. In terms of
the learning environment, both the comparison and experimental group were blended in the sense that they involved both face-to-face and
online elements. However, the TEL group did not employ digital games or simulations and relied upon alternative learning activities for
promoting higher order thinking and employment-related skills, such as searching and evaluating authentic online sources, organizing
information in the form of reports, and engaging in discussions and debates.
3.5. Data analyses
Quantitative data were collected from pretests and posttests for each of the dependent variables. Before analysis was conducted, outliers
in the data were identied by interquartile range (Q1 1.5 IQR and Q3 1.5 IQR) and removed in order to avoid experimental error
(Tukey, 1977). Boxplots for data from the four measures (TTCT, CTT-I, PS and AA) were constructed to identify and remove nine outliers. Each
of these outliers was from the comparison group, and all of them demonstrated abnormally low posttest results. Then, descriptive statistics
were employed to describe and compare the means and standard deviations for the four tests between the two groups. Based on signicant
Pearson's correlations among dependent variables (r between .27 and .52, p < .05) and acceptable homogeneity of covariance (insignicant
Box's M test .16), multivariate analysis of covariance (MANCOVA) was selected for analyzing the results (Tabachnick & Fidell, 2012).
4. Results
Complete and useable data were collected from 68 participants in the quasi-experiment. Descriptive statistics (shown in Table 4) are
provided for the dependent variables. One-way MANCOVA, using the four dependent variables' pretests as covariates, was adopted to
analyze the difference between pretest and posttest scores for all of the dependent variables.
The results reveal that the posttest scores for academic achievement, creative thinking, critical thinking and problem solving differed
signicantly between the two research groups, Wilks' L .37, F 25.07, p .00. Thus, univariate F tests were conducted as follow-ups for
creative thinking, critical thinking, problem solving and academic achievement, as shown in Table 5.
290
Table 5
Descriptive statistics for creative thinking (TTCT), critical thinking (CTT-I), problem solving (PS) and academic achievement (AA).
Comparison group (n 32)
Pretest
TTCT
CTT-I
PS
AA
Pretest
Posttest
SD
SD
Adj. M
SD
SD
Adj. M
432.41
17.91
108.22
43.91
105.30
3.04
13.53
10.68
486.59
17.19
103.03
61.41
47.55
3.16
14.55
11.93
482.71
17.12
102.89
60.73
443.00
17.61
101.39
31.58
71.66
3.33
17.73
11.51
565.58
19.56
117.67
86.25
56.08
2.25
9.49
11.24
569.04
19.62
117.80
86.85
291
Fig. 2. Creative thinking activity: Illustration of game play and integrated HOTS activity for Week 5.
unique, team-specic products (originality subscale) such as a logo and chant. Other creative dimensions, such as elaboration, abstractness
of title, and creative strengths, such as expressiveness and storytelling, were included as factors for group members to consider in determining the chant, motto, or logo selected for their team. These activities also served to motivate students, which is particularly important
since Taiwanese vocational learners often suffer from low motivation (Lee et al., 2010).
Since the comparison group participants explored multimedia and collaboratively designed presentations, created advertisements, and
organized materials digitally, an improvement in creative thinking was observed between the pretest and posttest. However, this
improvement was not signicant, suggesting that these activities did not have the same sense of authenticity and practicality as the
experimental group activities, which were directly linked to the challenge, fantasy, and curiosity evoked by the digital games (Malone, 1982).
Although game play itself likely provided some advantages over TEL in terms of opportunities for creativity, the use of a blended game-based
learning environment with HOTS integrative activities as aids and scaffolds for higher order thinking offered the opportunities necessary to
think creatively for and about game play while emphasizing the application of employment-oriented skills and knowledge.
5.2. Impact on critical thinking
Opportunities for critical thinking were provided by digital game play, which required players to evaluate the relative impact of market
conditions, competition, and their own decisions and strategies on game outcomes (induction and deduction subscales). Since vocational
learners often lack motivation and engagement, this study adopted several support mechanisms for learning: a) a blended environment
which offered personalization and feedback, b) scaffolding and teacher orchestration of activities, c) the provision of learning aids to assist in
acquiring and applying the course content, and d) peer support through a collaborative learning context. As such, groups of learners were
provided with the tools and support necessary to think critically about their decisions in the game, and reect upon their progress and any
difculties they faced, critical thinking factors which are important for employability (Kiener et al., 2014).
One example of an activity which supported learners in critically evaluating their decisions in the simulation is shown in Fig. 3. When
presented with a balance sheet and income statement for each of the different products selected by the group, the HOTS integrative activity
guided learners in evaluating how uctuations in cost associated with inventory and manufacturing variables impacted their protability
(interpretation subscale). Further, this form required learners to evaluate the most probable reason for cost uctuations before making a
nal decision as to which product to manufacture in the next stage of game play (evaluation of arguments subscale). In preparation for the
competition of the following week, this handout provided an external learning aid which enhanced learners' ability to remember and record
key information in order to make logical decisions (Amory et al., 1999). In other activities, the learners were prompted to consider a
consumer-oriented as compared to a producer-oriented approach when evaluating the role of the market in supply and demand (recognition of assumptions subscale).
292
Fig. 3. Critical thinking activity: Illustration of game play and integrated HOTS activity for Week 9.
The fact that critical thinking was not enhanced through TEL for the comparison group can be explained by the fact that fostering critical
thinking generally requires a fair amount of explicit instruction and practice in developing these skills. TEL activities did offer learners the
opportunity to engage in debate, compare and contrast alternative perspectives, and evaluate authentic case studies, however these activities were not translated into improved performance the CTT-I test of critical thinking. It appears that clearly labeling critical thinking
elements on the integrative learning activities used for the experimental group, in conjunction with the intrinsic challenges of digital game
competition, offered experimental group participants a signicant advantage in logically and reectively forming judgments. As Kiili (2005)
states, the challenging task of educational game designers is to develop the sort of game worlds that support reective thinking in the
private world (p. 22). The results of this study suggest that, for vocational students who are facing entry into the workplace, business
simulation games can provide this type of game environment, with the augmentation of teacher support and learning aids, in order to
address real-world needs and impact learners personally.
5.3. Impact on problem solving
Problem solving is the higher order thinking skill most directly associated with DGBL and most targeted by game designers (Li et al.,
2013). Opportunities for problem solving were plentiful in both CEO and Capitalism 2. Both gamers required learners to identify problems (whether in terms of demographic and geographic variables, market demands, resource prices, or competition) and solve these
problems through adjusting their gaming strategies. In addition, by repeating certain game levels, learners were required to prevent or avoid
future problems by relying upon information obtained from past decisions.
However, while these opportunities for problem solving during game play allowed for direct feedback in terms of the effectiveness of the
solution (i.e., the effectiveness category evaluated by the NTPS), learners' exibility in problem solving required prompting from integrative
activities. One example of such an activity was the SWOT analysis activity implemented during week 12 of the experiment. This activity,
illustrated in Fig. 4, asked learners to list as many potential strengths, weaknesses, opportunities, and threats identied during that semester
of game play. Thus, through the team rankings and reports provided by the game, players were further instructed to consider multiple
potential factors which led to their relative success or failure during the competition portion of the game.
The TEL condition also emphasized elements of problem solving through the exploration of case studies, the extensive use of discussion
and debate on issues related to store planning and management, and the requirement that learners collaborate in all learning tasks.
However, these tasks were not effective in promoting the development of effective and exible solutions to problems, since they were not
293
Fig. 4. Problem solving activity: Illustration of game play and integrated HOTS activity for Week 12.
grounded in an ongoing and sustained context, but were focused on textbook content which drew from multiple sources and topics.
Conversely, the digital games served as sustained narratives for contextualizing problems (Warren et al., 2012), causally linking game
strategies with game outcomes, and ensuring cognitive and affective engagement with the problem-solving tasks. Thus, in this study, the
use of DGBL as the principle instructional strategy provided an environment for simulating actual employment-related problems (Siewiorek
et al., 2012), which proved effective for promoting problem solving in vocational learners. This is signicant, since problem solving is a key
skill required by employers, and can be relevant to industries when students are seeking employment (Taiwan Ministry of Education, 2013).
294
Table 6
Summary of one-way MANCOVA results for creative thinking (TTCT), critical thinking (CTT-I), problem solving (PS) and academic achievement (AA).
DV
SV
SS
df
MS
Partial h2
TTCT
CTT-I
PS
AA
Group
91,640.46
76.79
2734.21
8389.02
1
1
1
1
91,640.46
76.79
2734.21
8389.02
38.22
13.85
19.93
61.95
.00*
.00*
.00*
.00*
.38
.18
.24
.50
*p < .01.
6. Conclusions
The results of this study demonstrate how a blended approach to DGBL which emphasizes teacher scaffolding and orchestration, provision of learning aids, and use of collaboration can empower vocational learners with improved higher order thinking and greater academic
achievement. While TEL also provided tangible benets for learners, who would otherwise have been taught with traditional, lecture-type
instruction, the synergy of DGBL with integrative activities for emphasizing HOTS was deemed more effective. Implications for both teachers
and researchers are provided in the following subsections (Table 6).
6.1. Implications for teachers
The effectiveness of the proposed approach to DGBL offers several suggestions for classroom teachers, particularly in light of the mixed
m
results for other types of DGBL approaches overall (Connolly et al., 2012) and specically, in terms of vocational education (Ha
al
ainen &
Oksanen, 2012). A review of the literature led to certain key suggestions related to the implementation of DGBL for fostering higher order
thinking and deep learning, which are summarized in the Methods section and linked to specic outcomes in the Discussion section. The
implication for instructors is that commercially available digital games can be an effective platform for fostering learner thinking and academic achievement, but that the implementation of DGBL in a blended environment, focused on scaffolding learners over an extended
period of time, offers greater possibilities for learning. Simple, integrative activities which challenge learners to think creatively, critically, or
with a problem-solving perspective can enhance the digital game experience. By blending digital game play with collaborative in-class
group activities, specic skills and knowledge (in this case HOTS) can be targeted. The fact that all three HOTS were signicantly
improved also suggests that a holistic approach to higher order thinking can be benecial, as the impact of DGBL on one thinking skill may
translate to improvements in terms of other thinking skills, when opportunities are presented for their application.
6.2. Limitations and implications for researchers
Suggestions for future research include the evaluation of traditional DGBL as compared to blended approaches, the adoption of more
precise and appropriate measures of employment-related skills, and the extension of these results to other contexts. First, while this study
demonstrates the effectiveness of the proposed model of DGBL, with signicant results for all three HOTS, the comparison group utilized TEL
rather than traditional DGBL. Therefore, the additive effect of the integrative learning activities over other forms of DGBL cannot be
quantied. Future studies adopting alternative forms of DGBL could provide further evidence of which features of the blended learning
environment are most benecial.
Additionally, while the purpose of this research was to foster learners' employability, this goal was only partially achieved. That is, while
HOTS are important to a learner's future employment, no direct measures of employability were available. Future studies can evaluate the
effects of DGBL on vocational learners' performance in actual workplace settings, possibly by including a form of practicum in the research
design or performing follow-up evaluation of graduates' employment status, salary, or employer evaluation. Likewise, the extension of this
type of DGBL to non-vocational contexts or vocational higher education could further the literature in this area of research.
Acknowledgments
The funding for this research was provided by the National Science Council of Taiwan, under grant NSC 102-2511-S-006 -004 -MY3.
References
Amory, A., Naicker, K., Vincent, J., & Adams, C. (1999). The use of computer games as an educational tool: Identication of appropriate game types and game elements. British
Journal of Educational Technology, 30(4), 311e321.
Boahin, P., & Hofman, A. (2013). A disciplinary perspective of competency-based training on the acquisition of employability skills. Journal of Vocational Education and Training,
65(3), 385e401.
Bourgonjon, J., De Grove, F., De Smet, C., Van Looy, J., Soetaert, R., & Valcke, M. (2013). Acceptance of game-based learning by secondary teachers. Computers & Education, 67,
21e35.
Chan, S. M., & Wu, W. T. (2007). New problem solving ability test. Taipei, TW: Psychological Publishing Press.
Chang, Y. C., Chen, W. C., Yang, Y. N., & Chao, H. C. (2009). A exible web-based simulation game for production and logistics management courses. Simulation Modelling
Practice and Theory, 17, 1241e1253.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games.
Computers & Education, 59(2), 661e686.
Ebner, M., & Holzinger, A. (2007). Successful implementation of user-centered game based learning in higher education: an example from civil engineering. Computers &
Education, 49(3), 873e890.
Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Computers & Education, 67, 156e167.
ma
l
Ha
ainen, R., & Oksanen, K. (2012). Challenge of supporting vocational learning: Empowering collaboration in a scripted 3D game e how does teacher' real-time
orchestration make a difference? Computers & Education, 59(2), 281e293.
295
ma
la
inen, R., Oksanen, K., & Ha
kkinen, P. (2008). Designing and analyzing collaboration in a scripted game for vocational education. Computers in Human Behavior, 24(6),
Ha
2496e2506.
Inayat, I., ul Amin, R., Inayat, Z., & Salim, S. S. (2013). Effects of collaborative web based Vocation Education and Training (VET) on learning outcomes. Computers & Education,
68(10), 153e166.
Kebritchi, M., & Hirumi, A. (2008). Examining the pedagogical foundations of modern educational computer games. Computers & Education, 51(4), 1729e1743.
Kiener, M., Ahuna, K. H., & Tinnesz, C. G. (2014). Documenting critical thinking in a capstone course: Moving students towards a professional disposition. Educational Action
Research, 22(1), 109e121.
Kiili, K. (2005). Digital game-based learning: towards an experiential gaming model. Internet and Higher Education, 8, 13e24.
Kiili, K. (2007). Foundation for problem-based gaming. British Journal of Educational Technology, 38(3), 394e404.
Lainema, T., & Nurmi, S. (2006). Applying an authentic, dynamic learning environment in real world business. Computers & Education, 47(1), 94e115.
Lee, T. H., Shen, P. D., & Tsai, C. W. (2008). Applying web-enabled problem-based learning and self-regulated learning to add value to computing education in Taiwan's
vocational schools. Educational Technology and Society, 11(3), 13e25.
Lee, T. H., Shen, P. D., & Tsai, C. W. (2010). Enhance low-achieving students' learning involvement in Taiwan's higher education: an approach via e-learning with problembased learning and self-regulated learning. Teaching in Higher Education, 15(5), 553e565.
Li, Z. Z., Cheng, Y. B., & Liu, C. C. (2013). A constructionism framework for designing game-like learning systems: Its effect on different learners. British Journal of Educational
Technology, 44(2), 208e224.
Lin, Y. L., & Tu, Y. Z. (2012). The values of college students in business simulation game: a means-end chain approach. Computers & Education, 58(4), 1160e1170.
Liu, C. C., Cheng, Y. B., & Huang, C. W. (2011). The effect of simulation games on the learning of computational problem solving. Computers & Education, 57(3), 1907e1918.
Malone, T. W. (1982). Heuristics for designing enjoyable user interfaces: Lessons from computer games. In Proceedings of the 1982 Conference on Human Factors in Computing
Systems (pp. 63e68). ACM.
Papastergiou, M. (2009). Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation. Computers &
Education, 52(1), 1e12.
Partnership for 21st Century Skills. (2009). Framework for 21st century learning. Retrieved from http://www.p21.org/our-work/p21-framework.
Pasin, F., & Giroux, H. (2011). The impact of a simulation game on operations management education. Computers & Education, 57(1), 1240e1254.
Ray, B., Faure, C., & Kelle, F. (2013). Using Social Impact Games (SIGs) to support constructivist learning: creating a foundation for effective use in the secondary social studies
education. American Secondary Education, 41(2), 60e70.
Siewiorek, A., Saarinen, E., Lainema, T., & Lehtinen, E. (2012). Learning leadership skills in a simulated business environment. Computers & Education, 58(1), 121e135.
Tabachnick, B. G., & Fidell, L. S. (2012). Using multivariate statistics (6th ed.). Boston, MA: Pearson.
Taiwan Ministry of Education. (2013). Reconstruction project for technical and vocational education: Second phase (Report No. 1020052561) Retrieved from http://www.ey.gov.
tw/News_Content.aspx.
Tsai, C. W., Lee, T. H., & Shen, P. D. (2013). Developing long-term computing skills among low-achieving students via web-enabled problem-based learning and self-regulated
learning. Innovations in Education and Teaching International, 50(2), 121e132.
Torrance, E. P. (1981). Predicting the creativity of elementary school children (1958e80) and the teacher who made a difference. Gifted Child Quarterly, 25, 55e62.
Torrance, E. P. (1998). The Torrance tests of creative thinking normsdtechnical manual gural (streamlined) forms A & B. Bensenville, IL: Scholastic Testing Service, Inc.
Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley Publishing Company.
Wall, J., & Ahmed, V. (2008). Use of a simulation game in delivering blended lifelong learning in the construction industry e opportunities and challenges. Computers &
Education, 50(4), 1383e1393.
Warren, S. J., Dondlinger, M. J., McLeod, J., & Bigenho, C. (2012). Opening the Door: an evaluation of the efcacy of a problem-based learning game. Computers & Education,
58(1), 397e412.
Williams, P. (2008). Assessing context-based learning: not only rigorous but also relevant. Assessment and Evaluation in Higher Learning, 33(4), 395e408.
Wu, W. H., Chiou, W. B., Kao, H. Y., Hu, C. H. A., & Huang, S. H. (2012). Re-exploring game-assisted learning research: the perspective of learning theoretical bases. Computers &
Education, 59(3), 1153e1161.
Yeh, Y. C. (2003). Critical thinking test-level I (CTT-I). Taipei, Taiwan: Psychological Publishing.