DOI: 10.1039/C8RP00248G (Paper) Chem. Educ. Res. Pract.

, 2019, Advance
Article

Does linking help? Effects and

student perceptions of a learner-
centered assessment
implemented in introductory
chemistry†
Alex Gilewski , Emily Mallory , Matthew Sandoval , Mikhail
ab a a

Litvak and Li Ye
c
*a

a
Department of Chemistry and Biochemistry, California State University,
Northridge, Northridge, California 91330, USA. E-mail: li.ye@csun.edu
b
Department of Chemistry, Glendale Community College, Glendale, California
91208, USA
c
Department of Biology, California State University, Northridge, USA

Received 8th October 2018 , Accepted 13th February 2019

First published on 13th February 2019

This study developed and implemented a learner-centered assessment

named Creative Exercises (CEs) in introductory chemistry courses at a four-
year university and a two-year community college. CEs were developed and
implemented as an intervention for treatment groups. The control groups
only used traditional assessments such as multiple-choice and short-answer
questions. A mixed-methods approach was employed for evaluating the
effectiveness of CEs in improving student learning and performance. First,
quantitative data included student exam scores, DFW rates, and percentages
of letter grades were analyzed and compared between treatment and control
groups. Second, student responses to CEs were coded as chemistry concepts
and then organized into chemistry topics. A series of visual maps were
plotted to show students’ linking of chemistry topics and progress made
throughout the semester. Lastly, student perceptions of the use of CEs were
investigated via a free-response survey. Quantitative results showed that CEs
improved students’ academic performance and retention in introductory
chemistry courses at both college settings. The implementation at the two
settings indicated that the frequency and quality of the use of CEs might
impact the effectiveness. The results from qualitative data analyses
converged with the positive effects of CEs. Students were able to connect
prior and newly-learned topics in response to CEs and made progress on
linking more topics over time. Open coding of the free-response survey data
identified four themes explaining why the use of CEs helped students:
knowledge integration, conceptual understanding, flexibility, and more
effective study habits. Best practices for implementation of learner-centered
assessments learned in this study and future directions for research are
discussed.

Introduction
Introductory chemistry courses taken by students in science and related
majors are focused on fundamental concepts, theories and laws of chemistry.
They usually cover a broad range of topics and the topics are built upon
themselves. Ideally, students learn the topics by relating and integrating
what is acquired previously to newly-learned knowledge. The links between
prior knowledge and new knowledge help students construct holistic
knowledge structures and learn meaningfully (Ausubel, 1960;Ambrose et al.,
2010). However, students may not make connections among chemistry
topics spontaneously. There is a concern that students treat chemistry topics
in introductory chemistry as discrete pieces of information. As such, students
may heavily rely on memorization, which would further hinder students’
learning (Francisco et al., 2002). Assessment plays an essential role in
curriculum as it not only measures students’ mastery of course content but
also serves as a means to send signals to students about what is considered
as important for instructors. Instructors might use the assessment to
emphasize particular content or promote certain skills they want students to
acquire. According to the nature of the assessment in the course, students
might orient themselves to allocate appropriate amounts of time and effort to
meet the important aspects of the assessment (Gibbs and Simpson, 2004).
Learner-centered assessment emphasizes the active role of students in the
process of creating assessment. It promotes a sense of ownership in learning
for students and shifts the purpose of assessment from assessing what was
learned to getting students to learn while they are completing the task that
was given (Webber, 2012; Rich et al., 2014). In order to promote students’
linking of chemistry concepts in introductory chemistry, we developed and
implemented a learner-centered assessment named Creative Exercises (CEs).
The purpose of CEs is to get students to think about connections among
chemistry topics and provide opportunities for them to demonstrate the
connections made (Lewis et al., 2010, 2011). Additionally, students’
responses to CEs can expand the rubric generated by instructors if they meet
the criteria, which bring students to be part of the answer-generating process
of the assessment. The overarching goal of the study is to investigate the
effectiveness of CEs as an intervention in improving student learning and
academic achievement in introductory chemistry.

Theoretical frameworks
With a clearer distinction between different types of learning, the assumptive
theory of learning introduced by Ausubel places learning in a continuum,
which rote learning on one end and meaningful learning on the other end
(Ausubel, 1960). Rote learning is described as learners learning knowledge in
isolation and simply relying on memorization. The knowledge learned is
easily forgotten without incorporating new knowledge to the existing
cognitive structures. In contrast, meaningful learning highlights the process
of linking new knowledge to the existing cognitive structures (Mayer,
2002; Taber, 2014). To learn meaningfully, both old and new knowledge
needs to be slightly modified to generate more complex and coherent
cognitive structures. Meaningful learning usually leads to longer concept
retention as compared to rote learning (Bretz, 2001; Novak, 2010). A graphic
representation of the assumptive theory of learning is shown in Fig. 1.
Additionally, the constructivist model of knowledge provides perspectives on
how learners acquire knowledge. Learners construct understanding in their
minds as a result of making sense of experiences. New knowledge in
learner's minds needs to be constantly tested and adapted to fit into learner's
existing knowledge structures. As such, learners do not construct
understanding until they learn to coordinate new knowledge into their
existing thoughts (Bodner, 1986).

Fig. 1 Ausubel’ assumptive theory.

 The two theoretical frameworks mentioned above both emphasize that
learners learn more effectively when they can make connections between
prior knowledge to newly-learned knowledge. The ideas that learning occurs
in the light of pre-existing knowledge and connection-making process leads
to meaningful learning provide the theoretical basis for designing this study.
Introductory chemistry contains a breadth of topics with which students may
not naturally make connections. As instructors, it is important that we
communicate with students the importance of linking knowledge. Linking
knowledge learned previously to new content is essential for students to build
meaningful knowledge structures. In Cooper and Stowe's recent chemistry
education research review article (Cooper and Stowe, 2018), they
summarized three criteria for meaningful learning to occur: “(1) the student
must have appropriate prior knowledge to which the new knowledge can be
connected. (2) The new knowledge must be perceived as relevant to this
prior knowledge. (3) The student must choose to make these connections.”
Creative Exercises provide students with a prompt designed based on new
knowledge and ask them to write as many as statements using what they
have learned. It requires students to retrieve prior knowledge associated with
the prompt and apply it to the new situation. In the process of generating
statements, students need to identify the relevant prior knowledge in the
prompt, transfer the knowledge into new problems, and choose to write
statements that they believe meet the criteria of CEs. These cognitive
processes satisfy the three criteria of meaningful learning stated above.
Therefore, using Creative Exercises regularly in introductory chemistry
courses has the potential to promote greater linking of chemistry concepts
and more meaningful learning to occur.

Learner-centered assessments for linking

concepts and effects
Concept mapping
Concept mapping is an assessment tool that has been reported extensively in
science education literature to help students build connections among
concepts. The origin of the concept mapping idea came from Novak and
colleagues (Novak, 1990). They developed the idea of hierarchical
frameworks and used them to represent how science concept meanings of
individuals changed over time in a longitudinal study. These frameworks later
evolved and were described as “concept maps”. Concept maps are two
dimensional diagrams that allow people to organize and connect information
among concepts. On a concept map, concepts are usually enclosed in boxes
or circles, a line drawn between two boxes or circles with a brief phase
indicates relationship between two concepts. The major component of a
concept map is designated a proposition, which contains two concepts and a
labelled line. Concept maps can be instructor-generated and used to
demonstrate to students how concepts are interrelated. Also, concept maps
can be used as learning and assessment tools via variable formats. For
example, students may be given a subject or a list of concepts and asked to
generate a concept map from scratch. In other cases, students may be given
a partially generated concept map and asked to fill in the blanks of concepts
or relationships (Ruiz-Primo and Shavelson, 1996; Ruiz-Primo et al., 2001).
A meta-analysis study (Nesbit and Adesope, 2006) analyzed 55 studies
reported the effects of concept mapping on students’ cognitive and affective
outcomes. The mean effect size of 0.60 was found for the effects of concept
maps as an intervention tool for improving student performance outcomes.
Of these 55, only 6 reported self-reported affective outcomes including self-
efficacy, motivation, and affect (anxiety, frustration, satisfaction). Positive
effects on all the affective measures were found. In 2009, another meta-
analysis study reported a medium effect size of 0.57 of six meta-analyses
regarding the effectiveness of concept mapping (Hattie, 2009). In a recent
study reported by Turan-Oluk and colleague (Turan-Oluk and Ekmekci, 2018),
concept maps were used to measure the learning gains on the topic of
gravimetric analysis in analytical chemistry. Concept maps were used as pre-
test after gravimetric analysis was introduced by a traditional lecture. Then,
the theory, objectives and application of concept maps were introduced as an
intervention. They administered a post-test using concept maps and student
opinions of the use of concept maps were investigated. The results showed
that the differences between pre and post-tests were statistically significant.
Students’ comments on the use of concept maps revealed that concept maps
enabled students to see the relations between concepts and learn chemistry
in a more effective way such as focusing more on understanding instead of
memorization.

Creative exercises (CEs)

One critique of using concept maps as a classroom assessment was the

complex and inconsistent scoring methods based on multiple organization
schemes (Ruiz-Primo and Shavelson, 1996; Lewis et al., 2011; Aguiar and
Correia, 2017). This makes concept mapping more useful as formative
assessments, and is less frequently used in summative assessments in
college-level chemistry courses (Schwendimann, 2015). Creative Exercises
were proposed as an alternative assessment technique that designed to
promote linking of prior and current knowledge student learned within a
course (Ye and Lewis, 2014). Trigwell and Sleet first introduced Creative
Exercises to assess chemistry knowledge in 1990 (Trigwell and Sleet, 1990).
Lewis and colleagues adopted and implemented CEs to promote linking of
concepts in college general chemistry settings (Lewis et al., 2010). Creative
Exercises were also developed and used in upper-division undergraduate
biochemistry course to assess linking of chemical and biochemical concepts
(Warfa and Odowa, 2015). In CEs, students are given a prompt that targets to
the current topics students learned in the course and asked to provide a
series of correct, distinct, and relevant statements according to prior topics
and newly-learned topics in the course (example can be found in Fig. 2). CEs
are preferable to concept mapping as a classroom assessment to prompt
linking of chemistry concepts because simpler grading method can be
employed (Ye and Lewis, 2014). A dynamic process forms the rubric of CEs.
At first, instructors brainstorm the possible statements meet the three criteria
of CEs, statements have to be scientifically correct, the concept used in each
statement needs to be unique, and the statements need to be relevant to the
given prompt in CEs and also the content presented in the course. Later,
when instructors grade CEs, if students think of statements that are not in
the rubric and meet the three criteria of CEs, these statements would be
added to the rubric for grading with subsequent students. As opposed to
closed-ended assessments where there are only one or a few correct
answers, the open-ended nature of CEs allows instructors to cover a wider
range of topics and integrate themes in the course. At the same time,
students can participate in the answer-generating process and are
encouraged to link concepts throughout the course.
Fig. 2 Example creative exercises and matched traditional assessment.

Research purposes
Evidence for the validity of CEs as a classroom assessment in chemistry and
linking chemistry concepts from students was obtained in the literature,
including content and structural validity, relationships with other chemistry
assessments, and generalizability among multiple graders (Lewis et al.,
2011; Lewis and Ye, 2014; Warfa and Odowa, 2015).
According to assumptive theory of learning and constructivist model of
knowledge, CEs have the potential to assist students in retrieving prior
knowledge and apply to new situations so as to foster meaningful learning in
introductory chemistry. However, the effects of CEs on students’ academic
achievement in chemistry courses haven’t been investigated yet. In this
study, we examined how CEs impact students’ academic performance and
retention in introductory chemistry courses at two different college settings.
The effectiveness of CEs can be used to compare to the effects of other types
of learner-centered assessments or those assessments designed to help
students make connections in chemistry, such as concept mapping.
Additionally, student perceptions of the use of CEs were also explored to help
understand why CEs are helpful or not for student learning in chemistry.
Another novel investigation of this study was that we showed students’
progress in linking of chemistry concepts for first time through visual maps.
These visual maps revealed the extent of current and prior topics used by
students while they were completing the CEs given at different time points
throughout the semester. This study was guided by the following research
questions:
(1) To what extent do Creative Exercises impact students’ performance and
retention in introductory chemistry?
(2) How do students link chemistry topics through answering Creative
Exercises across time?
(3) What are students’ perceptions of the use of Creative Exercises?

Methods
Research design and settings

This study employed a quasi-experimental mixed-methods research design. A

learner-centered assessment (i.e. Creative Exercises) was used as an
intervention in two introductory chemistry courses at two different types of
institutions in the southwestern United States. The first setting is a large
public, primarily undergraduate, four-year university. The second setting is a
public, two-year community college. For first-time freshman, attending
setting 1 requires high school diploma with appropriate high school grade
point averages and sufficient scores on standardized tests used for college
admissions in the United States such as SAT
(https://collegereadiness.collegeboard.org/sat) or American College Test (ACT)
(http://www.act.org/) while anyone with high school algebra or equivalent
background can attend setting 2. Comparison of course structures between
treatment and control groups at the two settings is listed in Table 1. At the
first setting, multiple classes of an introductory chemistry course are offered
each semester with class size of 70–80 except for Friday and Saturday
classes with fewer students. The course is a preparatory course for General
Chemistry I, intended for students who do not elect to take or do not pass the
chemistry placement test. The classes are coordinated with common
textbook and final exam. The textbook used for the classes is a custom
version of “Introductory Chemistry Essentials” (Tro, 2015). The custom
version of the textbook contains ten out of nineteen chapters from the
original version of the textbook and the content sequence taught in the
course was: the scientific method, measurement, matter and energy, atoms
and elements, molecules and compounds, chemical composition, chemical
reactions, quantities in chemical reactions, solutions, and electrons in atoms
and the periodic table. Two classes of this introductory chemistry course that
met the same time, but different weekdays were selected for the study in the
2017 Fall semester. Students attended the lectures twice a week for 75
minutes each. After introducing each chapter, the instructor engaged
students to practice the problems using in-class group activities. Students
were assigned into permanent groups (group size of 4–6). To promote
individual participation and group discussion, students in the same groups
received same grades for group activities based on consensus. Students were
asked to rotate roles of recorder (records the answers for the group),
manager (makes sure everyone in the group participants) and communicator
(is responsible to communicate with others for the group) within the groups.
Student grades were determined mostly by performance on two in-class
exams (16% each) and the cumulative final exam (42%). The rest of the
grade was attributed to three different effort-based measures: 12% from
online homework assignments using MasteringChemistry, 10% from group
activity, and 4% from class attendance. The instructor also provided four
extra-credit assignments (2%). The Creative Exercises were implemented in
one of the two classes. This class was considered as the treatment group.
The other class was served as control group, which only used traditional
assessment such as multiple-choice, true-false, and short-answer questions.
Table 1 Comparison of course structures between treatment and control
groups at the two settings
Setting 1 Setting 2
Treatment Control Treatment Control
a TD: traditional assessments including multiple choice, true-false, short-answer questions. CE: creative
exercises. MC: multiple-choice questions.
Semester Fall 2017 Fall 2017 Spring Spring
2018 2018
Class Lecture Lecture Lecture A Lecture A
format only only + lab A + lab B
Enrolment 73 71 25 19
Instructor Both groups were Lecture: instructor A
taught by the same (both groups were in
instructor the same lecture)
Lab A: instructor A;
Lab B: instructor B
Class time Twice a Twice a Lecture A: Lecture
week 75 week 75 twice a A: twice
min each min each week, 2 a week, 2
hours each hours
each
Lab A: Lab B:
once a once a
week, 3 week, 3
hours hours
Textbook 10 chapters 10 17 chapters 17
(Tro) chapters chapters
Activities Group Group Group Group
and activities: activities: activities: activities:
Examsa TD + CE TD CE n/a
Midterms: Midterms: Midterms: Midterm:
MC + CE MC TD + CE TD
Extra Extra Extra Extra
Credit: CE Credit: Credit: CE Credit:
MC TD
Final Final Final Final
 Setting 1 Setting 2
Treatment Control Treatment Control
exam: MC exam: exam: TD exam:
only MC only only TD only
Grading 12% Mastering online 10% weekly online
homework quizzes
10% Group activities 50% Four midterms
4% Attendance 20% Final exam
32% Two midterms 20% Lab
42% Final exam

At the second setting, multiple sections of an introductory chemistry

course are offered each semester. It is designed for students who have never
taken chemistry, took it in high school but did not pass the chemistry
placement exam, or those who are returning students. It satisfies general
education requirements and serves as a prerequisite for general chemistry
and biology courses. It uses the same textbook as course at the first setting
but covers seventeen chapters in the original version of “Introductory
Chemistry Essentials”. The content sequence taught was: the scientific
method, measurement, matter and energy, atoms and elements, molecules
and compounds, the mole concept, chemical reactions, oxidation–reduction
reactions, stoichiometry, electrons and atomic structure, chemical bonding,
intermolecular forces, liquids and solids, gas laws, solutions, chemical
equilibrium, acid–base chemistry, and nuclear chemistry. A laboratory
component is also required for the course. Students can only choose a
laboratory section associated with the lecture section for which they sign up;
as such, the same cohort is present in both the lecture and the linked
laboratory sections. Students had the option of two laboratory sections, one
with the lecture instructor and one with a separate instructor. The laboratory
section taught by the lecture instructor was the treatment group, and the
other section served as the control. Both groups attended the exact same
lectures, twice a week, for 2 hours each lecture. CEs were administered in the
laboratory and on lecture exams to maintain the separation between the two
groups. The week before each exam, students were tasked with completing a
CE as homework, which they brought to laboratory and collectively discussed
in permanent groups of 4 students before the experiment. The instructor
circulated the room and facilitated the group discussion by providing quick
feedback on the correctness of student responses to CEs. Student grades
were determined as follows: 10% for weekly online quizzes, 50% for four
midterm exams, 20% for a cumulative final exam, and 20% for the
laboratory.
At both settings, the same instructors taught the lecture for both
treatment and control groups. CEs were implemented for the treatment
groups multiple times over the semester via different formats, including in-
class group activity, midterm exam, and extra credit assignment. For
midterm exams, 90% of the content were the same for the treatment and
control groups. The treatment groups were given CEs worth 10% on each
midterm exam while the control groups were given the traditional
assessment questions matched to the prompt of CEs (see examples in Fig. 2).
The matched questions were written for the control groups so that students
have the opportunities to obtain equivalent points. Within each setting,
students took common final exam that didn’t include CEs. More details about
timelines of the implementation of CEs at the two settings can be found
in Fig. 3.

Fig. 3 Study design and timelines of the implementation of CEs at the two settings.
Data collection

Creative Exercises were used five times at the first setting and nine times at
the second setting. All the CEs used in the study can be found in Appendix 1.
To evaluate the effectiveness of the use of Creative Exercises on student
performance and retention, quantitative data from course records such as
exam scores, letter grades, and DFW rates were collected. Additionally,
qualitative data includes student written responses to CEs and an end-of-
semester free-response survey regarding student perceptions of the use of
CEs were collected. Student written responses to CEs were
collected via paper and the free-response survey was administered
online via a web-based management system.

Data analysis

Quantitative analysis.Analysis of the quantitative data was conducted using

software IBM SPSS Statistics (Version 24). Pearson product-moment
correlation coefficients were used to examine the relationship between
students’ CE scores and final exams scores within each setting.
Independent t-tests were used to compare the mean exam score differences
between treatment and control groups. Additionally, z-scores and effect sizes
(Cohen's d) were calculated to visualize and qualify the magnitude of the
differences in exam scores between the two groups.
Qualitative analysis – visual maps.Student responses to CEs given on four exams
for the treatment groups at the second setting were selected for analysis
because a larger breadth of topics was covered in that course. First, a list of
concepts in introductory chemistry was generated based on key terms listed
at the end of chapters of the textbook used in both courses. Then, the
concepts were categorized into seventeen major topics, and those in general
aligned with the chapters of the textbook. The list of major topics and the
concepts were categorized under each can be found in the Appendix 2.
Each student response to CEs was assigned a concept by the authors
based on what facet of the prompt the student attempted to answer. In many
cases, one response merited multiple concepts, as the student must have
utilized more than one to come up with their statement. For example, for a
prompt describing hydrochloric acid and sodium hydroxide reacting together,
a student wrote:
“HCl(aq) + NaOH(aq) → NaCl(aq) + H2O(l)”. This was coded as
‘nomenclature,’ ‘reactions,’ and ‘reactions’ because the student must have
translated names into chemical formulas and must have utilized two
concepts of reactions: states in chemical reactions and writing molecular
equations. Once the concepts were assigned to each statement, they were
then coded as the major topic to which the concepts belonged. To establish
reliability of the coding, two authors independently assigned statements to
concepts and major topics comprising 10% of a data set. An inter-coder
agreement of 76% was found, and any discrepancies were discussed and
resolved. One author then coded the rest of the data. After the coding was
completed, Gephi software (https://gephi.org/) was used to generate the
visual maps show students’ linking of topics and progress made in the
course. Each visual map was created by preparing and importing two files
into the software: nodes (topics and frequencies of the topics) and edges
(source topics, target topics, and frequencies of the links between topics).
The size of a node represents the frequency of the topic, which means
number of statements by all students coded as a particular topic. Larger
nodes mean more statements were written referencing that topic. For edges,
each student's coded statements were used to determine all the unique two-
code (topic) combinations possible between all topics used by that student.
These combinations were used to form the edges of the visual maps. Once
each student's combinations were generated, the frequency of each
combination was used to be represented as the thickness of the edge. The
thickness of an edge represents the number of students who used both of the
two topics connected by the edge. Visual maps were generated by Gephi
software using a metric called betweenness centrality, which quantifies the
number of times a node acts as a bridge along the shortest path between the
two other nodes (https://gephi.org/users/). Darker nodes in the maps have
higher betweenness centrality. In general terms, this means that darker
nodes display more connections to other nodes. The color of the edges
resonates with the color of the nodes they connected to. To summarize, the
nodes represent the frequency of each topic by all students, and the edge
connecting two nodes corresponds to how frequently students used both
topics in their responses. While this platform is widely used in sociology
network analysis (Bastian et al., 2009), Gephi visual maps are not unknown
in education literature. A study in healthcare education analytics describes
Gephi and similar programs which can be used to visualize data, allowing a
reader to more rapidly understand large and/or complex datasets (Vaitsis et
al., 2014). Another uses Gephi to demonstrate how science education
organizations are interconnected to schools, universities, museums, and
other educational institutions or groups (Falk et al., 2015). In chemical
education research, Galloway and colleagues use Gephi to visualize and
qualitatively compare how undergraduate students, graduate students, and
professors categorize organic chemistry reactions when completing card
sorting tasks (Galloway et al., 2018). In their Gephi maps, a node represents
a reaction card, two cards are placed in the same category are connected by
edges, the thicker edges indicate the more participants who sorted a card
pair together; then, visualizations for multiple groups are qualitatively
compared.
Qualitative analysis – student perceptions.Student responses to the free-response
survey were analyzed using open coding method (Corbin and Strauss, 2008).
The initial coding list was formed by having two authors assigned codes to
the student responses independently. Their codes were compiled and
modified based on discussion and were organized into categories. Then, 15%
of the student responses were selected and coded independently by the two
authors according to the initial code list. The inter-coder agreement was 70%.
The two coders discussed all the disagreements and refined the code list to
reach consensus. Using the resulting code list, one of the two authors
continued and finished all the coding. The survey questions and resulting
code list can be found in Appendix 3. The Institutional Review Boards at the
two institutions reviewed the research protocol and approved this study.

Results and discussion

Assessment validity

Validity and reliability of the scores on assessments lead to meaningful

interpretation of data (Arjoon et al., 2013). Validity of Creative Exercises has
been obtained and reported in college general chemistry settings (Lewis et
al., 2011; Lewis and Ye, 2014). However, we developed and implemented CEs
in different chemistry courses at different settings in this study. It is important
to collect the validity evidence at the current settings before interpreting the
data. According to Messick's framework of validity of assessments (Messick,
1995), the following three aspects of validity were examined: content validity,
external validity, and generalizability. Content validity was established by
having two chemistry instructors examine CEs and ensure the prompt of each
CE is scientifically correct, and the content coverage and level of difficulty are
appropriate for the students in the introductory chemistry courses at the
settings. To address external validity, correlations between CEs on midterm
exams and final exams at the two settings were calculated. Most of the
correlations were positive and significant except for CE3, CE4 and final exam
at the second setting were relatively low (see Table 2). The correlations for
CE3 and CE 4 may not be significant for several reasons. The topics covered
on CE3 constituted only 11% of questions on the final exam. CE4 was not
taken by several high-performing students, as the lowest examination score
was replaced with the final exam score (unless this lowered a student's
overall grade). This caused a general decrease in the mean CE 4 score. In
general, the correlations between average CE scores and final exam scores at
the two settings indicated moderate and significant relationships of CEs and
final exams. Generalizability was examined by inter-rater reliability of the CE
scores. Two chemistry instructors graded 10% of the statements from student
responses to CEs independently, and 86% inter-rater agreement was
reached.
Table 2 Correlations between CEs on midterm exams and final exam
CE1 CE2 CE3 CE4 Avg. CE
* Means correlation was significant at the 0.05 level. ** Means correlation was
significant at the 0.01 level.
Setting 1
Final 0.531* 0.544* N/A N/A 0.636**
exam
Setting 2
Final 0.558* 0.454* 0.152 0.356 0.584**
exam

Impact on student performance and retention

To ensure the treatment and control groups were comparable, Table
3 summarizes the demographics and backgrounds of the two groups at the
both settings. The first setting is a Hispanic-serving institution, the
percentages of underrepresented minorities (URMs) were relatively high.
URMs are defined as Black/African American and Hispanic/Latino according
to National Science Foundation (2007). Chi-square tests and t-tests were
used to compare the percentages of females, percentages of URMs, SAT
math scores, and high school GPA between two groups, no statistically
significant differences were found between two groups using with p = 0.05
significant level. At the second setting, the SAT is not required.
Table 3 Comparison between treatment and control groups
 Setting 1 Setting 2
Variables Treatment Control Treatment Control
Number of students 73 71 25 19
% Female 41 46 47 66
% URMs 61 59 53 30
SAT math 492 482 N/A N/A
High School GPA 3.34 3.42 3.32 3.42

The first research question was to investigate the effectiveness of the

learner-centered assessment (i.e. Creative Exercises) as an intervention on
student performance and retention in the introductory chemistry courses.
First, students’ exam scores in the courses were converted to percentages by
using the actual scores divided by the maximum scores that students could
obtain for the exams. At both settings, the mean exam score of students in
the treatment groups was higher than the control groups on every exam
(see Table 4).
Table 4 Mean values and standard deviations of student exam scores
Treatment Control
Mean (%) SD (%) Mean (%) SD (%)
Setting 1 Exam 1 61 16 58 16
Exam 2 60 17 54 19
Final exam 60 18 55 20
Avg. exam 61 15 56 17

Setting 2 Exam 1 88 8 80 22
Exam 2 80 23 57 34
Exam 3 81 19 66 28
Exam 4 52 33 41 28
Final exam 69 19 61 34
Avg. exam 75 14 61 23

The differences between the treatment and control groups were ranged
from 3% to 6% at the first setting, and 8% to 23% at the second setting. To
visualize the differences in exams scores between treatment and control
groups, student exam scores were also converted to z-scores and mean z-
scores of two groups were plotted in Fig. 4. The figure shows how far each
group was far away from the mean score of the class (z-score = 0). To
analyze whether the effect of the intervention was statistically significant on
exam scores, independent t-tests were conducted to compare the mean
differences in average exam scores between treatment and control groups
(see Table 5). Results of the independent t-tests showed that the mean
difference on average exam scores at the second setting was statistically
significant (t = 2.388, p = 0.024), but not for the first setting (t = 1.588, p =
0.115). Because the relatively small sample sizes in our study, we also
calculated and reported effect size (Cohen's d) to quantify the size of the
differences between treatment and control groups. Effect size takes into
account sample size and the amount of variation in scores. It is independent
of the inferential statistics and allows us to move beyond does it work or
not to how well does it work. Cohen suggested that d = 0.2, 0.5, and 0.8 to
be considered as small, medium, and large effect size, respectively (Cohen,
2005). As listed in Table 5, the average effect size was 0.31 for average exam
scores at the first setting and 0.75 at the second setting. The average effect
size was considered as small at the first setting and medium at the second
setting. More specifically, at the first setting, the average exam score of the
average person in the treatment group was 0.31 standard deviations above
the average person in the control group, and this difference between groups
increased more than twofold at the second setting.

Fig. 4 Mean z-scores of treatment and control groups at the two settings.
Table 5 Independent t-tests comparing average exam scores between
treatment and control groups
 Mean difference
(treatment – p-
control) Value Effect size (Cohen's d)
* Mean difference was significant at the 0.05 level.
Setting 1 (N = 144, treatment = 73, control = 71)
Avg. 5% 0.115 0.31
exam
Setting 2 (N = 44, treatment = 25, control = 19)
Avg. 14% 0.024* 0.75
exam

Finally, the distribution of letter grade percentages (Fig. 5) and DFW rates
between treatment and control groups were compared at the two settings. As
shown in Fig. 5, in general, the bars in red shifted more on the top as
compared to the blue bars. That is, students in the treatment groups earned
higher grades than the control groups overall. As a result, at the first setting,
the difference in DFW rates between the treatment and control groups was
7%, with 37% for the control group and 30% for the treatment group.
Similarly, at the second setting, in the treatment groups obtained more A, B C
grades and less D and F, leading the difference in DFW rates between the
two groups became 8% with 32% for the control group and 25% for the
treatment group. Chi-square analyses were carried out to determine if the
differences between letter grades (DFW vs. non-DFW) of the two groups were
statistically significant. The results indicated statistical differences with
moderate effect size at the second setting (χ2 = 4.565, p = 0.033,
Cramer's V = 0.322) but no difference was found at the first setting (χ2 =
1.350, p = 0.245, Cramer's V = 0.097).
Fig. 5 Letter grade comparison between treatment and control groups.
Comparing the effect of CEs on students’ cognitive outcomes to the
medium effect size of 0.57 using Cohen's d of six studies researched on the
effect of concept mapping conducted by Hattie (Hattie, 2009), we found a
similar average effect size (0.53) of the use CEs on exam scores at the both
settings. It is worth noting that the impact of CEs on average exam scores at
the second setting was more than twice compared to impact at the first
setting. We believe the difference between the two settings is mainly
because of the frequency and quality of the implementation of CEs. First of
all, the frequency of using CEs was almost doubled at the second setting. CEs
were used five times at the first setting while nine times at the second
setting. Relating to assumptive theory of learning, the higher frequency of
using CEs provides more opportunities for students to link concepts and
enable them to link topics with shorter time gaps within the course. These
experiences allow students construct more coherent understanding in minds
and undergo more meaningful learning through the more CE activities,
leading to a larger effect size. More importantly, for the treatment groups, the
time students spent in class for CEs at the second setting was probably with
higher quality. At the first setting, when students were given the CEs as in-
class group activities, they were given along with other traditional chemistry
problems. While at the second setting, the instructor pre-assigned the CEs to
students individually before the class as homework, and CEs were used as
the only problems that students had to complete during in class group
activities. As such, students had their own answers when they came to class
and were ready to share and discuss their answers with peers in groups.
Students were also given sufficient time to compile individual answers and
ask instructor for feedback on the correctness of group answers during the
class time. This mechanism allows adequate time for group discussion and
student–instructor interactions, which makes the implementation of CEs more
efficient. Also, the larger effect size at the second setting could be due to the
differences between control groups at the two settings. Students in control
group lacked opportunities for group activity at the second setting but similar
amount of group activities using traditional assessment were completed by
the students in control group at the first setting. In sum, the frequency and
quality of the implementation of CEs might be the important factors that
enhance the effectiveness of CEs on students’ performance.

Students’ linking of chemistry concepts and

progress
The second research question was to explore how students link chemistry
concepts when answering Creative Exercises. Through these visual maps, we
can see how frequently topics were applied in response to CEs, as well as
connections made among topics (Fig. 6). While students may not be explicitly
aware they are linking concepts when completing CEs, links must take place
as students apply multiple concepts to the same problem; at the minimum,
this demonstrates subconscious connection between concepts. Some survey
responses, however, do suggest students are aware they are building
linkages between topics: “It utilizes all your knowledge of chemistry at
once…” and “…everything we learned in the past can be applied to the new
concepts.” Further, the fact that some student responses incorporate multiple
topics simultaneously, as described in the analysis section, supports the
hypothesis that students are making connections.
The maps were generated from student responses to CEs on the four
exams implemented to the treatment groups at the second setting. The first
map shows the smallest number of topics mentioned among all maps.
Students focused on three topics: units and significant figures (sig. figs),
elements, and compounds. They chose compounds most heavily, but the
other two are not far off. The least mentioned topic was atoms. In exam 2,
the data shows a large contrast. By far, one topic was mentioned more than
any other topics: chemical reactions. Reactions were the target topic of the
prompt, indicating students used current topic more than prior topics.
Interestingly, stoichiometry was mentioned only once. This may be due to a
desire to avoid performing calculations on these types of problems or may
indicate a lack of understanding of stoichiometry by the students. In exam 3,
the largest number of topics was demonstrated, and topics used were more
spread out. Only one was much larger than the others: units and significant
figures. The major topics for that exam were gases, chemical bonding, and
intermolecular forces. Stoichiometry was again a rarely-used topic. In exam
4, which focused on solution, equilibrium, and acid–base chemistry, students
again chose units and significant figures most. Students tended to focus on
the topics presented on the exam, but similarly to exams 2 and 3,
stoichiometry was the least-used topic.
Overall, it appears that students prefer to apply units and significant
figures most often. This could be because of the “ease” of identifying how
many significant figures a number has, or simply identifying what a unit
represents. It could also result from the fact that students are simply
accustomed to using this topic and getting it marked correct on previous
exams, as “easy points.” However, students tended to not only apply
recently learned concepts but accessed previous topics as well. Finally,
stoichiometry and atoms are the least-used topics, which may due to their
relative difficulty or lack of understanding. Ralph and Lewis (Ralph and Lewis,
2018) recently reported that students with low SAT math scores struggled
disproportionately with stoichiometry across all general chemistry topics. This
may speak to our results here since the students enrolled in the selected
courses at the settings were in general less prepared for math. Atoms, as a
topic, in particular may be less obvious or applicable to students, or they may
have difficulty understanding the relevance of the topic.

Student perceptions of the use of CEs

The last research question was to understand students’ experiences of the
use of Creative Exercises in classroom from students’ perspectives. The free-
response survey was administered via a link in emails and sent out to all the
98 students who were in the treatment groups. There were 72 students (73%
response rate) answered all the questions in the survey. Among these
students, 56 (78%) answered “Yes” and 16 (22%) answered “No” to the first
question: “Do you think the Creative Exercises help you make connections
among content in this class?” Based on student responses to the free-
response questions, themes emerging from the data were divided into two
major categories: helpful and unhelpful themes (see Appendix 3).
There were four themes found under the helpful category: knowledge
integration, conceptual understanding, flexibility, and study habits. First and
foremost, the majority of the students stated that CEs helped them think
chemistry as a coherent theme instead of learning chemistry as separate
facts: “It helped in a way that allowed us to think about the whole concept of
chemistry and not just the specific topics we learn every week.”; “It makes
me think back to everything I've learned from every chapter and using all of
the information together to answer one question.” These comments are in
line with the purpose of using CEs as a learner-centered assessment, getting
students to learn the importance of linking chemistry concepts and promote
students making connections and build more coherent knowledge structures.
Moreover, CEs also assisted students in understanding chemistry concepts
conceptually and deeply: “It helps tremendously. These questions force me
to understand the concept and apply it rather than just memorize and
regurgitate it”; “You can explain why and how. You found such answers by
showing work or writing out sentences to explain your reasoning.” Reducing
the extent of using memorization in learning chemistry and being able to
internalize and explain chemistry concepts are the signs that show students
are adopting meaningful learning.
The third theme was flexibility, students felt that they had the freedom to
choose the topics they would like to present: “I could utilize the areas which I
felt the most comfortable in to answer the question.” Additionally, the open-
ended nature of CEs gave students a sense of ownership because they were
not restricted by a correct answer when answering CEs: “It gives you space
to think the way you want, and the answers are unlimited.” One of the most
noticeable themes throughout students’ responses was the impact of CEs on
students’ study habits. Many students indicated that they put more efforts to
learn prior knowledge because they need to use the knowledge consistently
throughout the semester: “It motivates me to keep the prior material learned
fresh in my mind”; “You had to remember the previous content long term
because you knew it would relate to other exams”. Also, students
appreciated the value of working with others through doing CEs in group
activities, “You connect with people…input from other classmates adds to my
own answer and thinking.” Lastly, CEs improved students’ metacognitive
skills such as reflecting and evaluating the content they learned in the past,
“It serves as a reminder of what areas I need to study more or seek help
from a professor and tutor if I'm still having difficulty understanding it”; “It
stimulated me to think critically and challenge myself to make sure I
understand past and current lecture”. These changes in students’ study
habits impacted by CEs, including reinforcement of prior knowledge,
engagement in teamwork, and better metacognitive skills, have been
reported as more effective study habits in chemistry (Cook et al.,
2013; Sinapuelas and Stacy, 2015; Chan and Bauer, 2016; Ye et al., 2016).
In addition to the helpful category, student responses also revealed
unhelpful sides of the use of CEs. Three themes were found: challenging,
need more instruction or feedback, and self-doubt. Some students thought
CEs were too challenging to answer or they couldn’t come up with enough
statements to meet the required number of statements for full credit: “to pull
a lot of concepts and apply it to the general question itself was beyond
difficult”; “I feel as if I run out of things to say and just end up getting
frustrated.” Second, students mentioned that they would like to have more
instruction on how to meet the criteria of CEs and more immediate feedback
on their answers to the CEs. The last theme was self-doubt, certain students
were not confident about their answers to CEs: “I would make connections
and they would be wrong”, which may cause additional anxiety during the
exams.
Fig. 6 Visual maps show student's linking of concepts and progress over time using CE responses (a) CE1 in
(c) CE3 in Exam 3, (d) CE4 in Exam 4.
In sum, the majority of the students think that the use of CEs is beneficial
for their learning in chemistry. Those helpful themes converge with the
positive effects of CEs. CEs encourage knowledge integration of topics in
introductory chemistry, promote conceptual understanding, and help
students form more effective study habits. These explain why the use of CEs
leads to better academic performance and retention.
Among these helpful themes, some viewpoints are consistent with
students’ views of the use of concept mapping, such as appreciating the
connections among chemistry concepts and improving conceptual
understandings (Turan-Oluk and Ekmekci, 2018). The unhelpful themes
provide insight into the implementation of CEs. They also resonate with the
effect of the CEs on exam scores were doubled at the second setting
compared to the first setting. Due to the improvement of instruction and
feedback provided at the second setting, students were assigned CEs
individually before class and given adequate time to work with each other in
groups in class, making them were clearer on how to answer CEs and better
at constructing responses to CEs through immediate feedback from
instructor.

Conclusions and implications

The learner-centered assessment (i.e. Creative Exercises) was designed to
promote connections and improve students’ academic performance and
retention in introductory chemistry courses at a large public, four-year
institution and a two-year community college. The visual maps show
students’ linking of prior and current topics at different time points and
progress made within the courses. Most students hold positive perceptions
about the use of Creative Exercises, which converge with the positive effects
of Creative Exercises on students’ academic performance and retention in the
courses.
For practitioners who might adopt Creative Exercises as an assessment
into their courses, the frequency and quality of implementing Creative
Exercises are important factors to be considered. First, implementing Creative
Exercises multiple times throughout the semester is necessary. Providing
example responses to Creative Exercises and explain them before assigning
Creative Exercises would be helpful for students to understand the criteria of
Creative Exercises. The more practices and feedback students have with CEs,
the better they are in response to Creative Exercises. Second, Creative
Exercises should be used as both formative and summative assessments.
Assigning Creative Exercises individually before class and having students
discuss their responses via in class group activity can be an efficient way to
implement Creative Exercises as formative assessment. In addition,
instructors should allocate sufficient in class time to give immediate feedback
on Creative Exercises. In order to maximize the effect of Creative Exercises
on student learning, using Creative Exercises along with traditional
assessment as summative assessment will help students value Creative
Exercises more and promote knowledge integration, conceptual
understanding and more effective study habits.
For researchers, the limitation of the study is the relatively small sample
sizes in the courses at the two settings. Future studies considering the
implementation of Creative Exercises into a larger sample would validate the
generalizability of the findings in this study. The authors in this study are
utilizing the student responses to Creative Exercises and developing a series
of assessments to measure linked chemistry concepts and evaluate the
effect of the follow-up assessment in a large scale study. Additionally, the
students in our settings are in general less prepared for chemistry courses
with poor math preparation and high proportions of underrepresented
minorities. Researchers need to be cautious when they have a very different
student body as compared to our settings. Moreover, visual maps show
students connect stoichiometry and atoms least to other topics. Future
research on developing instructional methods or interventions to improve
understanding on the relevance of those topics would be necessary. Another
insight of this study is from the visual maps of student responses to Creative
Exercises. We are able to show evidence of students’ linking of chemistry
topics and progress made across the semester through these visual maps.
Researchers who are interested in investigating student written responses to
chemistry assessments may utilize this method to show visual
representations of their data. Finally, positive effects on student performance
and encouraging student attitudes have been shown by implementing
concept maps and Creative Exercises separately in chemistry courses. Future
research might implement the two assessments simultaneously in one study
and examine whether coupling concept maps with Creative Exercises may
amplify the effects and explore similarities and differences of the two
assessments for meaningful learning.

Conflicts of interest
The authors claim no conflicts of interest.

Acknowledgements

The authors of this study would like to thank students who participated in this
study. Also, we appreciate the Office of Institutional Research provided the
demographic data our study and support from 2018–2019 Research,
Scholarship, and Creative Award at California State University, Northridge.

References
1. Aguiar J. G. and Correia P. R. M., (2017), From representing to modelling
knowledge: proposing a two-step training for excellence in concept
mapping, Knowl. Manage. E-Learn., 9(3), 366–379.

2. Ambrose S. A., Lovett M., Bridges M. W., DiPietro M. and Norman M. K.,
(2010), How learning works: seven research-based principles for smart
teaching, San Francisco, CA: Jossey-Bass.

3. Arjoon J. A., Xu X. and Lewis J. E., (2013), Understanding the state of

the art for measurement in chemical education research: examining
psychometric evidence, J. Chem. Educ., 90(5), 536–545.

4. Ausubel D. P., (1960), The use of advance organizers in the learning

and retention of meaningful verbal material, J. Educ. Psychol., 51, 267–
272.
5. Bastian M., Heymann S. and Jacomy M., (2009), Gephi: An Open
Source Software for Exploring and Manipulating Networks, Third Int.
AAAI Conf. Weblogs Soc. Media, pp. 361–362.

6. Bodner G. M., (1986), Constructivism: a theory of knowledge, J. Chem.

Educ., 63(10), 873–878.

7. Bretz S. L., (2001), Novak's Theory of Education: Human

Constructivism and Meaningful Learning, J. Chem. Educ., 78, 1107–
1115.

8. Chan J. Y. K. and Bauer C. F., (2016), Learning and studying strategies

used by General Chemistry students with different affective
characteristics, Chem. Educ. Res. Pract., 17, 675–684.

9. Cohen J., (2005), Statistical power analysis for the behavioral sciences,
2nd edn, Hillsdale: Lawrence Erlbaum Associates. de Leeuw, E.D.

10. Cook E., Kennedy E. and McGuire S. Y., (2013), Effect of teaching
metacognitive learning strategies on performance in General
Chemistry courses, J. Chem. Educ., 90(8), 961–967.

11. Cooper M. M. and Stowe R. L., (2018), Chemistry Education

Research – From Personal Empiricism to Evidence, Theory, and
Informed Practice, Chem. Rev., 118, 6053–6087.

12. Corbin J. and Strauss A., (2008), Basics of qualitative research:

techniques and procedures for developing grounded theory, 3rd edn,
Thousand Oaks, CA: Sage Publications.

13. Falk J. H., Dierking L. D., Osborne J., Wenger M., Dawson E. and
Wong B., (2015), Analyzing Science Education in the United Kingdom:
Taking a System-Wide Approach, Sci. Educ., 99, 145–173.

14. Francisco J. S., Nahkleh M. B., Nurrenburn S. C. and Miller M. L.,

(2002), Assessing student understanding of General Chemistry with
concept mapping, J. Chem. Educ., 79, 248–257.

15. Galloway K. R., Leung M. W. and Flynn A. B., (2018), A

Comparison of How Undergraduates, Graduate Students, and
Professors Organize Chemistry Reactions, J. Chem. Educ., 95(3), 355–
365.

16. Gibbs G. and Simpson C., (2004), Conditions under which

assessment supports students’ learning, Learn. Teach. High. Educ. 1,
3–31, ISSN 1742-240X.

17. Hattie J. A. C., (2009), Visible learning: a synthesis of over 800

meta-analyses relating to achievement, London, New York: Routledge.
18. Lewis S. E., Shaw J. L. and Freeman K. A., (2010), Creative
Exercises in General Chemistry: A Student-Centered Assessment,J. Coll.
Sci. Teach., 40(1), 48–53.

19. Lewis S. E., Shaw J. L. and Freeman K. A., (2011), Establishing

open-ended assessments: investigating the validity of creative
exercises, Chem. Educ. Res. Pract., 12, 158–166.

20. Mayer E. R., (2002), Rote Versus Meaningful Learning, Theory

Into Practice, 41, 4, 226–232.

21. Messick S., (1995), Validity of psychological assessment;

validation of inferences from persons’ responses and performance as
scientific inquiry into score meaning, Am. Psychol., 50, 741–749.

22. National Science Foundation, (2007), Women, Minorities, and

Persons with Disabilities in Science and Engineering, Arlington, VA:
National Science Foundation, Division of Science.

23. Nesbit J. C. and Adesope O. O., (2006), Learning with concept and
knowledge maps: a meta-analysis, Rev. Educ. Res., 76, 413–448.

24. Novak J. D., (1990), Concept mapping: a useful tool for science
education, J. Res. Sci. Teach., 27, 937–949.

25. Novak J. D., (2010), Learning, creating, and using knowledge,

New York, NY: Lawrence Erlbaum Associates, Inc.

26. Ralph V. R. and Lewis S. E., (2018), Chemistry topics posing

incommensurate difficulty to students with low math aptitude
scores, Chem. Educ. Res. Pract., 19(3), 867–884.

27. Rich J. D., Colon A. N., Mines D. and Jivers K. L., (2014), Creating
learner-centered assessment strategies for promoting greater student
retention and class participation, Front. Psychol., 5, 1–3.

28. Ruiz-Primo M. A. and Shavelson R. J., (1996), Problems and issues

in the use of concept maps in science assessment, J. Res. Sci.
Teach., 33(6), 569–600.

29. Ruiz-Primo M. A., Schultz S. E., Li M. and Shavelson R. J., (2001),

Comparison of the reliability and validity of scores from two concept-
mapping techniques, J. Res. Sci. Teach., 38(2), 260–278.

30. Schwendimann B. A., (2015), Concept maps as versatile tools to

integrate complex ideas: from kindergarten to higher and professional
education, Knowl. Manage. E-Learn., 7(1), 73–99.
 31. Sinapuelas M. L. S. and Stacy A. M., (2015), The relationship
between student success in introductory university chemistry and
approaches to learning outside of the classroom, J. Res. Sci.
Teach., 52(6), 790–815.

32. Taber K. S., (2014), Constructing active learning in chemistry:

concepts, cognition and conceptions, in Devetak I. and Glažar S. A.
(ed.), Learning with Understanding in the Chemistry Classroom,
Dordrecht: Springer, pp. 5–23.

33. Trigwell K. and Sleet R., (1990), Improving the relationship

between assessment results and student understanding, Assess. Eval.
High. Educ., 15, 190–197.

34. Tro N., (2015), Introductory chemistry essentials, 6th edn,

Boston, MA: Pearson Education, Inc.

35. Turan-Oluk N. and Ekmekci G., (2018), The effect of concept

maps, as an individual learning tool, on the success of learning the
concepts related to gravimetric analysis, Chem. Educ. Res.
Pract., 19(3), 819–833.

36. Vaitsis C., Nilsson G. and Zary N., (2014), Visual analytics in
healthcare education: exploring novel ways to analyze and represent
big data in undergraduate medical education, PeerJ, 2, e683.

37. Warfa A. R. M. and Odowa N., (2015), Creative exercises (CEs) in

the biochemistry domain: an analysis of students’ linking of chemical
and biochemical concepts, Chem. Educ. Res. Pract., 16(4), 747–757.

38. Webber L. K., (2012), The Use of Learner-Centered Assessment in

US Colleges and Universities, Res. High. Educ., 53(2), 201–228.

39. Ye L. and Lewis S. E., (2014), Looking for links: examining student
responses in creative exercises for evidence of linking chemistry
concepts, Chem. Educ. Res. Pract., 15, 576–586.

40. Ye L., Shuniak C., Oueini R., Robert J., Lewis S., (2016), Can they
succeed? Exploring at-risk students' study habits in college general
chemistry, Chem. Educ. Res. Pract., 17, 878–892.

Footnote

† Electronic supplementary information (ESI) available. See DOI: 10.1039/c8rp00248g

This journal is © The Royal Society of Chemistry 2019