Running Head: Complex Texts Module 1

An Easy Guide for Complex Texts Module

Giovanna Milena Rodriguez Uribe

California State University Monterey Bay

IST 622 Assessment and Evaluation

Dr. Bude Su

June 18, 2019

Running Head: Complex Texts Module 2

Table of Contents

I. Introduction……………………………………………………..……………1

II. Methodology …………………………………………………….…….…….1

Prototype…………………………………………………….…………….…1

Learners and Student’s survey….…………………….……………….….2

Tryout Process…………………………………………………….….……..4

1. Pre-Test………………………………………………………..……...5

2. Training…………………………………………….………………….5

3. Post-Test…………………………………………………..………….6

4. Observation……………………………………………..…………….6

5. Usability Test…………………………………………..……………..6

6. Tryout Conditions………………………………………….……...….7

III. Results…………………………………………….………………………...7

a. Entry conditions……………………………………………...…………7

b. Instruction………………………………………………...........………. 7

c. Outcomes……………………………………………….……………….8

d. Recommendations……………………………………………..………10

IV. Summary……………………………………………..……………… …….11

V. Appendices………………………………………………………………….12
Running Head: Complex Texts Module 1

Section I

Introduction

Second language learners encounter difficulties dealing with higher level

materials. Many because they are novice learning a second language, they do not have

the vocabulary needed to understand such materials or because they feel lost and do

not know what to look for. There are several approaches to work efficiently with these

materials. However, the most successful rely on three important aspects: vocabulary,

background knowledge and understanding the difficulties of these materials. This

training offers participants content and strategies to deal with higher level materials

regardless of the second language proficiency. Five students took part in this training

and answered several surveys and tests delivered through Google forms.

Section II

Methodology

Prototype

The training was designed in Adobe Captivate. It includes a series of interactive

activities with a duration of 10 minutes. At the beginning of the module, students

received information about the objectives of the training to attract their attention. Then,

they were presented with three different folders to access content to help students

understand the difficulties of higher-level materials and the importance of background

knowledge and vocabulary. After each section, students were required to answer

knowledge checks with the objective of helping them focus on the main ideas. Here

some examples of the training design. See Tables 1.1 to 1.4

Running Head: Complex Texts Module 2

Table 1.1 Table 1.2

Table 1.3 Table 1.4

Learners and student’s survey

The target audience is comprised of five DLI students who have been learning a

second language for at least 11 weeks of instruction. See Table 2.1. This training has

been designed to be taken from novice to advanced students. The students took a

learner’s profile survey with the purpose of collecting data about the participants’
Running Head: Complex Texts Module 3

preferences for the use of technology, language skills, motivation to learn new things

and reading. See Appendix A.

Table 2.1

The purpose of this student’s survey was to capture student’s expertise on basic

use of computers and to know more about their study habits and preferences towards

learning a second language as well as self-motivation and ability to learn independently

through a computer. Two different scales were used to collect the information.

Participants rated their responses to the questions about technology skills on a scale

ranging from below basic to advanced: strongly disagree to strongly agree to rate the

second part of questions. The technology skills from the participants vary: 40% of the

participants feel proficient using technology and the rest of the students feel their skills

are from basic to advance 20% each. See Table 2.2. Three students out of five

expressed to be self-motivated. Only one student of the five expressed to read a lot, the

rest of the students agree or are neutral. Three students like to learn new things and like
Running Head: Complex Texts Module 4

to use technology. Three students believe in their language skills but two disagree. See

Table 2.3

Table 2.2

Table 2.3

Tryout Process

The tryout process comprised of six parts: students’ survey, pre-test, training,

observation, post-test and usability survey. Students’ survey and pre-test were delivered

through email containing links to access the forms. During the one on one session with
Running Head: Complex Texts Module 5

the designer, students took the training and answered the post-test and the usability

survey. All the forms were delivered through Google Forms. Students’ email addresses

were recorded to the effects of studying the learning effects of the training on the pre

and post-tests. The observation checklist was filled by the designer during the in-person

session with the students.

1. Pre-Test

The pre-test questionnaire was shared to the students using the link

provided by Google forms. Previous presenting this test, students received a

session explaining the purpose of their participation in this project. The pre-test

gives an idea of what the learner does and does not know before attempting the

training. This part is critical to compare and contrast the results along with the

post- test. This test is comprised of ten multiple choice questions that measure

knowledge about text difficulties, background knowledge and vocabulary

strategies. The same questions were used for the post-test. See Appendix B.

2. Training

The training was designed in Adobe Captivate and accessed by the

participants via internet using the link provided by the designer.

Participants were able to copy and paste the link in their browsers.

The intended training was originally designed to take 10 minutes to complete.

Some students took two more minutes to complete it. Faster students could do it

in 8-9 minutes. See Appendix C.

3. Post-Test
Running Head: Complex Texts Module 6

After students finish the training, they were instructed to click on the post-

test link shared through email. The post-test contains the same questions as the

pre-test with the objective of evaluating the training efficacy. Having the same

questions for the pre and post-test is a requirement for the completion of a paired

analysis. See Appendix B.

4. Observation

During the observation, the facilitator filled out the checklist to record

students’ performance during the training. The facilitator looks for navigation

issues, general understanding of the module, buttons functionality and sound.

Also, students’ behaviors were recorded in the form. The participants were

engaged and wanted to obtain good scores thorough the session. See Appendix

C.

5. Usability Test

The purpose of this survey was to attain information about the student’s

impressions of the design as well as their opinions on the improvements that can be

made in terms of design. The usability test used a scale that ranged from strongly

disagree to strongly agree and asked two open ended questions about what they

enjoyed the most from the training and one about the improvements that are needed in

order to make the training better. The information collected from this test will be

considered in the design of the final capstone project. See Appendix E.

6. Tryout conditions

Students were invited to gather in one classroom with internet connectivity to

take the training. They were required to bring their laptops. The session began with the
 Running Head: Complex Texts Module 7

instructions given by me as the facilitator. The students accessed the training using the

link I sent to their emails and worked individually. Students could only see their scores

at the end of the post-test. Students made spontaneous comments about how the

training could teach them better study habits.

Section III

Results

a. Entry conditions

Despite the fact, one of students rated his technology skills as below basic,

learners didn’t have problems neither using the Google forms nor accessing the

training. Most of the learners had basic technological skills to navigate the training.

Students took approximately 30 minutes to finish which was ideal due to the training

conditions. None of the students took more than 30 minutes. All the content was

designed for students to work independently. However, one of the students had a

question about the interpretation of one of the questions and he was assisted

immediately.

b. Instruction

The intended instruction was designed to help learners achieve a higher

understanding of difficult materials by learning through a training module online.

Students would work independently for 10 minutes exposing themselves to the

materials and activities and would answer some surveys and a pre and post-tests. The

observed instruction showed that students were able to navigate through the training

and work independently as it had been envisioned by the design. It was observed that
 Running Head: Complex Texts Module 8

only one student had a question about the content, which was addressed during the

hands-on practice. At the end of the session, three students expressed their desire of

doing something similar during the first week of instruction at DLI.

c. Outcomes

The null hypothesis for this project was that there is no statistical difference

between the pre and post-tests scores after participants take the training. However,

after obtaining the scores from the statistical tests, it can be concluded there is a

statistically significant mean difference between the pre-test and post-test results. See

Table 3.1 and 3.2

The T-sat was 2.17. One tail T statistics was chosen because the design of the

project implies that post test scores should be distinctively higher than Pre- test. As

table 3.3 shows, T Stat was 2.18, which is higher than the Critical one tail of 2.13 with a

p-value of 0.048 smaller than the alpha critical 0.05. Therefore, the null hypothesis

should be rejected concluding that the results are statistically significant. Effect Size

was calculated to determine if the results are practically significant.

The pre-test scores (M=56, SD=20.8) and the post-test (M=86, SD=16.8) differed

significantly. [t(4)=2.1,p<.05, and d=1.4467]. The results showed that calculated p and

calculated T value are close to the critical corresponding values. By calculating the

Cohen's d value, it was possible to deduce that the design is valid due to the difference

between pre-test and post- test scores. The effect size Cohen's d= 1.4467 is significantly

larger than 0.8 threshold for large effect. This proves that even though the results showed

that calculated p and calculated T value are close to the critical corresponding values, the

study was effective and valid.

Running Head: Complex Texts Module 9

Table 3.1 Table 3.2

Table 3.3
 Running Head: Complex Texts Module 10

d. Recommendations

Based on the participants’ comments on the usability survey and the results of the

training, I recommend the following:

1. Add more visuals. Even though the training included a video, images and audio.

One of the participants commented on the usability survey that he was not

auditory, and he would have enjoyed more visuals,” movement” to maintain his

attention instead of audio. This is an important comment because attention is

required for learning.

2. Extend training length. Some participants felt the ten minutes was too short for a

training to really learn something.

3. Add more content. The design for ten minutes was too short to add more content.

By extending the time and adding more activities will allow participants to obtain

more information and expose to more learning.

4. Knowledge check works. As part of the design, participants had to answer a

question after they were done with each section of the lecture. This helped them

focus on the main ideas. I would recommend continuing doing it. See Table 4.1.

5. Add video controls. One participant mentioned the lack of controls to manipulate

the video rewind and fast forward bottoms. Adobe Captivate has a rewind and

fast-forward bar for videos. I suggest adding a comment for the learner about

how to navigate the video prior watching.

6. Add more questions in the student survey. One participant questioned the fact

the training did not address more styles. Therefore, knowing learners’ learning
Running Head: Complex Texts Module 11

preferences can be helpful for next designs. This could be done by adding an

extra question on the student’s survey.

Table 4.1

Section IV

Summary

The training module was designed to teach learner how to approach higher level

materials. In order to test that there was learning, participants presented pre and post-

tests. The research hypothesis for this project was that the post test scores would be

higher than the pretests. The results obtained after doing the paired two-sample t-test

supports the research hypothesis. This training module contributed significantly to

higher post test scores compared to the pretests. The results obtained corroborates the

need of this type of learning experiences for second language learners that continue to

improve their language skills as part of their lifelong learning.

Running Head: Complex Texts Module 12

Section V

Appendices
Appendix A Student’s Survey
Running Head: Complex Texts Module 13

Appendix B Pre and Post test

Running Head: Complex Texts Module 14

Appendix C Training
http://myspace.csumb.edu/~grodriguez-
uribe/ist526/Easy%20guide%20for%20complex%20topics/

Appendix D Observation Checklist

Checklist Yes No Comments

Does the prototype Y The prototype loads

load correctly? immediately

Buttons work Y Some students tried

properly to click on all the
buttons in a different
order just to check
where they took them
or if they worked

Is the transition Y No questions asked

smooth

Can the learner Y Facilitator answered

follow the just one question
instructions properly regarding content not
about the instructions.

Are there any One question related

questions about to the interpretation
content /instructions of a multiple choice
in the background
knowledge check

Does the sound Y No issues found

work?

Do the learners N
require assistance
with any difficulties?
Running Head: Complex Texts Module 15

Appendix E Usability Survey

Running Head: Complex Texts Module 16