5th National eLearning Conference

Taking IT to the Next Level: The Team Approach to eLearning Program Development

A Digital Courseware
Evaluation Rubric for
CICT Content Development Projects

Maria Melizza D. Tan

HEA, CICT-HCDG
Project Coordinator, eSkwela
mdtan@ncc.gov.ph
AGENDA

CICT’s Content Development Strategy
• BECTA Quality Principles

Digital Courseware Evaluation Rubric:
the eSkwela Experience

Commission on Information and Communications Technology

HCDG’s Programs and Projects

national ICT competency standards +
certifications

ICT4E: transformative role of ICT in
teaching and learning

ICT competency training for government
personnel

eSociety: digital inclusion + digital
communities

Commission on Information and Communications Technology

Comprehensive Approach
Best Practices
+ Standards
constant
Training Content clamor
from the
field

Infrastructure Applications

Community Involvement
Coordination + Ownership +
Sustainability

Commission on Information and Communications Technology

Digital Content Development
“After the first decade of ICT4D, a major lesson learned by the
development assistance sector, particularly the UN agencies
and the World Bank, is that investments on content and
capability building should be significantly larger than
investments on infrastructure (Walsh 2006). According to some
estimates, for every dollar invested on ICT4D, ten cents should
go to infrastructure. Another ten cents should go to software.
Still another ten cents should go to training. But the remaining
70 cents should be spent on content development.”

− DepEd’s Strategic Framework on ICT4E under BESRA

Commission on Information and Communications Technology

Content Development Strategy

Developing digital content that is
appropriate, relevant, accessible, and
meaningful to Filipinos is a priority of CICT.
• Interactive, engaging, motivating
• High Quality
• Well-designed
• Well-produced
• Easily replicated at low cost for greater access

Focus Areas:
• Educational Multimedia Content
• Digital Cultural Content
Commission on Information and Communications Technology
Content Development Strategy
1. Mapping the content landscape
2. Developing a Content Development
Policy Framework
3. Building the necessary technical
architecture for comprehensive
national access
4. Training: development & use

Commission on Information and Communications Technology

 Digital Content Devt phase 1

Mapping activities

Multi-media Learning
Materials for Teachers (Pre &
In)

Computer-Based Training
Modules (CBTs) for English
Proficiency

Web Competitions

Content and Application
Development for Rural
Communities and Enterprises:
Digitization of Replicable
Research Outputs of Partner
SUCs

Jeepney ni JenJen –-
Digitization of Tuklas Sining

eHealth: Health information
for emergencies website

Indigenous People’s (IP)
Knowledge

skoool.ph

eSkwela elearning modules

Islam in the Philippines

Commission on Information and Communications Technology

QA: BECTA Principles

British Educational Communications
and Technology Agency (BECTA)
Quality Principles for Digital Learning
Resources
1. Core Pedagogic Principles
2. Core Design Principles
http://partners.becta.org.uk/index.php?section=sa&catcode=_sa_cs_cf_03

Commission on Information and Communications Technology

 BECTA Principles

Core Pedagogic Principles
1. Match to the curriculum
2. Inclusion and access
3. Learner engagement,
challenge, and motivation
4. Effective learning
5. Assessment to support
learning
6. Robust summative
assessment
7. Innovative approaches
8. Ease of use
Commission on Information and Communications Technology

Core Design Principles
1. Digital learning resource
design
2. Robustness and support
3. Human-computer interaction
4. Quality and Suitability of
assets
5. Accessibility
6. Interoperability
7. Testing and verification
8. Effective communication
eSkwela

to use ICTs in broadening access to
basic education by tapping into
appropriate ICT-supported models to
improve the quality of learning and
enhancing the quality of teaching in the
non-formal education setting

target: out-of-school youth and adults

Commission on Information and Communications Technology

 eSkwela

Infrastructure
Deployment Materials Development

eSkwela
Community
Support for Stakeholders’
Program Training
Sustainability

Commission on Information and Communications Technology

eSkwela Content Development

roots: Smokey Mountain

conversion of existing ALS print modules
into digital modules
• Print modules: BALS + SEAMEO-INNOTECH
• Curriculum Review done by BALS (2005)

Commission on Information and Communications Technology

eSkwela
eSkwela
elearning
modules (A&E)
(beta) Learner outputs/
eSkwela portfolios
Learning +
Management Teacher-learner
System interactions
(alpha) +
Teacher-teacher &
BALS Session learner-learner
Guides & other interactions
materials

Involvement of Learning
Community
+
Livelihood Programs
Commission on Information and Communications Technology
Module Review for eSkwela

assumptions & constraints:
• content of print modules already reviewed
• HCDG-EDO-BALS combined effort: devt of
version 1
• English as medium of instruction
• review process requirement
• verbal discussions/agreements: problematic
• search for existing evaluation form

Commission on Information and Communications Technology

 Module Review Experience

Rubric: a work-in-progress

Briefing and discussion with reviewers/ raters
• Retooling on ICT-enhanced instructional design
• i.e. Session Guides also need to be revised!
• Clarifications on item definitions/ descriptions
• Targets: at least 3 per rubric item
• Not summed up/averaged
• Clarifications on developer limitations and scope of work

Clearer standards/ expectations for developers
• Currently in the module enhancement stage (1st 20)
• Due to time constraints: “minimums” were set; better to
have priority levels for detailed recommendations

Commission on Information and Communications Technology

Evaluation Rubric

Assumptions:
• The learners are not impaired in any way (i.e. not
disabled or psychologically dysfunctional).
• The modules are always accessible to the learners.

Acknowledgements:
• Some rubric definitions were taken from publicly-
available rubric-makers on Writing and Web Design,
located at Rubistar (http://rubistar.4teachers.org).

Initial Review: Dr. Lolit Suplido-Westergaard

Commission on Information and Communications Technology

 Evaluation Rubric: Content
4 3 2 1

The entire module Majority of the Module content Module content

is clearly matched module content is is loosely does not match
to /appropriate to matched to / matched to set the set
Matched to / set curricular appropriate to set curricular curricular
Appropriate standards. curricular standards. standards.
for the standards but
Curriculum may have one or
two elements that
are not
appropriate.

Commission on Information and Communications Technology

Evaluation Rubric: Content
4 3 2 1

The module has The module has a The purpose The module
a well-stated clearly stated and theme of lacks a purpose
purpose and purpose and the module is and theme.
theme that is theme, but may somewhat
Focus
maintained have one or two muddy or
consistently elements that do vague.
throughout the not seem to be
site. related to it.

Commission on Information and Communications Technology

 Evaluation Rubric: Content
4 3 2 1

Content is well Uses headings Content is There was no

organized using or bulleted lists logically clear or logical
headings or to organize, but organized for organizational
bulleted lists to the overall the most part. structure.
group related organization of
Organization material. topics appears
Subtopics flawed.
naturally
progress into
succeeding
ones.

Commission on Information and Communications Technology

 Evaluation Rubric: Content
4 3 2 1

Covers topic Includes Includes Content is

in-depth with essential essential minimal OR
details, extra knowledge information there are
information, about the topic. about the topic several factual
Thoroughness and examples. Subject but there are 1- errors.
and Accuracy Subject knowledge 2 factual errors.
knowledge is appears to be
excellent – no good – no
factual errors factual errors
detected. detected.

Commission on Information and Communications Technology

Evaluation Rubric: Content
4 3 2 1

The author has The author The author has The author has
made an has tried to put lots of provided only the
exceptional make the information in minimum amount
attempt to make content of this the module but of information and
the content of module there is little has not
this module interesting to evidence that transformed the
Interest interesting to the the people for the person tried information to
people for whom whom it is to present the make it more
it is intended. intended. information in interesting to the
an interesting audience (e.g.,
way. has only provided
a list of links to the
content of others).

Commission on Information and Communications Technology

 Evaluation Rubric: Content
4 3 2 1

Forms of Forms of Forms of There is only

assessment assessment assessment one form of
(formative and (formative and (formative and assessment,
summative) are summative) are summative) are and it is
Quality of varied, effective limited, but few, barely ineffective or
Assessment and appropriate effective and effective or inappropriate to
to the content. appropriate to inappropriate to the content.
the content. the content.

Commission on Information and Communications Technology

 Evaluation Rubric: Content
4 3 2 1

Module Module Module is Module is

consistently generally makes missing more lacking several
makes use of use of ICT than two key key elements
ICT effectively effectively vis-à- elements in its and has
vis-à-vis content vis content to use of ICT vis-à- inaccuracies
to enhance enhance vis content to that make it a
Effectiveness learning. It is a learning but is enhance poor elearning
highly effective lacking one or learning. It module.
elearning two key would make an
module. elements. It is incomplete
an adequate elearning
elearning module.
module.

Commission on Information and Communications Technology

 Evaluation Rubric: Content
4 3 2 1

Source information Source information Source information Very little or

collected for all collected for all collected for no source
graphics, facts, and graphics, facts, graphics, facts, and information
Sources quotes. All and quotes. Most quotes, but not was collected.
documented in the documented in the documented in the
desired format. desired format. desired format.

Commission on Information and Communications Technology

 Evaluation Rubric: Content
4 3 2 1

Fair use guidelines Fair use guidelines Fair use guidelines Borrowed
are followed with are followed with are followed with materials are
clear, easy-to- clear, easy-to- clear, easy-to-locate not properly
locate and accurate locate and and accurate documented
citations for all accurate citations citations for most OR material
borrowed material. for almost all borrowed material. was published
No material is borrowed material. No material is without
included from No material is included from obtaining
Copyright
sources that state included from sources that state permission
that publication is sources that state that publication is from copyright
not allowed unless that publication is not allowed unless owner.
permission has not allowed unless permission has
been obtained from permission has been obtained from
copyright owner. been obtained copyright owner.
from copyright
owner.

Commission on Information and Communications Technology

Evaluation Rubric: Design
4 3 2 1

Makes Makes good Makes use of Use of font,

excellent use use of font, font, color, color,
of font, color, color, graphics, graphics,
graphics, graphics, video, video,
video, video, animation, etc. animation, etc.
Aesthetics animation, etc. animation, etc. but but these often
to enhance the to enhance to occasionally distract from
presentation. presentation. these detract the
from the presentation
presentation content.
content.

Commission on Information and Communications Technology

 Evaluation Rubric: Design
4 3 2 1

Multimedia Multimedia Multimedia Multimedia

elements are elements are elements are elements seem
relevant and related and relevant and to be used
Multimedia appropriate to appropriate to appropriate to randomly, are of
(Graphics, the module, are the module, are the module, are poor quality or
Sound, Video, of high quality of good quality of good quality. distract the
etc.) and enhance the and enhance the learner from the
learner's interest learner's interest content.
or or
understanding. understanding.

Commission on Information and Communications Technology

 Evaluation Rubric: Design
4 3 2 1

Module allows Module allows Module allows a Module does not

learner a high an acceptable minimum degree contain any form
degree of degree of of interactivity by of learner
interactivity by interactivity by manipulating interactivity nor
manipulating manipulating objects in features for
objects in objects in animations/ immediate
animations/ animations/ simulations, feedback.
Interactivity
simulations, simulations, models, data
models, data models, data sets, or online
sets, or online sets, or online assessment.
assessment. assessment.
Module provides Module provides
immediate immediate
feedback. feedback.

Commission on Information and Communications Technology

 Evaluation Rubric: Design
4 3 2 1

All buttons and Almost all (90%- Most links Less than
links are clearly 99%) links are (75%-89%) 75%of the links
labeled, clearly labeled, links are are clearly
consistently consistently clearly labeled,
placed, and work placed, and labeled, consistently
properly. work properly. A consistently placed, work
A user does not user rarely placed, and properly. A user
Navigation
become lost. becomes lost. work properly. typically feels
Some needed lost.
links seem to
be missing. A
user
sometimes
gets lost.

Commission on Information and Communications Technology

 Evaluation Rubric: Design
4 3 2 1

Help and/or Help and/ or Help and/or No Help or

FAQ is easily FAQ is FAQ is hard to FAQ function.
accessible, accessible, find. Content is
User Support understandable understandable hardly
for the users and mostly understandabl
and comprehensive e or obviously
comprehensive. . lacking.

Commission on Information and Communications Technology

http://balsontheweb.wikispaces.com
5th National eLearning Conference
Taking IT to the Next Level: The Team Approach to eLearning Program Development

Thank you.

mdtan@ncc.gov.ph