Perspectives on how the quality of e-learning relate to its usability.

Neeraj Malik

ABSTRACT

The advent of e-learning (or technology enhanced learning) has opened up new frontiers of
knowledge in India as much as it has world over. Despite the vast advances made as a direct
consequence of this new learning experience, we do not yet have a consolidated methodology for the
evaluation of e-learning applications. When evaluating e-learning material (hardware, software,
content), it is paramount to consider the tools pedagogic effectiveness, efficiency and usability. This
opens up discussions around issues of quality relating to perception and usability of e-learning
applications. This paper aims to explore and review the myriad of perspectives arising in literature,
centered around the relevance of quality and its implications on usability in e-learning and uses
secondary data to explore existing compelling arguments and positions on the subject. The paper
opens future work using the identified quality and usability perspectives as a guideline to ascertain
their influence on end user perception with respect to a defined demography.

Keywords: Perspectives, E-Learning, Evaluation, Quality, Usability.

 Perspectives on how the quality of e-learning relate to its
relevance and usability

INTRODUCTION
E-LEARNING BACKGROUND

As the field of information and communication technology continues on its remarkable growth path,
the education sector has adopted a few IT related strategies, a novel advance which has seen a shifting
from the conventional way of obtaining and transferring academic knowledge. Technology
based/assisted learning from now on referred to as e-learning, which was initially framed as a tool to
help end users in distance learning set-ups has become a global game-changer with respect to all
manner of learning.

E-learning involves the remote communication between an instructor/teacher and student through
the use of information technology platforms. This method has been successfully adopted world-over
because of a few key factors:

- ICT technologies are universal in all countries. As a result many e-learning software products
have interfaces that carry a unified which can be understood independent of language or
nationality.
- Electronic exchange of information and data is now an integral part of the lives of this current
generation of instructors and learners. In all areas, including those in the most remote and
marginalized, information and communication technology tools are now regarded a part of
everyday life. This makes using these same tools for academic information transfer a
relatively easy shift.
- The modern world has evolved into a very fast and dynamic environment, a key adaptation
to thrive involves employing the cheapest and fastest ways to generate and transmit
knowledge, and e-learning is one of the tools that perfectly fit this bill.

In this light, e-learning can be viewed as a simple extension that can work independent of or side by
side with traditional, formal education methods (daily student/teacher face to face interactions). This
method allows end users to freely and independently acquire vast amounts of relevant information
from e-books, encyclopedias, social media, video repositories as well as online courses and open
educational resources. E-learning is applied in different form, specific to the environment of
operation but there are 3 general models:

- A web based instruction model where up to one third (33%) of the time allocated to mastering
the discipline is given in the form of an electronic course. Here the electronic environment
has a supplementary role to traditional learning. This can be achieved through the provision
of e-reading material, self-testing, online forums and webinars and other projects that could
utilize the electronic environment.
- A blended learning, which as the name suggests encompasses both traditional learning and e-
based endeavors. Classroom-based activities are complemented by e-learning activities and

Neeraj Malik, Research Scholar, FMS, MRIU

Page 2 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

the e-learning component can take anything between 20-80% of the total allocated time.
- Full-on e-learning, an approach where 90-100% of educational activities take place in an
electronic environment. To achieve this, the learning content has to be highly interactive and
also provide platforms for real time communication with the teachers and possibly other
learners. (Wisneski, Ozogul, Bichelmeyer, 2015), (Gillani, Eynon, 2014).

E-learning provides immediate access and endless options to all learners that access the platforms. It
provides easier and efficient learner/instructor environments by totally mitigating time and space
limitations. Ideally, internet access alone can guarantee one a world-class education at very minimal
costs, from the comfort of their different work stations. E-learning is a tool that will significantly
alter the future of education and there is a need to understand how to improve the performance,
efficiency and effectiveness: with this in mind it is key to develop uniform frameworks specific to
monitoring and evaluating the quality and usability of e-learning. These models should apply
globally.

STUDENT PROFILES AND CUSTOMIZATION

The learning process is a long-term engagement and outcomes can be seen after an investment of
effort and time. For this to occur, the subject, structure and material must have the ability to sustain
the interest of the learner. Different individuals have different profiles and the profile has an impact
on performance. (Duckworth, Gross, 2014). E-learning is a tool that can overcome this barrier
because e-learning can be adapted to a specific learner, customizing the learning environment.
Customization in educational content delivery has the potential to improve the overall quality of the
learning process (Liu & Yang., 2005), (Huang et al., 2007), (Chang et al., 2009), (Li et al., 2012),
(Chang & Ke., 2013). The content, sequence and pace can be adjusted accordingly to accommodate
the learners’ character traits as well as their learning style. It can be challenging to try and understand
particular needs of different learners in an attempt to personalize the learning experience accordingly,
but if achieved it will surely produce a more effective learning experience.
Customization/personalization is a significant quality and usability overlap that can be considered an
asset because it will improve most learners’ perception towards e-learning and result in a higher
uptake of the educational model (Chae, Shin, 2015)

E-LEARNING ADVANCES

Taking into account the broad definitions of e-learning, mobile learning can be seen as an advance
capable of achieving optimum educational advantages. The surfacing of smart phones provides
opportunity for both instructors and end users to access e-learning materials and services through
handheld mobile devices. This advance is not restricted to location as the devices are mobile. Many
new mobile devices have advanced features such as internet browsers, video streamers, high
definition colour display screens, this makes learning on these devices possible and practical. (Abachi

Neeraj Malik, Research Scholar, FMS, MRIU

Page 3 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

et al., 2014) Integration of mobile devices in e-learning service delivery carries a host of benefits,
namely ubiquity, mobility and freedom. It also provides a platform for real-time instructor-learner
interaction, communication and data sharing. (Motiwalla, 2007). These advances can only be usefully
exploited in the e-learning context when there is an understanding of end user’ perceptions and
perspectives of the mobile e-learning system. Acceptance is easy when one feels that the service and
the content are high quality. (Park et. al., 2012). In this light the usability of e-learning depends on
the quality factors that are considered when developing it. These factors for the most part can be
built off the learners perspectives and perception of e-learning through the use of mobile devices.
Quality factors also influence end users desire and intention towards continued use of the e-learning
platforms because satisfaction is critical for the establishment of long term client relationships
(Patterson et. al., 1997).

QUALITY IN E-LEARNING

Defining the ‘quality’ of education is a complex process. One would have to take a comprehensive
look at a number of factors like environmental suitability, adequate service provision, content,
availability, flexibility, speed, timeliness, these represent the process. In addition one needs to assess
the outcomes from the educational experience because they are also a reflection of the said quality
standards. (Agariya & Singh, 2012). The quantification of quality in an e-learning setup on the other
hand is to an extent complex because e-learning broadly defined, encompasses a number of varied
forms of technology supported learning and involves applying knowledge, educational technology
and information, linking people with educational resources as we as each other for the purpose of
formal or informal education. (Ehlers et al., 2012)

Quality in an e-learning system can be evaluated from many vantage points, one can consider the
product quality and another assesses the quality of the overall learning experience. The product
quality can be further broken down into 3 aspects, system quality framework, information quality
framework (content) and service quality framework. Quality can in e-learning can be viewed either
as the quality of education provided through the use of e-learning tools or it can also refer to the
quality of the combined inputs and subsequent outputs (Teodora et al., 2013)

E-LEARNING AND USABILITY

Usability refers to the extent that a system or product satisfies with effectiveness and efficiency, the
needs as well as specifications of users. Usability is the basis upon which users accept and adopt a
product or system. In an e-learning scene (with particular reference to human-computer interaction),
usability hinges on how well the learners can employ the full functionality of the system. (Nielson,
1993), (Teo et al., 2003). According to ISO 1998 usability can be deemed as defined above in a
specified context of use and is related to the following:

- Product use (efficiently, effectively to a point of satisfaction in a particular context of use.)

Neeraj Malik, Research Scholar, FMS, MRIU

Page 4 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

- The platform user interface and interaction modalities

- The product development process
- Organizational capability towards application of user-centered design.

In a nut shell, usability is primarily focused on making a system easy to learn as well as use.

LITERATURE REVIEW
DIFFERENT PERSPECTIVES ON E-LEARNING QUALITY

The term e-learning is used in reference to a number of different forms of technology supported
learning systems. It can be characterized as the application of information, knowledge and
educational technology. E-learning links people to each other or to educational resources all for the
purpose of education in an either formal or informal learning setup (Ehlers & Hilera, 2012).

(Teodora et al., 2013) suggest that e-learning quality should be examined in two contexts that are,
quality through e-learning and the quality of the e-learning itself.

(Dobre, 2012) puts forward the postulation adopted from Auvinen and Peltonen, which asserts that
we can define the quality of education from three perspectives and these are:

- Technological
- Economic
- Pedagogical
In the model, quality was evaluated in retrospect. The parameters used to evaluate quality included
course quality and pre-defined course outcomes. The approaches evolved to ones that were more
process focused. (Bremer, 2012) outlined an evaluation of the quality of the overall process in
addition to evaluating the outcome.

A more comprehensive assessment of quality in higher education was defined by (Thair et al., 2006)
where combinations of activities need to be carried out to support the quality model. These activities
include:

- Improving core activities (institutional)

- Aligning resource’s, budgets and activities within the bounds of the strategic plan
- A demonstration of innovation and leadership in all activities
- Active exploration of students and other stakeholders’ needs
- Intentional investment towards focused human resources development
- Making data driven decisions (using acquired information and knowledge)

Neeraj Malik, Research Scholar, FMS, MRIU

Page 5 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

- Achieve better outcomes.

Defining quality in e-learning is a complex endeavor and as a result when reviewing literature, one
will run into a number of alternate models, frameworks as well as recommendations for e-learning
quality assurance. A few of the frequently cited ones include (Zhang & Cheng, 2012), (Masoumi &
Lindstrom, 2012), (Saatz & Kienle, 2013), (Udo et al., 2011) and (Kirs, 2011)

Early initiatives on e-learning quality assumed that the quality principles of traditional classroom
education could be superimposed and apply to technology enhanced learning. The key deliverable in
this approach was based off well-designed learning activities being a precursor for ensuring success
regardless of the complimentary means (the presence or absence of and additional technology). This
approach falls short because e-learning is not an alternate implementation method for traditional
classroom teaching; E-learning overall is a new approach to education and quality assurance
frameworks must take this fact into account. (Masoumi & Lindstrom, 2012)

Assessing the level of quality can be done in one of two ways. It can be achieved through a
benchmarking process or by the specification of standards. Benchmarking compares the performance
and achieved results of an evaluated entity against the same for an entity that is operating under
comparable conditions. In a case where standards are defined, comparing them with set standards
sets performance. (Oliver, 2005)

Different perspectives on quality necessitate the need for uniform evaluation parameters. There are
a number of accepted standards and recommendations for quality in e-learning. A few examples
include the Learning Technology Standards Committee (LTSC), The Quality Assurance agency
(QAA), The Institute for Higher Education Policy, Learning Object Metadata (LOM). (Oliver, 2005).
Other workers, (Al-Mushasha & Nassuora, 2012), (Yee, 2013) and (Agariya et al., 2012) have written
on best practices in e-learning, these currently can potentially be used as performance standards.

PERSPECTIVES ON USABILITY

Usability in the e-learning context can be characterized as the effectiveness, adequacy and fulfillment
with which users can accomplish learning related objectives in a specific discipline through a specific
device or learning resource. Usability is closely related to accessibility, the more likely one is to
access a resource, the higher the likelihood that it will be more usable to them and the converse is
true. Usability and accessibility have a direct impact on the pedagogical adequacy of e-learning
suites, this applies to all learners but has a more dramatic impact on crippled learners. (Cooper et al,
2007)

The use of technology in learning should be a tool to empower to users and not a hindrance or barrier
to access. The fundamental aspects for usable design should be considered when developing e-
learning materials to facilitate better usability and uptake from all users. Usability is therefore a core

Neeraj Malik, Research Scholar, FMS, MRIU

Page 6 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

necessity to simplify life for all users of technology enhanced learning and should not be viewed as
a battle only set aside for those with compelling accessibility necessities (Jeffels, 2005)

The e-learning framework on usability takes into consideration, two core perspectives: technical
usability and pedagogical usability. Technical usability focuses on interventions that guarantee
inconvenience free interactions with the e-learning frameworks whereas pedagogical usability has a
hand in supporting the actual learning processes. The two aspects however are intricately interwoven
and they tap into the learner’s psychological assets (Melis et al., 2003)

Pedagogical usability takes into account whether or not the content, tools, software interface and the
assignments on the learning platforms facilitate the best outcomes for different learners in different
setting in accordance to the pre-set pedagogical goals (Tervakari et al., 2002)

THE ISO 9126 MODEL IN THE CONTEXT OF E-LEARNING

ISO/IEC 9126 can be adopted to evaluate the quality of e-learning software. In this regard, ISO 9126
defines quality, as ‘the totality of characteristics of an entity that bear on its ability to satisfy stated
and implied needs.’ This system puts forward a quality model that accounts for six quality
characteristics:

- Functionality: The software’s ability to fulfill the users stated and implied needs in a specific
context of use. (How the software operates to meet the perceived need)
- Reliability: The ability of the software product to perform consistently under stated
conditions for a stated period of time.
- Usability: The software, characteristic, which gives it visual appeal and ensures that it can be
understood, learned and used under specified conditions. (Effort required for use)
- Efficiency: The capability of the software to provide desired performance, related to amount
of resources used, under stated conditions
- Maintainability: The capability of the software product to be modified, corrected, improved
or adapted to environmental, functional specific general or requirement changes. (Effort
required to modify)
- Portability: The feasibility to transfer the software from one environment to another.
Environment could imply organizational, hardware or software.

These quality factors cannot be directly measured or evaluated and will have to be assessed in terms
of more objective sub-characteristics. Different process are put into evaluating the different
characteristics for example, functionality is highly specific to the particular educational domain,
maintainability on the other hand can only be evaluated by the software developer or a third party
with access to the source code (technical documentation), portability however does not require

Neeraj Malik, Research Scholar, FMS, MRIU

Page 7 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

domain specific expertise to assess (Valenti, 2002).

This suite of standards provides a framework for the evaluation of software quality, it does not
prescribe specific requirements but on the contrary describes a quality model that can be applied to
all software (e-learning software included). This standard introduces the quality concept, ‘quality in
use’ which refers to the users point of view.

Using this standard, two models of software quality are specified:

a) The quality characteristics, Functionality, reliability, effectiveness, usability, maintainability

and portability are used to model both internal and external software quality.
b) Usability/Quality in use is dependent on a different set of characteristics: effectiveness,
productivity, security and satisfaction.

E-LEARNING QUALITY FROM AN END USERS PERSPECTIVE

The complexity of the quality concept can be conceptualized in a systematic way. One has to
acknowledge that basis of the e-learning quality complexity; there are different meanings of quality,
different quality perspectives as well as different educational process levels to which quality applies.
(Ehlers, 2002), (Ehlers, 2003) (Ehlers, 2004). We can further distinguish quality into five different
levels, namely context, structure, process, output and impact quality. In each of these subsets, quality
applies differently. A simple scheme is shown below to demonstrate the quality complexity.

Figure 2: Different dimensions of the quality concept in e-learning (cf. Ehlers 2004)

Neeraj Malik, Research Scholar, FMS, MRIU

Page 8 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

Taking into account all the perspectives mentioned in this paper on quality, a definition of quality
would mean one positions themselves centrally in a multi-dimensional space as fig. suggests. We do
not have a universally applicable, standard perspective for the development or assurance of quality.
It is imperative that at every attempt, quality development has to take different perspectives into
account.

Figure 3: Different levels of the quality concept in e-learning (cf. Ehlers 2004)

When we focus on the end user perspective, we move away from standardized learning processes
and move towards customized training biographies (Robinson & Arthy 1998). A student suited e-
learning quality framework looks at the importance of knowledge and information, its acquisition
and application as well as the ability to use technological tools to generate, process and communicate
this knowledge and information. These are skills that are developed as per personal preference,
considering different usage contexts. (Castels, 1996)

Quality from the end user’s perspective must be viewed as a co-production process between the
learner and the learning environment. This means that the learning process is not something passed
on to the student by an e-learning provider but a product of their mutual interaction. With this view,
the e-learning processes do not exclusively influence the outcome of the learning process. Education
in this instance is not a commodity that can be purchased or traded; learning consists of a process
that the und users have to carry out by themselves. Quality in this regard will have to be defined at
the end point of the learning service where the system interacts with the student. This does not mean
the whole quality framework be built solely around the end user’s preference and perspectives, rather
take these as a starting point leading to quality development of all the other areas. The figure below
shows four different components that can be used to conceptualize a learning environment
incorporating different perspectives of quality from the end user vantage point. (Ehlers, 2004).

Neeraj Malik, Research Scholar, FMS, MRIU

Page 9 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

Figure 4: The End User position in a learning environment (Fricke, 1995)

A quality concept built around the learner’s perspective has to be comprehensive and should not only
focus on aspects of instructional and technological interface design. Future work on e-learning
quality development should not be built around a general criterion for all learners

Based on the above literature review, quality and usability in e-learning are two components that can
not be viewed in isolation of each other. Their inter-dependence has been a recurring theme cutting
through the different perspectives outlined by most of the reviewed work. When the sub-
characteristics of quality are explored, it becomes apparent that the two should be fused to give a
more holistic understanding on their relationship. To understand this picture one must step back and
make an assessment from the viewpoint of the end user. Quality influences usability, whereas,
usability is dependent on quality. These two converge at end user and can be assessed both
qualitatively and quantitatively, primarily using learner satisfaction algorithms. The next section of
this paper will focus on this association and how it helps us in formulating a framework for the
inclusive evaluation of quality from a usability standpoint.

Neeraj Malik, Research Scholar, FMS, MRIU

Page 10 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

RELATIONSHIP BETWEEN QUALITY AND USABILITY

Quality refers to the completeness of the features on a product, these features have a bearing on the
products ability to satisfy a user’s needs, (both stated and implied). Usability can be viewed as a
measure of the extent to which the said product actually meets the specific needs in an effective way
to the satisfaction of the user in a defined context. From a simpler perspective, usability is the
assessment of a quality aspect/s with relation to a specified end-user-task scenario.

From the literature review extract we can assert that usability is based on a specific user’s view on a
product’s quality and can be assessed from the point of actually using the said product whereas
quality is based on the properties of the product that make it usable.

This being said, usability is a user specific measurement. We can therefore state that:

The relationship between quality and usability depends on the profile of the user.

An e-learning application can only be termed usable to a particular end user if it meets a specific
need that is relevant to the user. From one user, the core deliverables could be portability because
they are frequently moving (porting) the product, whereas for another a key usability factor would
be maintainability because they are entrusted with the task of maintaining a software platform.

To effectively account for usability based of the quality factors, it is essential to break down the broad
sub-categories outlined in the ISO scheme represented in figure 1 above and decompose them down
to a specific context of use. The aspect addressed should be bot measurable and verifiable. Usability
can then be referred to as a measure of the effectiveness, relating to the accuracy and completeness
with which the said goals have been achieved.

The inclusion of the usability factor makes the ISO quality model more complete and eliminates the
weaknesses seen in comparable models such as McCall’s model, the Boehm model (Behkamal et al.,
2008). The table below explains and summarizes how quality and usability relate to each other.

Table 1: Characteristic and sub-characteristics of the ISO 9126 (Adapted from Chua, 2004)

Characteristic Quality Aspect Specific Usability Probe

Functionality Suitability, - Is the software able to perform required tasks?
Accuracy, - Does it give the expected results?
Interoperability, - Can it interact with other systems?
Compliance, - Does it meet compliance standards?
Security - Does the system prevent unauthorized access?
Reliability Maturity, - Have faults been identified and eliminated?
Fault tolerance, - Is the software able to handle errors?
Recoverability, - Can the system recover lost data and resume

Neeraj Malik, Research Scholar, FMS, MRIU

Page 11 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

Characteristic Quality Aspect Specific Usability Probe

working after failure?
Usability Understandability, - Can a first-time user understand how to use the
Learnability, system easily?
Operability, - Can a user learn how to use the system easily?
Attractiveness - Does the user require a lot of effort to use the
system?
- Is the interface pleasing to behold?
Efficiency Time, - How fast is the system response?
Behavior, - How efficient is the system resource
Resource utilization?
utilization
Maintainability Analyzability, - Can faults be easily diagnosed?’
Changeability - Can the software be easily modified?’
Stability, - Can the software continue functioning if
Testability changes are made?
- Can the software be tested easily?
Portability Adaptability, - Can the software be moved to other
Installability, environments?
Conformance, - Can the software be installed easily?
Replace ability. - Does the software comply with portability
standards?
- Can the software easily replace other software

DISCUSSION
Having high quality e-learning materials and applications relates directly to high usability,
subsequently leading to increased user satisfaction. Increased user satisfaction is related to elevated
levels of intention to use, and normally translates to actual use (Peter et al., 2008).

Satisfaction can be defined as an individual’s perception of the extent to which their needs goals and
desires have been sufficiently met (Sanchez-Franco, 2009). With respect to e-learning, satisfaction
refers to a users overall view of the experience (Wang and Wang, 2009). Whereas intention to use
can be defined as the likelihood that an individual will use a software application, in this case, e-
learning platforms. The role of intention is critical when it comes to the actual adoption and use of
new technology (Davis, 1989). We can also view the intention perspective as an attitude DeLone &
McLean, 2003).

Having considered the above factors, we can come up with an all encompassing quality framework
that attempts to weave the different parameters together in an attempt to reach a more holistic

Neeraj Malik, Research Scholar, FMS, MRIU

Page 12 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

evaluation on the effect of quality and usability in e-learning.

Proposed framework to be used to assess the impact of Quality and Usability perception on actual
uptake of E-learning.

Figure 5: A framework showing how internal and external quality influences usability

In coming up with this framework, we consider the processes involved in the development of e-
learning. The diagram, figure 5 considers three perspectives and how they are intricately related.

These perspectives include:

- Internal quality-refers to the programmatic make up. (Static Code properties).

- External quality-refers to how the software behaves when it is executed.
- Usability-the extent and ease to which the application meets the users need while in use

To measure internal quality, we can perform tests on static measures of the code, to test quality
externally; we can measure the behavior of the product while in use. Usability is then measured based
on the end users perspective on how adequate the software was in fulfilling set requirements of use.

There are measures taken at all three levels because internal quality influences external quality, which
in turn influences usability, which is dependent on external factors, which are dependent on internal
factors.

CONCLUSION
The term quality with reference to e-learning is very complex, as we have seen that we have to
consider and bring to agreement, many aspects, requirements as well as the perspectives of various

Neeraj Malik, Research Scholar, FMS, MRIU

Page 13 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

stakeholders. To achieve a high quality product in the e-learning (education and training) context,
both the learning process and the technology/innovation must be considered.

When a product is of a high quality, it addresses all the issues around its use (usability) and as a result
it is easy or practical to use and gives the overall feeling of user comfort and satisfaction. To best
achieve this, it is best to develop products based on the users view of quality. Usability must be the
key criteria in product development and a key to achieving this is adhering to the criteria for external
quality measures that are dependent on adherence to the internal quality measures.

The measurement of internal and external quality to examine its effect on usability will help systems
designers to make necessary alterations and or adjustments to design content and features to improve
quality from the users point of view. Perceptions on usability in turn influence satisfaction and intent
to use and these two factors have a direct bearing on actual use this therefore, necessitates the
development of an e-learning quality framework that considers all these aspects. This will also help
educators, educational administrators and policy makers to inform the decision-making processes
with regards to e-learning related investment.

REFERENCES
Abachi, H.R. & Muhammad, G. (2014). The impact of m-learning technology on students and
educators, Comput. Hum. Behav. 30, 491–496.
Abran, A., Khelifi, A., Suryn, W., Seffah, A. (2010). Consolidating the ISO Usability Models. École
de Technologie Supérieure, Montréal, Canada Available from:
http://profs.logti.etsmtl.ca/wsuryn/research/SQE-
Publ/Consolidating%20ISO%20Usability%20Models%20% 28SQM03%29.pdf [Accessed: 1
March 2010].
Abran, A., Khelifi, A., Suryn, W. & Seffah, A. (2003). Usability Meanings and Interpretations in
ISO Standards. Software Quality Journal, 11(4), 325-338.
Abel, D. E. & Rout, T. P. (1993). The Mapping of Software Quality Engineering to ISO 9126.
Australian Computer Journal, 25(August), 1-14.
Behkamal, B., Kahani, M., Akbari, M. K. (2008). Customizing ISO 9126 quality model for
evaluation of B2B applications. Information and Software Technology. 51, 599-609.
Berge, Z. L. (1998). Conceptual Frameworks in distance training and education.
Bhatti, S.N. and Kepler, J. (2005). Why Quality? ISO 9126 Software Quality Metrics (Functionality)
Support by UML Suite. ACM SIGSOFT Software Engineering Notes. 30, 2 (March 2005).
Bremer, C. (2012). Enhancing e-learning quality through the application of the AKUE procedure
model. Journal of Computer Assisted Learning, 28(1), 15-26. doi: 10.1111/j.1365-2729.2011.00444.
Castells, M. (1996). The rise of the network society. Blackwell. Oxford.
Chang, Y.C., Li, J.W., Ku, H.H. (2012). A Genetic Algorithm (GA)-Based Personalized Learning
Service in Cloud Learning Environments. IEEE Technology and Engineering Education. 7(2), 28-
32.
Chae, S.E., & Shin, J.H. (2015). Tutoring styles that encourage learner satisfaction, academic

Neeraj Malik, Research Scholar, FMS, MRIU

Page 14 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

engagement, and achievement in an online environment. Interactive Learning Environments, 15.

Chatterjee, S. (2014). Why e-learning has a promising future in India.
http://www.financialexpress.com/article/industry/jobs/why-e-learning-has-a- promising-future-in-
india/19204/
Chua, B.B. & Dyson, L.E. (2004). Applying the ISO 9126 model to the evaluation of an e-learning
system. ASCLITE Conference Proceedings.
Dobre, I. (2012). An Overview of the Most Important Aspects Related to Quality Assurance in
Computer Supported Collaborative E-Learning. International Journal of Computer Science Research
and Application, 2(1), 25-30.
Ehlers, U. (2004). Quality in E-Learning. The Learners Perspective. In: European Journal of
Vocational Training. CEDEFOP. Thessaloniki.
Ehlers, U. D. & Hilera, J. R. (2012). Special Issue on quality in e-learning. Journal of Computer
Assisted Learning, 28(1), 1-3. doi: 10.1111/j.1365-2729.2011.00448.x
E-learning market in India - Rearing to grow, http://ictpost.com/e-learning-market-in- india-rearing-
to-grow
E-Learning Market Trends & Forecast 2014 – 2016 (2014), A report by Docebo,
https://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&
uact=8&ved=0ahUKEwiwp7KM2YDKAhXQHI4KHfJhD-
UQFggcMAA&url=https%3A%2F%2Fwww.docebo.com%2Flanding%2Fcontactfor
m%2Felearning-market-trends-and-forecast-2014-2016-docebo- report.pdf&usg=AFQjCNE
SQixHv01LrIwxLQa8ehfCHrHhzg&sig2=GRKlB4peEx0 bn8wvnwzMYA
Fricke, R. (1995). Evaluation von Multimedia. In Issing, L.J., Klimsa, P. (ed.), Information 
und
lernen mit Multimedia. Weinheim.
Huang, M.J., Huang, H.S., Chen, M.Y. (2007). Constructing a Personalized e-Learning System based
on Genetic Algorithm and Case based Reasoning Approach. Expert Systems with Applications.
33(3), 551-64.
ISO/IEC 9126 Software Product Evaluation - Quality Characteristics and Guidelines for the User.
Gillani, N. & Eynon, R. (2014). Communication patterns in massively open online courses. Internet
and Higher Education, 23, 18-26.
Li, J.W., Chang, Y.C., Chu C.P., Tsai, C.C., (2012). A self-adjusting e-Course Generation Process
for Personalized Learning. Expert Systems with Applications. 39(3), 3223-3232.
Losavio, F., Chirinos, L., Matteo, A., Levy, N., Ramdane-Cherif, A. (2004). ISO quality standards
for measuring architectures. The Journal of Systems and Software. 72, 209– 223.
Martin, M. & Jennings, A. (2002). E-Learning Technology Evaluation Report.
http://www.elearningalliance.org/uploads/attachments/eLearning%20Technology%20Evaluation%
20Report.pdf
Massy, J. (2002) Quality of eLearning Must Improve. BizMedia Ltd,
http://www.learningcitizen.net/articles/QualityofeLearningmu.shtml.
Motiwalla, L.F. (2007). Mobile learning: a framework and evaluation, Comput. Educ. 49 (3), 581–
596.
Pappas, C. (2015). Top e-learning statistics And Facts For 2015.
http://elearningindustry.com/elearning-statistics-and-facts-for-2015

Neeraj Malik, Research Scholar, FMS, MRIU

Page 15 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU
 Perspectives on how the quality of e-learning relate to its
relevance and usability

QED, Quality Initiative E-Learning in Germany (Q.E.D.), Universität Duisburg-Essen, (2004).

http://www.qed-info.de/index_en.php.
Quality for E-Learning. Journal of Knowledge Management, Economics and Information
Technology, 3(1)
Robinson, C. & Arthy, K. (1998). Lifelong learning, developing a training culture: Creating our
future: A new training culture for Australia, NCVER. Adelaide.
Saatz, I. & Kienle, A. (2013). Increasing Quality in large scale University Courses. E-flashcards as
an approach to support active learning and individual facilitation. eleed, 9.
Sarkar, S. (2012). An Introduction to Perceptual Organization. Computer Science and Engineering,
University of South Florida, Tampa, Florida.
Schreiber, D. A. & Berge, Z. L. (1998). Distance training: How innovative organizations are using
technology to maximize learning and meet business objectives. (pp.19-36). San Francisco: Josser-
Bass.
Sharma, S. (2015). 10 Key Insights On PC Usage In India.
http://www.huffingtonpost.in/2015/06/12/what-indian-pc-users- want_n_7551024.html.
Smart, K. L. & Cappel, J. J. (2006). Students’ Perceptions of Online Learning: A Comparative Study.
Journal of Information Technology Education, Vol. 5, 2006, Central Michigan University, Mount
Pleasant, MI, USA.
Spencer, R. (2015). 8 Fatal Interaction Errors That Kill eLearning. http://elearningindustry.com/8-
fatal-interaction-errors-that-kill-elearning.
Sun, P., Tsai, R. J., Finger, G., Chen, Y. & Yeh, Dowming. (2007). What drives a successful e-
Learning? An empirical investigation of the critical factors influencing learner satisfaction.
doi:10.1016/j.compedu.2006.11.007
Telecom Regulatory Authority of India. (September 2015). Press Release No. 47/2015,
www.trai.gov.in.
Teodora, V., Mioara, U. & Magdalena, N. (2013). Quality through E-Learning and Quality for E-
Learning. Journal of Knowledge Management, Economics and Information Technology, 3(1).
Thair, A., Garnett, P. & King, S. (2006). Quality assurance and change in higher education. In L.
Hunt, A. Bromage & B. Tomkinson (Eds.), The Realities of Change in Higher Education:
Interventions to Promote Learning and Teaching (pp. 52–63). London: Routledge.
Udo, G. J., Bagchi, K. K. & Kirs, P. J. (2011). Using SERVQUAL to assess the quality of e-learning
experience. Computers in Human Behavior, 27(3), 1272-1283. doi: 10.1016/j.chb.2011.01.009.
Valenti, S., Cucchiarelli, A. & Panti, M. (2002). Computer Based Assessment Systems Evaluation
via the ISO9126 Quality Model. Journal of Information Technology Education. 1, 3, 157-175.
Van Zeist, R.H.J. & Hendricks, P.R.H. (1996). Specifying software quality with the extended ISO
model. Software Quality Journal. 5, 273-284.
Wisneski, J.E., Ozogul, G. & Bichelmeyer, B.A. (2015). Does teaching presence transfer between
MBA teaching environments? A comparative investigation of instructional design practices
associated with teaching presence. The Internet and Higher Education, 25, 18-27.

Neeraj Malik, Research Scholar, FMS, MRIU

Page 16 of 16
Dr. Nandini Srivastava, Supervisor, FMS, MRIU