i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

journal homepage: www.intl.elsevierhealth.com/journals/ijmi

An evaluation framework for Health Information

Systems: human, organization
and technology-t factors (HOT-t)

Maryati Mohd. Yusof a, , Jasna Kuljis b ,

Anastasia Papazafeiropoulou b , Lampros K. Stergioulas b
a Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia
b School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK

a r t i c l e i n f o a b s t r a c t

Article history: Background and purpose: The realization of Health Information Systems (HIS) requires rig-
Received 15 December 2005 orous evaluation that addresses technology, human and organization issues. Our review
Received in revised form indicates that current evaluation methods evaluate different aspects of HIS and they can
12 August 2007 be improved upon. A new evaluation framework, human, organization and technology-t
Accepted 12 August 2007 (HOT-t) was developed after having conducted a critical appraisal of the ndings of existing
HIS evaluation studies. HOT-t builds on previous models of IS evaluationin particular, the
IS Success Model and the IT-Organization Fit Model. This paper introduces the new frame-
Keywords: work for HIS evaluation that incorporates comprehensive dimensions and measures of HIS
Information Systems and provides a technological, human and organizational t.
Health Information Systems Methods: Literature review on HIS and IS evaluation studies and pilot testing of developed
Evaluation framework. The framework was used to evaluate a Fundus Imaging System (FIS) of a primary
Framework care organization in the UK. The case study was conducted through observation, interview
Human factors and document analysis.
Organizational factors Results: The main ndings show that having the right user attitude and skills base together
with good leadership, IT-friendly environment and good communication can have positive
inuence on the system adoption.
Conclusions: Comprehensive, specic evaluation factors, dimensions and measures in the
new framework (HOT-t) are applicable in HIS evaluation. The use of such a framework is
argued to be useful not only for comprehensive evaluation of the particular FIS system under
investigation, but potentially also for any Health Information System in general.
2007 Elsevier Ireland Ltd. All rights reserved.

1. Introduction tem [1]. The more technology, human and organization t

with each other, the greater the potential of HIS. Most existing
The benets derived from a Health Information Systems (HIS) evaluation studies of HIS focus on technical issues or clinical
require rigorous evaluation. It is claimed that organizational processes, which do not explain why HIS works well or poorly
and social issues are the main components of such a sys- with a specic user in a specic setting [27].

DOI of original article:10.1016/j.ijmedinf.2007.08.004.

Corresponding author. Tel.: +603 8921 6649.
E-mail addresses: mmy@ftsm.ukm.my, maryati226@yahoo.com (M.Mohd. Yusof).
1386-5056/$ see front matter 2007 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.ijmedinf.2007.08.011
 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
The central purpose of this paper is to show how we
derived a framework for HIS evaluation that incorporates com-
prehensive dimensions and measures of HIS and provides
a technological, human and organizational t. A case study
demonstrates the application of the proposed framework for
describing the critical adoption factors of a particular HIS
with a particular user in a particular setting. This proposed
framework, human, organization and technology-t (HOT-t)
is constructed so that it is capable of being useful in conduct-
ing a thorough evaluation study. It is also possible that it will
assist researchers and practitioners to unfold and understand
the perceived complexity of HIS evaluation. The new frame-
work builds on previous work on the review of HIS evaluation
[8,9]. It also makes use of two models of IS evaluation, namely
the IS Success Model [10,11] and the IT-Organization Fit Model
[12].
This paper is organized as follows. Section 2 discusses
the theoretical background of the proposed framework. The
aforementioned models are presented to explore their appli-
cability in improving on those used in Health Informatics. This
then forms the basis for our rst proposed evaluation frame-
work for HIS presented in Section 3. Section 4 discusses an
existing Fundus Imaging System in a primary care organiza-
tion that is used as a research case study. Section 5 presents
the research methodology for applying our framework to this
case study. The case study ndings are presented in Section
6. Finally, discussion and conclusions are given in the last
section.
2. Theoretical background
The proposed human, organization and technology-t evalu-
ation framework was developed after a critical investigation
of the existing ndings of HIS and IS evaluation stud-
ies (discussed in the preceding paper in this issue of the
journal, [13] and from feedback gained from two confer-
ence presentations of earlier stages in the research [8,9]).
A review of success determinants of Inpatient Clinical IS
indicates that the categories for success in the IS Suc-
cess Model can be used to assess HIS [14]. The IS Success
Model has then been identied as being complementary
to another model in fullling the limitations of existing
HIS evaluation frameworks, namely the IT-Organization Fit
Model. As a result, both models are utilized in constructing
HOT-t.
The IS Success Model is adopted because of its com-
prehensive, specic evaluation categories, extensive val-
idation and its applicability to HIS evaluation [10,11].
HOT-t makes use of the IS Success Model in catego-
rizing its evaluation factors, dimensions and measures.
In addition, the IT-Organization Fit Model [12] is used
to complement the IS Success Model by integrating its
featured organizational factors and the concept of t
between the human, organizational and technological fac-
tors. The two models are discussed in the last two
sub-sections of this section prior to HOT-ts development
being explained in Section 4. But rst some background to
the human and organizational aspects is necessary for later
discussions.
387
2.1. Human and organizational factors
The importance of human and organizational factors in the
development and implementation of IS has been advocated in
the IS literature. Rigorous evaluation of HIS can be undertaken
by addressing these factors as well as the t or alignment
between them. According to Willcocks [15], the alignment of
organization, technology and human is an important starting
point in IT implementation as it is one of the strategies that
affect IT investment.
Barriers to using HIS are also important to consider in HIS
evaluation as they explain the failure and success of these
systems. Culture and process changes are reported to be the
barriers to the wider use of health care systems [16]. Studies
cited in Anderson [17] identied a number of barriers to direct
physician use of HIS including low level of expertise, lack of
acceptance, lack of medical staff sponsorship and alteration
of traditional workow patterns. Examples of organizational
challenges include hospital culture, such as being risk adverse,
reluctance to invest much in IT and resistance to change
[18,19].
A study on factors inuencing success and failure of HIS by
a group of medical informaticians identied 110 success fac-
tors and 27 failure criteria from a wide range of socio-technical
issues [20]. These factors are assessed for six types of HIS.
Highest success factors for Clinical Information Systems (CIS)
are: collaboration and cooperation, setting goals and courses;
while for educational system is user acceptance. Highest fail-
ure criteria for CIS is response rate; while for administrative
system is not understanding the organizational context and
not foreseeing the extent to which new HIS affects the orga-
nization, its structure and/or work procedures.
In short, human and organizational factors are as impor-
tant as technical issues with regards to system effectiveness
[4]. Human, organizational and technical elements should also
have a mutual alignment or t in order to ensure successful
HIS implementation. It is crucial that HIS t organizational
aspects as well as align with work routines, management
assumptions, patient care philosophies and users needs as
the introduction of a system affects different dimensions
of t in complex ways [3]. A number of studies in Health
Informatics have included the concept of t in explaining
the interdependent relationship between human, organiza-
tion and technology factors [21,22]. Aarts et al. [21] propose a
model which illustrates the stages of information and system
changes and their relative specic personal requirements.
They argued that changes in both technology and health prac-
tices affect each other to a similar extent. Berg [22] makes
use of the socio-technical approach in HIS evaluation; work
practices are seen as integrated networks of various related
elements such as people, tools, organizational processes,
machines, and documents.
Southon et al. [23] found that the lack of t among main
organizational elements contributes to a large number of
system failures in public health. The t between techni-
cal, organizational, and social factors is analyzed to identify
gaps between current health care systems and new sys-
tem features [24]. Kaplan [3] shows that poor t between
system developers goals and clinicians cultural values con-
tributes to user reluctance to use Clinical Decision Support
388 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

Systems (CDSS). Executive support, understanding the busi-

ness, IT-business relations, and leadership are identied as
both enablers and inhibitors of the t of IT and business
[25].

2.2. IS Success Model

In order to structure numerous results from research on IS

success categories; a comprehensive taxonomy is introduced
[10]. A model is constructed which consists of six success cate-
gories or dimensions; they are linked causally and temporally
as success is viewed as a dynamic process instead of a static Fig. 2 The MIT90s (IT-Organization Fit Model) (adapted
state. The multidimensional relationships among the mea- from Scott Morton, [12]).
sures of IS success have been tested extensively in a number
of IS studies [26]. Based on these studies, an updated version
of this model is presented [11] (see Fig. 1).
2.3. IT-Organization Fit
These measures are included in the six system dimensions:
System Quality (the measures of the information processing
Management in the 1990s (MIT90s) is a well-known IT-
system itself), Information Quality (the measures of IS output),
Organization Fit Model, which includes both internal and
Service Quality (the measures of technical support or service),
external elements of t [12]. Fig. 2 illustrates the concept
Information Use (recipient consumption of the output of IS),
of t between the main organizational elements. Internal t
User Satisfaction (recipient response to the use of the output
is accomplished by a dynamic equilibrium of organizational
of IS) and Net Benets (the overall IS impact).
components including business strategy, organizational struc-
Because of its circularity, the framework might lead to spi-
ture, management processes, and roles and skills. External
ral behaviors in both positive and negative directions. For
t is achieved by formulating organizational strategy based
example, effective use of the system will result in higher net
on environmental trends and changes such as market, indus-
benets which lead to more intensive use of the system. In
try and technology. Within this internal and external t as its
contrast, insufcient system use will yield to lower net ben-
enabler, IT is expected to affect the management process, thus
ets; thereby acting as a disincentive to system use. In both
impacting on organizational performance and to some degree,
cases, the IT-Organization Fit Model highlights the important
its strategy.
role of the organization in managing the process of changes
In order to realize the benets of IT, three prerequi-
that take place during system introduction [12]. Effective pro-
sites are required for successful IT transformation. First,
cess management can lead to increased user acceptance and
organizational vision and the reasons behind it have to be
participation in system use, which will in turn help achieve
clear to organizational members to get them prepared for
high net benets and subsequently increase system use.
organizational changes and hence reduce the challenges in
In comparison with existing HIS frameworks, DeLone and
managing transformation. Second, organizational corporate
McLeans IS model illustrates clear, specic dimensions of IS
strategy (business and IT), information technology and organi-
success or effectiveness and the relationships between them.
zational dimensions have to be aligned with each other. Third,
However, it does not include organizational factors that are
a robust IT infrastructure such as an electronic network and
pertinent to IS evaluation. In addition, Van der Meijden et
understood standards should be equipped within the organi-
al. [14] discovered that a number of measures such as user
zation. These three prerequisites as well as the internal and
involvement during system development and organizational
external t may be used to identify the problems in IT imple-
culture do not match any of the dimensions of the framework.
mentation. This model was relatively new and has not been
The extension of this framework is recommended by adding
extensively utilized in healthcare [27]. The model was also
the organizational factors, their dimensions and clinical mea-
identied as being capable of identifying the main organiza-
sures related to the healthcare domain.
tional elements which can affect IS as well as emphasizing the
essential alignment or t between them. Moreover, the model
is comprehensive as it includes the following factors: technol-
ogy (IT), human (roles and skills) and organization (strategy,
structure and management process). However, these factors
can be categorized into more detailed dimensions to provide
more specic evaluation dimensions. For instance, IT can be
further classied into system quality and information quality,
as proposed by DeLone and McLean [10]. Similarly, roles and
skills can be associated with use and user satisfaction.
Based on the strengths and limitations pointed out in both
models, IT-Organization Fit and the IS Success Model comple-
Fig. 1 Information System Success Model (Source: DeLone ment each other in presenting a comprehensive evaluation
and McLean [11]). framework. Organizational factors, which are lacking in the IS
 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
Fig. 3 Proposed human-organization-technology t (HOT-t) framework.
Success Model, are featured in the IT-Organization t. Simi-
larly, specic evaluation dimensions and measures which are
lacking in the IT-Organization t, are featured in the IS Suc-
cess Model. Based on the two models explained above, a new
evaluation framework is presented in the next section.
3. Proposed evaluation framework
Building on previous studies on the evaluation approach [8,9],
the proposed evaluation framework was developed after a crit-
ical appraisal of the existing ndings of HIS and IS evaluation
studies. It makes use of the IS Success Model in categorizing its
evaluation factors, dimensions and measures. The IS Success
Model is adopted based on its comprehensive, specic evalua-
tion categories, extensive validation and its applicability to HIS
evaluation. In addition, the IT-Organization Fit Model is also
used to incorporate the concept of t between the evaluation
factors: human, organization and technology. The IS Success
Model was extended by the addition of the following features
which are explained in the following part of this section (see
Fig. 3):
Organization factors, their dimensions (Structure and Envi-
ronment) and evaluation measures (listed in Table 1).
Fit between technology, human and organization factors.
Two-way relationships between these dimensions: Informa-
tion Quality and System Use, Information Quality and User
Satisfaction, Organizational Structure and Environment, Organi-
zational Structure and Net Benets, Organizational Environment
and Net Benets.
One-way relationship between these dimensions: Structure
and System Use.
New evaluation measures pertinent to HIS and IS in general.
Human, organization and technology are the essential
components of IS; the impacts of HIS are assessed in the net
benets. These three factors and the impacts of HIS corre-
spond to eight interrelated dimensions of HIS success: System
Quality, Information Quality, Service Quality, System Use, User
389
Satisfaction, Organizational Structure, Organizational Environment
and Net Benets. Each of these dimensions is associated with a
number of evaluation measures. Examples of evaluation mea-
sures according to their corresponding dimension and factor
are listed in Table 1.
These evaluation dimensions inuenced each other in a
temporal and causal way:
System Quality, Information Quality and Service Quality singu-
larly and jointly affect both System Use and User Satisfaction.
Organizational Structure and Organizational Environment
affects System Use.
Some of these relationships are two ways:
System Use, which relies on user knowledge and training,
can inuence the Information Quality, since the users knowl-
edge in using the system can affect reports, images and
prescriptions produced by the system.
The level of System Use can affect the degree of User Satis-
faction and vice versa, for both positive and negative cases.
Effective System Use yield to higher User Satisfaction as user
is able to explore and make full use of system features
and functions; higher User Satisfaction subsequently moti-
vate/lead user to increase System Use.
Similarly, the Organizational Environment factors such as
government policy and politics can affect Organizational
Structure while factors in Organizational Structure will affect
the population served in the Organizational Environment.
System Use and User Satisfaction are direct antecedents of
Net Benets. Net Benets subsequently affect System Use and
User Satisfaction. Similarly, Organizational Structure and Envi-
ronment are direct antecedents of Net Benets. Net Benets
subsequently have impact on organizational Structure and
Environment.
The concept of t is perceived as complex, abstract and
subjective. It can be viewed in terms of strategic planning
(formulating IS plan according to organizational plan) and
strategic alignment (managing IT closely with organizational
needs) perspectives [23]. In the HOT-t context, t is concerned
with the ability of HIS, human (HIS stakeholders and clinical
390 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398

practices) and setting to align with each other. Thus, t can be

communication, clinical
efciency, effectiveness

outcomes (patient care,

measured and analyzed from multiple compatibility between

morbidity, mortality,)
Clinical practice (Job

participation), error
human, organization and technology (human-organization,

(analysis, accuracy,
(goal achievement,
productivity, work
Net benets

service), decision

time, condence,
volume, morale,)
human-technology, organization-technology) using a number

making quality
performance,
of measures dened in the three factors including system
effects, task

reduction,
exibility, systems ease of use, system usefulness, informa-
tion relevancy, user attitude, user training, user satisfaction,

cost
organizational culture, planning, strategy, management and
communication. For example, an effective use of a HIS can be
politics, localization,

inter-organizational

attributed to the t between system exibility and clinical pro-

population served,
Environment

Financing source,

cess (system-human). The t between human, organization

communication

and technology is illustrated by the bold arrows in Fig. 3.

competition,
government,

relationship,

Based on its comprehensive dimensions and outcome

external
Organization

measures, the framework could be used to evaluate the

performance, effectiveness and impact of HIS or IT in health-
care settings. Effectiveness refers to the accomplishment of
specic goals with accuracy and completeness, as well as
Nature, (type, size)
culture, planning,

support, medical
communication,
clinical process,

the correct utilization of appropriate resources [28]. In this

Structure

leadership, top
management,

management

sponsorship,

research, effectiveness is dened as the ability of a health-

champion,
autonomy,

teamwork
mediator,

care organization to continuously accomplish goals using

strategy,

optimum resources within a specied time. The three evalua-

tion factors can be evaluated through out the whole system
development life cycle namely planning, analysis, design,
User satisfaction

specic functions,

implementation, operation and maintenance. Meanwhile, Net

Satisfaction with

decision making

benets can be anticipated before implementation and appar-

satisfaction,

satisfaction,

satisfaction
enjoyment,
usefulness,

ently evaluated after implementation. As mentioned above,

perceived

software

each phase focuses on different issues. This framework can

overall

be applied using qualitative, quantitative or a combination of

both approaches. Subsequently, any data collection methods
from both approaches can be employed while conducting the
intended purpose, appropriate use,
reported use, nature of use (use for
number of functions used, number

type of information used,) purpose

inquiries, amount of connect time,

knowledge/expertise, acceptance,
use, attitude, expectations/belief,
generated), use by whom? (direct
of records accessed, frequency of
Human

voluntaries of use, motivation to

evaluation.
vs. chauffeured use,) actual vs.

resistance/reluctance, training
of use, level of use (general vs.
specic,) recurring use, report
Amount/duration:(number of

acceptance, percentage used,

The following subsections explain the evaluation factors,

requests, number of reports
access, frequency of report

dimensions and measures in detail.

System use
Table 1 Examples of the evaluation measures of the proposed framework

3.1. Technology

The studies on System Quality are often associated with system

performance. System Quality in a healthcare setting measures
the inherent features of HIS including system performance
and user interface. Examples of system quality measures
Service quality

empathy, follow
responsiveness,

are ease of use, ease of learning, response time, usefulness,

availability, reliability, completeness, system exibility, and
up service,
assurance,

technical

security [2933]. Ease of use assesses whether healthcare pro-

support
Quick

fessionals regard HIS as satisfactory, convenient and pleasant

to use. Availability refers to the up time of HIS while exibility
is concerned with the ability of HIS to adapt to a healthcare
Information quality

setting and integrate with other systems. Even systems that

legibility, format,

often work are often not used as anticipated. Thus, it is impor-

timeliness, data
entry methods
completeness,

tant to determine whether the system (1) meets the need of

conciseness,
Importance,

usefulness,
relevance,

reliability,

the projected users, (2) is convenient and easy to use, (3) ts

accuracy,

the work patterns of the professionals for whom it is intended

and the overall health system [30].
Measures of Information Quality are concerned with infor-
support, security,
System quality

contents, ease of

mation produced by HIS including patient records, reports,

turnaround time
system features

response time,
and functions,
Data accuracy,
data currency,

images and prescriptions. Information quality measures can

usefulness of
Technology

use, ease of

availability,

utilization,

be subjective, as they are derived from the user perspec-

efciency,
reliability,
exibility,
Database

technical
learning,

resource

tive. Criteria that can be used for HIS quality are information
completeness, accuracy, legibility, timeliness, availability, rel-
evancy, consistency and reliability [23,29,30,33,34].
 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
Service Quality is concerned with the overall support deliv-
ered by the service provider of HIS or technology, regardless of
whether the service is delivered by the internal department of
healthcare organization or outsourced to external providers.
Service quality can be measured through technical support,
quick responsiveness, assurance, empathy and follow up
service.
3.2. Human
The IS Success Model includes the use of system and user sat-
isfaction while MIT90s incorporates roles and skills as part of
human factors elements. HOT-t combines those human fac-
tors in both frameworks in addition to other factors of human
factors as mentioned earlier.
System Use is concerned with the frequency and breadth
of HIS inquiries and functions. The use of information out-
put such as reports appears to be one of the most frequent
measures to assess the success of IS. The actual use of the sys-
tem as a measure of IS success refers to voluntary instead of
mandatory use. System Use also relates to the person who uses
it, their levels of use, training, knowledge, belief, expectation
and acceptance or resistance [30,33].
Knowledge is concerned with computer literacy and skills
[7,17]. Expectation refers to the anticipation of improved
patient care delivery from the use of HIS [7]. Jiang et al. [35]
regard resistance as an important factor of system success.
As different types of systems are usually related with a par-
ticular type of function and user, the reasons for resistance
might differ among system types. Resistance can be viewed
from one of the following theories: (1) people-oriented, (2)
system-oriented and (3) interaction-oriented. People-oriented
theory describes resistance to system results from users
(groups or individuals) internal factors. Personal characteris-
tics such as age, gender, background, value and belief have
been suggested as inuencing individuals attitude towards
technology. System oriented theory suggests that resistance
results from system design factors or relevant technology
including user interface and system characteristics. Interac-
tion theory explains resistance from the interaction between
people and system factors; thus, assessment of a system
varies across settings and users. Job insecurity and fear are
some examples of interaction resistance.
User Satisfaction is often used to measure system success.
It is subjective in nature as it depends on whose satisfac-
tion is measured. User satisfaction is dened as the overall
evaluation of a users experience in using the system and
the potential impact of the system. User Satisfaction can be
related to users perceived usefulness and attitudes towards
HIS which are inuenced by his/her personal characteristics.
3.3. Organization
The nature of a healthcare institution can be examined
from its structure and environment [36]. Organization struc-
ture consists of nature including type and size (number of
beds), culture, politic, hierarchy, autonomy, planning and
control systems, strategy, management and communication.
Leadership, top management support and medical staff spon-
sorship can also be measured from the organization factors
391
[17,23]. The environment of a healthcare organization can be
analyzed through its nancing source, government, politics,
localization, the type of populations being served, competi-
tion, inter-organizational relationship, population served, and
communication.
3.4. Net benets
A system can benet a single user, a group of users, an
organization or an entire industry. Net Benets capture the
balance of positive and negative impacts on user, which
includes clinicians, managers and IT, staff, system devel-
opers, hospitals or the entire healthcare sector. Individual
impact is the effect of information on the behavior of
the recipient. It is associated with performance as well as
changes in user task (clinical practice) such as job perfor-
mance, change in work activity and improved productivity
[7,30]. Thus, individual Net Benets can be assessed using job
effects, efciency, effectiveness, decision quality, and error
reduction.
Organizational impact is the effect of information on orga-
nizational performance. In the healthcare context, clinical
outcomes can be used as a means of measurement. Exam-
ples of these measures include costs reduction, which is due
to fewer medication errors and adverse drug effect (ADE);
improved efciency in patient care delivery, specically per-
taining to tests and drug orders and increased use of generic
drug brands and number of consultations and length of wait-
ing lists [7,37]. Clinical outcomes are also measured through
two criteria: morbidity (the rate of incidence of a disease) and
mortality (death rate). Apart from these quantitative mea-
sures, clinical impacts can also be assessed qualitatively using
these measures: quality of care, impact on patient care and
communication, such as change in communication style and
facilitation of information access [38].
4. The case of Fundus Imaging System (FIS)
Our research design consists of a case study strategy. The case
study serves dual purposes: (1) to evaluate the adoption fac-
tors of HIS in the context of the phenomena under study; (2) to
validate the proposed HOT-t evaluation framework. The case
study is also undertaken to obtain a comprehensive view and
understanding of the development process of a HIS described
in this section. The case study facilitated the conceptualisa-
tion of HIS adoption tending towards success and failure, in
association to the factors involved and their relationships. We
make use of the case study to answer our research question,
as well as to test the applicability of the proposed framework
in being a useful evaluation tool.
A case study is conducted in the clinical settings of a pri-
mary care organization (PCO) and two of its collaborating
specialist hospitals, all members of the UK National Health
Service (NHS). The GP practice was established in the 1920s
and now serves around 6000 patients who are mainly the
elderly. The practice is actively engaged in medical and nurs-
ing research, training and education. A number of HIS were
already in place, including telemedicine and General Practice
Information Systems (GPIS).
392 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
This study evaluates the development of a digital Fundus
Imaging System (FIS) for diabetic retinopathy. While evaluat-
ing FIS, a GPIS was also studied in order to investigate the
overall adoption factors of HIS of the specic users in the
particular setting. FIS consists of a Fundus camera and dig-
ital imaging software that is used to capture the eyes images
of patients with diabetes. FIS has functions for image cap-
ture, image manipulation, patient data management, image
transfer, viewing and temporary storage. For each captured
eye image, the system assembles the image, patient data,
and timer data into a data le. These data les are trans-
ferred to the GPIS by attaching each image to associated
patient.
The UK government has recommended that all patients
with diabetes have their annual retinal screening using digital
retinal fundoscopy [39]; the FIS is part of this initiative. FIS is
recognized for its importance in preventing blindness among
a large population of diabetic patients and in improving the
control of diabetic care.
Traditionally, retinal fundoscopy for such patients has been
carried out in a hospital or by an optometrist. A GP had to refer
a patient to a hospital to be invited for annual screening. The
recall system in hospital is seen as inefcient in terms of trav-
eling cost and accessibility, particularly to the elderly patient
who represents a signicant population of the PCO. The inef-
ciency of traditional screening motivated the partners at the
PCO to implement a retinal fundoscopy screening within their
practice.
The idea to develop the system was initiated by the senior
partner, who is a GP himself, after a Fundus camera was
purchased. The primary care screening aims to improve the
patient pathway by providing better patient educationGPs
could educate patients during the screening by showing
them their eye images and immediately discuss the clin-
ical implications and subsequently improve their diabetic
control. In addition, the patient pathway could be more
cost effective through reduced time, effort and expense of
travel and double appointments, which allow patients to
have both retinal screening and diabetic health check on the
same day.
The retinal screening of the FIS is done by a GP, Dr. ABC, who
is also the sole user of the system. The screening took place in
a separate examination room equipped with a digital Fundus
camera with a touch screen display, and PCs to transfer patient
eye images into patient records which are featured in the GPIS.
However, the retinal screening is temporarily discontinued, as
the existing camera does not comply with the guideline of the
National Screening Committee, which requires more advance
equipment.
The attempt to implement the FIS was continued by col-
laborating with a leading specialist hospital. A group of eye
specialists who include Consultant Ophthalmologists and a
Reading Centre Advisor was liaised with to obtain expert
advice in terms of selecting alternative patient pathways,
purchasing digital camera and relevant software, and train-
ing for using the system and grading the eye images. During
this study period, major system development and improve-
ment decisions with regards to the technology procurement,
stafng, quality assurance and possible number of patient
pathways are still under negotiation.
5. Methodology
The approach used in this study was that of a subjectivist, case
study strategy employing qualitative methods. A formative
evaluation was undertaken of the adoption of FIS to identify
system problems as they emerged and to improve the system
as it was developed [40]. A subjectivist approach was employed
in order to gain an extensive understanding of the health-
care context surrounding the FIS through detailed, insightful
explanation of the study [34]. Further, qualitative methods
were employed to generate a fuller description of the health-
care setting and its cultural issues and to understand why the
system functioned well or poorly in a particular setting. The
eld study was conducted between April and September 2005
by one of the authors. A number of data collection methods
were employed, including interviews, participant observation
and document/artifact analysis.
A purposeful, snowball sampling method [41] was used in
order to gain in-depth information from key informants about
the development of the FIS. Participants were identied from
the researchers initial contact with individuals known to staff
members. After identifying an initial group of participants, a
network was built by asking these rst participants to suggest
additional participants for interview. In the end 15 participants
were found among clinicians, staff and patients who were
associated with FIS, both from the primary care organization
and from the two specialist hospitals involved in the study. In
particular, the list of participants is as follows (Table 2).
Our research approach to the evaluation of the FIS
consisted of six iterative phases, which included problem
identication, the development of an initial evaluation frame-
work, the selection of a research strategy and methods, system
evaluation, framework validation, and renement of the eval-
uation framework (Fig. 4) [34,36].
All phases were completed. Evaluation problems (issues,
questions and concerns) were identied through a literature
review as well as observations made during an immersion. The
immersion was carried out to set the general context of the
research, as well as to establish rapport with relevant stake-
holders. During this period, in addition to the FIS, the use of a
General Practitioner Information Systems was also observed
in order to obtain an overall view of the general application
and the attitudes of the PCO staff towards Health Informa-
Table 2 List of participants for the case of FIS
Participant Total (N) Initial used in
data analysis
User (a GP) 1 Dr. ABC
Senior partner 1 Dr. MNO
GP 2 Dr. DEF
Nurse 5 STU
Physiotherapist 1 GHI
IT staff 1 PQR
Ophthalmologist Consultant 1 Mr. VWX
Reading Centre Advisor 1 Dr. JKL
Patients 2
Total 15
 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
Fig. 4 Research design (adapted from Friedman and Wyatt [33]; Kaplan [35]).
tion Systems. Initial data collection was gathered during the
immersion. An initial evaluation framework was constructed
based on the ndings from this rst phase. The research
strategy and methods were selected based on the research
problem. The original framework in Part I of this paper has
been employed initially for the FIS case study. After the con-
clusion of the study, the framework was eventually validated
as well as rened in terms of its evaluation measures based on
an appropriate mix of current literature and case study results.
The HOT-t framework was used as a guideline in the
evaluation of the FIS. During system evaluation, participant
observations of daily clinical routines, meetings, discussions
and social events took place in different departments, clin-
ics and nursing homes of the primary care organization and
specialist department of the collaborating specialist hospitals.
A good rapport between the researcher and participants was
established at both formal and informal occasions. Partici-
pants were aware of the role of the researcher in their clinical
settings.
During observations and face-to-face interviews, individu-
als including users, clinicians and IT staff that were involved
with the system were queried about their system use and
patient pathways. General background information on the pri-
mary care organization and its clinical and administrative
staff were gathered to obtain a holistic and in-depth view of
the clinical setting. This view included aspects of manage-
ment, facilities, vision, culture, politics, conict, leadership
and of the staff in terms of their way of working, communi-
cation, relationships and rapport, as well as attitudes towards
computer and IT literacy. Meanwhile, patients were queried
about their perception about the system. Data were collected
on planned occasions as well as spontaneously in a number
of iterative cycles. The data were audio- and hand-recorded,
transcribed into eld notes, and analyzed.
Based on the HOT-t framework, four techniques were used
to analyze the results: coding, analytic memos (such as reec-
tion notes, displays, and concept maps), and contextual and
393
narrative analysis [40]. The eld notes were fully transcribed
on margin-marked paper. The margins were used to note any
reections, themes relevant to HOT-t and statements which
were unclear or needed to be conrmed with the participants.
The data were coded and categorized under similar themes
or concepts of the HOT-t framework and rened through
out a series of analyses. These codes corresponded to each
factor, dimensions and measures described in the HOT-t
framework (see Fig. 3). Further, texts under the same cate-
gory were compared to identify variations and nuances in
meanings. Categories were compared to discover connections
between themes. Concept maps were drawn to understand
the relationship between the number of concepts involved in
the evaluation of FIS. Contextual and narrative analyses were
done based on the themes assigned to the codes and these are
further covered in the next section of this paper.
Two tests were used to establish the quality of this empir-
ical research [42]:
1. Construct validity: establishing correct operational mea-
sures for the concepts being studied. This is concerned with
exposing and reducing subjectivity, by linking data collec-
tion questions and measures to research questions [43]. In
this study, the evaluation measures in the interview ques-
tions were identied based on the proposed framework
and then linked with the research objectives. The results
showed that the interview questions addressed each of the
research questions to a reasonable degree.
2. Reliability: demonstrating that the operation of the study
such as the data collection produced can be repeated
with the same results. This was achieved through detailed
documentation of procedures and appropriate record keep-
ing [43]. Activities during immersion were recorded in a
detailed eldwork log.
Potential bias generally acknowledged in qualitative
research approach was overcome by conducting a reliability
394 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
test and data triangulation. Triangulation was done through
the use of multiple evidences from different sources to con-
rm the same fact or nding [42]. For example, a certain fact
obtained from a GP was conrmed with a different GP as well
as the organization documentation and observation. Feedback
from participants regarding a particular conclusion in the set-
ting studied is also gathered to reduce bias.
6. Discussion of the ndings from the
Fundus Imaging System case
Based on observations that took place during the immer-
sion and pilot study, a number of emerging themes from the
framework, namely human, organizational and technological
factors and net benets have been identied. These issues are
presented in the next sections, which are organized in terms
of the three themes.
6.1. Technology
The general practice where the study took place is some-
what advanced with its use of modern technology and is well
equipped with computing and telecommunications devices.
Currently, all staff members are using a General Practice Infor-
mation System which features electronic patient records as
well as service and management for GPs. A number of the
associate GPs dislike the GPIS. According to Dr. DEF, it is com-
plicated and hard to learn - I hated it! was her literal response.
In comparison to the GPIS, the old system that she used in
her previous workplace was template based and made typ-
ing and data entering much easier. In contrast to the GPIS,
the ease of use of the Fundus Imaging System (FIS) can be
seen immediately as a user friendly, simple to use interface
and straightforward data entry mode. Captured images can
be manipulated, stored and attached to the patient record.
However, the inefciency of FIS is viewed from its incon-
venient storage capacity. The frequent need to delete and
transfer les from a small size memory card (attached to the
camera) to a hard drive when it gets full was seen as tedious
and time consuming. The user, Dr. ABC said it is a pain to keep
on deleting the [images on the] memory card when it is full and
it gets full quickly because it has small storage capacity. She
had to take the card out and upload the patients images to the
PC and linked them with the patient records and saved them
in a specic folder. The contents of the memory card will be
deleted for future use.
In addition to system inefciency, slow response time is also
seen as a disincentive for using the GPIS. For example, physio-
therapist GHI has to recall a number of exercises appropriate
to a patients condition in order to suggest which of them
should now be followed by the patient. Although she could
access their details on the system, she does not use the sys-
tem because from her point of view, it takes too long for her to
print the list of exercises for the patient (2 min). So she requires
patients to remember their own lengthy list of exercises.
With regards to the system under investigation (FIS), there
are some problems related to the less mature technologies
involved in its development. For example, the existing cam-
era had to be replaced, and the use of the current FIS was
temporarily halted because the camera did not comply with
the National Screening Committee guideline, which was pub-
lished after the camera was purchased.
In terms of Information Quality, two aspects were analysed:
relevancy and completeness. The relevancy of the eye images
lies in their usefulness in educating patients to take control
of their diabetes. In addition to educating patients effectively,
it is very important to produce high-quality, accurate images
to avoid erroneous diagnosis. However, images taken with the
existing camera are not as detailed and as accurate compared
to those provided by hospitals. According to the Reading Cen-
tre Advisor JKL, in terms of screening standards, the images
are not good enough to detect eye abnormalities; however, in
practice and for the intended purpose, the images are good
enough.
Service Quality can be observed and determined from the
usage of GPIS. The service providers of the primary care organi-
zation comprise of external vendors and a member of internal
IT staff. The internal IT specialist is trusted by the senior part-
ner of the practice, Dr. MNO who argued, He is difcult to deal
with but he is always right about something. It seems that
the service providers give little empathy to the clinicians that
they are serving. According to the senior partner, the exter-
nal service providers do not care about doctors needs at all;
they just care about their businesses. It is also perceived that
the user also receives too little support from the IT staff when
technical assistance is needed. When she requested technical
support for a particular camera feature, a member of the IT
staff, Mr. PQR responded by saying Theres nothing that can
be done about it, adding He started giving me all the jargon.
6.2. Human
The Fundus Imaging System (for diabetic retinopathy) was
developed for a very specic clinical purposeto have a GP
user educate patients with diabetes in improving their con-
trol of diabetic care. This purpose was viewed as impractical
by the collaborating specialists in terms of cost since it is much
cheaper to employ technicians to screen the patient via the
FIS. This view was not taken into serious consideration until
months after FIS was in place, where a large amount of time
and effort were already allocated by the GP in planning, devel-
oping, training and using the system.
According to nurse STU, although initially most of the users
of the primary care organization have minimal IT skills, their
willingness to use the system was a key contributing factor
that put the system in place. As mentioned earlier, some of the
GPs were not supportive of the system because, in their view,
it was difcult to use. The remaining staff believe that the use
of GPIS and other HIS had assisted them in performing their
jobs better; tasks were completed faster and communications
between staff and doctors were improved.
The user of FIS on the other hand has acknowledged her
limited IT skills. However, the user was motivated to use the
system with the assistance provided by a staff member at the
outset of FIS implementation. The barrier to using the sys-
tem can be seen from the lack of familiarity of the user in
using basic le organization functions, such as copying les,
and in performing time consuming and tedious tasks such as
deleting and transferring les. This process resulted in user
 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
dissatisfaction because in her view, it is taking a lot of her lim-
ited time. Another example of unfamiliarity was seen when a
pop up window was displayed and the user said, I hate these
messages, I dont understand what they are talking about.
Despite her busy schedule, the user attended a series
of training organized outside the practice. She needs to be
trained as an accredited screener to be able and eligible to
examine the images and inform patients of the initial screen-
ing results.
6.3. Organization
The senior partner played a major role in shaping the orga-
nizational culture and establishing the use of technology in
the organization. He was the leader with a long-term vision
and strategy to keep abreast with technology advancement.
He also formed a number of collaborations and partnerships
with other primary and secondary care organizations, and
universities to exploit new technologies in medicine. Organi-
zational readiness has been established in several occasions.
For instance, research meeting is held on a regular basis to
discuss existing and new technology adoption and clinical
concern pertinent to technology, including the FIS. Informal
discussions during lunchtime between various staff mem-
bers, which includes the senior partner, associate GPs and
nurses, have build a strong rapport and good communication
between them. Everyone is respected and treated equally
regardless of their position. Everyone is also encouraged to
work as a team in performing their daily tasks. As nurse STU
put it, This place is different from other places. The senior
partner really made a difference by creating a teamwork
spirit. The teamwork spirit among the staff members can
be seen in the decision making process. They tend to seek
advice from their colleagues who have expertise in an area
they themselves do not feel very condent with. For instance,
a GP seek advice from the nurses regarding the critical case
of a home care resident and a physiotherapist consulted a GP
to check on the effect of a drugdrug interaction.
In addition to the rather advanced computing facilities, the
organization has an environment that is conducive to work
in; for example, manual and electronic records are kept in
an orderly manner. According to the user, motivation, person-
ality, and encouragement are the catalytic factors that made
possible the adoption and use of the system. After the senior
partner assigned the role of FIS user to her, he continued to
encourage the GP to change her perception about her role as
a sole service provider and convincing her that she is capa-
ble of using the system. The initiative to use the system was
also encouraged by other staff members. Without these fac-
tors, Dr. ABC pointed out that . . .. I would not take the role.
It was not easy for me to accept that role initially, it was a big
change. I was very reluctant because I have a different pic-
ture of a GPs role. In her view, a GP only comes in, sees 30
patients, writes the letters, checks the results, and sends them
to hospital if necessary. She believed that the senior partners
leadership has changed her perception, which made her real-
ized that she can develop professionally and change. Dr. ABC
stated that change can be done in two ways: nurtures a per-
son to make him feel that he can do it or it puts him down; for
example, you start to criticise your colleague by saying you are
395
not good enough, you should do more. Usually encouragement
will work. Its like the parenting role. She used this example
to refer to the initiation of the FIS in this general practice.
The uptake of the FIS is also attributable to medical spon-
sorship in the practice. The user recalled the time when the
camera was not being used after it was purchased. The senior
partner kept on teasing the user, using his gentle persuasive
skill, prompting her to use the system by saying something
like Why dont you have a play with it? After a few reluctant
initial attempts, the user nally took pictures of a nurses eyes,
which was the starting point of diabetic retinopathy screening
in this practice.
Three types of communication were observed in this
clinical setting: patientphysician, physicianphysician and
staffphysician. In terms of working style, although each
GP and nurse consults their patients in different ways,
a commonality in their communication is made obvious:
patientphysician relationships in terms of interpersonal
communication and eye contact are essential during consul-
tations.
The communication between the user and eye specialist
is limited by their busy schedule. On several occasions, a joint
meeting was postponed by a few months and this had affected
the speed of FIS implementation. This lack of communication
has resulted in a difference of understanding about project
implementation. In addition, the conict of interest between
the project leader, the user and the eye specialist team in terms
of technology procurement, training cost and the role of user
has also delayed the uptake of FIS. After months of meeting
and training, the PCO has considered assigning the role of cur-
rent user, the GP, to an external screener since the time spent
by the GP for training, screening and grading would be more
expensive than hiring the screener. This situation, however,
has been envisaged by one of the Consultant Ophthalmolo-
gists earlier and he had informed the user but it was not taken
seriously.
Meanwhile, although the communication between staff is
good, there is a common lack of communication between IT
staff and clinicians, which results in conicts. This is an indi-
cation of a typical problem of communication gap between
technical staff and users. The IT staff also uses jargon that
makes the communication more difcult and confusing to the
doctors. The senior partner commented that one of the IT staff
always leaves us in a mess. He does not tell us exactly what
is going on and when the service provider meets the staff, we
look like idiots! He also said that, All service providers are
rubbish!
Focusing on the external environment of this organization,
there are some problems with the communication between
this primary care and secondary care organization, which
include miscommunication and error in reporting. For exam-
ple, a patient received the wrong medical reportthe pain was
reported to be in the left shoulder instead of the right one. In
another instance, physiotherapist GHI stated, The communi-
cation between hospital and the primary care organization is
dreadful in terms of patient medical history. The hospital did
not give any referral letter or information that explains the
previous medical diagnoses and it is difcult and time con-
suming to get hold of these documents. I have to guess to
identify the causes of the patients pain.
396 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
6.4. Net benets
In terms of the net benets of the new system, it seems that
most of the practitioners have acknowledged certain effects
on patient care and clinical practice. According to Dr. ABC, the
benets of primary care screening can be assessed through: (1)
patients control in their diabetic care, (2) equivalency or better
treatment in primary care than the traditional screening in
terms of speedy treatment and (3) low error rate and patients
satisfaction.
It has been recognized that the FIS has the potential to
improve the delivery of patient care by providing better patient
education. For example, Dr. ABC said to one of the patients
that these are your blood vessels, they are so tiny; if theres
any problem with them, it will cause major problems with
your vision. The cause of the problem is poor diabetic con-
trol. She believes that getting patients see their eye pictures
may enable them realize that their diabetic control is not
good and they are at a risk of losing their vision. Accord-
ing to Dr. ABC, one way to nd out whether the diabetic
control is improved by this sort of patient education is by
a blood test in the diabetic clinic. The GP quoted an exam-
ple of an obese patient who took a lot of liberty with his
diet and had very bad diabetic control. The GP changed the
patients attitude by showing to the patient pictures of his
eyes and discussing possible problems that may occur in rela-
tion to his diet. As a result, the patient came back, having
lost a couple of kilos and told the GP that she effectively
made him change the whole way he thought about his dia-
betes condition. In addition, the senior partner, Dr. MNO
said, By her (the GPs) involvement in taking pictures of
the patients eyes, she took up a marvellous opportunity to
change her patients attitude disorder. That is a fantastic way
of inuencing patients behaviour about managing their own
health.
The partners of this GP practice envisaged that the imple-
mentation of FIS will benet patient by saving their time
and cost of travelling, thus enable them to be screened ear-
lier. This shorter patient pathway will also reduce possible
error rates as data are stored and access directly within the
practice.
On the other hand, some negative effects have also been
recognized. The use of FIS has increased the GPs work vol-
ume. She has to spend additional hours in the retinal clinic
capturing the eye images, as well as storing and trans-
ferring them to the patient notes. In addition, she has to
spend a few hours per month for regular training sessions.
As a result, there have been certain occasions when, due
to accessibility problems in the FIS, she was late for her
surgeries.
6.5. Fit between human, organization and technology
Based on these three factors, their t with each other has
been recognized. The uptake of FIS was contributed by the
users strong acceptance and personality to learn using the
system (t between human and technology) and the technical
support provided by a staff member that acted as a system
champion. However, a lack of internal t can also be seen
between technology and human: storage inconvenience and
time and effort required from the busy GP user, as well as
poor technical support and user need for assistance. Moreover,
the mismatch between clinical processes with technology was
illustrated by the GPs busy schedule with time required for
training. External t of the practice is achieved by formulat-
ing its strategy according to current IT trends and advances.
However, the lack of external t in FIS was shown in the
non-compliance of current FIS equipment with the National
guideline, which caused the system to be discontinued
temporarily.
7. Further discussion and conclusions
The case of FIS has demonstrated the importance of having
the right user attitude and knowledge in order to be able to
use the system effectively and efciently. Arguably, choos-
ing the right people to ll the user role is more important
than possessing the required skills, as skills can be acquired
later. This can be seen from the use of FIS, which was pri-
marily driven by the willingness of the GP to learn and use
it after being persuaded by the senior partner and despite
her limited computing literacy. Knowledge can be acquired
through appropriate training; however, physicians have a very
busy schedule. Thus, alternatives such as having replacement
physicians can be taken to enable the user to attend this
training.
The practice is distinguished by being a research practice
and having a good leadership. Such organizational culture and
leadership has created awareness of technical advancements
among the staff members as well as expedited the adoption
of HIS, as shown by the uptake of telemedicine, GPIS and FIS.
The alignment of the organizational strategy with IT and the
existence of up-to-date computing infrastructure have also
facilitated the implementation of HIS in the practice.
The barriers to system use are contributed by the user per-
ception, ease of use, response time and clinical process. The
user was initially reluctant to use the system due to her per-
ception of her role solely as a service provider. GPs have such a
heavy daily workload. This explains why response time is cru-
cial in the adoption decision. The same goes with the training
issue. It is quite impossible for a GP in the PCO to attend a
series of time consuming training sessions while still having
to attend her regular surgery sessions.
Meanwhile, the communication gap between clinicians
and the IT staff is obviously caused by the knowledge gap as
well as individual characteristics towards being more sensitive
to different stakeholders needs.
Communications between all staff at all levels are crucial
to ensure that the purposes and benets of an HIS are under-
stood since communication problem can be costly. One way to
achieve effective communication is through leadership. In this
case study, leadership as well as top management support has
proven to be an important starting point and has major inu-
ence in the realization of HIS. In addition to leadership and
top management support, medical sponsorship also played a
signicant role in changing user perception and encouraging
system use.
Furthermore, external factors such as government policy
can largely affect the viability of HIS. For example, although
 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
FIS had already been in place and operational, its use had to be
halted due to the fact that the camera did not comply with the
latest National Guidelines. Thus, it is very important to plan
and develop a system and keep it up-to-date in accordance
with the latest policy.
FIS can potentially improve healthcare delivery by provid-
ing better patient education. Other potential benets of FIS
can be identied in terms of patient control of their diabetic
care and the provision of equivalent or better treatment in
primary care than secondary care organizations in terms of
speedy treatment and patient satisfaction.
A number of challenges have been encountered while
applying the HOT-t framework in data analysis. First, there
are a number of data that share the same evaluation mea-
sures; a careful consideration has to be taken in selecting the
most appropriate measures for each data. For example, com-
munication can be classied under organizational structure
and environment; problems in using the system can be cat-
egorized under of technology factors (ease of use) or human
factors (system use). Otherwise, a repetition of the same data
in different category can be used where necessary. Second, the
classication of data analysis according to the three evalua-
tion factors has affected the ow of the narrative approach of
presenting the data and can result in confusion to the reader.
In order to validate its usefulness, the proposed framework
was tested in a real clinical setting. The system put on the test
was a Fundus Imaging System in a NHS primary care orga-
nization. The description of the case study and its ndings
offer a strong indication of the applicability of the framework
to HIS evaluation. A number of critical factors to the adop-
tion of FIS have been identied; factors that had inuenced
the adoption negatively include: system usefulness, response
time, technical support, empathy of service quality, user per-
ception and user skills. Meanwhile, factors contributing to
the positive adoption of FIS include: information relevancy,
user attitude, leadership, medical sponsorship, organizational
readiness, clinical process and external communication with
the inter-organizational system (the eye specialist). The align-
ment of IT and organizational strategy has led to the initiation
of a number of systems, including FIS. The strong willingness
of the user to change her perception and clinical practices
resulted in the uptake of FIS. However, the adoption of FIS
was disrupted with the incompatibility of the system with
the National Guidelines, as well as a lack of technical support
and limited communication between the technical staff and
the collaborating partners of the specialists hospital. We thus
conclude that human, organizational and technological fac-
tors and the t between them are essential in the realization
of FIS.
Although our case study focused on a specic setting,
the proposed evaluation framework is potentially useful to
researchers and practitioners for conducting thorough eval-
uation studies of other HIS or IT applications in healthcare
settings. As proposed here, the framework can and should be
applied in a exible way, taking into account different con-
texts and purposes, stakeholders point of views, phases in
system development life cycle, and evaluation methods. The
framework is not the solution to any problem; it is a struc-
tured debating tool that stakeholders can access in order to
know their own system health better.
397
Summary points
What was known before the study:
There is a large number HIS evaluation frameworks
looking at different aspects of these systems.
The existing evaluation methods do not provide
explicit evaluation categories.
More work on human and organizational issues is
called for as most existing evaluation studies of HIS
focus on technical issues or clinical processes which do
not explain why HIS work well or poorly with a specic
user in a specic setting.
What the study has added to the knowledge:
Previous work on the evaluation of Information Sys-
tems is reviewed.
An evaluation framework for HIS, which incorpo-
rates the concept of t between human, organization
and technology (HOT-t), is proposed using a multi-
disciplinary approach.
The application of the proposed evaluation framework
is demonstrated in a real-life, practical context where
formal evaluation methods have not been or could not
have been used.
Insights shed from the ndings of the case study that
can be used to inform decision making.
Acknowledgements
We gratefully acknowledge the funding received from both the
Public Service Department of Malaysia and from the Universiti
Kebangsaan Malaysia (National University of Malaysia) that
helped sponsor this study.
references
[1] D.F. Sittig, B.L. Hazlehurst, T. Palen, J. Hsu, H. Jimson, M.C.
Hornbrook, A clinical information system research agenda
for Kaiser Permanente, Permanente J. 6 (3) (2002).
[2] B. Kaplan, Evaluating informatics applicationsclinical
decision support systems literature review, Int. J. Med. Inf.
64 (1) (2001) 1537.
[3] B. Kaplan, Evaluating informatics applications-some
alternative approaches: theory, social interactionism, and
call for methodological pluralism, Int. J. Med. Inf. 64 (1)
(2001) 3956.
[4] B. Kaplan, N. Shaw, People, organizational and social issues:
evaluation as an exemplar, IMIA 2002 Yearbook (2002)
91102.
[5] B. Kaplan, N.T. Shaw, Future directions in evaluation
research: people, organizational, and social issues, Methods
Inf. Med. 43 (3) (2004) 215231.
[6] J. Brender, C. Nohr, P. McNair, Research needs and priorities
in health informatics, Int. J. Med. Inf. 5859 (2000) 257289.
[7] E. Coiera, Guide to Health Informatics, Hodder Arnold, 2003.
398 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386398
[8] M.M. Yusof, R.J. Paul, L. Stergioulas, Health Information
Systems Evaluation: A focus on Clinical Decision Supports
System. Proceedings of the XIX International Congress of
the European Federation for Medical Informatics (MIE2005),
Geneva, Switzerland, 2005, pp. 855860.
[9] M.M. Yusof, R.J. Paul, L.K. Stergioulas, Towards a Framework
for Health Information Systems Evaluation. Proceedings of
the 39th Hawaii International Conferences on System
Sciences, Kauai, Hawaii, USA, 2006.
[10] W.H. DeLone, E.R. McLean, Information systems success: the
quest for the dependent variable, Inf. Syst. Res. 3 (1) (1992)
6095.
[11] W.H. DeLone, E.R. McLean, Measuring e-commerce success:
applying the DeLone & McLean Information Systems
Success Model, Int. J. Electron. Commerce 9 (1) (2004) 3147.
[12] M.S. Scott Morton, The Corporation of the 1990s, Oxford
University Press, New York, 1991.
[13] M.M. Yusof, A. Papazafeiropoulou, R.J. Paul, L.K. Stergioulas,
Investigating evaluation frameworks for health information
systems, Int. J. Med. Inf. (2007).
[14] M.J. Van der Meijden, H.J. Tange, J. Troost, A. Hasman,
Determinants of success of inpatient clinical information
systems: a literature review, J. Am. Med. Inform. Assoc. 10 (3)
(2003) 235243.
[15] L. Willcocks, Managing technology evaluationtechniques
and processes, in: R.D. Galliers, B.S.H. Baker (Eds.), Strategic
Information Management: Challenges and Strategies in
Managing Information Systems, Oxford, 1994, pp. 365381.
[16] K.A. Kuhn, D.A. Giuse, J.L. Talmon, The Heidelberg
Conference: setting an agenda for the IMIA working Group
on Health Information Systems, Int. J. Med. Inf. 69 (23)
(2003) 7782.
[17] J.G. Anderson, Clearing the way for physicians use of
clinical information systems, Commun. ACM 40 (8) (1997)
8390.
[18] K. Bottles, Critical choices face healthcare in how to use
information technology, Medscape Gen. Med. 1 (1) (1999).
[19] N.M. Lorenzi, R.T. Riley, Organizational ISSUES = change, Int.
J. Med. Inf. 69 (23) (2003) 197203.
[20] J. Brender, E. Ammenwerth, P.N. Nyakanen, J. Talmon,
Factors inuencing success and failure of Health
Information System a pilot delphi study, Methods Inf. Med.
45 (1) (2006) 125136.
[21] J. Aarts, V. Peel, G. Wright, Organizational issues in health
informatics: a model approach, Int. J. Med. Inf. 52 (13) (1998)
235242.
[22] M. Berg, Patient care information systems and health care
work: a sociotechnical approach, Int. J. Med. Inf. 55 (2) (1999)
87101.
[23] F.C.G. Southon, C. Sauer, C.G. Grant, Information Technology
in Complex Health Services: organizational impediments to
successful technology transfer and diffusion, J. Am. Med.
Inform. Assoc. 4 (2) (1997) 112124.
[24] R. Heeks, D. Mundy, A. Salazar, Why Health Care Information
Systems Succeed or Fail, Institute for Development Policy
and Management, University of Manchester, 1999.
[25] J. Luftman, Assessing business-IT alignment maturity,
Commun. AIS 4 (14) (2000) 150.
[26] W.H. DeLone, E.R. McLean, Information systems success
revisited, in: Proceedings of the 35th Hawaii International
Conference on System Sciences (HICCS), Big Island, HI, USA,
2002.
[27] F.C.G. Southon, C. Sauer, K. Dampney, Lessons from a failed
information systems initiative: issues for complex
organizations, Int. J. Med. Inf. 55 (1) (1999) 3346.
[28] N.K. Roderer, Outcome measures in clinical information
systems evaluation, in: Medinfo. 2004, San Francisco, CA,
USA, 2004, pp. 10961100.
[29] A.P. Stoop, M. Berg, Integrating quantitative and qualitative
methods in patient care information systems evaluation:
guidance for the organizational decision maker, Methods
Inf. Med. 42 (4) (2004) 458462.
[30] J.G. Anderson, C.E. Aydin, Overview: theoretical perspectives
and methodologies for the evaluation of health care
information systems, in: J.G. Anderson, C.E. Aydin, S.J. Jay
(Eds.), Evaluating Health Care Information Systems; Methods
and Applications, Sage, Thousand Oaks, CA, 1994, pp. 529.
[31] B. Doebbeling, Information Technology and Primary Care at
VA: Interdisciplinary Partnership Opportunities for
Providers, Managers, and Researchers, Health Service
Research & Development Series, 2003.
[32] J. Smith, Health Management Information Systems: A
Handbook for Decision Makers, Open University Press,
Buckingham, 2000.
[33] T. Lippeveld, Routine Health Information Systems: The Glue
of a Unied Health System, Workshop on Issues and
Innovation in Routine Health Information in Developing
Countries, Bolger Centre for Leadership Development
Potomac, Maryland, 2001.
[34] C.P. Friedman, J.C. Wyatt, Evaluation Methods in Medical
Informatics, Springer-Verlag, New York, 1997.
[35] J.J. Jiang, W.A. Muhanna, G. Klenin, User resistance and
strategies for promoting acceptance across system types,
Inf. Manage. 37 (1) (2000) 2536.
[36] B. Kaplan, Organizational evaluation of medical information
resources, in: C.P. Friedman, J.C. Wyatt (Eds.), Evaluation
Methods in Medical Informatics, Springer-Verlag, New York,
1997, pp. 255280.
[37] A. vant Riet, M. Berg, F. Hidemma, K. Sol, Meeting patients
needs with patient information systems: potential benets
of qualitative research methods, Int. J. Med. Inf. 64 (1) (2001)
114.
[38] N.T. Shaw, CHEATS: a generic information communication
technology (ICT) evaluation framework, Comput. Biol. Med.
32 (3) (2002) 209220.
[39] The National Screening Committee (NSC), Transfer and
Management of Patient Information in Diabetic Retinopathy
Screening Programmes Version 2.0, July 2005. Retrieved 2
August, 2005 from: http://www.nscretinopathy.org.uk (2004).
[40] B. Kaplan, J. Maxwell, Qualitative research methods for
evaluating computer information systems, in: C.E. Aydin,
J.G. Anderson, S.J. Jay, S.J. (Ed.), Evaluating Health Care
Information Systems; Methods and Applications, Sage,
Thousand Oaks, 1994, pp. 4567.
[41] M.Q. Patton, Qualitative Research & Evaluation Methods,
Sage Publications, Thousand Oaks, 2002.
[42] R.K. Yin, Case Study Research: Design and Methods, Sage
Publications, Thousand Oaks, 2003.
[43] J. Rowley, Using case studies in research, Manage. Res. News
25 (1) (2002) 1627.