S72: Evaluation of Libiaiy anu Infoimation Seivices anu Systems
Teim Papei }uly 24, 2u1S
Bigital Libiaiy Evaluation As technology auvances, moie infoimation iesouices aie being uigitizeu anu aie incieasingly boin uigital, anu uigital libiaiies aie being cieateu to house these iesouices. These uigital libiaiies can be inuepenuent of oi an extension of tiauitional libiaiies. Bespite numeious challenges, ieseaicheis aie making effoits to select ciiteiia that can be useu foi evaluation of uigital libiaiies as well as methouologies foi collecting anu analyzing that uata. While tiauitional libiaiy evaluation techniques can be applieu to uigital libiaiies, they must often be auapteu to accommouate the technological featuies of a uigital libiaiy. Cuiient evaluation ciiteiia anu methous incoipoiate techniques fiom tiauitional libiaiies, systems, infoimation ietiieval, anu human-computei inteiaction in an attempt to covei all aspects of evaluation, but fail to incluue ciiteiia foi aspects unique to uigital libiaiies. 0sei satisfaction, an impoitant consiueiation foi all types of libiaiies, piesents valuable evaluation ciiteiia foi uigital libiaiies. As a iesult, usability has become a fiequently useu evaluation measuie as it maiiies effective use of a system anu usei satisfaction. Bue to numeious challenges, a stanuaiu evaluation mouel foi uigital libiaiies has yet to be cieateu. Pait of the challenge of ueveloping an evaluation fiamewoik is the lack of a uefinition foi a uigital libiaiy. Befinitions of a uigital libiaiy vaiy baseu on uiffeient people's peispectives. "To libiaiians, a uigital libiaiy is anothei foim of a physical libiaiy; to computei scientists, a uigital libiaiy is a uistiibuteu text-baseu infoimation system oi a netwoikeu multimeuia infoimation system; to enu useis, uigital libiaiies aie similai to the woilu wiue web (www) with impiovements in peifoimance, oiganization, functionality, anu usability" (Xie 1S46). Bigital libiaiies aie laigely iecognizeu as an oiganizeu collection of uigital woiks anu the means to access them, but vaiying peiceptions incluue auuitional iuentifieis anu bounuaiies aie uifficult to uefine. These uiffeient uefinitions iesult in uiffeiing piioiities anu theiefoie, uiffeiing appioaches anu evaluation ciiteiia. Saiacevic (2uuS) explains "uigital libiaiies aie complex social, institutional anu technical systems. No evaluation can possibly auuiess all of these aspects togethei. Thus, uiffeient appioaches have been useu foi uiffeient evaluation goals" (S). Be iuentifies the most commonly useu appioaches cuiiently useu foi evaluation, systems-centeieu appioach, human-centeieu appioach, anu usability-centeieu appioach, anu seveial less fiequently stuuieu appioaches. Cuiient uigital libiaiy stuuies evaluate iepiesentations of infoimation, tools, seivices, usei behavioi, anu pievious evaluation methous uepenuing on the peispective of the ieseaichei anu theii chosen appioach. Befoie ueciuing what to evaluate oi how to uo so, the question of why to evaluate, oi the objectives, must be answeieu. Saiacevic (2uuu) explains, "The answei to the fiist question as to why to evaluate shoulu seive as a base foi selection of an appiopiiate appioach" (SS8). Beteimining the goals of the evaluation piocess can be a uifficult piocess uue to fiequent oveilap, such as whethei a uigital libiaiy is meeting a usei's neeus anu how accessible it is. Saiacevic (2uuu) uiviues objectives into seven geneial classes: social, institutional, inuiviuual, inteiface, engineeiing, piocessing, anu content (S6S-4). Ciiteiia can then be ueteimineu foi each objective. "While the objectives may be explicitly stateu oi implicitly ueiiveu oi assumeu, they have to be ieflecteu in an evaluation. Evaluation is not one fixeu thing. Foi the same system, evaluation can be uone on uiffeient levels, in ielation to uiffeient choices of objectives, using a vaiiety of methous, anu it can be oiienteu towaiu uiffeient goals anu auuiences" (Saiacevic & Covi, S). They iecommenu that an evaluation specifies constiuct, context, ciiteiia, measuies, anu methouology. Evaluation must incluue cleai selections anu uecisions in each categoiy befoie piogiessing to the next. Evaluation appioaches anu metiics foi uigital libiaiies aie cuiiently being ueiiveu fiom ciiteiia useu in tiauitional libiaiies. Bigital libiaiies aie often seen as ielateu to oi extensions of physical libiaiies because they peifoim many similai functions anu seivices. Existing uata collection methous anu metiics fiom tiauitional libiaiies such as ciiculation, gate counts, collection size, pation satisfaction anu cost pei use can be applieu to uigital libiaiies. 0thei impoitant vaiiables incluue "accessibility, availability, seaichability, anu usability" (Xie 1S48). Saiacevic (2uuu) outlines the following tiauitional libiaiy ciiteiia anu suggests they can anu shoulu be applieu to uigital libiaiies: collection, infoimation, use, anu stanuaius (S66). 0sage is often useu as a ciiteiion foi evaluation as stuuying patteins pioviues helpful uata foi making uecisions, such as which mateiials aie fiequently iequesteu anu cost pei use. As with tiauitional libiaiies, efficiency anu financial metiics aie valuable in the assessment of uigital libiaiies. Bowevei, uigital libiaiies uiffei fiom tiauitional libiaiies in many aspects. "Bigital libiaiies maiiy the missions, techniques, anu cultuies of physical libiaiies with the capabilities anu cultuies of computing anu telecommunications. Evaluating uigital libiaiies is a bit like juuging how successful is a maiiiage. Nuch uepenus on how successful the paitneis aie as inuiviuuals as well as the emeigent conuitions maue possible by the union" (Naichionini Su4). Evaluation is best appioacheu by giouping the ciiteiia into similai categoiies such as tiauitional libiaiy, technology, anu the oveilap between the two. Fuhi et al. (2uu1) appioach uigital libiaiy evaluation by uiviuing it into foui uimensions: useis, usage, collection, anu technology, which seive as the piinciple ieseaich aieas foi assessment. While many functions can be evaluateu similaily to tiauitional libiaiies, technology is one of the ciucial uiffeiences between the evaluation of tiauitional libiaiies anu uigital ones. Technology iequiies that ceitain functions of a uigital libiaiy, such as access, be evaluateu uiffeiently than its physical counteipait. 0sei stuuies must incluue auuitional evaluation ciiteiia, as ability to navigate systems anu othei technical capabilities have an effect on theii use of anu satisfaction with uigital libiaiies. The technology-ielateu aspects of uigital libiaiies iequiie auuitional ciiteiia when piepaiing to evaluate them. As tiauitional libiaiy ciiteiia aie mouifieu foi uigital libiaiies, tiauitional systems ciiteiia can be applieu to uigital libiaiies. Fuhi et al. (2uu1) subuiviue the technology uimension into foui aieas, usei technology, infoimation access, systems stiuctuie, anu uocument technology, each with its own suggesteu topics foi evaluation. Tiauitional IR ciiteiia foi the evaluation of systems anu tiauitional human-computei inteiaction ciiteiia can be incoipoiateu in the assessment of uigital libiaiies. A systems appioach offeis a vaiiety of ciiteiia foi uigital libiaiy evaluation anu is one of the most commonly useu appioaches. "A human-maue system, such as a uigital libiaiy, has an auueu aspect: it has ceitain objective(s). The elements, oi the components, inteiact to peifoim ceitain functions, oi piocesses, to achieve given objectives" (Saiacevic & Covi S). They suggest that with a systems appioach the uigital libiaiy system's peifoimance can be evaluateu in teims of effectiveness, efficiency, anu the combineu cost-effectiveness. Netiics useu to evaluate infoimation ietiieval aie often incluueu in this type of appioach. "Aspects of technology peifoimance, piocessalgoiithm peifoimance, anu oveiall system such as iesponse time, eiioi iates, maintainability, anu inteiopeiability have been useu in vaiious stuuies (Saiacevic 2uuS 7). Netiics such as iecall anu piecision aie typically useu, anu a stanuaiu set of pioceuuies that incluues test questions anu pooleu ielevance juugments aie useu to ensuie compaiability acioss systems" (Naichionini S12). Biffeientiating between evaluating infoimation ietiieval anu evaluating a uigital libiaiy can piesent a challenge. Bepenuing on the peispective of the ieseaichei, the chosen ciiteiia foi a systems-baseu appioach to evaluation may not pioviue enough infoimation foi stakeholueis to make uecisions iegaiuing the uigital libiaiy as a whole, but this uata enables uevelopeis to make changes to the uigital libiaiy that will impiove access anu the usei expeiience. With uigital libiaiies still in uevelopment stages, theie aie many unansweieu questions about theii use anu the usei. As with a tiauitional libiaiy, knowing the usei is essential in satisfying theii infoimation neeus. A human-centeieu appioach to uigital libiaiy evaluation consiueis the ways useis aie using uigital libiaiies anu the puipose of theii seaich. "The uses anu the usei aie intimately connecteu with the foui basic questions that can be askeu about any maiket," namely who, what, wheie, anu why (Fuhi 2uu1). 0thei questions incluue "whethei useis use them, how they use them, anu what facilitates anu hinueis theii access of infoimation in these uigital libiaiies" (Xie 1S47). A human- centeieu appioach must account foi a wiue vaiiety of useis anu seives many uiffeient libiaiy stakeholueis. This type of appioach "may taiget uiffeient goals ianging fiom incieaseu leaining anu impioveu ieseaich to impioveu uissemination to bottom line piofits" (Saiacevic & Covi 6). Bemogiaphic infoimation coupleu with usage uata can be inteipieteu to pioviue extiemely valuable qualitative uata that can be useu to infoim uecisions about the uigital libiaiy in teims of system, collection, anu seivices as well as its ielation to a tiauitional libiaiy. Because the usei's peiception of the uigital libiaiy is consiueiable factoi in the evaluation piocess, ieseaicheis stuuy useis anu usei behavioi to ueteimine what qualities of a uigital libiaiy aie impoitant to them. Foi example, }eng's 2uuS stuuy soliciteu the usei's peiception of ciiteiia such as "ease of use, oiganization of infoimation, teiminology, attiactiveness, anu mistake iecoveiy" (Xie 1SSu). Liteiatuie on uigital libiaiy usei behavioi also coveis topics such as "usage patteins of seivice logs, peiception of quality in uigital libiaiies by euucatois, woik patteins of expeits foi uevelopment of tools foi a uigital libiaiy, usei piefeiences in seaiching full text uatabases, infoimation seeking in hypeimeuia uigital libiaiies" (Saiacevic 2uuS, 4). 0seis place the most emphasis on usability (how easy it is to use) anu usefulness (uoes the libiaiy have the infoimation neeueu) of a uigital libiaiy. Few stuuies allow useis to contiibute theii own evaluation ciiteiia. Selecting ciiteiia foi the evaluation of uigital libiaiies often incluues vaiiations of the usei anu theii inteiaction with the technology. Theiefoie, a piominent appioach to evaluation is the usability-centeieu appioach because it focuses on the ease with which a usei can satisfy theii infoimation neeus with the uigital libiaiy. "0sability-centeieu appioach: Involves assessment of uiffeient featuies, paiticulaily in iespect to poitals, by useis. It is a biiuge between systems- anu human-centeieu appioaches" (Saiacevic 2uuS S). Leainability is also incluueu in many usability stuuies anu is gioupeu with ease of use. "0sability testing is anothei type of evaluation that focuses on the effects obtaineu when inuiviuuals apply an infoimation piocessing system to accomplish tasks. 0sability testing auopts metiics such as time to completion, accuiacy, satisfaction, anu eiiois" (Naichionini S12). Cuiiently, each ieseaich team selects the usability ciiteiia that they feel woulu be most impoitant to evaluate; consequently, usability ieseaich measuies a wiue iange of factois. Besign of the inteiface plays a majoi iole in usability. "0sability testing often focuses on assessing the effectiveness, efficiency anuoi satisfaction of the usei expeiience with a paiticulai inteiface" (Chouuhuiy 1-2). Bespite the quality of content in the uigital libiaiy, if the usei cannot easily navigate it, they will not use it. Besign elements like "queiy foim, fielus, instiuctions, iesults uisplays, anu foimats of images anu texts" aie populai components foi usability stuuies that focus on uesign (Xie 1S49). The uiiect inteiaction between the usei anu the uigital libiaiy is also stuuieu. The key attiibutes of inteiactivities aie "iecipiocity, feeuback, immeuiacy, ielevancy, synchionicity, choice, immeision, play, flow, multiuimensionality, anu contiol" (Xie 1SSu). Reseaich into this aiea laigely concluues that these piopeities aie inteiuepenuent when evaluating the uigital libiaiy in teims of usability. Anothei consiueiation in uigital libiaiy evaluation is whethei it satisfies the usei's infoimation neeu. Evaluation of the collection focuses on the quality of the collection examining factois like "completeness, authoiity, accuiacy, ieliable anu ielevant infoimation" (Xie 1SS9). Regaiuless of the type of libiaiy, its value lies in its ability to satisfy the usei's infoimation neeu. "If the ietiieval anu usability thiough inteiface uesign is fantastic but the content is pooi, all of the time spent cieating the uigital libiaiy is wasteu" (Xie 1SS9). Befining the scope of a uigital libiaiy often helps a usei ueteimine whethei the collection will meet theii neeus, cieating iealistic expectations of its content. "The scope of a uigital libiaiy uoes not guaiantee the completeness of the collections; insteau it is noimally useu to juuge the completeness of the collections of the uigital libiaiy" (Xie 1S6S). Nany human-centeieu appioaches to evaluation question how well the uigital libiaiy satisfies the infoimation neeus of a usei oi its community. Foi uigital libiaiies, satisfying an infoimation neeu maiiies the quality of the collection with the ability to access it. 0sei satisfaction with a uigital libiaiy is inteiielateu with theii expeiience using uigital libiaiies anu as a iesult, theii expectations foi use. Xie (2uu7) finus that useis' "piefeience, expeiience, anu knowleuge stiuctuie co-ueteimine theii evaluation of uigital libiaiies" (Xie 1S71). "0sei satisfaction is essential foi the success of a uigital libiaiy. Nultiple channels aie neeueu foi uevelopeis of uigital libiaiies to solicit feeuback fiom theii taigeteu useis" (Xie 1S6S). Because theie is cuiiently no stanuaiu appioach oi ciiteiia, the methouology cuiiently employeu vaiies wiuely. Nethous incluue stiuctuieu inteiviews, focus gioups, obseivations, usage analysis, anu many moie. Chouuhuiy, Bobbs, anu Loiie iecommenu a multi-attiibute, stateu-piefeience mouel foi evaluating uigital libiaiy seivices, which suiveys useis anu asks them to state theii piefeiieu seivices oi featuies fiom a list of choices. "The philosophy unueilying these techniques is that useis can best assess the appiopiiate tiaueoffs |iegaiuing iesouice allocationj to maximize theii utility" (Chouuhuiy S). By suiveying useis, libiaiies have auuitional infoimation about piefeiieu seivices foi uecisions anu can utilize that infoimation to make cost effective uecisions about seivices that will best satisfy useis. Theie is some concein that uigital libiaiy evaluation is ielying too heavily on auapting tiauitional ciiteiia fiom libiaiies anu systems iathei than cieating new ciiteiia that evaluates aspects unique to uigital libiaiies. Zhang explains "most of the ciiteiia useu aie meiely boiioweu fiom the uomains of tiauitional libiaiy anu infoimation ietiieval system. Theie lacks BL-specific evaluation measuies foi examining, foi example, how well BL infoimation anu collections aie integiateu with each othei, to what extent uiffeient BLs aie compatible with each othei, how well BLs suppoit socialgioup inteiaction among heteiogeneous useis utilizing hypeimeuia infoimation, anu whethei theie aie any changes in useis' uaily woik anu lives that aie associateu with BL applications" (89). Zhang's mouel incluues ciiteiia that have yet to be incluueu in pievious evaluation stuuies, such as secuiity, sustainability, anu inteiopeiability, all of which aie consiueieu impoitant conceins foi uigital mateiials. These ciiteiia waiiant fuithei ieseaich anu inclusion in futuie evaluation mouels, paiticulaily inteiopeiability, which is only possible with stanuaiuization. Tiauitional libiaiies aie similaily oiganizeu to one anothei, which allows foi a stanuaiuization of ciiteiia. Challenges in ueveloping stanuaiuizeu ciiteiia foi uigital libiaiies incluue the vaiiety of technology that compiises the uigital libiaiy expeiience anu the many uiffeient types of useis that the uigital libiaiy may seive. While the vaiiety in uigital libiaiies cieates uifficulties, it is impoitant to consiuei the neeu foi stanuaiuization as evaluation techniques aie being uevelopeu. "Nuch moie has to be specifieu anu agieeu upon befoie uigital libiaiy evaluation can be caiiieu out in a consistent mannei, a mannei that woulu allow even foi compaiisons" (Saiacevic 2uuu S66). "No agieement exists not only on ciiteiia, measuies, anu methouologies foi uigital libiaiy evaluation, but even on the 'big' pictuie, the constiuct anu context of evaluation. The evaluation of uigital libiaiies is still in a foimative stage. Concepts have to be claiifieu fiist. This is the funuamental challenge foi uigital libiaiy evaluation" (Saiacevic & Covi, S). Noving foiwaiu, uefining uigital libiaiies anu agieeing upon the fiamewoik oi possible objectives foi evaluation will help cieate stanuaiuization foi uigital libiaiy evaluation anu potential foi long-teim evaluation. Bigital libiaiy evaluation has met with many challenges in effoits to uevelop a fiamewoik foi consistent evaluation. vaiying peispectives have iesulteu in a lack of a uefinition foi uigital libiaiies, which in tuin, makes uefining objectives uifficult. Bepenuing on the ieseaichei's chosen objective, the appioach may be systems-baseu, human- centeieu, oi a combination of the two, such as usability. Similaily, iueal ciiteiia foi evaluation woulu incluue tiauitional libiaiy ciiteiia, tiauitional IR ciiteiia, anu tiauitional human-computei inteiaction ciiteiia as well as ciiteiia unique to uigital libiaiies. Befining uigital libiaiies, appioaches, ciiteiia, anu methouology piepaie uigital libiaiies foi on- going, stanuaiuizeu evaluations that enable peei compaiison. Evaluation of uigital libiaiies ovei time will piesent challenges as technology evolves at a iapiu pace, anu global accessibility anu inteiopeiability will cieate laigei, moie uiveise usei populations. Bigital libiaiy evaluation ciiteiia anu methous, once establisheu, will have to iemain flexible anu evolve to accommouate new technologies anu its useis.
Woiks Citeu
Chouuhuiy, S., Bobbs, B., & Loiie, N. (2uu2). A fiamewoik foi evaluating uigital libiaiy seivices. B-Lib Nagazine, 8 (78). Retiieveu fiom http:www.ulib.oigulibjulyu2chouuhuiyu7chouuhuiy.html.
Fuhi, N. et al. (2uu7) Evaluation of uigital libiaiies. !"#$%"&#'("&) +(,%"&) (- .'/'#&) 0'1%&%'$2, 8 (1), 21-S8. Retiieveu fiom http:static.sciibu.comuocsjSytg1pmxnikf.swf.INITIAL_vIEW=9u Naichionini, uaiy. (2uuu) "Evaluating Bigital Libiaiies: A Longituuinal anu Nultifaceteu view." 0'1%&%3 4%$"526 49 (2), Su4-SSS. Saiacevic, T. (2uuu). Bigital libiaiy evaluation: Towaiu evolution of concepts. 0'1%&%3 4%$"526 78 (2), SSu-S69. Special issue on Evaluation of Bigital Libiaiies. Retiieveu fiom http:comminfo.iutgeis.euu~tefkoLibiaiyTienus2uuu.puf. Saiacevic, T. anu Covi, L. (2uuu). Challenges foi Bigital libiaiy Evaluation. 9&:$% 2,1;'##$5 #( #<$ ="",&) >$$#'"/ ?@@@ (- #<$ =;$%'A&" B(A'$#3 -(% !"-(%;&#'(" BA'$"A$C Retiieveu fiom http:www.is.inf.uni-uue.uecouisesul_ssu4foliensaiacevicuu.puf Saiacevic, T. (2uuS). Bow weie uigital libiaiies evaluateu. Piesentation at the couise anu confeience Libiaiies in the Bigital Age (LIBA 2uuS), Su Nay - S }une, Bubiovnik, Cioatia. Retiieveu fiom http:comminfo.iutgeis.euu~tefkoBL_evaluation_LIBA.puf. Xie, B. I. (2uu8). 0seis' evaluation of uigital libiaiies (BLs): Theii uses, theii ciiteiia, anu theii assessment. !"-(%;&#'(" 9%(A$22'"/ D >&"&/$;$"#6 44 (S), 1S46-1S7S. Zhang, Y. (2u1u). Beveloping a Bolistic Nouel foi Bigital Libiaiy Evaluation. +(,%"&) (- #<$ =;$%'A&" B(A'$#3 -(% !"-(%;&#'(" BA'$"A$ &"5 4$A<"()(/3, 61(1), 88-11u.