i

Usability Testing for World Cat Discovery

By Justin Orders, Malcolm Harris, Evan Siler
To Gregory Wickliff

https://uncc.on.worldcat.org/discovery

ii

Executive Summary
In order to measure the quality and effectiveness of the beta version of WorldCat search
interface, the J. Murrey Atkins Library conducted a series of usability tests. These tests were
constructed based on perceived everyday tasks that are carried out by the average student at the
University of North Carolina at Charlotte. In our series of testing, we were able to gather a total
of six participants. Each participant was an undergraduate at the University, and had accessed the
library and its resources at least once in the past. Lastly, the participants varied in the amount of
time they currently access the library or any of its resources.
In summary, the test users all identified key features of the interface that they either liked or
disliked. Some users gave constructive feedback in regards to why they liked a particular feature
and if they disliked a feature, what they would do to change it. Listed here are some of the key
findings and highlights:
-

Almost all of the users expressed the need for clear filters. The majority of the users felt
that they overlooked the filters on the side initially, but were expressed great satisfaction
with the filters once they began incorporating them with their searches.

83% of the users believed that they had to jump through too many hoops to gain access
to eBook sources. They stressed that the links should be direct with little redundancy.

One user identified an issue with an eBook resource. The book was displayed to be
checked out even though it was an eBook.

Users expressed a specific expectation with eBooks. They expect that the full text be
available if it is listed in the search results and also expect the copy to be free.

If OCLC wishes to incorporate eBooks for purchase, there should be an added filter to
distinguish between free sources and paid sources.

iii

Table of Contents
Executive Summary----------------------------------------------ii
Introduction and Technique------------------------------------1
Usability Test

Discussion Procedures and Research Outcomes-----------2

Key Findings
Pre-Test Questions Results
Tasks Results

Conclusion---------------------------------------------------------5
Recommendations
Appendices--------------------------------------------------------7
References---------------------------------------------------------9

List of Figures and Ideas

Figure 1: Test Completion vs. Time Spent in Library
Video 1: User Frustration
Video 2: Finding eBook
Picture 1: Filter Settings
Picture 2: Cite Record Button

Introduction
For this project we as a group had to test a new online library interface created by the
Online Computer Library Center (OCLC). The interface is a beta version of an search engine
called WorldCat which catalogs online versions of books and texts as well as locates physical
versions in libraries. While it is a beta version, it is still available to all students who use the J.
Murrey Atkins Library database for research at the University of North Carolina at Charlotte
(UNCC). By giving students access to it OCLC can assess how it needs to improve the site
before it is completed.
In order to measure the usability and quality of the new WorldCat Discovery interface.
The J. Murrey Atkins Library usability facility conducted a series of tests on the beta version of
the interface (https://uncc.on.worldcat.org/discovery). Five UNCC students were gathered and
tested separately to see how they used the new interface. Their actions and reactions to it were
recorded using video and voice capture technologies. These recordings were then analyzed to
find important issues that the users came across that hindered their ease of use. The results of this
analysis was then used to advise WorldCat on how to improve their beta interface. Information
was gathered through both qualitative and quantitative measures with a different focus for each
area of interest.

Usability Test
This test was designed so that the feedback we got from the users could be seen and
understood easily through our recording techniques. By giving our testers a set of tasks to do
using the interface we could monitor their actions in ways that could find subtle but still
significant patterns in their performances that could lead to better improvements to the interface.
Attention was also given to the major, age and experience with the site for each user. After
finding this information the users were given the following tasks.

Task 1: Search for a book related to a class in your major that you are currently taking.
Is this book available in the Atkins Library at this time?
Create a citation, in any format, for this book.
Task 2: Find an eBook for Hunger Games Catching Fire.
Is this available for download?
Task 3: What floor is the book Curious George by H.A. Ray located on?
Is it available for checkout?
If not, when is it due back?
Task 4: Find a copy of the book Les Miserables by Victor Hugo written in French.
Where is the closest place to get this book that is NOT Atkins Library?
Task 5: Locate a peer reviewed copy of The Scam by Hochstein R.A.
Are you able to view the entire text?

In case we could not acquire enough valuable information from their recordings, the users were also
asked the following post-test screening questions:
1.
2.
3.
4.

What changes or improvements would you make to the current library interface if any?
Was there anything that you found to be too challenging or difficult? Explain.
Can you see yourself using this over your preferred search engine now? Why or Why not?
Was there anything that you learned from this test?

Discussion Procedures & Research Outcomes

In this report you will find the methods for how we conducted our test along with the reasoning
behind why we thought that the prompts and questions we chose to ask of our testers was
necessary to acquire useful results for WorldCat. You will also find graphs and charts
representing correlations that we found were important for the improvement of this interface.
The key problems we found through our testing are also thoroughly examined in this document
as we assessed their frequency and detriment to the testers ability to complete the test. The We
then conclude this report with reasonable solutions for how we chose to fix these problems.

Pre-test screening results

Starting with our pre-questions, the question about how many times the user used the lab in a
week was correlated with how long it took the user to complete the test.We found that the total
time decreases as amount of times the library was accessed increases

Argument: More experienced users were generally able to navigate the tasks with more
expertise. In one scenario that we were interpreting as an outlier, a user who had accessed the
library about two times per week took longer to complete the set of tasks than users who did not
access the library on a weekly basis or only once a week. As a result, we returned to analyze this
test case. We found that this user had become fairly familiar with the older version of the search.
Throughout the test, the user would anticipate certain features to appear in locations that appear
to have changed in the Beta version of the interface. Though this was a statistical outlier, it still
holds meaning. It demonstrates that in migrating to the new interface, experience users will face
a small window of transition a natural period of time for adapting to any new features and
changes added to the new version.

Task results
Problem #1: eBook Filter
The first problem we noticed through our testing was concerning the filter settings. This came
about when we noticed that many users were finding it difficult to locate the filter settings when
they were asked to find an eBook-an electronic version of the book located on the site.
Problem #2: Cite Record
The next problem ties in with this first as it is another one that deals with the difficulty of finding
something needed to perform the task given to the users. In this case it was the citation maker
that was difficult to locate when we asked the users to create a citation using the Cite Record
feature. Almost all of the users could not locate this feature as it was an icon that only read what
it did when you hovered over it. This is a problem that we, the test makers, found to be in need of
fixing from our own experience with the interface.
Problem #3: View Online
The third problem relates to first as it again deals with the difficulty of finding an online version
of a book. Now when the users would click on the View Online feature it would take them to a
page that listed a variety of options to chose from to view the book online, though almost all of
the users were unable to select the correct one.
Video 1: User Frustration
The student anticipates that a full version of the eBook will be available online. Though this is
technically the case, most students assume that library resources will generally be free. They do
not anticipate extra costs or fees. We recommend adding a filter that separates eBooks into
paid and free categories for access if this is the case. Moreover, if a link is created with the

label See Full Text or Full Text Online, the user should be able to access this text upon
clicking the link.
https://www.youtube.com/watch?v=V_RbTt2INBw
Video 2: Finding eBook

The student anticipates that a full version of the eBook will be available online. Though this is
technically the case, most students assume that library resources will generally be free. They do
not anticipate extra costs or fees. We recommend adding a filter that separates eBooks into
paid and free categories for access if this is the case. Moreover, if a link is created with the
label See Full Text or Full Text Online, the user should be able to access this text upon
clicking the link. In a number of scenarios, our tests revealed that the user had to jump through
too many loops to reach their desired destination.
https://www.youtube.com/watch?v=V_RbTt2INBw

Problem #4: Filter Menu

The final problem discovered through our testing was one that needs a more or a preference
problem which we feel a solution for would help users understand what all is happening on the
page as they search for things. The problem is that the users couldnt tell which filters they were
using for their search and sometimes didnt know how to change them.

Conclusion
Recommendations
Noticing these problems and correlations between the users experiences we came up with a few
basic solutions and recommendations for World Cat to fix their interface. For the first problem
we decided the most reasonable solution would be to make the eBook filter more obvious to
users. Since this is a popular query for most users who will use this site it should be more
apparent.

The image seen above shows the current top 6 formats to choose from when searching for a
book. Only when you click on the Book filter does the selection for eBook come up. Just as
Print Book is seen just below Book, we feel that the eBook filter should be shown here as
well. Since this is the easiest way to find a ready-to-read book, users are more likely to use this
option then Print Book we feel this would fix a lot of confusion future users will have with this
site.
The second solution we offer regards to citation maker. While the icon (seen in the image
below) for this feature appears concise, it does not appear to be the obvious choice for users to
click on when looking for a citation

The current layout does not contain buttons with an explicit message. When attempting to make
citations, users would often scroll the length of the page in both directions looking for a more
obvious button. After it was not found, they began clicking on the highlighted areas.
1. The second button is used to add the book to a list. However, the unexperienced user has
no inkling of this.
2. The first button allows the user to make citations, amongst other features.
In most test cases, users overlooked the citation feature in this dropdown menu. Moreover, one
user added that, although a plethora of citation formats are supported, the APSA format is not
listed as an option. The Cite record selection is the correct way to get a citation for the book
selected. Our solution would be to have a button that says Citation instead of a record icon.
The other two options available through this icon would still be available.
The third solution we provide again deals with the ambiguity of this sites current format.
Clicking the View Online option brings users to a page where there are many options to
choose from and most of them do not bring the user to what they are looking for. The option that
brings the user to the quickest online source for the book is the View PDF option which allows
the user to see the entire book without having to download anything. This View PDF option
should be more straightforward by having it read View Online on the first page of the search
instead of the button that currently has it. If this cannot be done then another option is to give a
list of hints or tips in a menu or that come up when you hover over the button.
The final solution to the key problem of fixing the filter menu is to design or employ an
entirely new filter menu. This menu will now allow users to see which filters they are using at a
given time by having them on the header of the page they are currently viewing instead of on the
side where they are hidden. Errors also come about when going back the browser as some of the
filters are still up blocking content that users might want to see. Having filter breadcrumbs
display at the top of the page will let users know which filters they are using at all times and
allow them to change them with ease.

Appendices

The majority of user testing results conveyed that test two took the longest of the tests on
average. This test required the user to find an eBook version of The Hunger Games: Catching
Fire and determine whether or not it was available for download. Of the six user tests, test two
and test five which required users to find full electronic text version of a peer-reviewed version
of The Scam proved to be the most difficult. Test two had a success rate of 33 percent.
Though the task was not particularly difficult, users encountered various issues with accessing
the eBook. The primary issue according to user feedback was that the method used to access the
eBook required them to jump through too many loops. Most users used the Google Preview
button to attempt to access the eBook. From this link, users came to the conclusion that they
would need to purchase the book as a Google Book in order to access a full version.

Test five also had a success rate of 33 percent. The great majority of users experienced a similar difficulty
to those expressed in test two. Through a combination of video capture and user feedback, we were able
to identify the primary issue to be misleading links. Although there are several buttons that prompt the
user to View Online, the majority of the links take the user to the same landing page. Moreover, from
this page the user needs to download the book. To do so, they are first required to enter their credentials.
However, there is a way to access the text without downloading it. The link that is used to do so, should
be the primary link that a user is taken to when they click the View Online button.

References
"J. Murrey Atkins Library | UNC Charlotte." J. Murrey Atkins Library | UNC Charlotte. Web. 29
Feb. 2016.
Kuniavsky, Mike. Observing the user experience: a practitioner's guide to user research.
Morgan kaufmann, 2003.