COMPUTER SCIENCE AT THE UNIVERSITY OF TORONTO

CSC318H1F THE DESIGN OF INTERACTIVE COMPUTATIONAL MEDIA

Assignment 2: Heuristic Evaluation

We encounter designed objects everywhere: the doors we open, the train seats we use,
the student content management systems were forced to interact with we all have
personal experience with designed objects of varying quality. One important method for
systematically finding and evaluating usability problems is heuristic evaluation. This
method involves a small set of evaluators (typically usability experts) critically
examining a designed object with respect to its compliance with recognized usability
principles (the heuristics). In this assignment, you will conduct a heuristic evaluation of
the MarkUs, an interface most of you are familiar with. You must work on this
assignment in pairs; individual and 3+ member teams are not allowed.

Instructions:
See the end of this document for the list scenarios and tasks, the list of heuristics you
must use, how to rate severity, and the reporting template.
1) First, pair up with someone in the class, and submit your UTORids to the
Assignment 2 Part 1: Pairs submission by October 27th at 9pm. If you cant find
a partner, email the instructor. If you dont, youll be randomly paired.
2) CDF will then create MarkUs instances per pair. You will be notified when these
are ready. You cant begin the next part until this is done, so please be patient.
You will be able to access the admin and instance URL from here:
https://daphne.teach.cs.toronto.edu/admin
3) When your MarkUs instance is ready, you can conduct an individual evaluation.
Spend ~1 hour independently going through the provided scenarios and tasks
(see Appendix below), identifying as many usability problems as you can and
which heuristics were violated for each problem. You should not discuss your
results with others. Take good notes, as you will be submitting these later.
a. Pro Tip 1: By default, your account is the instructor view. You can
log in as other users (TAs and students in your MarkUs instance)

CSC318 ASSIGNMENT 2

by clicking on the people icon on the top right. Use the other accounts ID
but your own password. Logout and re-login with your UTORid when done.
b. Pro Tip 2: You can delete all data or reset your MarkUs instance to the
default data on the admin page. IMPORTANT: This will clear your
partners edits and additions as well as your own, so be sure to check with
them before you do anything drastic. Additionally, your instance may be
inaccessible for about 5 minutes as the processes are processing.
4) Meet up with your partner. Together, describe the usability problems you found
for MarkUs to each other, and then combine these into an overall list (resolving
duplicates and discussing differences in opinion). For each problem:
a. Illustrate with a screenshot. You can list multiple problems under one
screenshot, if applicable.
b. Indicate what heuristic(s) are violated using the short heuristic codes,
e.g., H1 (Status).
c. Rate the severity of the problem.
5) Report your results for MarkUs in the Assignment 2 Part 2: Report submission
using the provided template.

Rubric:
Note that both partners will receive identical marks for this assignment, which will be
calculated based on your combined report and the weaker of the two individual
heuristic evaluations. This is done to ensure that you both put in a reasonable effort.
Criteria

Weight

Individual report from each partner. Each includes the new scenarios/tasks
you came up with.

30%

Combined report: List of usability problems with screenshots.

20%

Combined report: Violated heuristics per problem.

20%

Combined report: Severity rating per problem.

20%

Proper formatting, spelling, and grammar.

10%

CSC318 ASSIGNMENT 2

Deliverables:
Three PDF files:

Two PDF files from each evaluator in PDF files named A2-LASTNAME.pdf. These
should contain, in order, with these headings:
a. Additional Scenarios/Tasks: The new scenarios/tasks you created for the
instructor, TA, and student roles.
b. Evaluation Notes: The raw evaluation notes, problems identified,
descriptions, and screenshots from each evaluator. These are a process
documents that can be handwritten and scanned or digital. They do not
need to be polished or perfect.
One PDF file named A2-LASTNAME-LASTNAME-Report.pdf containing the
combined, finalized report. Include a title page with your names, UTORids, and
preferred contact emails. Describe all of the usability problems, violated
heuristics, and severity ratings in the format asked for.

Each PDF file should have 1 margins, Arial font, 12 pt, 1.5 line height.

Due through MarkUs on:

Both Sections:

CSC318 ASSIGNMENT 2

November 9th (Wednesday of Week 9) at 11:59pm.

Appendix:

Scenarios & Tasks

Note that these are categorized by role (instructor, TA, student) and therefore may not
be completed in the order listed below. For instance, you cant assign TAs if there are
no assignment submissions from students. You will have to switch back-and-forth
between roles to complete all scenarios and tasks for each role.
Instructor:
Scenario 1: Create an assignment for a tutorial activity on usability testing.
Task 1: Make it due on some future date at 11:59pm. Allow a late period of
one day, with a 20% reduction.
Task 2: Create a marking scheme rubric where 50% of the mark is the
testing, 30% is the report, and 20% is formatting, so that the total is out of
100%.
Task 3: Edit the rubric: change the 50% testing to 25% testing procedure
and 25% recording of testing.
Task 4: Assign the TAs evenly among all submissions.
Scenario 2: Review the marks your TAs have applied and release these to
students. Be sure that penalties are applied correctly.
Scenario 3: Create a marks spreadsheet for attendance. Create columns for 3
tutorials at 1 mark each.
Scenario 4: You decide to make a grade adjustment on the assignment you
created. Add a 5% adjustment to everyone's grade.

Your turn: After completing the above, come up with at least two more scenarios
or tasks based on your experience so far. You can use the default testing content
or the content youve created. Describe and evaluate them.

CSC318 ASSIGNMENT 2

Teaching Assistant:
Scenario 1: Mark all assignments assigned to you, using the rubric and providing
summary feedback as well as in-text annotations.
Scenario 2: Mark attendance for the 1st tutorial using the spreadsheet.

Your turn: After completing the above, come up with at least two more scenarios
or tasks based on your experience so far. You can use the default testing content
or the content youve created. Describe and evaluate them.
Student:
Scenario 1: Sign in as 3 different students and submit 3 PDF files as assignment
work.
Scenario 2: Get your current mark for the course.

Your turn: After completing the above, come up with at least one more scenario
or task based on your experience so far. You can use the default testing content
or the content youve created. Describe and evaluate it.

Heuristics
For this assignment, use the following ten heuristics identified by Nielsen (1994):

H1 (Status), Visibility of system status: The system should keep users informed
about what is going on through appropriate feedback within a reasonable time.

H2 (Real World), Match between system and the real world: The system should
speak the user's language, with words, phrases, and concepts familiar to the user,
rather than systemoriented terms. Follow real-world conventions, making
information appear in a natural and logical order.

H3 (Control), User control and freedom: Users often choose system functions by
mistake and will need a clearly marked "emergency exit" to leave the unwanted
state without having to go through an extended dialogue. Support undo and redo.
H4 (Consistency), Consistency and standards: Users should not have to wonder
whether different words, situations, or actions mean the same thing. Follow
platform conventions.

CSC318 ASSIGNMENT 2

H5 (Prevention): Even better than good error messages is a careful design which
prevents a problem from occurring in the first place.

H6 (Recognition), Recognition rather than recall: Make objects, actions, and options
visible. The user should not have to remember information from one part of the
dialogue to another. Instructions for use of the system should be visible or easily
retrievable whenever appropriate.

H7 (Flexibility), Flexibility and efficiency of use: Acceleratorsunseen by the

novice usermay speed up the interaction for the expert user to such an extent that
the system can cater to both inexperienced and experienced users.
H8 (Design), Aesthetic and minimalist design: Dialogues should not contain
information that is irrelevant or rarely needed. Every extra unit of information in a
dialogue competes with the relevant units of information and diminishes their
relative visibility.

H9 (Recovery), Help users recognize, diagnose, and recover from errors: Error
messages should be expressed in plain language (no codes), precisely indicate the
problem, and constructively suggest a solution.

H10 (Help), Help and documentation: Even though it is better if the system can be
used without documentation, it may be necessary to provide help and
documentation. Any such information should be easy to search, focused on the
user's task, list concrete steps to be carried out, and not be too large.

The Severity Rating

After you identify all of the usability problems, you must assess the relative severity of
each individual issue to the rest. Doing this allows you to prioritize allocation of
resources towards fixing the most serious problems in the next design iteration.
The severity of a usability problem is a combination of three factors:
1. The frequency with which the problem occurs: Is it common or rare?
2. The impact of the problem if it occurs: Will it be easy or difficult for the users to
overcome?
3. The persistence of the problem: Is it a onetime problem that users can
overcome once they know about it or will users repeatedly be bothered by the
problem?

CSC318 ASSIGNMENT 2

Although the severity of a usability problem has several components, it is common to

combine all aspects of severity into a single severity rating. In this assignment, we will
use a five-point scale for rating the severity of usability problems:
0 I don't agree that this is a usability problem at all
1 Cosmetic problem only need not be fixed unless extra time is available on project
2 Minor usability problem fixing this should be given low priority
3 Major usability problem important to fix, so should be given high priority
4 Usability catastrophe imperative to fix this before product can be released

Note: You may wonder why there a 0 (I dont agree that this is a usability problem at
all) option if you have marked it as a usability problem. The reason is that you are often
rating usability problems found by other evaluators, and youre allowed to disagree.
Rating usability problems for severity happens after all of the evaluators have finished
their evaluations. This is for two reasons. First, during evaluation, the evaluator is more
focused on inspecting the interface and finding new usability problems. Second, each
evaluator will only be able to find a subset of the usability problems, so severity rating
of the problems found by that evaluator will be incomplete. The method is strengthened
by the combined analyses of multiple evaluators.

Reporting Template:
The following is an example of a standard usability reporting template for one usability
problem identified. Use this template in your final, combined report.

CSC318 ASSIGNMENT 2

Problem: There is no indication of what elements on the screen are clickable or what
they do. The user has to move the mouse around in order to find out what to do next.
Violated heuristics: H3 (Control), H4 (Consistency), H10 (Help)
Severity (Evaluator A): 3

CSC318 ASSIGNMENT 2

Severity (Evaluator B): 4