User Observation | Learnability with NUI / iPad

Observe a user, for at least 30 minutes, learning a new program that the participant
has never encountered/used before. Frame up the observation as a training session
but try to provide as little guidance as needed, without imperiling the observed.
Take notes and report on findings.

Cameron Friedlander

2010-05-11
 LEARNABILITY AND THE NUI [C
Friedlander]

Contents
User Observation | Learnability with NUI / iPad..........................................................1
Objective.................................................................................................................3
Observed Tools........................................................................................................3
Observation Tools....................................................................................................3
Volunteer Persona / Attributes................................................................................3
Research Method.....................................................................................................4
Results....................................................................................................................4
Discussion...............................................................................................................6
Did the user embrace experimentation and or trial-and-error without fear of
repercussion?.......................................................................................................6
What feedback did the participant provide during the observation, either
positive or negative?............................................................................................6
What specific problems did the user encounter?.................................................7
What issues did the participant encounter with the interface as they relate to
Norman’s Principles of Design?............................................................................7
If you were to repeat the observation with another user, would you do anything
differently?...........................................................................................................8
Conclusion...............................................................................................................8
Appendix...............................................................................................................10
Observation Output............................................................................................10
Tools...................................................................................................................12
Tools

2
 LEARNABILITY AND THE NUI [C
Friedlander]

Objective
Observe whether the iPad and or NUI (Natural User Interfaces) has more learnability
and accessibility compared to a traditional desktop interface. Specifically,
complicated programs programmed for the devices environment / framework. For
the observed it two-fold; does the iPad interface inherently breed learnability and is
a complicated paint program easier to learn within this environment? All of this
within 30 minutes of observation “guised” as training. This research should be
followed up with a comparable persona and study using Adobe Photoshop on a PC.
The final objective being to see whether it is easier to learn within a natural user
interface vs. traditional taxonomic interface.

Observed Tools
1. Use the iPad.
2. Learn SketchBook Pro.

Observation Tools
1. Flip Mino Video Recorder.
a. Task 1 – Learning – 23.56 minutes
b. Task 2 – Free Draw – 5.27 minutes

Volunteer Persona / Attributes

1. Female
a. 32 years old.
b. PC user for business and pleasure.
i. Familiarity with Microsoft Office tools
1. Word
2. PowerPoint
3. Excel
4. Outlook
5. Access
c. Mobile priorities
i. iPhone for the following only:
1. Phone
2. Email
3. Voicemail
ii.Very little application use.
2. Primary method for computer interaction.
a. Trackpad
b. Mouse
3. Has never used the iPad.
a. Is familiar, at a basic level, with the iPhone.
b. Understands basic gestures only.
4. Never used a professional paint program.
a. Beginner experience in the past with sketch programs.
b. No experience with Adobe Creative Suite.
i. No professional or industry experience within the creative fields.
c. Does not consider herself to be an artist, in any way shape or form.
3
 LEARNABILITY AND THE NUI [C
Friedlander]

d. Has never heard of or used Sketchbook Pro or any other similar

program on:
i. PC
ii.Mac OSX
iii.iPhone
iv.iPad

Research Method
Using direct methods the observer and the observed were within the same room,
set up as a participatory contextual inquiry using semi-structured interview /
training methods to guide the observation and final report. The training was set up
into two parts: the first objective was to learn the program at a basic level, the
second was to utilize the learned skills to create a piece of artwork. This information
was not disclosed to the participant. Even though the program came with a wizard /
tutorial this was skipped to see how quickly the user could learn the basic interface
of Sketchbook Pro. The training was task based vs. instructional. Asking the user to
try to complete a certain task versus how to complete this task specifically. There
was never a moment where the observer actually told the user how to do
something, occasionally a task might be put on hold to do something else to limit
any sort of frustration and or interrupt the training feel of the observation.

Results
Task based result set:

Task Name Time (secs) Issues Notes

Find Sketchbook 20 Users knew left

Pro and right gestures
for navigating
screens.

Interface 20 Asked if this would User could draw

Exploration be guided. with finger with
ease.

User found menu

(which was
hidden).

Create New Page 7 Immediately

opened new page.

Draw a Cat 25 User did not have Discovered the pen

the correct pencil selection module.
queued up.

Black Pen 10 Selected color and

pen

Drawing cat outline 40 Learned that

tapping creates

4
 LEARNABILITY AND THE NUI [C
Friedlander]

depthness of color.

Make eyes blue / 80 Pointed out where

erase black eyes the “brushes”
were.

Draw yellow sun 14

Explore brushes 90 Instructed user to Wanted user to

and textures (cats investigate screen discover other
fur) pressure. items without help.

User discovers
thickness for
brushes and
opacity.

Draw whiskers 72 Prompted user for See if user could

alternate way to find pencil thin
define brush transparent brush.
thickness –
manually.

Blend textures in 120 Actually discovered User had to find

body / smudge tool multiple other the “smudge” tool
discovery tools, such as without any idea of
mirroring, patterns, what it could look
and shapes. like.

User did not find Without prompting

smudge tool. the user would call
out what they saw
Prompted to turn and learned “wow
off mirroring. look, flowers”.

70 seconds in user
discovers mirror
brush.

User discovers
splatter paint
brush.

User sees how

eraser can be used
to help enhance
the textures.

Turn mirroring off 120 User was not aware User comments
as to how they set that they are “lost”
it. after 30 seconds
but continues
NOTE: observer
5
 LEARNABILITY AND THE NUI [C
Friedlander]

didn’t know how to forward.

get rid of it either
but this was not User discovers “fill”
disclosed. option and turns
most of the screen
Mirroring not black.
turned off due to
“fill” task. Tried to have her
find “undo” but
instead she used
white fill to reverse
black fill in screen.

Reverse fill from 60 Needed prompting User kept hitting fill

black to white to remind her what but with black
color she has
selected.

Find the smudge 190 Mirroring magically

tool – part 2 off now.

Still discovering
new brushes and
patterns while
looking for
smudge.

Add background 90 Looking at tool

pattern(s) recall after
discovery

Background fill 15 Turned cats face

blue as well as
background.
Needed to instruct
her on how to
reverse just the
cat’s face from
blue black to white.

Draw background 30 Prompted to find User recalled the

pattern leaf pattern. leaf pattern.
Commented “this
is cool”.

Add horizon / 82 Recalled how fill

background fill works by
completing lines
and making sure
there were no
gaps; turning cats

6
 LEARNABILITY AND THE NUI [C
Friedlander]

face blue.

Save picture 12 User completed

first picture (see
appendix).

Free draw 300 User could create

anything they
wanted.

User recalled every

single item learned
and used them
creatively to go
technically farther
than first picture in
a lot less time.

User found the

actual “undo” tool
without prompting.

See appendix for

drawing 2.

Discussion
Did the user embrace experimentation and or trial-and-error without fear
of repercussion?
The interface in and of itself encourages exploration; with its simplicity, free flowing
gestures, and the fact that this was an “art” program instead of a true business
task/program. There was no fear of doing something wrong and the participant
appeared to be fully engaged in the exploration of the observation and tasks.

What feedback did the participant provide during the observation, either
positive or negative?
Occasionally the user would sigh and or say that they were lost. This was very
infrequent and was either subdued via their own discovery or minor prompting from
the observer. Since there was no right and wrong the positive truly outweighed the
negative within this experience. Multiple times throughout the session the user
would comment on how “cool” the program was. By being able to paint with the
hands / finger tips it brought out a natural child like play and exploration that should
probably be encouraged more in life.

What specific problems did the user encounter?

Though the “problems” were minimal it can be noted that the NUI/gesture interface
is very forgiving. The participant would double-tap for the “fill” even though a single
tap was needed. Another example is when scrolling brushes, simple left and or right
taps would suffice but the user would drag left or right to activate the menu, again
the interface understood this and interrupted it correctly either way.
7
 LEARNABILITY AND THE NUI [C
Friedlander]

On task specific items such as “smudge” and turning off “mirroring” the user had no
context and or framework as to what this iconography would look like. They needed
to explore the brushes offered and essentially learn by trial-and-error to figure out
how to activate or deactivate these options. Though this may have been a
“problem” it actually allowed the user time to explore the capabilities of the
program beyond the observers prompting and the guided training. The user would
later reflect on this and utilize these skills that they gained above and beyond the
“tutorial”.

What issues did the participant encounter with the interface as they relate
to Norman’s Principles of Design?
1. "Visibility. By looking, the user can tell the state of the device and the
alternatives for action. "
a. Contrary to the gist of “visibility” the entire menu and options are
hidden from the user. In fact the first state of the application is a blank
screen; all menu items are hidden within natural gestures. There is one
“dot” at the bottom of the screen that can pull up the initial high-level
menu. But by using different gestures the user can pull up more
specific menus rapidly.
b. The user seemed to not struggle with the hidden menu system at all,
this would argue against Norman’s rule suggesting that as interfaces
become more friendly and natural having persistent options may not
be needed at all.
2. "A good conceptual model. The designer provides a good conceptual model
for the user, with consistency in the presentation of operations and results
and a coherent, consistent system image. "
a. The breakdown of the gestural menu taxonomy seemed consistent
with the tasks / metaphors that one could relate to outside of the
program itself. The menu was as simple or as complicated as the task
at hand.
b. Affordance
i. By allowing the user to use their hands to grasp of the overall
conceptual model came inherently with the system.
1. Press hard for a darker imprint.
2. Lightly touch for a soft stroke.
3. The menu / colors / brush selector worked the same way.
a. Gestures to flip.
b. Tapping to select.
3. "Good mappings. It is possible to determine the relationships between actions
and results, between the controls and their effects, and between the system
state and what is visible. "
a. For the most part this was very transparent within the application /
interface; again this could be because the application itself is very
visual in nature.
b. There was one instance where the user was lost within the system and
this was while they were performing the fill option, in place of the
undo.
i. The task was to revert the background back to white

8
 LEARNABILITY AND THE NUI [C
Friedlander]

1. The correct way was to use the “undo” (very clear arrows
on the menu; even simpler than Adobe Photoshop).
2. However the user opted to fill the background again but
they kept making it black because they forgot to change
the color.
3. Once the user discovered the undo this type of issue
became null and void.
4. "Feedback. The user receives full and continuous feedback about the results
of actions."
a. There was never a time where the user would select something and
not see and or understand that they had done something.
b. Because the menu / options were not exposed by rule, the user would
always need to tap (or hit the dot on the bottom of the page) to start
an action. This meant that the user always expected something
happen after doing so. So they were prepared to have things change
because the requested it. The feedback would be immediate as soon
as they hit “pencil to paper” so to speak.
5. Affordance. A clue or visual model that inherently lets the user know how to
use it.
a. The tools themselves, paint, brush, strokes, are obvious and lend
themselves to a certain level of affordance as to their use.

If you were to repeat the observation with another user, would you do
anything differently?
To be perfectly honest I think the observation was a success and accomplished the
goals intended. The only thing I would change is my own level of skill with the
program; Sketchbook Pro. Since I am well versed with Adobe Photoshop I knew the
language and terminology needed but since the program itself was new to me as
well I did not always know the correct answer (i.e. how do you turn off mirroring).
But because of this it could have made this even more successful than originally
intended since there was never a point where I needed to grab the iPad and look for
myself. I think that this study could be greatly enhanced by doing this with multiple
participants and timing how long it takes the average user to “learn” the program to
the level suggested in this task/training. Comparing it, under the same time
constraints, to Adobe Photoshop would probably underscore my hypothesis greatly
that this interface / gestures and application are much easier, much more
approachable, and ultimately the future of interaction.

Conclusion
While I can’t say that this proves my hypothesis this first step does seem support
the idea that a NUI interface has better learnability and perhaps is easier to
navigate. While Sketchbook Pro is not quite at the level of Adobe Photoshop, it is a
complicated program that contains layering, and advanced design tools but is, of
course, missing some other capabilities. I do think that as we move into the future
the NUI will soon replace the single-input device and we will find that menu
structures fall into memories of the past. What struck me as interesting was the
testing for Norman’s good design principles specifically visibility; and the hidden
menu / gesture structure of SketchBook Pro. While we would encourage visibility in
certain instances, say a “stop” sign, it seems to be of little use within the context-
9
 LEARNABILITY AND THE NUI [C
Friedlander]

specific hidden menus. Of course we can apply this principle to the hidden menu
when it is displayed and we have visibility with items grouped within a context that
makes sense. But I do see the tablet and this test as an initial step forward to
eliminating the menu system altogether.

10
 LEARNABILITY AND THE NUI [C
Friedlander]

Appendix

Observation Output

Figure 1 - Learning the program

11
 LEARNABILITY AND THE NUI [C
Friedlander]

Figure 2 - Free draw - using the tools learned

12
 LEARNABILITY AND THE NUI [C
Friedlander]

Tools
1. iPad – First Generation – 16GB – WiFi
2. SketchBook Pro - Autodesk

13