Received: 12 April 2017 Revised: 20 July 2017 Accepted: 30 December 2017

DOI: 10.1111/jcal.12238

ORIGINAL ARTICLE

ViDA: A virtual debugging advisor for supporting learning in

computer programming courses
V.C.S. Lee1 | Y.T. Yu1 | C.M. Tang2 | T.L. Wong3 | C.K. Poon4

1
Department of Computer Science, City
University of Hong Kong, Hong Kong Abstract
2
Department of Information Systems, City Many students need assistance in debugging to achieve progress when they learn to write com-
University of Hong Kong, Hong Kong puter programs. Face‐to‐face interactions with individual students to give feedback on their pro-
3
Department of Computing Studies and grams, although definitely effective in facilitating their learning, are becoming difficult to achieve
Information Systems, Douglas College, BC, with ever‐growing class sizes. This paper proposes a novel approach to providing practical auto-
Canada
4
mated debugging advice to support students' learning, based on the strong relationship observed
School of Computing and Information
between common wrong outputs and the corresponding common bugs in students' programs. To
Sciences, Caritas Institute of Higher Education,
Hong Kong implement the approach, we designed a generic system architecture and process, and developed
Correspondence a tool called Virtual Debugging Advisor (ViDA) that was put into use in classes in a university. To
Chung Keung Poon, School of Computing and evaluate the effectiveness of ViDA, a controlled experiment and a survey were conducted with
Information Sciences, Caritas Institute of
first year engineering students in an introductory computer programming course. Results are
Higher Education, Hong Kong.
Email: chungkpoon@gmail.com encouraging, showing that (a) a higher proportion of students could correct their faulty code
themselves with ViDA enabled, (b) an overwhelming majority of respondents found ViDA helpful
Funding information for their learning of programming, and (c) most respondents would like to keep ViDA enabled
Research Grants Council of the Hong Kong when they practice writing programs.
Special Administrative Region, China, Grant/
Award Number: Ref.: UGC/FDS11/E02/15

KEY W ORDS

computer science education, debugging, novice programmers

1 | I N T RO D U CT I O N and Higgins (2005) found that although the majority of competent

Learning of computer programming is known to be difficult for many
beginners (Katai, 2015; Kelleher & Pausch, 2005; Lahtinen, Ala‐Mutka,
& Järvinen, 2005). Students in introductory programming classes are
usually required to do a lot of hands‐on practices in writing programs
in order to comprehend the key concepts and develop the relevant
skills (Choy, Nazir, Poon, & Yu, 2005; Guzdial & Guo, 2014). During
practice, students invariably make mistakes in some of their programs
and, hence, have to debug and correct the faults in their code.
However, debugging is often non‐trivial and can be very frustrating
to many students. Indeed, debugging is also known to be tedious and
time‐consuming even for professional software developers (Parnin &
Orso, 2011; Tang, Chan, Yu, & Zhang, 2017). Ahmadzadeh, Elliman,
debuggers are competent programmers, the converse is not necessar-
ily true. Therefore, to learners of computer programming, debugging is
“harder” than coding, in the sense that debugging is hard even to those
who can do coding well. There is a practical need to provide assistance
to students on debugging so that students' learning of computer pro-
gramming will not be severely obstructed by their weakness in finding
bugs in programs. This is especially true in introductory classes when
many students' coding ability is still at its infancy.
Conventionally, it is believed that the competence of students in
detecting and fixing bugs depends heavily on their own experience,
logical thinking ability, and coding skills, but novice learners are typi-
cally weak in all these aspects. Practice is a major learning activity to
acquire these abilities, and so it is very important for learners to do

Dr. V.C.S. Lee is an Assistant Professor in the Department of Computer Science, City University of Hong Kong. His research interest includes data management in
mobile computing systems and computing education. Dr. Y.T. Yu is an Associate Professor in the Department of Computer Science, City University of Hong Kong.
His research interest includes software engineering, software testing, and computers in education. Dr. C.M. Tang is a Graduate Teaching Assistant in the Department
of Information Systems, City University of Hong Kong. His research interest includes software engineering and programming education. Dr. T.L. Wong is a Faculty
Member in the Department of Computing Studies and Information Systems, Douglas College, BC, Canada. His research interest includes data analytics, information
extraction, programming education, and e‐learning. Prof. C.K. Poon is a Professor and the Dean of School of Computing and Information Sciences, Caritas Institute
of Higher Education, Hong Kong. His research interest includes databases and information retrieval, scheduling and planning, and computers in education.

J Comput Assist Learn. 2018;1–16. wileyonlinelibrary.com/journal/jcal © 2018 John Wiley & Sons Ltd 1
2 LEE
more practice in a computer programming course (Law, Lee, & Yu,
2010). However, students nowadays will easily lose enthusiasm and
interests in doing programming exercises due to the obstacles encoun-
tered during program debugging, especially when they experience
repetitive failures in coming up with the right code and do not know
whether they are working in a right direction. Hence, a major challenge
in programming classes is to upkeep the motivation of students to
overcome these obstacles. One way to address the challenge is to pro-
vide timely feedback and advice to students during the debugging pro-
cess. Of course, the presence of an experienced mentor offering face‐
to‐face assistance and guidance to individual students will make the
learning process much more effective and pleasant. Unfortunately, such
an arrangement is increasingly difficult to achieve in practice and is close
to impossible with ever‐growing sizes of introductory programming
classes against limited manpower and resources (Ala‐Mutka, 2005;
Kelleher & Pausch, 2005; Wang & Wong, 2007). Moreover, human
mentors cannot operate 24 hr a day to provide on‐demand assistance
to students while the latter are learning at their own pace after hours.
A great deal of research has been conducted in the past with
regard to offering automated assistance to students in the learning of
writing programs (see a brief review in Section 2), but relatively little
work has been done on the provision of automated support on
debugging for students in introductory programming classes. One pos-
sible reason is that automated debugging, although being an intensive
research topic, is not yet mature enough to offer precise and practical
assistance to programmers (Parnin & Orso, 2011; Tang et al., 2017). To
precisely identify bugs automatically from general programs remains a
long‐term research goal that is unlikely to be reachable soon. On the
other hand, there is indeed an imminent practical need to assist stu-
dents in debugging their own faulty programs, at least at the beginning
stage of their learning of computer programming.
In this paper, we propose a novel approach to providing practical
debugging advice to support students' learning when they do exercises
in introductory programming classes. Our goal is neither to teach
debugging per se, nor to automatically locate bugs for students, but
to provide practical and useful advice when they locate bugs them-
selves, based on the analysis of common bugs found in past students'
programs. We have developed a tool, called Virtual Debugging Advisor
(ViDA), to implement the proposed approach. ViDA was used in conjunc-
tion with an existing automated system called Programming Assignment
aSsessment System (PASS) for facilitating student learning of computer
programming (Choy et al., 2005; Law et al., 2010; Yu, Poon, & Choy,
2006). ViDA was put into use in classes in a university with strongly
encouraging evaluation results. This paper reports the development of
ViDA and the results of our evaluation. Our contributions are threefold:
• We exploited the strong relationship between common wrong
outputs and the corresponding common bugs to provide guidance
for students when they debug their programs.
• We developed a novel approach, designed a generic system archi-
tecture and process, and built a tool to help improve teaching and
learning in introductory computer programming classes by facili-
tating students to receive instant debugging advice.
• We evaluated both the effectiveness of the tool empirically and
students' perception of its usefulness via a survey.
ET AL.
The organization of the rest of this paper is as follows. We first
briefly review related work in Section 2. Section 3 introduces the
notions of common wrong outputs and common bugs that form the
basis of the debugging advice provided by ViDA. In Section 4, we dem-
onstrate how ViDA works with its underlying automated student pro-
gram submission and assessment system, PASS, and then delineate the
generic system architecture and process of ViDA. Sections 5 and 6
present the experiment and survey, respectively, which we conducted
to evaluate ViDA, and discuss the corresponding data analysis and
results. Finally, Section 7 concludes the paper.
2 | R E L A T E D WO RK
Learning computer programming is difficult for beginners because it
involves the simultaneous acquisition of several pieces of new knowl-
edge and skills such as program structure, abstraction of problem, for-
mulation of solution, and verification of the program (Guzdial & Guo,
2014; Kelleher & Pausch, 2005). Cognitive Load Theory (Atkinson &
Shiffrin, 1968; Baddeley & Hitch, 1974) explains that the nature of
learning programming can easily overload the limited working memory
of novice programmers (Moons & De Backer, 2013). Lahtinen et al.
(2005) conducted an international survey of over 500 students and
teachers, which indicated that students found difficulties in learning
program structure and concepts. Their study suggested that students
had to do programming exercises by themselves, with the aid of care-
fully designed course materials and guidance. Moreover, timely feed-
back is important to guide the students when they work on
programming exercises themselves (Moons & De Backer, 2013;
Odekirk‐Hash & Zachary, 2001; Paiva, Ferreira, & Frade, 2017; Rae &
Samuels, 2011; Choy et al., 2005; Wang & Wong, 2007) and to keep
them motivated in learning (Law et al., 2010). As it is increasingly unre-
alistic to expect one‐to‐one interactions and tutoring for typically large
programming classes, researchers have been building various auto-
mated systems to facilitate student learning and to offer timely feed-
back to students (Gómez‐Albarrán, 2005; Kelleher & Pausch, 2005;
Odekirk‐Hash & Zachary, 2001). For example, Satratzemi, Dagdilelis,
and Evagelidis (2001) adopted visualization to reduce the cognitive
load of learning programming. They developed an education learning
environment called AnimPascal to help students understand the differ-
ent phases (coding, verifying, debugging, etc.) of program develop-
ment. Other than visualization systems, some researchers built
interactive development environments that were customized for stu-
dent learning in introductory programming classes based on various
educational theories. For example, Moons and De Backer (2013)
recently designed an interactive learning environment for introductory
programming that was influenced by the cognitive load theory and the
constructivist paradigm. Hooshyar, Ahmad, Yousefi, Yusop, and Horng
(2015) developed the Flowchart‐based Intelligent Tutoring System,
which is an interactive system to visualize the solution developed by
individual students to programming problems.
In a recent literature survey on the teaching of introductory pro-
gramming, Pears et al. (2007) identified four major research areas
(i.e., curriculum, pedagogy, languages, and tools) and noted that the
most researched one was on teaching tools, predominantly
LEE ET AL.
visualization tools (such as AnimPascal), automated program assess-
ment systems (APASs), tutoring, and programming environments,
including “microworlds” such as the Robocode framework (Gómez‐
Albarrán, 2005; Kelleher & Pausch, 2005). They further observed that
only very few tools have seen wide adoption in practice.
Among the more widely adopted tools for facilitating the learn-
ing of computer programming, APASs has grown rapidly in popular-
ity with the advent of GUI and Internet technologies. Automatic
grading tools have been developed as early as in the 1960s, initially
for economic reasons, but soon it was discovered that students did
learn better with automatic graders (Hollingsworth, 1960). Over the
years, the tools have evolved from rudimentary job control programs
to Unix scripts, shell utilities, client–server systems, and Internet‐
based systems with distributed architectures (Choy et al., 2005;
Higgins, Gray, Symeonidis, & Tsintsifas, 2005; Ihantola, Ahoniemi,
Karavirta, & Seppälä, 2010). Some well‐known APASs include BOSS
(Joy, Griffiths, & Royatt, 2005; Luck & Joy, 1999), CourseMarker
(Higgins et al., 2005), HoGG (Morris, 2003), PASS (Choy et al.,
2005; Law et al., 2010), Oto (Tremblay, Guérin, Pons, & Salah,
2008), and many others (Gómez‐Albarrán, 2005). Also available now-
adays are commercial deployment (Pearson Education Inc., 2015)
and open source releases, such as ASAP (Douce, Livingstone, &
Orwell, 2005) and Web‐CAT (WebCAT, 2015). Many systems cur-
rently in use worldwide have been found to be educationally bene-
ficial in a variety of ways. In particular, PASS was developed in
City University of Hong Kong in early 2004 with the primary aim
of facilitating student learning of computer programming (Choy
et al., 2005). The work reported in this paper was built on the
success of PASS by incorporating a novel approach of providing
debugging advice through an add‐on subsystem, ViDA.
A contemporary APAS generally supports several core tasks:
• The instructor uploads a programming problem with predefined
test cases.
• A student reads the problem from the APAS, works to produce a
solution, and submit to the system.
• The APAS assesses the student's solution, typically by executing it
with instructor‐prepared test cases, and returns the assessment
results to the student, including the correctness of the student's
solution and other information.
Existing APASs differ in additional capabilities to meet the assess-
ment needs of different universities, but none of them provide
debugging advice to students when their programs are incorrect, as
what ViDA is aimed at.
One early attempt to improve students' debugging skills was the
development of a practicing system called DebugIt (Lee & Wu,
1999). DebugIt contains a bank of 20 problems and a buggy pro-
gram for each problem. Students using DebugIt are given a faulty
program to debug. Hints are provided upon request. DebugIt was
designed to train students on debugging skills using preselected
manually prepared problems and buggy programs. By contrast, ViDA
seeks to assist students by providing debugging advice when they
debug their own code. Recently, Dereferee (Allevato & Edwards,
2014) was developed to generate diagnostics to assist students in
3
tracking down logical errors that are specifically related to C++
memory management and pointer usage. Caso, Garbervetsky, and
Gorín (2013) integrated a verifying compiler as a plug‐in for the
Eclipse integrated development environment to aid the analysis of
programs written in an experimental programming language. In com-
parison, ViDA is designed to deal with common bugs found in stu-
dent programs that produce common wrong outputs. Its application
is not limited to particular types of bugs or languages in which the
program is written.
Another attempt to provide automated feedback to students on
program bugs was the development of the Independent Student
Tutoring by Examining Programs (InSTEP) system by Odekirk‐Hash and
Zachary (2001). InSTEP is a Web‐based application that presents to
the student a partial solution of a given programming problem, and
the student is required to fill in the blanks in the partial solution.
InSTEP then reports the student's mistakes (if any), together with sug-
gestions for repairing them. The major limitation of InSTEP is that it
provides a code template extracted from a small set of preselected
problems to students and only allows students to complete the code
by filling in certain missing pieces (typically one or two lines or state-
ments) instead of writing a whole program. By contrast, ViDA is
designed to provide debugging advice to whole programs that are
entirely written by students as complete solutions to programming
exercises that are assigned by the instructor in an introductory pro-
gramming course.
As stated in Section 1, ViDA is designed based on the analysis of
common bugs found in students' programs. In this regard, some previ-
ous research studies (e.g., Ahmadzadeh et al., 2005) have been done to
analyse students' bugs or debugging patterns, but not to provide auto-
mated debugging advice to students. Lam, Chan, Lee, and Yu (2008)
initiated the idea and a preliminary design framework of an automatic
debugging assistant that inspired ViDA, but the latter was built after
substantial redesign and enhancement, integration into PASS, archival
analysis, field tests, and adoption in class. Poon, Wong, Yu, Lee, and
Tang (2016) designed a hierarchical model for the program output
and developed an approach to automatically analysing the output
pattern of students' programs. However, their study focuses on the
detection of admissible program output instead of utilizing the
debugging advice to enhance students' learning.
3 | C O M M O N WR O N G O U T P U T S A N D
COM M ON BU GS
Programs written by different students may contain different bugs.
Debugging is also prone to human errors, which vary among dif-
ferent individuals. More bugs may be inadvertently introduced
and accumulated while revising the program if the original bugs
cannot be correctly located and fixed during debugging. Our expe-
rience, however, tells us that bugs produced by novice program-
mers can be very similar across different cohorts of students.
This information may be exploited so that current students may
learn from the mistakes made by past students, thereby accelerat-
ing their self‐learning and improvement. To facilitate such learning,
we have developed a tool to accumulate and analyse the common
4 LEE ET AL.

mistakes of past students, relating the frequently occurring bugs to
the outputs of the faulty programs when the latter were executed
with instructor predefined test cases. We further utilize such infor-
mation produced from the above archival analysis and incorporate
it into the existing student program assessment system, PASS.
When another student later makes a similar mistake in his/her
program and then submit it to PASS for assessment, ViDA can
immediately provide customized feedback in the
debugging advice based on the outputs of the program, thereby
helping this student to make progress towards revising his/her
program to a correct one.
For example, consider the following exercise, Convert_time
(Problem 1), in an introductory computer programming course.
Problem 1. Convert_time
Write a program, convertTime.c, that reads in the
number of seconds and converts it into hours, minutes,
and seconds. For example, when the input is 5000, the
output should be as follows.
5000 second(s) = 1 hour(s) 23 minute(s)
20 second(s)
and “Wrong use of the same variable for the number of seconds
in both input and output,” respectively. Table 2 shows the code of
a correct program and two sample faulty programs (Program‐A
and Program‐B) with these two common bugs (underlined), together
with their outputs and the corresponding annotations provided
by ViDA.
form of
4 | VIRTUAL DEBUGGING ADVISOR
ViDA is designed to be built on PASS, a Web‐based application that
can be seamlessly integrated with popular e‐learning platforms such
as Blackboard to provide detailed statistics of students' performance
(e.g., success rate) to the instructor and instant feedback to students
(Choy et al., 2005; Yu et al., 2006). In this section, we first present a
brief overview of PASS in Section 4.1, then demonstrate how ViDA
works with an example in Section 4.2, present the system architecture
of ViDA in Section 4.3, and describe, in Section 4.4, the process
through which the instructor devises debugging advice to be automat-
ically provided to students through ViDA.

Table 1 summarizes our collected past statistics of each wrong

output of the whole class when the input is 10,000. We observed that
the majority of wrong programs produced the first two outputs, which
we call common wrong outputs for this test case.
Next, we examined all the faulty programs and found that
Program‐A and Program‐B in Table 2 were typical in producing the
first two common wrong outputs shown in Table 1, respectively.
Specifically,
• all programs giving the first common wrong output contain the
same bug as in Program‐A, and
• 90% of all programs giving the second common wrong output con-
tain the same bug as in Program‐B.
Thus, a strong relationship between a particular common wrong
output and its corresponding bug can be observed in this example.
We call these bugs common bugs. Therefore, an annotation in
response to a common wrong output that hints at the correspond-
ing common bug may help most, if not all, students to identify their
own bugs. In this example, the annotations can be “Forgot to sub-
tract the number of hours in calculating the number of minutes”
4.1 | Programming assignment assessment system
(PASS)
PASS was first developed in 2004 with the primary aim to assisting
beginners in learning computer programming. It allows students to
submit their code as many times as needed. For every student pro-
gram submission, PASS instantly returns the results of testing the
program against a large set of test cases, and pinpoints the exact
position where the expected output differs from the actual program
output. This was unlike many other APASs built at the time that typ-
ically return feedback to students after the submission deadline until
the marking process had completed. Experience shows that, with the
instant results from PASS, students are motivated to practice more
extensively and, hence, can learn much more effectively and indepen-
dently (Law et al., 2010).
Furthermore, PASS provides a variety of comprehensive online
information of students' performance to the instructor (Choy et al.,
2005; Choy et al., 2008). For example, the bar charts in Figure 1
show the number of students who have attempted or successfully
solved a particular programming problem or a certain number of
problems, respectively. The instructor may make use of such

TABLE 1 Wrong outputs for Convert_time when the input is 10000

Wrong outputa Percentage
b
1. 10000 second(s) = 2 hour(s) 166 minute(s) 40 second(s) 37.9
2.c 40 second(s) = 2 hour(s) 46 minute(s) 40 second(s) 19.0
3. 10000 second(s) = 2 hour(s) 0 minute(s) 40 second(s) 14.7
4. 10000 second(s) = 2 hour(s) 46 minute(s) 3640 second(s) 9.5
5. 10000 second(s) = 46 hour(s) 46 minute(s) 40 second(s) 9.5
6. (others) 9.5
a
The correct output is: 10000 second(s) = 2 hour(s) 46 minute(s) 40 second(s)
b
All programs giving the first wrong output contain the same bug as in Program‐A, shown in Table 2.
c
90% of all programs giving the second wrong output contain the same bug as in Program‐B, shown in Table 2.
LEE ET AL. 5

TABLE 2 Sample programs for Convert_time and their outputs when the input is 10000.
Programs Code, outputs, and annotations

A correct program Code (correct) void main( )

{
int n_sec, hr, min, sec;
cout << “Please enter the number of seconds: ”;
cin >> n_sec;
hr = n_sec / 3600;
min = (n_sec / 60) % 60;
sec = n_sec % 60;
cout << n_sec << “ second(s) = ” << hr << “ hour(s) ” << min << “ minute(s) ” << sec <<
“ second(s) ” << endl;
}
Output (correct) 10000 second(s) = 2 hour(s) 46 minute(s) 40 second(s)
Annotation (None)
Program‐A Code (incorrect) void main( )
(faulty) {
int temp, h, m, s;
cout << “Please enter the number of seconds: ”;
cin >> temp;
h = temp / 3600;
m = temp / 60;
s = temp % 60;
cout << temp << “ second(s) = ” << h << “ hour(s) ” << m << “ minute(s) ” << s << “ second(s) ”
<< endl;
}
Output (incorrect) 10000 second(s) = 2 hour(s) 166 minute(s) 40 second(s)
Annotation Forgot to subtract the number of hours in calculating the number of minutes
Program‐B Code (incorrect) void main( )
(faulty) {
int h, s, m;
cout << “Please enter the number of seconds: ” ;
cin >> s;
h = s / 3600;
m = (s % 3600) / 60;
s = (s % 3600) % 60;
cout << s << “ second(s) = ” << h << “ hour(s) ” << m << “ minute(s) ” << s << “ second(s) ” <<
endl;
}
Output (incorrect) 40 second(s) = 2 hour(s) 46 minute(s) 40 second(s)
Annotation Wrong use of the same variable for the number of seconds in both input and output

information innovatively, not only to better understand the degree of
engagement and performance of their students, but also to provide
additional stimuli that motivate students to learn (Wang & Wong,
2007). He/she may, for instance, boost the competitive atmosphere
of a laboratory class by showing them how many students in the class
have currently solved the given problem, or he/she may encourage
students to do more practice by revealing to them the real‐time sta-
tistics of submissions. Indeed, the statistics can be drilled down to
each specific student such that the instructor can easily identify
exactly which students are lagging behind and, accordingly, offer
timely personal advice and individual assistance. Such specific tailored
assistance, however, requires substantial manpower resources, which
has become increasingly difficult to achieve in practice as the class
size grows. It is partly due to such needs that ViDA was developed
to extend the capability of PASS.
4.2 | Overview of ViDA
In this section, we give a demonstration of how ViDA works. Figure 2
shows the interface of ViDA integrated in PASS. To assign an exercise
for students to practice, the instructor may select an existing
programming problem from the problem repository of PASS for
students to solve. Students can then attempt to solve the problem by
writing programs and submitting them to PASS for testing their
correctness (Figure 2(a)). PASS will automatically test the submission
using a number of instructor predefined test cases and, for each test
case, compare the program output and the expected output. Referring
to the sample Submission History screen in Figure 2(a), for example, the
student (with UserID 51112233) has attempted the problem (Problem
2) called Classify_triangle four times. Classify_triangle is a well‐known
programming problem used by Myers (1979) to illustrate the common
inadequacy of test suites constructed by professional programmers at
the time when he authored his classic book on software testing.
Figure 2(a) shows that, out of a total of 16 test cases, the student's
program produces correct outputs for 0, 10, 14, and 16 test cases,
respectively, in four different attempts. If enabled, ViDA will be auto-
matically invoked when the student's program fails a test case. If the
program's output matches a common wrong output, ViDA will provide
the feedback in the Test Results table, as shown in Figure 2(b). Each row
below the header row of the Test Results table corresponds to a test
case. Reading from the left, each column represents, respectively, the
(a) test case number, (b) input values, (c) annotation with hints of the
corresponding common bugs (where applicable), (d) expected output
according to the problem specification, (e) actual output produced by
the student's program, and (f) result of correctness of the actual
output. Note that a character of the expected output in each row of
6 LEE
FIGURE 1 Some online information provided by Programming Assignment aSsessment System (PASS) to the course instructor [Colour figure can
be viewed at wileyonlinelibrary.com]
a failed test case will be highlighted by PASS (in red with yellow
background), indicating exactly the first position where the expected
output deviates from the actual output.
Problem 2. Classify_triangle
Write a program, classifyTriangle.c, which
reads three integer values from the user. The three
input values are interpreted as representing the
lengths of the sides of a triangle. The program prints
a message saying whether the triangle is equilateral
(all sides equal), isosceles (only two sides equal),
scalene (all sides unequal), or impossible (cannot
form a triangle). A triangle can be formed only if the
lengths of all the sides of the triangle are positive and
it satisfies the triangle inequality, that is, the sum of
the length of any two sides is greater than the length
of the third side. For example, when the input values
are “2 3 4,” the output should be the word “scalene”
(excluding the quotation marks in both the input and
output).
In Figure 2(b), the actual output of the submitted program for the
first test case (No. 0) is the same as the expected output “scalene”
and hence, the result is correct (“Accepted”). For the second test
case (No. 1), the expected output is “equilateral” and the actual
output is “isosceles” (Note that PASS automatically highlights the
first character of the expected output “e” which is the first one that
differs from the actual output.). From our past statistics, the top com-
mon bug causing a program to fail this test case and produce this
wrong output lies in the comparison of all three lengths of the trian-
gle within a single condition. ViDA therefore displays the following
debugging advice to the student in the Annotation column of the Test
Results table:
ET AL.
Common bugs causing a program to fail this test case:
*** comparing more than two numbers in a single condi-
tion, e.g.,
“if (a == b == c)” should be written as “if (a == b &&
b == c)”
In this case, the annotation not only advises what the bug proba-
bly is, but also tells the student a specific way of how the bug can be
corrected. With this debugging advice, the student can then examine
the program code and debug accordingly if such a bug indeed exists
in the code.
For the third test case (No. 2), the program produces the correct
output “isosceles” and therefore, no annotation is displayed. For
the fourth test case (No. 3), the program fails again, producing the
incorrect actual output “isosceles” and the expected output is
“impossible” (This time, the two outputs first differ in the second
character, and hence, PASS highlights the character “m” of the
expected output.). From our past statistics, several common bugs
may cause the program to fail this test case and produce exactly this
wrong output. Thus, ViDA displays hints of the top three common
bugs to the student as follows, using the number of asterisks to indi-
cate the relative frequency of each bug's occurrence in the past:
Common bugs causing a program to fail this test case:
*** Missing or extra equality sign in the condition for
validation of triangle inequality
** Missing validation of triangle inequality
* Classified and outputted the triangle type before
validation of triangle inequality
If the wrong output produced by the student's program does not
match any of the common wrong outputs in the repository of ViDA,
the instructor may optionally leave the annotation blank or, perhaps
LEE ET AL. 7

FIGURE 2 Sample screenshots from PASS with Virtual Debugging Advisor (ViDA) enabled or disabled. (a) Sample Submission History screen: The
student may submit a program many times for testing its correctness. (b) Sample Test Results screen: The student's program fails some test
cases* and ViDA is enabled. (c) Sample Test Results screen: The student's program fails some test cases* and ViDA is disabled. (d) Sample Test Results
screen (the same with ViDA enabled or disabled): The student's program is correct*. *Only the results of executing the first four test cases are shown
here [Colour figure can be viewed at wileyonlinelibrary.com]
8 LEE
better still, state the specific purpose of the test case to advise the stu-
dent which part of the program logic was likely to fail, and therefore,
should be carefully examined in detail. The annotation is also left blank
if ViDA has not been enabled, as shown in Figure 2(c).
Finally, for those test cases for which the student's program pro-
duces the correct output, ViDA will not be invoked (no matter whether
ViDA is enabled or disabled) as PASS will inform the student that the
result is “Accepted” as shown in Figure 2(d).
4.3 | System architecture of ViDA
Figure 3 shows the generic system architecture of ViDA. Built on top
of the Web‐based PASS (Yu et al., 2006), ViDA includes several mod-
ules that interact closely with two modules (namely, Program Execution
Manager and Submission History Manager) of PASS. Whereas PASS
currently supports C, C++, and Java, ViDA and its architecture are both
independent of the programming languages used. Hence, ViDA
supports any new programming language as long as the corresponding
compiler is incorporated in PASS. Interested readers may refer to the
paper in Choy et al. (2008) for a description of the architecture and
modules of PASS. The modules of ViDA and their interactions are
depicted in Figure 3 and described as follows.
• Test Case Editor/Upload provides an interface for instructors to
edit or upload test cases.
• Result Processor processes the result of testing the student's sub-
mitted program by Program Execution Manager in PASS, extracts
from Annotation Repository the annotations associated with the
wrong outputs, and returns instant feedback to the student (as
shown in Figure 2).
• Submission History Accumulator assembles all submission details
(e.g., timestamp, program name, test results) for storage and later
processing by Submission History Manager in PASS.
• Submission Post‐Processor analyses the submission records (which
include the corresponding test results) that are retrieved by Sub-
mission History Manager to extract common wrong outputs for
instructors to identify common bugs and devise appropriate anno-
tations as debugging advice.
FIGURE 3 System architecture of Virtual Debugging Advisor (ViDA). *The two modules, Program Execution Manager and Submission History
Manager, are part of Programming Assignment aSsessment System (PASS) on which ViDA is built [Colour figure can be viewed at
wileyonlinelibrary.com]
ET AL.
• Annotation Repository collects and stores, for each test case, the
sets of common wrong outputs and associated annotations
devised by the instructor for automatic retrieval by Result
Processor.
4.4 | Devising debugging advice with ViDA
We now describe the process through which ViDA helps the instructor
identify common bugs from the records of past submissions for devis-
ing annotations that serve as debugging advice to students.
4.4.1 | Preparation of problems and test cases
During the semester, through Test Case Editor/Upload, the instructor
may select or modify an existing programming problem from the prob-
lem repository of PASS, or upload a new problem, together with an
associated set of test cases, for students to work for practice.
4.4.2 | Program output analysis
When a student submits a program to solve a problem, PASS tests it
and sends the test results to Result Processor. For each test case, Result
Processor compares the program output with the expected output pro-
vided by the instructor. If they match, the program is said to pass this
test case, that is, the test result is “Accepted” as shown in Figure 2(d).
Otherwise, Annotation Repository tries to match the wrong output
against the stored common wrong outputs of this test case. When a
match is found, the annotation associated with the wrong output is
displayed by Result Processor to the student together with the
correctness result, that is, the result is “Wrong Answer” as shown in
Figure 2(b). Meanwhile, Submission History Accumulator records this
submission and its wrong outputs for subsequent analysis.
4.4.3 | Analysis and extraction of common bugs
At the end of a semester, the instructor may use ViDA to analyse
students' submissions and wrong outputs of each problem. For each
test case, the system groups the submissions with the same or similar
wrong outputs and analyses the number of submissions in each group,
which is done by Submission Post‐Processor. For each group with a large
LEE ET AL.
number of submissions, the instructor has to determine the existence
of a common bug among all these submissions in the group. This can
be done manually while the instructor is reading the code during the
grading process. For each identified common bug, the instructor
devises a useful annotation for guiding students to fix the bug. The pair
of common wrong output and its associated annotation is kept by
Annotation Repository. Instructors may also use Annotation Repository
to find out any trends in program bugs for them to prepare courseware
and further practice that meet students' need.
To summarize this section, ViDA is designed to improve teaching
and learning by facilitating
• instructors to analyse student submissions to identify common
wrong outputs and common bugs,
• instructors to devise relevant and useful annotations (based on the
association of common wrong outputs and common bugs) and
store them for automatic retrieval, and
• students to test their programs against the set of instructor‐pre-
pared test cases, and then receive instant feedback (debugging
advice) when the program fails certain test cases.
5 | EVALUATION: EXPERIMENT
To evaluate the success of ViDA, we pose two specific research
questions:
• Did ViDA assist more students to write correct programs after
reading the debugging advice?
• Did students find ViDA helpful in their learning?
To answer the first question, we conducted an experiment in the
controlled setting of practical laboratory classes in a course. To answer
the second question, we conducted a post‐experiment questionnaire
survey to study students' perception. We describe the experiment
setup and present its results in this section, and report the design
and results of the questionnaire survey in Section 6.
5.1 | Experiment setup
Our experimental subjects were first year engineering students of the
university who were taking an introductory computer programming
course. The students had no formal training in programming before
attending the course. The classes were conducted by the same instruc-
tors as those in the past years. As before, during the first two labora-
tory classes, students spent some time to get familiar with the
operations of the program development tools (such as the editor,
compiler, and PASS) through writing “Hello world” programs and the
like. In the third and fourth laboratory classes (each lasting for an hour),
students were asked to work on the two typical programming prob-
lems selected from the problem repository of PASS, namely, Problem
1 Convert_time (see Section 3) and Problem 2 Classify_triangle (see
Section 4.2), respectively.
9
Students were required to solve the problems and submit their
programs to PASS as soon as they thought their programs were correct
within the 1‐hr laboratory classes. This controlled setting could elimi-
nate, as far as possible, other factors that might affect the evaluation.
For example, we could ensure that the submissions were entirely the
students' own work within the same amount of class time. PASS would
verify the correctness of the submitted programs using the predefined
test cases and provide instant feedback to students. When there were
failed test cases, students could submit their revised programs again
after debugging and repeat the submit–debug cycle as long as they
needed (see Figure 2(a)). No special arrangements were made to pre-
pare students for the possible use of the ViDA tool. When invoked,
ViDA was simply seen as one of many functionalities of PASS.
In order to investigate the effect of the use of ViDA, we adopted a
crossover experiment design (Senn, 2002). We logically divide the stu-
dents into two groups, namely Groups A and B, as follows (see Table 3).
This arrangement was designed to contain the possible effects of the
confounding factors of the different students involved and different
problems used in the study.
• Group A worked on Problem 1 Convert_time with ViDA enabled
and Problem 2 Classify_triangle with ViDA disabled.
• Group B worked on Problem 1 Convert_time with ViDA disabled
and Problem 2 Classify_triangle with ViDA enabled.
In the course, students were enrolled into different laboratory ses-
sions at their own will, of which we had no control. For practical class
administrative reasons, all students would have to work on the same
problem in the same week. Also, all students within the same labora-
tory session would have to work with the same ViDA setting (enabled
or disabled) in the PASS environment to avoid confusion and complica-
tions, such as a student finding another student's screen different from
his/hers. Under these practical constraints, we randomly assigned
seven laboratory sessions to Group A and the other eight laboratory
sessions to Group B. The total number of students enrolled in the lab-
oratory sessions of Group A was 159 and that of Group B was 210,
respectively. They were required to work on Problem 1 Convert_time
in Week 3 and Problem 2 Classify_triangle in Week 4. Students in
Group A worked on PASS with ViDA enabled in Week 3 but disabled
in Week 4, and vice versa for those in Group B. All submissions and
records were collected in PASS for analysis. Table 3 summarizes the
(logical) arrangements of Groups A and B in the 2 weeks.
When a group worked with ViDA enabled, we refer it as the exper-
imental group to compare the effect of ViDA in contrast to the other
group, the corresponding control group, who worked with ViDA dis-
abled. Thus, in Week 3, Group A was the experimental group, and
Group B was the control group, and vice versa in Week 4. Note that
both the grouping into A and B or experimental versus control were
TABLE 3 Logical grouping of students in the experiment
Number of enrolled (Week 3) (Week 4)
students Convert_time Classify_triangle
Group A 159 ViDA enabled (✓) ViDA disabled (✗)
Group B 210 ViDA disabled (✗) ViDA enabled (✓)
Note. ViDA = Virtual Debugging Advisor.
10
purely done logically for the purpose of analysis of data and results.
Students themselves were not aware of which groups they belonged
to as they simply attended the laboratory sessions and worked on
the two problems as class exercises in the course.
5.2 | Experiment results
At the end of the 2 weeks, 89 students of Group A and 149 students
of Group B submitted programs for both problems in class. Some
enrolled students were absent in one or both of the laboratory ses-
sions and some did not submit their solutions owing to a variety of
possible reasons (such as insufficient time or inability to keep to the
progress) within one or both of the laboratory sessions. These stu-
dents were asked to continue working on the problems for practice
after classes, but their late submissions were not included in our sub-
sequent analysis of results. In any case, the two exercises were for
practice purposes only and did not contribute to their formal assess-
ment in the course, that is, there was no penalty for not submitting
their work.
During the laboratory classes, when a correct program was sub-
mitted by the student, no annotation would be provided regardless
of whether ViDA was enabled or not (see Figure 2(d)). On the other
hand, in case of incorrect programs submitted, only students who
worked with ViDA enabled would be provided relevant debugging
advice in the Annotation column (see Figure 2(b)), and students who
worked with ViDA disabled would still see a blank Annotation column
(see Figure 2(c)).
By virtue of the experiment design, if a student produced a correct
program on the first submission to PASS, then he/she had done so
without reading any annotation provided by ViDA even if it was
enabled (Figure 2(d)). Indeed, in such a case, as the submission was cor-
rect already, the student would actually not need the help of ViDA.
Because we aimed at evaluating the effectiveness and usefulness of
ViDA, we restricted our analysis to students whose first submissions
were incorrect, so that ViDA might have had some effect. There were
33 such students out of 89 in Group A and 89 out of 149 in Group B
(Table 4).
With the aid of ViDA, students could benefit from the debugging
advice; otherwise they would not get any hints on debugging as they
TABLE 4 Summary of number of students who submitted correct/
incorrect programs
ViDA
enabled (✓)
or
Problem disabled (✗) Group s x y y/x (%)
1. Convert_time ✓ A 89 33 28 84.8
1. Convert_time ✗ B 149 89 67 75.3
2. Classify_triangle ✓ B 149 89 59 66.3
2. Classify_triangle ✗ A 89 33 20 60.6
Note. ViDA = Virtual Debugging Advisor; s = Number of students who
finally submitted programs for both problems; x = Number of students
who initially submitted incorrect programs; y = Number of students who
finally submitted correct programs after initially submitted incorrect pro-
grams; y/x = Proportion of students who finally submitted correct programs
after initially submitted incorrect programs.
LEE ET AL.
were asked to work individually. To evaluate whether ViDA assisted
students to write correct programs after reading the debugging advice,
we further examined the number of students who submitted a correct
solution at the end of the laboratory session after having submitted an
incorrect solution the first time. Among the 33 students in Group A
whose first submitted solutions were incorrect, 28 of them finally sub-
mitted correct solutions at the end of the laboratory session for
Problem 1 Convert_time with ViDA enabled, but only 20 for Problem
2 Classify_triangle with ViDA disabled. For Group B, among the 89 stu-
dents whose first submitted solutions were incorrect, 67 of them
finally submitted correct solutions at the end of the laboratory
session for Problem 1 Convert_time with ViDA disabled, and 59 for
Problem 2 Classify_triangle with ViDA enabled. All these numbers
are summarized in Table 4, together with the percentage of students
who finally submitted correct programs after initially submitted incor-
rect programs. Table 5 reorganizes these percentages and explicitly
shows that the percentages for the experimental groups (who worked
with ViDA enabled) are consistently higher than those for the corre-
sponding control groups (who worked with ViDA disabled). The differ-
ences in percentages are 9.5% for Problem 1 Convert_time and 5.7%
for Problem 2 Classify_triangle, respectively. Note that because Prob-
lem 2 Classify_triangle is considerably harder than Problem 1
Convert_time, particularly for the weaker students, a smaller percent-
age of students submitted a correct Classify_triangle program at the
end of the laboratory sessions. Figure 4 depicts the percentages in
Table 5 visually in the form of bar charts.
By way of our experimental design, the consistently positive effect
of ViDA was evidently independent of the particular problems or
TABLE 5 Percentages of students who finally submitted correct pro-
grams after initially submitted incorrect ones
Experimental group Control group with
Problem with ViDA enabled (E) ViDA disabled (C) E–C
1. Convert_time 84.8 (Group A) 75.3 (Group B) 9.5
2. Classify_triangle 66.3 (Group B) 60.6 (Group A) 5.7
Note. ViDA = Virtual Debugging Advisor.
FIGURE 4 Percentages of students in Table 4 shown as bar charts
[Colour figure can be viewed at wileyonlinelibrary.com]
LEE ET AL. 11

students involved. Hence, the experimental results confirmed that a
higher proportion of students could correct their initially faulty pro-
grams when ViDA was enabled.
6 | EVALUATION: SURVEY
To answer the research question (“Did students find ViDA helpful in
their learning?”), we requested students to complete a questionnaire
anonymously after the laboratory sessions. In this section, we first
presented the questions in the questionnaire and then a summary
and analysis of the responses.
6.1 | Survey questions
The questionnaire consists of 10 questions as follows. Each of the first
nine questions contains a statement, and the responses are in a 5‐point
Likert scale, in which scores of 5, 4, 3, 2, and 1 refer to strongly agree,
agree, neutral, disagree, and strongly disagree, respectively. The 10th
question was set to solicit students' general comments in an open‐
ended format.
1. I read most of the annotations.
2. The annotations are easy to understand.
3. The annotations make me aware of common bugs/mistakes.
4. I made similar bugs/mistakes before.
5. The annotations make debugging easier.
6. The annotations help me understand the purpose of each
test case.
7. The annotations help me understand the program requirements.
8. The annotations are helpful for learning programming.
9. I want PASS to provide annotations on the common bugs/
mistakes.
10. Other comment(s).
Some of the students did not encounter the annotations provided
by ViDA (e.g., if their first submissions to PASS were already correct).
Because the questionnaires were anonymous, we had no way of
knowing which students did read the annotations and which did not.
Question 1 was therefore designed as a control to distinguish between
students who did or did not read the annotations. Questions 2 to 8
were designed to find out whether students perceived that the anno-
tations might have been helpful to them in various possible specific
ways. For example, Questions 3 and 4 were designed to find out
whether the annotations had increased their awareness of others' bugs
and their own bugs. Question 9 served as an overall indicator of
whether students would like ViDA enabled, regardless of in what spe-
cific way they might find it useful or not. Question 10 provided space
for open‐ended comments of any issue students would like to raise.
Our subsequent analysis of students' responses only included the
questionnaires in which the students responded with strongly agree or
agree to Question 1. We received 166 completed questionnaires with
such responses. Because the questionnaire was anonymous, there was
no way to identify which respondents submitted a correct solution the
first time, at the end of the class, or none at all. As such, the question-
naires were essentially analysed independently of the experiment.
6.2 | Summary and analysis of survey responses
Table 6 summarizes the individual percentages of each score for each
of Questions 2 to 9, whereas Figure 5 depicts these percentages
graphically. First, we note that for all the questions, the responses
were mostly strongly agree and agree. In particular, we observe that
• 57.8% of the respondents agreed that they made similar bugs/mis-
takes before (Question 4).
• 78.4% of the respondents agreed that the annotations made
debugging easier (Question 5).
• 83.2% of the respondents agreed that the annotations were help-
ful for learning programming (Question 8).
• 77.7% of the respondents wanted PASS to provide annotations on
the common bugs/mistakes (Question 9).
We conducted statistical tests on the data collected from the sur-
vey. We first calculated the reliability coefficient (α; also known as
Cronbach's α) of the survey. It essentially measures the internal consis-
tency and is commonly used when multiple Likert questions are set in a
survey to determine if the scale is reliable. The α of the survey was
0.821. Values of α greater than 0.70 are considered statistically

TABLE 6 Percentages of responses to each question

Percentages of responsesa of each scoreb
Question 5 4 3 2 1

2. The annotations are easy to understand. 12.7 71.1 16.3 0.0 0.0
3. The annotations make me aware of common bugs/mistakes. 13.3 63.3 19.9 3.6 0.0
4. I made similar bugs/mistakes before. 10.2 47.6 34.9 6.6 0.6
5. The annotations make debugging easier. 13.3 65.1 18.1 3.0 0.6
6. The annotations help me understand the purpose of each test case. 13.3 54.2 28.9 3.6 0.0
7. The annotations help me understand the program requirements. 13.3 62.7 22.3 1.2 0.6
8. The annotations are helpful for learning programming. 15.7 67.5 16.3 0.0 0.6
9. I want PASS to provide annotations on the common bugs/mistakes. 22.3 55.4 21.1 1.2 0.0
a b
The total percentages for each question may not be equal to 100 due to rounding. The scores are: 5 = strongly agree, 4 = agree, 3 = neutral, 2 = disagree,
1 = strongly disagree.
12 LEE ET AL.

FIGURE 5 Summary of responses to

individual questions in the survey [Colour
figure can be viewed at wileyonlinelibrary.
com]

significant (Johnson & Wichern, 1998). Hence, the results from the sur-
vey are reliable.
In addition, we conducted statistical hypothesis testing to inves-
tigate the students' perception of ViDA. For each question in the sur-
vey, we conducted one‐sample t test, using 3.5 as discerning point
(test value) for mean score, to evaluate significance. The null hypoth-
esis is that the mean score is equal to 3.5. Table 7 shows the results
of the tests, including the mean, standard deviation, t value, and p
value. A small p value (<.05) indicates that the null hypothesis should
be rejected.
There were several interesting findings from the results. First, the
mean score of every question is statistically significantly higher than
3.5 (with p value less than .05 or even .001). In particular, Questions
8 and 9 receive the two highest scores of 3.98 and 3.99, respectively.
This shows that students found ViDA helpful for learning and would
like to have ViDA enabled when working on their programming exer-
cises. Second, the results of Questions 6 and 7 show that students
agreed that ViDA could help them better understand the purpose of
the test cases and program requirements. Overall, the majority of the
respondents agreed with the statements in Table 7.
Finally, regarding Question 10, we received a large number of
written comments. Consistent with their responses to the preceding
questions, an overwhelming number of students wrote favourable
welcomed ViDA. A complete list of written comments is included in
Appendix A of this paper.
• As a computer programming new learner, it's quite hard for me to
correct my mistakes if no annotation is provided. I can easily find
out [from ViDA] what my common mistakes are and benefit a lot
from it.
• Can try to understand the mistakes if I am at home.
• I can understand and [am] aware [of] more about the error that I will
make.
• If PASS [ViDA] can provide annotation, we can know the common
mistake even [if] we didn't make this mistake this time.
• Some of the mistakes can hardly be found by ourselves.
• The annotations help students to [be] aware [of] common bugs/mis-
takes. Furthermore, the annotations help students easier to finish
the program to match the requirements by PASS [ViDA].
• PASS [ViDA] annotation can help me understand the program and
check the program.
A handful of students explicitly chose to stay ambivalent:
• Neutral

comments on ViDA. Below are some of their comments on why they • No comment

TABLE 7 Statistical t test analysis of survey responses

Statisticsa of the scoresb
Question Mean Standard deviation t valuea p valuea

2. The annotations are easy to understand. 3.96 0.54 11.11 <.001**

3. The annotations make me aware of common bugs/mistakes. 3.86 0.68 6.87 <.001**
4. I made similar bugs/mistakes before. 3.60 0.79 1.68 .047*
5. The annotations make debugging easier. 3.87 0.69 6.98 <.001**
6. The annotations help me understand the purpose of each test case. 3.77 0.72 4.86 <.001**
7. The annotations help me understand the program requirements. 3.87 0.67 7.11 <.001**
8. The annotations are helpful for learning programming. 3.98 0.61 10.01 <.001**
9. I want PASS to provide annotations on the common bugs/mistakes. 3.99 0.70 9.03 <.001**
a
The t value and p value are computed by using the one sample t test. bThe scores are: 5 = strongly agree, 4 = agree, 3 = neutral, 2 = disagree, 1 = strongly
disagree.
*p < .05, **p < .01.
LEE ET AL.
On the other hand, a few other students would like to see that the
system improve in efficiency and usability.
• PASS system should be fast more.
• The system is a little stubborn sometimes.
It is clear that many students did require assistance in debugging
and, unlike ViDA, the helpdesk cannot provide instant service around
the clock. In this regard, we believe that ViDA would be especially
helpful to the weaker students so that they would not be undesirably
discouraged during their course of progress in learning to write code.
Overall, there is supportive evidence from both numerical statis-
tics and qualitative comments that students perceived that the use of
ViDA helped to improve their learning experience.
7 | C O N CL U S I O N
This paper describes a ViDA built on an automated PASS to provide
debugging advice to students to facilitate their learning of computer
programming. ViDA enables students to learn effectively on their
own by analysing their program outputs automatically and, in light of
common bugs of the past, suggesting probable mistakes they might
have made. Such feedback not only helps students identify and correct
bugs, but also allows them to better understand the program require-
ments and the purpose of each test case by knowing the cause of
errors. The latter ability is also essential for both program testing and
debugging. Although ViDA is a customized tool that integrates with
PASS only, its architecture and process are generic that can be applied
to other automated program assessment tools and environments in
other universities.
This paper also reports the evaluation of the effectiveness and
usefulness of ViDA by means of a controlled experiment and a student
survey. The findings are encouraging, showing that a higher proportion
of students could correct their faulty programs with ViDA enabled and
the majority of students found ViDA helpful for learning computer pro-
gramming. Indeed, instructors have made anecdotal observations that
students generally seem to be better motivated and more persistent
in working with the programming exercises, given the instant
debugging guidance now available when they learned on their own.
ViDA was designed to offer students automated individual assis-
tance so that their learning of computer programming will not be
unduly obstructed by their weakness in bug finding. Precisely what
form and amount of debugging advice are conducive to proper learning
may need further research, but ultimately, the instructors, who know
the students best, have to decide what exact information should be
put into the annotation to be provided by ViDA. The proposed frame-
work of ViDA allows sufficient flexibility for the instructors to custom-
ize the annotations to their best need.
Our study provides several implications in practice of learning and
teaching. Our study shows that effort should be made to help students
in learning debugging, so as to upkeep students' motivation in learning
programming. Providing timely and useful advice is proved to be effec-
tive to engage students in their learning in many different literatures.
Our study substantiates the findings of the usefulness of providing
13
timely and customized debugging advice to students when learning
programming. Instead of simply checking the “all‐or‐nothing” correct-
ness of students' programs by comparing the actual program output
of students' submission to the expected output in test cases, instruc-
tors should consider adopting automated or semi‐automated means
of generating customized feedback based on individual students' work.
This customized feedback can significantly enhance students' learning
experience and avoid them from getting frustration during debugging.
Another implication is that ViDA can be utilized in Massive Open
Online Courses (MOOCs) related to programming learning
(Vihavainen, Luukkainen, & Kurhila, 2012). People from over the world
can enrol to any MOOCs anytime and anywhere. However, the high
dropout rate of MOOCs attracts much attention to researchers, course
providers, developers, and instructors. Onah, Sinclair, and Boyatt
(2014) discussed that one of the possible reason for the dropout from
a computer programming MOOC is the lack of support. ViDA can pro-
vide guidance for learners to practice their programming skills in a vir-
tual environment.
To conclude, we believe that, by providing instant and customized
feedback, our system is helpful for students to overcome the initial
hurdles they encounter in learning computer programming, rendering
their learning experience more enjoyable and increasing their
motivation.
ACKNOWLEDGEMENT
The work described in this paper is fully supported by a grant from the
Research Grants Council of the Hong Kong Special Administrative
Region, China (Project UGC/FDS11/E02/15).
ORCID
V.C.S. Lee http://orcid.org/0000-0003-2596-1257
Y.T. Yu http://orcid.org/0000-0002-6548-5446
C.M. Tang http://orcid.org/0000-0003-2842-6196
T.L. Wong http://orcid.org/0000-0002-4581-9912
C.K. Poon http://orcid.org/0000-0002-8508-5060
RE FE RE NC ES
Ahmadzadeh, M., Elliman, D., & Higgins, C. (2005). An analysis of patterns
of debugging among novice computer science students. Proceedings of
the 10th Annual SIGCSE Conference on Innovation and Technology in
Computer Science Education, pp. 84‐88.
Ala‐Mutka, K. M. (2005). A survey of automated assessment approaches for
programming assignments. Computer Science Education, 15(2), 83–102.
Allevato, A., & Edwards, S. H. (2014). Dereferee: Instrumenting C++
pointers with meaningful runtime diagnostics. Software: Practice and
Experience, 44(8), 973–997. https://doi.org/10.1002/spe.2184.
Atkinson, R. C., & Shiffrin, R. M. (1968). Human memory: A proposed
system and its control processes. The Psychology of Learning and
Motivation: Advances in Research and Theory, 3, 89–195.
Baddeley, A., & Hitch, G. J. (1974). Working memory. Recent Advances in
Learning and Motivation, 8, 47–90.
Caso, G., Garbervetsky, D., & Gorín, D. (2013). Integrated program
verification tools in education. Software: Practice and Experience, 43(4),
403–418. https://doi.org/10.1002/spe.2143.
Choy, M., Nazir, U., Poon, C. K., & Yu, Y. T. (2005). Experiences in using
an automated system for improving students' learning of computer
programming. Lecture Notes in Computer Science, 3583, 267–272.
14
Choy, M., Lam, S., Poon, C. K., Wang, F. L., Yu, Y. T., & Yuen, L. (2008).
Design and implementation of an automated system for assessment
of computer programming assignments. Lecture Notes in Computer
Science, 4823, 584–596.
Douce, C., Livingstone, D., & Orwell, J. (2005). Automatic test‐based
assessment of programming: A review. ACM Journal of Educational
Resources in Computing, 5(3), Article 4.
Gómez‐Albarrán, M. (2005). The teaching and learning of programming:
A survey of supporting software tools. The Computer Journal, 48(2),
130–144.
Guzdial, M., & Guo, P. (2014). The difficulty of teaching programming
languages, and the benefits of hands‐on learning. Communications of
the ACM, 57(7), 10–11.
Higgins, C. A., Gray, G., Symeonidis, P., & Tsintsifas, A. (2005). Automated
assessment and experiences of teaching programming. ACM Journal
on Educational Resources in Computing, 5(3), Article 5.
Hollingsworth, J. (1960). Automatic graders for programming classes.
Communications of the ACM, 3(10), 528–529.
Hooshyar, D., Ahmad, R. B., Yousefi, M., Yusop, F. D., & Horng, S.‐J. (2015).
A flowchart‐based intelligent tutoring system for improving problem‐
solving skills of novice programmers. Journal of Computer Assisted
Learning, 31(4), 435–449.
Ihantola, P., Ahoniemi, T., Karavirta, V. & Seppälä, O. (2010). Review of
recent systems for automatic assessment of programming assignments.
Proceedings of the 10th Koli Calling International Conference on Comput-
ing Education Research, pp. 86–93.
Johnson, R. A., & Wichern, D. W. (1998). Applied multivariate statistical anal-
ysis. USA: Prentice‐Hall International Inc.
Joy, M., Griffiths, N., & Royatt, R. (2005). The BOSS online submission and
assessment system. ACM Journal on Educational Resources in Computing,
5(3), Article 2.
Katai, Z. (2015). The challenge of promoting algorithmic thinking of both
sciences‐ and humanities‐oriented learners. Journal of Computer
Assisted Learning, 31(4), 287–299.
Kelleher, C., & Pausch, R. (2005). Lowering the barriers to programming: A
taxonomy of programming environments and languages for novice pro-
grammers. ACM Computing Surveys, 37(2), 83–137.
Lahtinen, E., Ala‐Mutka, K. & Järvinen, H. (2005). A study of the difficulties
of novice programmers. Proceedings of the 10th Annual SIGCSE Confer-
ence on Innovation and Technology in Computer Science Education,
pp. 14‐18.
Lam, M. S. W., Chan, E. Y. K., Lee, V. C. S., & Yu, Y. T. (2008). Designing an
automatic debugging assistant for improving the learning of computer
programming. Lecture Notes in Computer Science, 5169, 359–370.
Law, K. M. Y., Lee, V. C. S., & Yu, Y. T. (2010). Learning motivation in
e‐learning facilitated computer programming courses. Computers &
Education, 55(1), 218–228.
Lee, G. C., & Wu, J. C. (1999). Debug It: A debugging practicing system.
Computers & Education, 32(2), 165–179.
Luck, M., & Joy, M. (1999). A secure on‐line submission system. Software:
Practice and Experience, 29(8), 721–740.
Moons, J., & De Backer, C. (2013). The design and pilot evaluation of an
interactive learning environment for introductory programming influ-
enced by cognitive load theory and constructivism. Computers &
Education, 60(1), 368–384.
Morris, D. S. (2003). Automatic grading of student's programming assign-
ments: An interactive process and suite of programs. Proceedings of
the 33rd ASEE/IEEE Frontiers in Education Conference, pp. S3F‐1–6.
Myers, G. J. (1979). The art of software testing (1st ed.). New Jersey: Wiley.
Odekirk‐Hash, E. & Zachary, J. L. (2001). Automated feedback on programs
means students need less help from teachers. Proceedings of the 32nd
SIGCSE Technical Symposium on Computer Science Education, pp. 55–59.
Onah, D. F. O., Sinclair, J. & Boyatt, R. (2014). Dropout rates of massive
open online courses: Behavioural patterns. Proceedings of the 6th
LEE ET AL.
International Conference on Education and New Learning Technologies,
pp. 5825–5834.
Paiva, R. C., Ferreira, M. S., & Frade, M. M. (2017). Intelligent tutorial sys-
tem based on personalized system of instruction to teach or remind
mathematical concepts. Journal of Computer Assisted Learning, 33(4),
370–381.
Parnin, C. & Orso, A. (2011). Are automated debugging techniques actually
helping programmers? Proceedings of the 2011 International Symposium
on Software Testing and Analysis, pp. 199–209.
Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., …
Paterson, J. (2007). A survey of literature on the teaching of introduc-
tory programming. ACM SIGCSE Bulletin, 39(4), 204–223.
Pearson Education Inc. (July 2015). MyProgrammingLab. Retrieved from
http://www.myprogramminglab.com/.
Poon, C. K., Wong, T. L., Yu, Y. T., Lee, C. S. & Tang, C. M. (2016). Toward
more robust automatic analysis of student program outputs for assess-
ment and learning. Proceedings of the 2016 IEEE 40th Annual Computer
Software and Applications Conference (COMPSAC), pp. 780–785.
Rae, A., & Samuels, P. (2011). Web‐based personalised system of instruc-
tion: An effective approach for diverse cohorts with virtual learning
environments? Computers & Education, 57, 2423–2431.
Satratzemi, M., Dagdilelis, V. & Evagelidis, G. (2001). A system for program
visualization and problem‐solving path assessment of novice program-
mers. Proceedings of the 6th Annual Conference on Innovation and
Technology in Computer Science Education, pp. 137–140.
Senn, S. (2002). Cross‐over trials in clinical research (Vol. 5). New Jersey:
John Wiley & Sons.
Tang, C. M., Chan, W. K., Yu, Y. T., & Zhang, Z. (2017). Accuracy graphs of
spectrum‐based fault localization formulas. IEEE Transactions on Reli-
ability, 66(2), 403–424.
Tremblay, G., Guérin, F., Pons, A., & Salah, A. (2008). Oto, a generic and
extensible tool for marking programming assignments. Software: Prac-
tice and Experience, 38(3), 307–333. https://doi.org/10.1002/spe.839.
Vihavainen, A., Luukkainen, M. & Kurhila, J. (2012). Multi‐faceted support
for MOOC in programming. Proceedings of the 13th Annual Conference
on Information Technology Education, pp. 171–176.
Wang, F. L. & Wong, T. L. (2007). Effective teaching and learning of
computer programming with large class size. Proceedings of the 1st
Symposium on Hybrid Learning, pp. 55‐65.
WebCAT. (July 2015). The Web‐CAT Community. Retrieved from http://
web‐cat.org/.
Yu, Y. T., Poon, C. K., & Choy, M. (2006). Experiences with PASS:
Developing and using a Programming Assignment aSsessment System.
Proceedings of the 6th International Conference on Quality Software
(QSIC), pp. 360–365.
How to cite this article: Lee VCS, Yu YT, Tang CM, Wong TL,
Poon CK. ViDA: A virtual debugging advisor for supporting
learning in computer programming courses. J Comput Assist
Learn. 2018;1–16. https://doi.org/10.1111/jcal.12238
APPENDIX A
The following are the written comments received from the survey.
Each bullet point refers to the written comment by a student:
• A good software that help student learn independent.
• Amazing!
• As a computer programming new learner, it's quite hard for me to
correct my mistakes if no annotation is provided. I can easily find
out what my common mistakes are and benefit a lot from it.
LEE ET AL. 15

• Because I can know what's wrong with my program. • It makes life easier.
• Because I can learn much easier. • It makes little difference.
• Because I want to know what the mistakes are and I want to • It makes the job of debugging by hand easier.
improve my skills. • It makes users debug the program more easily.
• Because it helps me make debugging easier. • It will lag during in the tutorial lesson.
• Can try to understand the mistakes if I am at home. • It's easy to use.
• Cheer up. • It's good to learn from mistakes.
• Easy to correct my mistakes. • It's useful for me to avoid some common bugs.
• Easy to test. • Let me easy to know what I am done wrong.
• Easy to understand. • Neutral.
• Facilitate learning. • No comment.
• Finish the tutorial. • Not enough time to do classwork.
• For the sake of user friendliness, this makes programming, • Not just to know what the program is wrong, but to revise the
debugging exactly, more easy. detail of programming.
• Give me comparison. • PASS annotation can help me understand the program and check
• Help me to find out my mistakes quickly. the program.
• Helpful. • PASS is useful to check the problems of the program.
• I agree because I can notice the mistake I made very soon. • PASS system should be fast more.

• I can find my mistakes easily. • Prevent the common mistakes perform again.
• I can find out my mistake easier. • Quite user friendly.
• I can know where the problems are. • So I'll not make that mistakes again.
• I can understand and aware more about the error that I will make. • Some of the mistakes can hardly be found by ourselves.

• I could know more about the common bugs. • Students can learn from mistakes and will not make those mistakes
• I don't mind if PASS do so. next time.

• I forgot to, or I don't even know the logic behind like a + b <= c • Take reference and avoid making those mistakes in the future.

• I know the mistakes.
• If PASS can provide annotation, we can know the common
mistake even we didn't make this mistake this time.
• It can give me more opinions.
• It can help me to learn more common mistakes.
• It can help me understand the program logic.
• It can make me easier to understand my mistakes.
• It can suggest me to make corrections.
• It can tell me what is exactly required in the task.
• It gives details about my mistakes and help me to improve.
• It helps me to learn faster.
• It is because some careless mistakes that may not cause a bug to
the program cannot debug by the program, but PASS can do it
and find it out. This lead us to write more accurate and perfect
programs.
• It is comparatively abstract for debugging details.
• It is convenient.
• It is easier for me to remember.
• It is helpful for learning programming.
• The PASS do help me to find out what's wrong with my
programme.
• The PASS is quite not user friendly. It always told me I am wrong
because of a space.
• The PASS system help me make the format of program better.
• The PASS system input the difference situation to test the
program. Let me find bugs easily.
• The PASS system is useful for checking outcome.
• The annotation always remind me of spaces which are useless in
code writing.
• The annotation is more understandable and user friendly.
• The annotations can let me know easily what bugs would be made.
• The annotations help students to aware common bugs/mistakes.
Furthermore, the annotations help students easier to finish the
program to match the requirements by PASS.
• The helper is very useful, as doing a program face a lot of
problem, by reading the notes which cannot tell you the practical
situation.
• These annotations give useful advise which really helpful for
debugging.

• It is helpful in debugging. • This make me easier to understand the problems.

• It is useful to improve my skill. • To make sure not make mistakes again.

16 LEE ET AL.

• To prevent simple mistakes. • We usually make the alike common mistakes.

• Tutor can help me. • Because it is convenient.
• Useful. • Easy to debug.
• Useful. • Easy to identify the bugs.
• User friendly, and correct my ans if I don't know the answer. • Easy to understand the mistakes & take the correction.

• Very good. • Easy to understand what is my mistake.

• Very lag. • Help to understand.
• We can avoid making the common mistakes. • The system is a little stubborn sometimes.
• We need to know how we make mistakes more clearly. • Very good.