See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/259723661

Levels of agreement between student and staff

assessments of clinical skills in performing
cavity preparation in artificial teeth

ARTICLE in EUROPEAN JOURNAL OF DENTAL EDUCATION FEBRUARY 2014

Impact Factor: 0.94 DOI: 10.1111/eje.12059 Source: PubMed

READS

57

5 AUTHORS, INCLUDING:

Jonathan P San Diego Jonathon Timothy Newton

King's College London King's College London
62 PUBLICATIONS 70 CITATIONS 321 PUBLICATIONS 3,317 CITATIONS

SEE PROFILE SEE PROFILE

Barry Quinn Mark Woolford

King's College London King's College London
97 PUBLICATIONS 42 CITATIONS 54 PUBLICATIONS 211 CITATIONS

SEE PROFILE SEE PROFILE

Available from: Jonathon Timothy Newton

Retrieved on: 29 January 2016
 European Journal of Dental Education ISSN 1396-5883

Levels of agreement between student and staff assessments

of clinical skills in performing cavity preparation in artificial
teeth
J. P. San Diego, T. Newton, B. F. A. Quinn, M. J. Cox and M. J. Woolford
Dental Institute, Kings College London, London, UK

keywords Abstract
levels of agreement; self-assessment; cavity
preparation in artificial teeth; dental chair Objective: To determine the level of agreement between staff and students assess-
simulators; clinical skills; haptics. ment of clinical skills in performing tasks related to cavity preparation on a traditional
dental manikin.
Correspondence
Jonathan P. San Diego
Methods: Two studies were conducted with two successive student cohorts: Study
Dental Institute
1138 year 1 BDS students in a 2009/10 cohort and Study 2135 students in a 2010/11
Kings College London
cohort. Staff members and students rated the students performance in preparing a
Floor 18, Guys Hospital, Tower Wing, London
cavity on a traditional dental manikin using artificial teeth: hand-held (Study 1) and
SE1 9RT, UK
Tel: +44(0)20 7188 1831
located in a lower jaw (Study 2).
Fax: +44 (0)20 7188 1159 A 5-item criterion-related scoring rubric was developed. The rubric assessed students
e-mail: j.p.san_diego@kcl.ac.uk abilities to hold the instrument correctly, determine the angle of entry to the tooth,
remove the caries, conserve healthy tissues and avoid pulp exposure.
Accepted: 28 May 2013
Results: Agreement between the students self-assessment and the staffs assessment
doi: 10.1111/eje.12059
was high for three of the five criteria (i.e. removal of artificial caries on the cavity wall,
removal of artificial caries from the cavity floor and avoidance of pulp exposure). Lev-
els of agreement for the remaining two criteria were moderate. A change in task diffi-
culty affected the levels of agreement between staff and students, such that the more
difficult the task, the greater the discrepancy in ratings.

Conclusions: Students tend to overrate the quality of their performance when com-
pared with staff ratings. Task difficulty has an impact on levels of agreement.

curriculum, it is often infrequent and lacking in appropriate

Introduction
Formative assessment is important in helping learners to
acquire conceptual and practical skills. As Black and Wiliam
state:
There is a body of firm evidence that formative assessment
is an essential component of classroom work and that its
development can raise standards of achievement
(1; p.12)
However, formative assessment can be time-consuming for
teaching staff, and although, as shown by Black and William (i)
and many others, it should be ongoing and frequent and is
central to effective teaching, in the Dental undergraduate
application. An important aspect of formative assessment is to
encourage students to assess their own performance as part of
an ongoing process of reflection on their learning. The advanta-
ges of such an approach would be to provide ongoing forma-
tive feedback during the learning process.
The majority of assessments in undergraduate dental pro-
grammes are undertaken by the tutor or lecturer; therefore,
to maximise the benefits of students self-assessment and to
know what types of self-assessment will complement the
tutors assessments, it is important to compare students self-
assessments and teachers assessments of their learning (1).
Sullivan and Hall (2) provide guidance for the introduction

58 2013 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd
Eur J Dent Educ 18 (2014) 5864
San Diego et al.
of students to self-assessment. Similarly, Mattheos et al. (3)
state that it is vital that students understand that self-assess-
ment plays an important role in identifying their own learn-
ing needs, understand the criteria for making the assessments
and the anticipated standards of achievement. Through facili-
tating these processes, learning can be enhanced by using
self-assessment.
It has already been established by previous studies that
teachers assessment of students performance can be used as a
baseline for the objectiveness of students self-assessments
(46). Although Holmboe (7) has shown that comparing staff
and students assessment could provide a form of validation,
there are inconsistencies among different studies regarding the
levels of agreement found between staff and students (4, 8).
Some of these disagreements can be attributed to the methods
the researchers used when measuring students and staff assess-
ments, the inconsistencies of which lead to ambiguous results.
For example, Satterthwaite and Grey (8) investigated peer-
group assessment of two pre-clinical operative skills in restor-
ative dentistry and compared these with experienced assessors
scores. No statistically significant difference was found. In
comparison with the above, Mattheos et al. (3), who com-
pared students self-assessment with that of the instructors
through the use of a questionnaire, did find a difference
between these groups. The students were asked to self-assess
their competence in 11 different knowledge and skill areas
corresponding to the learning objectives of the clinical peri-
odontology course in the third semester of the curriculum
matched to the judgment of their instructors. A total of 214
(42%) student scores were found to be higher than those
given by the clinical instructors, while 95 (19%) were lower
and 197 (39%) were in agreement. On an individual basis, the
assessments made by 18 students (39%) were significantly
higher than those made by their instructors (P < 0.05), while
the assessments made by five students (11%) were lower.
These authors concluded that student agreement with clinical
instructors judgments appears to be lower in the present
study than that reported in previous research. In addition, in
the present study, students appeared to overestimate their
competence rather than underestimate it, in contradiction to
several previous studies in the examination and student self-
assessment patterns (3, p. 387).
A possible explanation for the different findings between
these two studies and others is that each study used different
methods for the students and instructors to assess the stu-
dents learning and achievements, and the students themselves
had different levels of prior experience in self-assessment tech-
niques. Furthermore, other studies have shown that the abili-
ties of the students in the discipline will affect their abilities
to self-assess accurately. The self-assessment skills of students
being dependent on their general abilities has been confirmed
in an earlier study by Boud and Falchikov (9) who showed
that able students working in a new subject are likely to be
aware of their own deficiencies, and thus, the results of
underrating which we see are also not unexpected. Also,
unsurprising is the fact that weaker and less mature students
tend to overrate themselves, and the weaker they are, in
terms of teacher ratings, the greater the degree of overrating
(9, p. 544).
2013 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd
Eur J Dent Educ 18 (2014) 5864
Levels of agreement of student and staff assessments
The research reported in this article is part of the hapTEL
projects1 research work (10). Amongst a range of investiga-
tions, the project explored, in part, how self-assessment of clin-
ical skills can be embedded in a virtual learning system to
enhance assessment of learning. However, the research reported
here is based upon student and staff assessments of perfor-
mance on a traditional manikin dental simulator. This article
addresses the following questions:
Are there differences in staff and students self-rating with
tasks that have varying levels of difficulty?
Is there a difference between students assessments of differ-
ing tasks?
Do the rating criteria and scoring rubrics affect the staff and
student assessments?
Methods
Data on student and staff assessments of performance on a cav-
ity preparation task on a traditional dental manikin were col-
lected as part of a cluster randomised controlled trial
comparing the learning of cavity preparation skills in a phan-
tom head laboratory vs. a haptic technology simulator to work
on virtual teeth (10). The results reported here refer to the
comparison of staff and student performance on cavity prepa-
ration on a plastic tooth and in the traditional phantom head.
The results for the haptic technology use are being published in
other papers. Data were collected from two successive cohorts
of Year 1 dental students, 2009 entry and 2010 entry.
Ethical approval was granted from the Kings College Lon-
don College Research Committee (Reference CREC 06/07-222).
Design
For the whole hapTEL project (10), Year 1 dental undergradu-
ate students were divided into two groups. One group (control
group) was assigned to learn cavity preparation using phantom
head facilities with plastic artificial teeth; the other group
(experimental group) was simultaneously assigned to learn cav-
ity preparation using technology-enhanced learning on virtual
teeth, using haptic technology. Both learning environments
provided first: a single tooth held by the student in the tradi-
tional environment and displayed on the screen in the virtual
environment for the first parallel session and then a tooth
located in a lower jaw for subsequent sessions. Both cohorts of
students participated in the assessment research presented in
this article. From a total of 277 students, 269 participated from
2 year one cohorts. The 2009/2010 (Study 1) cohort comprised
138 students, and 131 were from the 2010/2011 cohort (Study
2). We assumed that the students did not have any previous
experience in performing the task involving removing artificial
caries.
In both cohorts, students were given three sessions to prac-
tise their skills. The practice sessions involved the removal of
an artificial carious lesion from an artificial tooth held in the
hand (single-tooth) or mounted in a jaw (tooth-in-jaw) either
1
hapTEL has been a 1.5 million research project funded jointly by the
two UK research councils. The projects aim was to design, develop and
evaluate synthetic, haptic and computer-based technologies.
59
Levels of agreement of student and staff assessments
Fig. 1. The practice and assessment teeth (carious lower mandibular molar).
in the traditional phantom head laboratory or in the hapTEL
lab using the virtual dental work stations as explained earlier.
In all cases, the artificial tooth was the mandibular lower molar
tooth. The size of caries in the three practice teeth ranged from
3 to 8 mm occlusal area and 2 to 3.5 mm depth. The carious
lesion was simulated in the plastic tooth by a density of the
plastic mimicking the consistency of the feel and appearance of
soft dental caries (in comparison with the harder plastic simu-
lating the enamel). After the practice sessions (traditional and
haptic) at the end of the 3-month term, the students perfor-
mance on removing artificial caries was assessed through pre-
paring the final cavity on a similar plastic tooth hand-held
(Study 1) and located in a lower jaw (Study 2). The test tooth
had a size and depth within the range of the practice teeth used
during the practice sessions (see Fig. 1). The teeth have three
layers representing the enamel, dentine and pulp (indicated by
a red colouring of the material). The teeth were all constructed
by one manufacturer (Frasaco Corp, Tettnang, Germany) to
the specification defined in the study by the clinical experts.
There was a slight variation in the procedure conducted
between Study 1 and Study 2 as described in Table 1 below.
The assessment rubric
An assessment rubric was devised based on a tool devised by
Jokstad and Mj or (11) but simplified following discussion with
four experienced dental tutors (Table 2).
The evaluation criteria and rating descriptors given in Table 2
were piloted with a small group of students then refined and
used to assess the students performance of the assessment task.
The students cavity preparations in the plastic teeth were
assessed in groups of 24 by two tutors while the students were
completing their assessment task. During the test, the tutors
rated each student on Criteria A and B by circulating around
each group and recording their performance, for example angle
of entry (Criterion A), holding the instrument correctly (Crite-
rion B). The remaining Criteria C, D and E were rated post hoc,
after the test session on the teeth removed from the jaw, by a
third tutor, because these required measuring the teeth
objectively and could not be conducted during the dental test
60
San Diego et al.
procedure. The criteria descriptors for these, for example,
Approximately a total 2 mm area or more black material left
enabled an objective assessment to be performed by one assessor
compared with the more subjective assessments for Criteria
A and B. Study 1 and Study 2 students gave their self-assessment
scores on all criteria on their individual teeth (hand-held
Study 1) and removed from the lower jaw after the cavity prepa-
ration (Study 2) immediately after completion of the post-test.
To check the reliability of the third tutors assessment, the
two laboratory tutors were also asked to assess the students
test teeth for Criteria C-E after the test sessions. The level of
agreement between these three tutors was assessed initially on a
set of 60 test teeth chosen at random of the 138 teeth rated. In
addition, the third tutor rated the same 60 teeth for a second
time (intra-rater reliability). There was about a 3-month period
difference before the first and the second rating for both stud-
ies. There was a 100% intra-rating agreement. The three clinical
tutors also demonstrated a 100% inter-rating agreement. It
took an average of 45 min to 1 h to rate 60 teeth.
Statistical analysis
For both studies, the degree of agreement between staff and
student ratings was calculated by cross-tabulating the scores
given by students with the scores given by tutors. From these
tables, the percentage agreement was calculated and a weighted
kappa with simple linear weights. Kappa is preferred as a mea-
sure of agreement as it corrects the agreement statistic for
chance levels of agreement (12). For both studies, the teeth
were assessed during the sessions for Criteria A and B and
hand-held for Criteria C-E.
Results
Study 1: Assessment of performance when
tooth models for the assessment task were
hand-held (2009/2010 cohort)
Table 3 below gives the numbers of students who gave each
rating level and the corresponding percentages of students and
2013 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd
Eur J Dent Educ 18 (2014) 5864
San Diego et al. Levels of agreement of student and staff assessments

TABLE 1. Descriptions of the practice and assessment tasks

Study 1 (2009/10) Study 2 (2010/11)

Mount and instruments Description of caries Mount and instruments Description of caries

Practice Task 1 Hand-held tooth using the
handpiece (TG). single tooth
on the screen HG)
Practice Task 2 Hand-held tooth using the
handpiece. (TG). single tooth
on the screen HG)
Practice Task 3 Tooth located in a lower jaw
held on a tray. (TG). Tooth
embedded in lower jaw (HG)
Assessment Artificial plastic tooth located
in a lower jaw held on a tray
(both groups)
Caries in the central pit of the Hand-held tooth using the Same as Study 1
occlusal. Occlusal, circular shape. handpiece. (TG). single
Through the enamel into dentine. tooth on the screen HG)
No more than approximately 0.5 mm
just into dentine
Caries start at the middle of central Mounted in phantom head, Same as Study 1
fissure (45 mm mesiodistally on occlusal but only three teeth (TG).
surface, 34 mm buccolingually on vvTooth embedded in
occlusal surface) Through the enamel lower jaw (HG)
about half-way through the dentine
thickness (23 mm into dentine) Cruciate
shape as per picture
Occlusal caries of larger cruciate shape. Tooth located in a lower jaw Same as Study 1
About 2 mm from the pulp centrally but held on a tray. (TG). Tooth
shallower going mesially and distally (Fig. 1) embedded in lower jaw (HG)
plus use of mirror
Occlusal caries of larger cruciate shape. Lower-left 6 tooth in a Same as Study 1
Almost touching the pulp centrally, phantom-head
mesially and distally (Fig. 1)

TG, Traditional phantom head group; HG, hapTEL virtual learning group.

TABLE 2. Evaluation checklist

Indicators

Evaluation criteria Poor Adequate Good

A Choosing the angle of entry to
the surface used in gaining
access to the artificial caries on
a lower-left 6 in a jaw
B Holding the instruments
appropriately
C Removing of artificial caries
from the cavity wall
D Removing of artificial caries
cavity floor
E Avoiding pulp exposure
The bur is never positioned at
90 angle to the occlusal surface
The handpiece is held using
a palm grip
Approximately a total 2 mm
area or more black
material left
Approximately a total 2 mm
area or more black
material left
Approximately a total of 2 mm
area or more exposed pulp
The bur is sometimes positioned
at 90 angle to the occlusal surface
The handpiece is held using a pen
grip but the hand is far away from
the head of the handpiece
Approximately a total of <2 mm
area of black material left
Approximately a total of <2 mm
area of black material left
Approximately a total of <2 mm
area of exposed pulp
The bur is always positioned at
90 angle to the occlusal surface
The handpiece is held using a
pen grip and the hand is close to
the head of the handpiece
No black material left
No black material left
No pulp exposed

kappa levels. As the table shows, only one Criterion, E, pulpal Study 2: Assessment of performance when
exposure, demonstrated adequate inter-rater agreement (97.1%) models for the assessment task were mounted
between staff and students. For Criteria C (caries removal on in a phantom head (2010/2011 cohort)
the wall of the tooth) and D (caries removal on the floor of
In this second, more difficult, activity, the students first
the tooth), the overall level of agreement was high, but the
inserted the plastic tooth into their individual phantom head
weighted kappa was low. For Criterion A (gaining access) and
jaws and then carried out the caries removal and cavity prepa-
Criterion B (holding the drill), there was low agreement. Apart
ration. For Criteria C-E, students and tutors removed the tooth
from Criteria A, the trend was always for students to rate their
from the lower jaw for these assessments. In this situation, we
performance as better than the ratings given by the staff
found that there was greater variability in the assessment of the
members.

2013 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd 61
Eur J Dent Educ 18 (2014) 5864
Levels of agreement of student and staff assessments San Diego et al.

TABLE 3. Comparison between students and staff ratings of performance on cavity preparation in a tooth on a traditional dental manikin.
(Study 1 hand-held tooth; n = 138)

Frequency of rating by students Frequency of rating by staff

Weighted
Evaluation criteria Poor Adequate Good Poor Adequate Good Kappa (Kw) % Agreement

A (Angle of Entry) 2 (1.4%) 120 (87%) 16 (11.6%) 5 (3.6%) 92 (66.7%) 41 (29.7%) 0.08 67.7
B (Holding instrument) 0 (0.0%) 16 (11.6%) 122 (88.4%) 4 (2.9%) 49 (35.5%) 85 (61.6%) 0.19 67.39
C (Removal of caries from 0 (0.0%) 12 (8.7%) 126 (91.3%) 0 13 (9.4%) 125 (90.6%) 0.16 86.23
cavity wall)
D (Removal of caries from 0 (0.0%) 5 (3.6%) 133 (96.4%) 0 9 (6.5%) 129 (93.5%) 0.25 92.75
cavity floor)
E (Avoiding pulp exposure) 0 (0.0%) 8 (5.8%) 130 (94.2%) 1 (0.7%) 4 (2.9%) 133 (96.4%) 0.7 97.10

TABLE 4. Comparison between students and staff rating of performance on cavity preparation in a tooth on a traditional dental manikin. (Study 2;
tooth in lower jaw. n = 131)

Rating by students Rating by staff

Weighted
Evaluation criteria Poor Adequate Good Poor Adequate Good Kappa (Kw) % Agreement

A (Angle of Entry) 11 (8.4%) 98 (74.8%) 22 (16.8%) 3 (2.3%) 73 (55.7%) 55 (42.0%) 0.01 47.9
B (Holding instrument) 0 (0.0%) 29 (22.1%) 102 (77.9%) 9 (6.9%) 87 (66.4%) 35 (26.7%) 0.01 37.7
C (Removal of caries from 2 (1.5%) 31 (23.7%) 98 (74.8%) 10 (7.6%) 13 (9.9%) 108 (82.4%) 0.48 80.5
cavity wall)
D (Removal of caries from 2(1.5%) 8 (6.1%) 121 (92.4%) 5 (3.8%) 3 (2.3%) 123 (93.9%) 0.28 89.4
cavity floor)
E (Avoiding pulp exposure) 1(0.8%) 16 (12.2%) 114 (87.0%) 1 (0.8%) 2 (1.5%) 128 (97.7%) 0.16 87.8

performance between the staff and students. The students and
staff tended to agree on only three of five of the criteria as is
shown in Table 4.
Criteria C (caries removal on the wall of the tooth), D (car-
ies removal on the floor of the tooth) and E (pulpal exposure)
showed high levels, of agreement but the level of agreement
was only moderate as indicated by the kappa statistic for C.
Again, Criteria A and B showed low levels of staffstudent
agreement. Apart from criterion A, as with Cohort 1, the stu-
dents rated their performance better than when assessed by
staff. For Criterion A, the majority of students (74.8%) rated
their performance at level 2 (the bur is sometimes positioned
at 90 angle to the occlusal surface) which as explained above
would be the optimum angle compared with the staff who
rated only 55.7% of the students at this level.
Discussion
This discussion focuses on three aspects of the findings: first,
the explanation for the poor agreement between staff and stu-
dents for some of the criteria, and why students tended to rate
themselves higher than when rated by tutors; second, the way
in which variation of task procedures can affect the scores
given by both students and staff on each criterion; and, third,
the implications of introducing measureable and directly obser-
vable descriptors in assessing students performance in remov-
ing artificial caries.
There were rating discrepancies and differing levels of agree-
ment between the students and staff on two of the criteria in
the first study (i.e. gaining access to the caries and instrument
grip) and a third criterion in the second study (i.e. removal of
caries on the wall of the tooth). Compared with the students in
the study who have acquired some learning experience, the
tutors are likely to have a clearer understanding of the learning
goals and the concepts attributed to cavity preparation (cf. 7).
Because of the limited period associated with the conduct of
the research study, concepts and skills that students might have
acquired in assessing cavity preparation might not be sufficient
to be able to judge a correct performance criterion (e.g. see 2)
and therefore would tend to rate themselves more highly. This
has confirmed findings from other studies that non-experts, in
this case, non-dentists (the students) are likely to be less critical
in marking, if they are not familiar with the learning objectives
and outcomes (5, 9, 13).
Although, the results of this study confirmed previous evi-
dence that students (learners) tend to give themselves higher
ratings than staff, in three criteria (A-C), the results showed
good levels of agreement on two other criteria; that is, removal
of caries of cavity floor (D) and pulpal exposure (E) and mod-
erate level of agreement on C. These three criteria were the
least ambiguous in terms of the three ratings (13) because
they could be assessed from the finished preparation [e.g. C
and D: no black material (caries) left; 3; E: no pulp exposed
(red material showing); 3].

62 2013 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd
Eur J Dent Educ 18 (2014) 5864
San Diego et al.
Regarding the first two criteria (A and B), the tutors gave
their rating synchronously (whilst the students were doing the
tasks), whilst students rated themselves retrospectively (after
performing the task). In this case, students retrospective recall
of their performance might have confounded the results (14).
These two criteria might have required recall of their actions
which need to be associated with the recall of the concepts and
knowledge imparted by the tutors (e.g. gaining access to the
caries and instrument grip), whilst the other might have
required pure memory recall of performance (e.g. caries
removal indicated by remembering whether there was black
staining left in the tooth; and pulpal exposure indicated by
remembering whether there was an exposure of red material in
the tooth). Furthermore, it is possible that tutors found it diffi-
cult to observe all students on these criteria, thus leading to
recall bias.
Aside from the differences in levels of agreement on the five
criteria, the two studies showed that the extent to which the
task varies in process can have an impact on these assessment
comparisons. The assessment task introduced in Study 2 could
be perceived as more difficult than the assessment task given
in Study 1 due to the variation in the task procedure. In
Study 1, the teeth were prepared hand-held, whilst in Study 2,
the teeth were prepared whilst mounted in a phantom-head.
Although, the results shown in Tables 3 and 4 showed a slight
difference in the agreement between students and staff on the
five criteria, these corroborate the findings of Satterthwaite
and Grey (8), who attributed the differences in assessments to
the shape of the crown preparation task denoting that one
task was more difficult than the other. In our study, it is also
likely that the complexity could be attributable to the context
and the situation of the task. It is probable that the introduc-
tion of the phantom head made the task more real than the
tooth task which is hand-held. The closer that task is to the
context of real practice, the more critical the assessors
might be.
The rubric was designed to encourage objective assessment
amongst tutors and students. However, the results for the com-
plicated correct procedure for Criterion A (the bur is some-
times positioned at 90 angle to the occlusal surface) also could
be interpreted subjectively by each student and tutor because
sometimes level 1 (the bur is never positioned at 90 angle to
the occlusal surface) would be correct for a specific caries lesion
and sometimes level 3 (the bur is always positioned at 90 angle
to the occlusal surface) might be correct.
The evidence that the students were still marking themselves
higher than the tutors for some criteria confirms the findings
of earlier studies (1, 46, 9) that it is likely that self-assessment
skills require practice and expertise, and for these Year 1 under-
graduate students, more training and practice might narrow the
difference between theirs and their tutors assessment ratings. It
is also possible that student demographic characteristics such as
age, gender, ethnicity and educational background will have an
impact on the accuracy of their self-ratings in relation to their
tutors. Future research should explore the relationship between
these factors and concordance in ratings. As this research also
involved some students learning during the sessions in the
Phantom Head Laboratory and the remaining groups learning
in the virtual laboratory, it is possible that a further level of
2013 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd
Eur J Dent Educ 18 (2014) 5864
Levels of agreement of student and staff assessments
analysis might reveal differences between these two groups;
although for this article, the assessment tasks for both tradi-
tional and virtual student groups were identical. Such results
will be presented in following papers.
Conclusions
This study confirms previous reports that students assessing
their own performance tend to rate themselves as more compe-
tent than tutors. Furthermore, levels of agreement are depen-
dent upon the clarity of the assessment criteria and their
relevance to the skills being assessed. This may relate to the
degree of conceptual understanding of the learner the
relatively inexperienced students have a narrower concept of
adequate performance in comparison with their more experi-
enced tutors. If student self-assessment is to play a key role in
formative assessment, attention will need to be paid to the
processes through which students develop a conceptual
understanding of skilled performance.
Acknowledgements
This project has been funded by two UK Research Councils.
hapTELTM is part of the Technology-Enhanced Learning Pro-
gramme (TEL) funded by the United Kingdom, Economic and
Social Research Council (ESRC) and the Engineering and
Physical Sciences Research Council (EPSRC) Award Number:
RES-139-25-0387 (http://www.haptel.kcl.ac.uk). Our thanks to
the hapTEL project team, Kings College London: Prof.
A. Banerjee, Dr. S. Banerji, Prof. M.J. Cox, Prof. S. Dunne,
Mrs. C. Gray, Ms. T-A. Green, Mr. J. Harper, Prof. J. Hind-
marsh, Mr. L. Hyland, Prof. B. Millar, Prof. T. Newton, Dr. B.
Quinn, Prof. P. Reynolds, Dr. B. Robinson, Dr. J.P. San Diego,
Mr. A. Shahriari-Rad, Prof. N. Wilson, Prof. M. Woolford;
University of Reading: Prof. W. Harwin, Dr. A. Barrow, Mr. B.
Tse; Birmingham City University: Mr. B. Elson.
References
1 Black P, Wiliam D. Inside the black box: raising standards through
classroom assessment. Phi Delta Kappan 1998: 80: 139148.
2 Sullivan K, Hall C. Introducing Students to Self-assessment. Assess
Eval High Educ 1997: 22: 289305.
3 Mattheos N, Nattestad A, Falk-Nilsson E, Attstrom R. The
interactive examination: assessing students self-assessment ability.
Med Educ 2004: 38: 378389.
4 Boud D. The role of self-assessment in student grading. Assess Eval
High Educ 1989: 14: 2030.
5 Black P, Wiliam D. Assessment and classroom learning. Assess Educ
Princ Pol Pract 1998: 5: 774.
6 Nicol D. E-assessment by design: using multiple-choice tests to good
effect. Journal of Further and Higher Education 2007: 31: 5364.
7 Holmboe E. Faculty and the observation of trainees clinical skills:
problems and opportunities. Acad Med 2004: 79: 1622.
8 Satterthwaite JD, Grey NJA. Peer-group assessment of pre-clinical
operative skills in restorative dentistry and comparison with
experienced assessors. Eur J Dent Educ 2008 May: 12: 99102.
9 Boud D, Falchikov N. Quantitative studies of student self-
assessment in higher education: a critical analysis of findings. High
Educ 1989: 18: 529549.
63
Levels of agreement of student and staff assessments
10 San Diego JP, Cox MJ, Quinn BFA, Newton JT, Banerjee A,
Woolford M. Researching haptics in higher education: the
complexity of developing haptics virtual learning systems and
evaluating its impact on students learning. Comput Educ 2012: 59:
156166.
11 Jokstad A, Mj or I. Cavity designs for class II amalgam restorations:
a literature review and a suggested system for evaluation. SODE
1987: 45: 257273.
12 Cohen J. Weighed kappa: nominal scale agreement with provision
for scaled disagreement or partial credit. Psychol Bull 1968: 70:
213220.
64
San Diego et al.
13 Wetherell J, Mullins G, Hirsch R. Self-assessment in a problem-
based learning curriculum in dentistry. Eur J Dent Educ 1999 Aug:
3: 97105.
14 Gaba D, Howard S, Flanagan B, Smith B, Fish K, Botney R.
Assessment of clinical performance during simulated crises using
both technical and behavioral ratings. Anesthesiology 1998: 89:
818.
2013 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd
Eur J Dent Educ 18 (2014) 5864