THE DIRECT CONSENSUS

METHOD
THE DIRECT CONSENSUS
METHOD
“Improve upon some of the perceived
shortcomings of the Angoff method and
to give panelists more direct control in
recommending where the passing score is
set.” (2004, p. 21)
PROCEDURE FOR THE DIRECT
CONSENSUS METHOD
1st Phase of Rating
1) Qualified participants are selected and they are introduced to the purpose of the
testing program and are familiar with the content standards or objectives covered
by the test.
2) Participants do not proceed item – by – item through a test form. Rather, an whole
test form is reorganized into sections and entire sections are reviewed and judged.
3) Participants begin with the 1st subarea, and they are instructed to indicate those
items that they believe the just – qualified candidate will answer correctly.
4) The sum of each participant’s subarea score is taken as that participant’s
recommended passing score for the total test.
 Hypothetical Data & example of Direct Consensus standard Setting Method

Rater ID Number
1 2 3 4 5 6 7 8
Participants Recommended Number of Items within a Percentage of
Subset Area (number Subset Area
Subset Area that the Just – Qualified Examinee Should Number of Items in
of items) Means (S.D)
Answer Correctly Subarea
A (9) 7 7 6 7 7 7 7 6 6.75 (0.46) 75.0
B (8) 6 6 7 6 5 7 7 7 6.38 (0.74) 79.8
C (8) 5 6 7 6 6 6 5 6 5.88 (0.64) 73.5
D (10) 7 8 8 7 7 8 7 7 7.38 (0.52) 73.8
E (11) 6 5 7 7 7 7 8 6 6.63 (0.92) 60.2
Grand Mean Percentage
(S.D)
Raters Sum 31 32 35 33 32 35 34 32 33.00 (1.51) 71.7

1) 46 Item test comprising of 5 subareas of varying lengths and labeled A through E.

2) 8 participants marked the number of items.
TO BE CONTINUED….
The process often includes open discussion in which participants are asked to support, defend or
elaborate on the rationale for their judgments.
1. Participants are encouraged to discuss their ratings and the reasons for their ratings. The
discussion remain focused on the content of that subarea and on the characteristics of the just
qualified examinee.
2. Participants discuss to revise their subtest area ratings. During this round, participants can
clearly see how changes in their subtest area judgments directly affect their individual total
test cut score.
3. Participants are encouraged to indicate their rationales for support of the group mean as a
final cut score.
4. Final discussion would foster participants’ agreement on a consensus cut score.
ALTERNATIVE PROCEDURES
1. The reality information be provided to participants at
the end of one of the rounds of ratings. According to
authors, “it would be possible to show the average item
performance of the examinee in each content area” (p.
22). This suggestion to provide item difficulty data.
2. It can be conducted by proceeding from the first round
of ratings directly to discussion of the final cut score.
ADVANTAGES
1. Time Saving as participants have to rate a
subtest area, rather than all the items one by
one.
2. It allows participants to exert more direct
influences on the final cut score
recommendation.
LIMITATIONS
1. The method has only been tried with tests that can be
divided into distinct content based subarea.
2. The failure of a group of panelists to agree at the end of
the session on a single cut score may be quite probable,
particularly when the group of participants is larger or
more diverse, or when the test contains a large number of
items.
3. Skilled facilitators may be able to accomplish quite well.