Central Examination Board Report August 2015

THE BAR STANDARDS BOARD
CENTRAL EXAMINATIONS BOARD

CHAIRS REPORT AUGUST 2015
First Sit 2014/15
Executive summary
The Central Examination Board (CEB) has now completed its fourth cycle of
overseeing first sit assessments in the three knowledge areas of the Bar Professional
Training Course (BPTC).
The confirmed post-scale outcomes of the 2014/2015 first sit centralised assessments
following review of cohort performance by the CEB are as follows:
Professional
Ethics
Number of
Candidates
Passing MCQ
Passing SAQ
Passing Overall
Criminal
Litigation,
Evidence and
Sentencing
Number of
Candidates
Passing MCQ
Passing SAQ
Passing Overall
Civil Litigation,
Evidence and
Sentencing
Number of
Candidates
Passing MCQ
Passing SAQ
Passing Overall
2015 First
Sit *
2014 First
Sit *
2013 First
Sit *
2012 First
Sit *
Change
2014 to
2015
1572
1649
1722
1591
-77
91.5%
58.0%
56.7%
81.0%
65.6%
59.6%
94.3%
89.5%
86.4%
92.6%
88.5%
84.9%
10.5%
-7.6%
-2.9%
1483
1586
1719
1569
-103
83.3%
64.2%
62.5%
84.1%
78.2%
72.8%
88.9%
69.9%
68.2%
88.7%
77.8%
74.7%
-0.8%
-14.0%
-10.3%
1595
1663
1768
1568
-68
71.3%
65.0%
58.0%
68.6%
67.8%
57.4%
73.2%
61.5%
56.2%
83.7%
73.5%
68.0%
2.7%
-2.8%
0.6%
(*Although the first sit assessment, a number of candidates will have undertaken the
assessments on a deferred or referred basis.)
The first sit all-Provider post scale pass rates for 2014/15 present a mixed picture
when compared with the previous year. Professional Ethics MCQ all-Provider pass
rates and Civil Litigation MCQ all-Provider pass rates are up compared to 2013/14,
2
but all other MCQ and SAQ all-Provider pass rates are down. In terms of combined
pass rates (MCQ and SAQ) for the three knowledge areas, all-Provider pass rates are
largely unchanged on the previous year in Professional Ethics and Civil Litigation, but
are noticeably lower in respect of Criminal Litigation. A more detailed analysis is
contained in the main report.
A number of general points should be borne in mind when considering the results data:
Knowledge area assessment pass rates in the above table indicate the
aggregated percentage of candidates passing an assessment across all
Provider institutions.
All data set out in the body of the report has been presented so as to preserve
the anonymity of Provider institutions.
There is no pre-scale data for 2010/11 as this was pre-CEB and all knowledge
area assessments were set and marked by individual Provider institutions.
Where Provider trend analysis data is used in this Report it will only include
Providers who have had cohorts across all four assessment cycles. This
impacts on trend analysis data for Kaplan (last full intake in 2013/14) and BPP
Manchester where the BPTC was offered for the first time in 2013/14. Where
such data is excluded the commentary on the data table will note this.
1. Context
1.1 The 2010/11 academic year saw the first round of assessments under the BPTC
regime (replacing the BVC) in the wake of the Wood Report (July 2008). For 2010/11
all Providers were required to assess candidates in Professional Ethics, Civil Litigation,
Remedies & Evidence, and Criminal Litigation, Evidence & Sentencing (often referred
to as the knowledge areas) by means of MCQs and SAQs. Together these three
modules represent 25% of the BPTC (i.e. 30 credits out of 120). For 2010/11 the
knowledge area assessments were set and marked by the Providers. Centralising
these assessments was a key recommendation of the Wood Report, and the Central
Examinations Board (CEB) was established to oversee this change on behalf of the
Bar Standards Board (BSB). 2011/12 was the first year of operation for the system of
centralised examinations for the knowledge areas on the BPTC. No changes were
made to the format of assessment, but the setting of the assessments was undertaken
independently of the Providers by a team of CEB examiners appointed by the BSB.
1.2 In each knowledge area candidates are required to attempt a multiple choice
question (MCQ) test, and a short answer question (SAQ) test. In the Civil and
Criminal papers the MCQ comprises 40 questions and the SAQ 5 questions. In
Professional Ethics the MCQ comprises 20 questions and the SAQ 3 questions. All
questions are compulsory and the pass mark in each paper is 60%. The marks for the
MCQ and SAQ papers are combined to provide a candidate with an overall mark in
each of the knowledge areas. Candidates have to achieve the pass mark of 60% in
both elements of assessment, there being no scope for the aggregation of marks
below 60% between MCQ and SAQ scores to achieve the minimum 60% pass mark
overall.
3
1.3 The assessment process is overseen by the CEB whose members are appointed
by the BSB. The CEB comprises a Chair, teams of examiners (a Chief Examiner and
two Assistant Examiners for each knowledge area), an independent observer, an
independent psychometrician and senior staff from the BSB. The Chair and the
examiners contribute a mix of both academic and practitioner experience.
1.4 From the academic year 2011/12 onwards the CEB has had responsibility for
setting the knowledge area assessments and confirming the knowledge area cohort
marks for each of the Provider institutions. In confirming marks for cohorts of
candidates the CEB is concerned to ensure that a consistent measure of achievement
has been applied across all Providers, and that proper account has been taken of any
relevant factors that may have had a bearing on the performance of a cohort of
candidates. As a result the CEB has the discretion to scale cohort marks (upwards or
downwards) if it feels there are issues relating to all candidates, or a statistically
relevant sub-cohort of candidates, that justify such intervention. In confirming the
marks for MCQ and SAQ papers the CEB does not address the combined effect of the
marks in terms of whether or not a candidate has passed a knowledge area overall,
or where a candidate falls in terms of grade boundaries.
1.5 Once the CEB has confirmed the MCQ and SAQ marks for each cohort of
candidates at each Provider the marks are distributed to the Providers where they feed
into the individual BPTC student profiles considered at the Provider award and
progression examination boards. It is at the Provider examination boards that issues
relating to individual candidates such as extenuating circumstances or academic
misconduct are considered.
1.6 For 2014/15 there were 8 Provider institutions offering the BPTC across 11
centres:
BPP London, Manchester and Leeds

Cardiff
City University Law School
University (formerly College) of Law London and Birmingham
Manchester Metropolitan University
Nottingham Trent Law School
UNN
UWE
BPP Manchester came on stream as an assessment centre for the first time in
2013/14, and Kaplan recruited its last intake in 2013/14 (although it had a very small
number of referred and deferred candidates in the 2014/15 first sit cohort).
Provider cohort results contained in the report have been anonymised. Providers are
identified by numbers in the graphs of cohort performance featured in this report, and
the allocation to a Provider of a number will vary from one graph to another depending
on the baseline adopted for the comparison Provider cohort performance.
1.7 Terms used in this report:
All-Provider refers to the aggregated data bringing together cohort

performance across all Providers centres
By Provider refers to data showing the performance of each of the Providers
compared with each other
First sit refers to the March/April exam cycle note that some candidates
undertaking these examinations may be doing so on a referred or deferred
basis
Resit refers to the August exam cycle some candidates undertaking these
examinations may be doing so on a deferred basis (i.e. as if for the first time)
Combined refers to the result following aggregation of the MCQ and SAQ
scores. Combined pass rates can be lower than either MCQ or SAQ pass rates
that feed into them depending on the pattern of candidate failure across the two
elements).
2. The setting and conduct of assessments

2.1 All Provider institutions were required to supply their 2010/11 knowledge area
MCQ and SAQ papers (first sit and resit), along with solutions, to the BSB. The
supplied material was used by the CEB examiner teams, with necessary amendments
and additions, to devise the 2011/12 first sit and resit assessments and related
solutions. Additional material was supplied by the Provider institutions for the 2012/13
and 2013/14 assessments, and this was further supplemented by questions and
solutions devised by members of the CEB examining teams and suitably qualified
individuals recruited specifically as question writers. Assessment scrutiny meetings
were attended by relevant CEB examiners, the CEB Chair and BSB staff.
Assessments were also checked by independent technical readers and any issues
raised referred back to the scrutiny meetings.
2.2 Candidates at all Provider institutions attempted the assessments in each of the
knowledge areas on the same dates.
Professional Ethics 23 March 2015

Civil Litigation, Remedies & Evidence 8 April 2015
Criminal Litigation, Evidence & Sentencing 10 April 2015
2.3 All examinations had a two oclock start time. In any case where a Provider
identified candidates as having special assessment arrangements necessitating a
start time earlier than that of the main cohort, the relevant candidates were not allowed
to leave their assessment area until the commencement of the main cohort
assessment, and secure delivery and collection arrangements were put in place for all
examination materials.
2.4 In exceptional circumstances candidates can be allowed to attempt the
assessments at locations overseas. The onus is placed on the Provider institution to
ensure that a secure assessment centre is available in these cases, and the BSB
requires the start time of the examination at the overseas centre to be the same (or
5
subject to similar to arrangements in 2.3) as the UK start time (regardless of time

differences). To ensure the complete security of the examination papers the BSB
dispatches all examinations to the overseas contacts directly.
2.5 Provider institutions were given guidance on examination arrangements by the
BSB. Exam invigilation reports were submitted by Providers to the BSB, detailing any
major issues Providers believed may have had a bearing on the conduct of the
examination itself at their assessment centres (for example, transport difficulties, bomb
hoaxes, fire alarms, building noise).
2.6 Providers were also required to complete a pro-forma on each MCQ and SAQ
question in each of the knowledge areas indicating their view as to the level of difficulty,
relevance to the syllabus, whether a question was technically defective in some way,
and the appropriateness of the solution. The pro-forma also provided an opportunity
to comment on the paper as a whole. External examiners also submitted comments
on the assessments.
CEB examiners were invited to respond to the pro-formas, as required, by issuing
revised post-exam marking guidance in relation to the SAQ papers. All Provider
comments were taken into account at the CEB meetings held to review MCQ and SAQ
marks in each knowledge area.
3. The marking process
3.1 Candidates attempting the MCQ papers in each of the knowledge areas marked
their answers on machine-readable answer sheets. Provider institutions were required
to take copies of the MCQ answer sheets and return the originals to the BSB. The
MCQ scanning was undertaken by BSB support staff, using Speedwell scanners and
software, who had undergone training provided by Speedwell.
3.2 SAQs were marked by staff at the Provider institutions following (revised) marking
guidance provided by the CEB examiners. Provider marking teams were required to
observe annotation conventions when marking SAQ scripts to show the impact of
internal marking and moderation. Each Provider was required to ensure that an
appropriately robust system of internal SAQ moderation was in place with particular
regard being given to the pass/fail borderline candidates. The external examiner for
the relevant knowledge area at each Provider was also required to sample SAQ
papers as appropriate and require knowledge area teams to remark where it was felt
guidelines had not been adhered to. From 2013 an enhanced sampling system was
put in place whereby randomised samples in appropriate marking bands were selected
by Providers, as directed by the BSB, for review by external Examiners, with copies of
any scripts supplied to the External Examiners also being sent to the CEB examiners
at the BSB. The range of scripts within the sampling bands was adjusted to reflect the
fact that the effect of upward scaling by the CEB could result in candidates who, on
first marking, appear to be clear fails in SAQ papers becoming borderline pass/fail
candidates once the effect of scaling is applied.
3.3 Reports were provided by each Provider and the relevant BSB external examiner
assigned to the Provider, commenting on the SAQ marking process in each of the
6
knowledge areas, with a requirement to flag any issues that should be considered by
the CEB as possible grounds for intervention in determining the marks to be confirmed.
4. How the CEB reviewed the results
4.1 Two sub-boards were constituted to review the results in each of the three
knowledge areas, one to look at MCQ results and one to look at SAQ results. Each
sub-board was chaired by the CEB Chair, and attended by the relevant knowledge
area examiners, and an independent psychometrician.
4.2 At each sub-board the pre-scale MCQ and SAQ results for the current, and
previous first sit rounds were noted, in particular the aggregated all Provider pass
rate (i.e. the percentage of candidates passing across all Providers combined), and
the pass rates at each Provider.
4.3 Other key sources of information available to the sub-boards included:
Representations made by Providers in the MCQ/SAQ feedback pro-formas

(see 3.3 above) and comments from CEB examiners.
A spreadsheet showing the pass rate for each MCQ, and each component of
each SAQ, achieved at each of the Providers cross-referenced to the
representations made in the assessment pro-formas returned by the Providers
(see 3.3 above) thus flagging up any correlation of Provider criticisms and
concerns with systematic poor performance by candidates.
For both the MCQs and SAQs, Manhattan diagrams indicating the extent to
which each question discriminated effectively between strong and weak
candidates.
For the SAQs, Provider feedback, BSB External Examiner comments, and any
amended marking guidance issued by the CEB examiner teams.
Evidence provided by routine random sampling any further requested sampling
of SAQ scripts undertaken at the direction of the CEB
4.4 On the basis of the above evidence, and as advised by the independent
psychometrician, the CEB sub-boards formulated a series of proposals, where
required, for the scaling of cohort marks to be considered at the final CEB meeting.
The CEB recognises that there are a multiplicity of approaches to cohort scaling and
operates within the general principles set out in the scaling protocol adopted following
consultation with the Providers. Amongst the options open to the CEB are:
Crediting more than one correct answer to an MCQ

Disregarding an MCQ or part of an SAQ entirely if defective or inappropriate
(e.g. no correct answer) no candidate credited and maximum score
recalculated
Crediting all candidates with the correct answer if an MCQ or part of an SAQ is
defective or inappropriate
Crediting all candidates with a mark or marks in respect of a part of an SAQ if
there is headroom to do so (i.e. provided a candidate has not already achieved
the maximum marks available)
7
Adjusting marks globally for an SAQ as a whole, an SAQ paper as a whole, or

an MCQ paper as a whole to compensate for generic factors if there is
headroom to do so (i.e. provided a candidate has not already achieved the
maximum marks available)
Adjusting marks for each candidate in a sub-cohort due to local assessment
issues (not arising from the delivery of the course by a Provider, or matters
related to the conduct of the assessment that can be dealt with through a
Providers extenuation processes) provided the sub-cohort constitutes a
statistically reliable sample for scaling purposes.
4.5 The final CEB Board meeting for the 2014/15 first sit examinations, held on 26
June 2015, considered the recommendations of the sub-boards in respect of the
knowledge area MCQ and SAQ cohort marks. The meeting was attended by the
relevant examiners, key BSB officers, an independent psychometrician and an
independent observer. The function of the final CEB board is to test the
recommendations of the sub-boards, consider whether further sampling of SAQs may
be necessary, and to confirm the MCQ/SAQ cohort marks subject to any outstanding
QA issues related to on-going sampling. Once cohort marks are confirmed by the CEB
they cannot subsequently be altered by Provider institutions. The process for
challenging marks confirmed by the CEB is outlined in section 16 (below).
5. 2014/15 first sit results in Professional Ethics: MCQs
5.1 MCQ pre-scale first sit pass rates all Providers 2011/12 to 2014/15
Professional Ethics
MCQ pre-scale All-Provider pass
rate
2014/15
2013/14
2012/13
2011/12
91.5%
81.0%
86.4%
92.6%
The pre-scale all-Provider pass rate for Professional Ethics MCQ is significantly up on
2013/14 showing a rise in excess of 10%, the highest performance level since the
2011/12 first sits.
5.2 MCQ pre-scale 2014/15 first sit histogram question by question
Professional Ethics 2014/15 First Sit Pre-Scale

MCQ All Provider question by question histogram
100.0
Pass rate %
90.0
80.0
70.0
60.0
50.0
40.0
30.0
1
9 10 11 12 13 14 15 16 17 18 19 20
MCQ item number
This table indicates the range of pre-scale pass rates for the Professional Ethics MCQs
across all Providers, showing no MCQs with a pre-scale pass rate of below 40%
(compared with 3 in 2013/14).
5.3 Final exam board review of the Professional Ethics MCQ results
Question
MCQ 4
MCQ 9
Intervention
applied
No
intervention
No
intervention
CEB rationale
48.4% all-Provider pass rate, very good discrimination
between candidates. The feedback from Providers was
considered and the Board agreed no further action was
required.
The Board decided there was no reason for intervention.
40.5% all-Provider pass rate, with strong discrimination.
It was noted that 6 Provider cohorts had pass rates
below 40% for this MCQ. The Provider feedback was
considered and addressed by the Chief Examiner, who
explained that the question covered the ethical
principles according to the Code, rather than what the
Providers noted to be common practice. The feedback
from Providers did not give grounds for intervention as
students have not gained the experience in the field to
enable them to be aware of these practices. The Board
agreed that distractor A worked well.
9
45.7% all Provider pass rate, very good discrimination

between candidates. It was noted that Providers did not
submit feedback on this question.
No
intervention
MCQ 16
The Final Exam Board noted the careful consideration of the statistical data, the written
feedback from Providers, and the written feedback from the external examiners that
had taken place at the Professional Ethics MCQ exam board, and the
recommendations therefrom. The comments from the Chief Examiner for Professional
Ethics regarding the impact of the new handbook on the setting of MCQs were noted.
Having considered this evidence and the deliberations of the MCQ Board the Final
Board determined that no interventions were justified.
5.4 MCQ post-scale first sit pass rates all Providers 2011/12 to 2014/15
Professional Ethics
MCQ post-scale All-Provider pass
rate
2014/15
2013/14
2012/13
2011/12
91.5%
81.0%
94.3%
92.6%
Given the absence of intervention, the post-scale MCQ first sit pass rate for 2014/15
remains at 91.5%. As indicated above, this is a noticeably higher figure compared
with the post-scale MCQ rates for the previous year, but in line with 2012/13 and
2011/12.
5.5 MCQ first sit pass rates across all Providers 2014/15
Professional Ethics First Sit 2014/15

MCQ pass rates per Provider
100
Pass rate %
95
90
85
80
75
MCQ 2014/15 Pass rate
70
1
Provider
10
10
11
Providers are ranged in order of pass rates (high to low) according to their
performance. Hence Provider 1 had the highest pass rate at 98%, Provider 11 the
lowest at just under 79%, a range of 18% between top and bottom.
5.6 MCQ post-scale first sit pass rates by Provider 2011/12 to 2014/15
Pass rate %
Professional Ethics MCQ Post-Scale pass rates

First Sit 2011/12 to 2014/15
100.0
95.0
90.0
85.0
80.0
75.0
70.0
1
10
Provider
MCQ 2011/12 post-scale
Providers are ranked 1 to 10 based on their 2011/12 post-scale first sit Professional
Ethics MCQ pass rates, showing how their 2012/13, 2013/14 and 2014/15 pass rates
compare. Hence Provider 1 achieved a pass rate of over 98% in 2011/12, over 96%
in 2012/13, dropped to 80% in 2013/14, and recovered with a pass rate of 95% in
2014/15. Provider 7 was the only Provider to record a lower cohort pass rate for
2014/15 compared to 2013/14. Provider 10 recorded a notable 20% increase in pass
rates compared to 2013/14. Provider 5 has the best 4-year average pass rate at over
93%, whilst the weakest, on average, over the same period is Provider 7 with an
average pass rate just under 88%. Note that results for the BPP Manchester cohort
are not reflected in this table as there is no data pre 2013/14 and results for Kaplan
have been excluded, as their last intake was in 2013/14.
6. 2014/15 first sit results in Professional Ethics: SAQs
6.1 SAQ pre-scale first sit pass rates all Providers 2011/12 to 2014/15
Professional Ethics
SAQ pre-scale All-Provider pass
rate
2014/15
2013/14
2012/13
2011/12
58.0%
65.6%
86.7%
88.5%
Following a significant decline in Professional Ethics SAQ pre-scale pass rates in

2013/14 compared to the two previous rounds of first sit assessments (over 20%), the
all-Provider pass rate for 2014/15 is a further 7.6% down on 2013/14.
11
6.2 SAQ pre-scale first sit histogram question by question (showing sub-parts)
Pass rate %
Professional Ethics 2014/15 First Sit Pre-scale

SAQ All Provider question by question histogram
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
SAQ sub-parts and totals
This table indicates the range of pre-scale pass rates for the Professional Ethics SAQs
across all Providers, clearly showing the relatively poor cohort performance in respect
of SAQ2. Compared to 2013/14, where three out of eleven SAQ sub-elements
reported a pre-scale pass rate of below 40%, the 2014/15 SAQ paper shows only one
sub-element out of six with a pass rate below 40%. Sub-element SAQ2(b) clearly
presented a significant challenge to candidates as it is something of an outlier with an
all-Provider pass rate of 30%, resulting in an all-Provider pass rate for SAQ2 as a
whole of just over 50%, but this is not of itself remarkable. SAQ 1 in the 2013/14 first
sit produced a pre-scale pass rate of less than 40% overall and was not regarded by
the examination boards as warranting any further intervention.
6.3 Final exam board review of the Professional Ethics SAQ results
Question
SAQ 1 (a)
Intervention
applied
No
intervention
CEB rationale
SAQ1(a) had 6 marks available and the all-Provider pass
rate for the component was 49.0%. The Board
considered the comments from Providers and noted one
comment suggesting the word unregistered should be
given a mark. The Chief Examiner explained that this was
incorrect and should not warrant an additional mark. The
Board also noted comments suggesting an extra mark
should be awarded for candidates stating the 2 core
duties rather than only 1. The Board agreed only one
mark was required for this question.
The Chief Examiner confirmed that the question was
written in a very clear way. The range of performances
across the question suggested good candidates did well
12
Question
Intervention
applied
CEB rationale
and the question had good discrimination. All comments
from Providers were considered and the Board agreed
that no further action or interventions were required.
The Board decided that there was no reason for
intervention.
SAQ 1 (b)
No
intervention
SAQ1(b) had 4 marks available and the all-Provider pass

rate for the component was 69.8%. The comments from
Providers were considered and amendments were made
to the mark scheme to reflect this. The Board agreed no
interventions in respect of this question were warranted
and noted the good discrimination data for the question.
intervention.
SAQ2(a) had 4 marks available and the all-Provider pass
rate for the component was 52.4%.
SAQ 2 (a)
SAQ 2 (b)
No
intervention
No
intervention
The Board considered comments from Providers and

decided that there was no reason for intervention.
The Board noted the relatively low all-Provider pass rate,
and the rate achieved by the cohort at the best performing
Provider across the 2014/15 first sit assessments as a
whole. The Board noted that amendments had been
made to the mark scheme in light of Provider and EE
feedback. The Board considered whether the extent to
which reliance on knowledge from the academic stage
(criminal law) required to answer this question was fair
and agreed that it was. It was noted that the amended
marking scheme permitted 6 ways to get the full 4 marks
for question 2(b). The Board considered the statistical
information and noted that this was a more challenging
question that discriminated well between weaker and
stronger candidates.
intervention.
SAQ 3 (a)
No
intervention
SAQ3(a) had 5 marks available and all-Provider pass

rate for the component was 71.3%. The Board noted that
13
Question
Intervention
applied
CEB rationale
some comments from Providers had been reflected in the
amendments to the final marking scheme.
intervention.
SAQ 3 (b)
No
intervention

The Board noted that the marking scheme permitted 8
possible ways to get the 5 available marks for SAQ3(b),
and that the comments from Providers had been
considered and the final mark scheme amended
accordingly.
intervention.
There were no other global or generic issues relating to the SAQ assessment.
6.4 SAQ post-scale first sit pass rates all Providers 2011/12 to 2014/15
Professional Ethics
SAQ all-Provider pass rate
2014/15
2013/14
2012/13
2011/12
58.0%
65.6%
86.7%
88.5%
Given the absence of intervention, the post-scale SAQ first sit all-Provider pass rate
for 2013/14 remains at 58%, 7.6% down on the previous year. The Final examination
board endorsed the recommendations of the SAQ Board.
6.5 SAQ pre and post-scale first sit pass rates across all Providers 2014/15
14

SAQ pass rates per Provider
90
Pass rate %
80
70
60
50
SAQ 2014/15 Pass rate
40
30
1
10
11
Provider
Providers are ranged in order of pass rates (high to low) according to their
performance. Hence Provider 1 had the highest pass rate at 81.6%, Provider 11 the
lowest at just under 37%. There is clearly a very significant range in the performance
for Provider cohorts across this assessment (44% between best and worst), but this
is largely consistent with 2013/14 where the range was 39%, and suggests that the
assessment is discriminating effectively between weak and strong cohorts. The
biggest difference between the 2013/14 and 2014/15 cohort data sets is that in
2014/15 there are three very weak cohorts compared to the rest as the table below
indicates.
Cohort pass rate
range
80%+
70% to 79%
60% to 69%
50% to 59%
40% to 49%
Below 40%
2014/15
(11 Providers)
2013/14
(12 Providers)
1
2
3
1
2
2
2
3
3
4
0
0
This suggests that performance at the higher end has held up and that the weaker
cohorts in 2014/15 have been significantly weaker than in 2013/14.
6.6 SAQ first sit pass rate by Provider 2011/12 to 2014/15
15
Professional Ethics SAQ Post-scale

Pass rates for First Sit 2011/12 to 2014/15
100.0
90.0
Pass rate %
80.0
70.0
60.0
50.0
40.0
30.0
SAQ 2011/12 post-scale
20.0
1
10
Provider
Providers are ranked 1 to 10 based on their 2011/12 post-scale first sit Professional
Ethics SAQ pass rates, showing how their 2012/13, 2013/14 and 2014/15 pass rates
compare. Hence Provider 1 achieved a pass rate of 95.5% in 2011/12, 92.4% in
2012/13, and 71% in both 2013/14 and 2014/15. Provider 2, by contrast, has seen a
very worrying year-on-year decline in cohort performance from 93% in 20111/12, to
83% in 2012/13, 59% in 2013/14 dropping even further to 37% in 2014/15. Three
Providers recorded cohort pass rates for the Professional Ethics SAQs that were
higher than their 2013/14 pass rates Provider 2 recorded the biggest decline in pass
rates from 2013/14, a drop of over 22%. Provider 1 has the best 4-year average pass
rate at over 81%, whilst the weakest, on average, over the same period is Provider 7
with an average pass rate just over 65%. Note that results for that BPP Manchester
cohort are not reflected in this table as there is no data pre 2013/14, and results for
Kaplan have been excluded, as their last intake was in 2013/14
16
6.7 MCQ and SAQ first sit post-scale pass rates by Provider 2014/15 high to low

MCQ and SAQ cohort performance comparison
100
90
Pass rate %
80
70
60
50
40
30
1
10
11
Provider
This table compares 2014/15 Professional Ethics MCQ and SAQ all-Provider postscale pass rates. The order of Providers is determined by the Providers combined
post scale pass rate for the MCQ and SAQs taken together (i.e. the percentage of
candidates at the Provider pass both sections). The data shows that cohorts typically
perform better in the MCQ element compared to the SAQ element. As in previous
years there is some evidence to suggest that the gap in pass rates between the two
forms of assessment tends to widen where cohorts are weaker overall (this is very
much the case with Providers 9 to 11).
17
7. 2014/15 first sit in Professional Ethics: combined MCQ and SAQ post-scale
results
7.1 Combined MCQ and SAQ all-Provider first sit post-scale pass rates and grade
boundary distribution 2014/15
Combined (MCQ and SAQ) All Provider post-scale results
Total number sat (All Providers)
Pass Combined
Fail Combined
1572
892
680
%
56.7%
43.3%
Combined Grade Boundaries

Outstanding
Very Competent
Competent
Not Competent
139
557
196
680
8.8%
35.4%
12.5%
43.3%
It should be noted that the CEB does not review, adjust or confirm the combined pass
rate for any of the three knowledge areas. The combined results are simply the product
of the relevant confirmed MCQ and SAQ marks which may, or may not, have been
subject to adjustment by the CEB at an earlier stage. No further scaling is applied once
MCQ and SAQ marks are combined to produce the pass rate for a knowledge area as
a whole. Hence the overall knowledge area marks for cohorts and individual
candidates, including grade boundaries, lie where they fall.
7.2 Combined MCQ and SAQ first sit pre and post-scale pass rates by Provider high
to low

Combined pass rates by Provider
90
Pass rate %
80
70
60
50
Combined 2013/14 pass rate
40
30
1
Provider
18
10
11
Providers are ranged in order of pass rates (high to low). Cohort performance in the
SAQ element of the assessment largely drives the outcome of the combined cohort
result, hence this performance profile is very similar to the SAQ table at 6.5.
7.3 Combined MCQ and SAQ first sit pass rates all Providers 2011/12 to 2014/15
Professional Ethics
Combined post-scale All-Provider pass
rate
2014/15
2013/14
2012/13
2011/12
56.7%
65.6%
86.4%
85.0%
7.4 First sit 2014/15 post-scale grade boundaries by Provider

Post-scale grade boundaries by Provider
% student per category
100.0
80.0
60.0
40.0
20.0
0.0
1
10
11
Provider
Not Competent
Competent
Very Competent
Outstanding
Providers are ranked according to their 2014/15 post-scale combined Professional

Ethics first sit pass rate. Hence Provider 1 has the highest combined first sit pass rate
and, predictably the lowest level of Not Competent candidates. The level of Not
Competent candidates rises in line with the drop in the Provider pass rate. By contrast,
the number of Outstanding candidates presents a profile less closely linked to overall
pass rates, with Provider 5 showing a higher proportion of Outstanding candidates
than Provider 1.
19
7.5 Combined MCQ and SAQ all-Provider first sit post-scale pass rates trend
analysis 2011/12 to 2014/15
Professional Ethics First Sit Combined (MCQ & SAQ)

Post-scale pass rated by Provider 2011/12 to 2014/15
100.0
Pass rate %
90.0
80.0
70.0
60.0
50.0
40.0
30.0
1
10
Provider
% Pass 2011/12
% Pass 2012/13
% Pass 2013/14
% Pass 2014/15
Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit combined
Professional Ethics pass rate, showing how their 2012/13, 2013/14 and 2014/15 pass
rates compare. Hence Provider 1 achieved a 92% pass rate in 2011/12, 83% in
2012/13. 54.7% in 2013/14, and 41.7% in 2014/15 and, along with Providers 4 and 8
records a drop in combined pass rates year on year across all 4 cycles of first sit
assessments in Professional Ethics. Five Providers show a decline in their combined
pass rate compared to 2013/14 (Provider 8 showing the largest drop at over 18%),
whilst 5 Providers improve their combined pass rates (Provider 3 up 8%). Provider 3
has the best 4-year average pass rate at over 78%, whilst the weakest, on average,
over the same period is Provider 4 with an average pass rate just over 63%. Note that
results for that BPP Manchester cohort are not reflected in this table as there is no
data pre 2013/14, and results for Kaplan have been excluded, as their last intake was
in 2013/14.
20
7.6 All-Provider first sit post-scale grade boundaries trend analysis 2011/12 to
2014/15
Professional Ethics First Sit post-scale

Grade boundaries trend analysis 2011/12 to 2014/15
60.00
% students
50.00
40.00
30.00
20.00
10.00
0.00
Outstanding
Very Competent
Competent
Not Competent
Grade category
2011/12
2012/13
2013/14
2014/15
No significant movement in classifications between 2013/14 and 2014/15.

8. 2014/15 first sit results in Criminal Litigation: MCQs
Criminal Litigation
rate
2014/15
2013/14
2012/13
2011/12
80.4%
74.6%
85.3%
86.8%
A 6% rise in the pre-scale MCQ all-Provider pass rate compared to the previous
year.
21
8.2 MCQ pre-scale first sit histogram question by question
Pass rate %
Criminal Litigation 2014/15 First Sit Pre-scale

100.0
90.0
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
1
11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
MCQ item number
This table indicates the range of pre-scale pass rates for the Criminal Litigation MCQs
across all Providers, showing 2 MCQs with a pre-scale pass rate of below 40%
(compared to 3 in 2013/14 first sit).
8.3 Final exam board review of the Criminal Litigation MCQ results
Question
MCQ 2
Intervention
applied
No
intervention
CEB rationale
32.1% all-Provider pass rate with high discrimination.
The Board noted the relatively low pass rate and the lack
of comments from Providers and External Examiners. It
was agreed this was a harder question but one that was
appropriate for the paper.
MCQ 9
No
intervention
45.9% all-Provider pass rate with very good

discrimination. The Board noted the low pass rate and
considered the comments from Providers, especially
referring to the query over the correct answer. The
Board agreed there was no possibility distractor C was
the correct answer and the correlation figure supported
this. The Board considered the issue of syllabus
coverage and determined that there were no matters of
concern.
22
MCQ 13
MCQ 16
MCQ 33
MCQ 39
No
intervention
No
intervention
The Board decided there was no reason for

intervention.
48.9% all-Provider pass rate and good discrimination.
That the pass rate was below 50% was noted and
Provider comments were considered. The statistics
were positive and showed this was a harder question in
the paper. The distractors were working.
48.1% all-Provider pass rate and very strong
discrimination. The pass rate below 50% was noted and
comments from Providers were considered. The Board
agreed the question was covered in the syllabus.
The Board decided there was no reason for

intervention.
65.3% all-Provider pass rate and good discrimination.
The comments from Providers were considered. There
was a concern with distractor D being wrong which would
Credit D as
make it the correct answer. The Board discussed and
well as B.
agreed to intervene and credit those candidates who had
answered D as well as those who had chosen B.
Remove
question 39
from the
paper.
The Board decided to intervene and credit D as well as

B.
22.4% all-Provider pass rate with very poor
discrimination. The Board considered the comments
from Providers and External Examiners as well as the
very poor discrimination and low pass rate. The Board
agreed to remove the question, as the answers offered
did not have the required level of clarity to fall safely in
to either a right or wrong category. The Board agreed
that the rationale for removing an entire question was
satisfied: because of the defect in the question, there
was no correct answer hence no candidate lost
anything by virtue of the questions removal. Further
there was statistical evidence that candidates may have
been guessing which answer was correct (i.e. weak
correlation with strong candidate scores)
The Board decided to remove question 39 from the
paper.
There were no other global or generic issues relating to the MCQ assessment
23
Criminal Litigation
2014/15
2013/14
2012/13
2011/12
83.3%
84.1%
88.9%
88.7%
MCQ Post-scale All-Provider pass

rate
The post-scale MCQ first sit pass rate for 2014/15 of 83.3% shows the impact of the
CEB intervention in raising the pre-scale pass rate of 80.4% by 2.9. The trend data for
Criminal Litigation MCQ national pass rates over these three cycles of first sit
assessments indicates a broadly consistent pattern of all-Provider cohort performance
within a 6% range.
8.5 MCQ pre and post-scale first sit pass rates across all Providers 2014/15
Criminal Litigation First Sit 2014/15

MCQ pass rated per Provider
95
Pass rate %
90
85
80
75
MCQ 2014/15 pre-scale
70
65
60
1
10
11
Provider
The graph shows the effect of the interventions in respect of MCQs 33 & 39 across all
Providers. Providers are ranged in order of pass rates (high to low) according to their
2014/15 first sit pre-scale cohort performance. Hence Provider 1 had the highest pass
rate both before and after CEB intervention. Generally the intervention favoured the
weaker cohorts, particularly Provider 10, showing an uplift of 10%. The effect of the
intervention on the range of pass rates across Providers shows a compression from
25% pre scale to 19% post-scale.
24
8.6 MCQ post-scale first sit pass rate by Provider 2011/12 to 2014/15
Criminal Litigation MCQ Post-scale pass rates

First Sit 2011/12 to 2014/15
95.0
Pass rate %
90.0
85.0
80.0
75.0
70.0
1
10
Provider
Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit Criminal
Litigation MCQ pass rate, showing how their 2012/13, 2013/14 and 2014/15 pass rates
compare. Hence Provider 1 achieved a 93% pass rate in 2011/12, 89% in 2012/13.
89% in 2013/14, and 82% in 2014/15. Six Providers record a lower cohort pass rate
for 2014/15 compared to 2013/14. Provider 2 has the best 4-year average pass rate
at over 90%, whilst the weakest, on average, over the same period is Provider 10 with
an average pass rate just over 81%. Note that results for that BPP Manchester cohort
are not reflected in this table as there is no data pre 2013/14, and results for Kaplan
have been excluded as their last intake was in 2013/14.
9. 2014/15 first sit results in Criminal Litigation: SAQs
9.1 SAQ pre-scale all Provider first sit pass rate 2011/12 to 2014/15
Criminal Litigation
rate
2014/15
2013/14
2012/13
2011/12
64.2%
75.7%
54.5%
73.9%
Data for 2014/15 shows an 11.5% drop in pre-scale all-Provider pass rates compared
to 2013/14 but sitting in the middle of the range for the 4 year cycle as a whole.
25
9.2 SAQ pre-scale first sit histogram question by question (showing sub-parts)
Criminal Litigation First Sit 2014/15 Pre-scale

90
80
Pass rate %
70
60
50
40
30
20
10
0
This table indicates the range of pre-scale pass rates for the Criminal Litigation SAQs
across all Providers, with a relatively strong cohort performance in respect of all the
SAQs, with only one sub-element, SAQ2(c) recording an all-Provider pass rate below
40% (compared to 4 in the 2013/14 SAQ first sits).
9.3 Final exam board review of the Criminal Litigation SAQ results
Question
SAQ 2(c)
Intervention
applied
No intervention
CEB rationale
SAQ2(c) had 2 marks available and the allProvider pass rate for the component was 19.8%.
There were no comments from Providers regarding
this question. The Board considered the relatively
low pass rate and considered how many
candidates only scored 1 mark. The Independent
Psychometrician confirmed the percentage of
candidates scoring 0 was 27%, candidates scoring
1 mark was 54% and scoring 2 marks was 20%.
The Board noted that the sampling of scripts
suggested that it was the second point that
candidates did not pass and felt it was a harder
question. The Board considered the operation of
the marking scheme, in particular the fact that
candidates only capable of securing 1 mark out of
2 would not be achieving the 60% rate required to
pass the paper as a whole. The Board considered
the options for crediting an extra mark or removing
the marks for the question, but concluded that the
assessment had performed as it would have
26
wanted, and this conclusion was reinforced by the

absence of Provider comments on the question.
SAQ4(c)

intervention.
SAQ4c had 3 marks available and the all-Provider
pass rate for the component was 47.1%. The
Board noted that two Providers in particular had
not performed well on this question. The comments
from Providers were considered and the Board
concluded that no changes were required.
No intervention

intervention.
There were no other global or generic issues relating to the SAQ assessment
9.4 SAQ post-scale first sit pass rates all-Providers 2011/12 to 2014/15
Criminal
Litigation
SAQ postscale AllProvider
pass rate
2014/15
2013/14
2012/13
2011/12
64.2%
78.2%
69.9%
77.8%
With no interventions in respect of the Criminal Litigation SAQ assessments the allProvider pass rate remains unchanged at 64.2%, 14% down on the previous round
of first sit assessments, and the lowest level recorded across the four cycles of first
sit assessment.
9.5 SAQ first sit pass rates across all Providers 2014/15

95
Pass rate %
85
75
65
SAQ 2014/15 Pass rate
55
45
1
Provider
27
10
11
The graph shows the range of performance across all 11 Providers in respect of the
Criminal Litigation SAQ assessment. As noted above, there were no interventions
sanctioned by the examination boards, hence there is no pre and post scale
comparison. There is a 36% range between the top performing cohort at Provider 1
and the weakest at Provider 11, an increase on the 27% post-scale range reported in
2103/14. The data for 2014/15 suggests a challenging assessment that the Criminal
Litigation SAQ discriminated effectively across a range of cohort ability.
Criminal Litigation SAQ Post-scale

Pass rates for First Sit 2011/12 to 2014/15
100
Pass rate %
90
80
70
60
50
40
1
10
Provider
Providers are ranked 1 to 10 based on their 2011/12 post-scale first sit Criminal
Litigation SAQ pass rates, showing how their 2012/13, 2013/14 and 2014/15 pass
rates compare. Hence Provider 1 achieved a pass rate of 91.1% in 2011/12, 89% in
2012/13, and 92.9% in 2013/14 and 80.8% in 2014/15. Provider 2, by contrast, has
seen a very worrying year-on-year decline in cohort performance from 90% in
20111/12, to 59.3% in 2014/15. None of the Providers reported in this graph recorded
cohort pass rates for the Criminal Litigation SAQs that were higher than their 2013/14
pass rates. Provider 10 recorded the biggest decline in pass rates from 2013/14, a
drop of over 28.7%. Provider 1 has the best 4-year average pass rate at over 88%,
whilst the weakest, on average, over the same period is Provider 7 with an average
pass rate just over 67%. Note that results for that BPP Manchester cohort are not
reflected in this table as there is no data pre 2013/14, and results for Kaplan have
been excluded, as their last intake was in 2013/14.
28
9.7 MCQ and SAQ first sit post-scale pass rates by Provider 2014/15

100.0
Pass rate %
90.0
80.0
70.0
60.0
50.0
40.0
1
10
11
Provider
This table compares 2014/15 Criminal Litigation MCQ and SAQ all-Provider post-scale
pass rates. The order of Providers is determined by the Providers combined post
scale pass rate for the MCQ and SAQs taken together (i.e. the percentage of
candidates at the Provider passing both sections). The data shows that cohorts
typically perform better in the MCQ element compared to the SAQ element (no cohort
performs better in the SAQ element). As in previous years there is some evidence to
suggest that the gap in pass rates between the two forms of assessment tends to
widen where cohorts are weaker overall (this is very much the case with Provider 11,
where the differential is over 34%).
10. 2014/15 first sit post-scale results in Criminal Litigation: combined MCQ
and SAQ results
10.1 Combined MCQ and SAQ all Provider first sit post-scale pass rates and grade
boundary distribution
Combined (MCQ and SAQ) All Provider post-scale results
Total number sat (All Providers)
Pass Combined
Fail Combined
1483
927
556
62.5%
37.5%

Outstanding
Very Competent
Competent
Not Competent
259
498
170
556
17.5%
33.6%
11.5%
37.5%
29
of the relevant confirmed MCQ and SAQ marks which may, or may not, have been
MCQ and SAQ marks are combined to produce the pass rates for a knowledge area
as a whole. Hence the combined knowledge area marks for cohorts and individual
10.2 Combined MCQ and SAQ first sit pre and post-scale pass rates by Provider
2014/15

Combined (MCQ and SAQ) Pre and Post scale pass rates
per Provider
90
Pass rate %
80
70
60
Combined 2014/15 pre scale
50
Combined 2014/15 post scale

40
1
10
11
Provider
The graph shows the effect of the interventions in respect of the MCQs (there were no
SAQ interventions) across all Providers. Providers are ranged in order of combined
pre-scale pass rates (high to low). Hence Provider 1 had both the highest pre-scale
and post-scale pass rates. Given the very limited level of intervention this year there
is very little difference in the pre and post scale result profiles.
10.3 Combined MCQ and SAQ post-scale first sit pass rates all Providers 2011/12
to 2014/15
Criminal Litigation
rate
2014/15
2013/14
2012/13
2011/12
62.5%
72.8%
68.2%
74.7%
The combined all-Provider pass rate is 14% down on the previous round of first sit
assessments, and is the lowest level recorded across the four cycles of first sit
assessment.
30

Post Scale grade boundaries by Provider
% students per category
100.0
90.0
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
1
10
11
Provider
Not Competent
Competent
Very Competent
Oustanding
Providers are ranked according to their 2014/15 post-scale first sit pass rates in
Criminal Litigation. Hence Provider 1 has the highest percentage of Outstanding
candidates and lowest percentage of Not Competent. The distribution of Not
Competent candidates follows the profile of declining cohort pass rates fairly closely,
whilst the distribution of Outstanding candidates across Providers 2 to 9 seems less
closely related to overall cohort performance. For example, Provider 10 is fifth best in
terms of the number of Outstanding candidates in its cohort.
10.5 Combined MCQ and SAQ all-Provider first sit post-scale pass rates trend
analysis 2011/12 to 2014/15
Criminal Litigation Combined (MCQ and SAQ)

Post-scale pass rates by Provider 2011/12 to 2014/15
100.0
% Pass 2011/12
%Pass 2012/13
% Pass 2013/14
% Pass 2014/15
Pass rate %
90.0
80.0
70.0
60.0
50.0
40.0
1
Provider
31
10
Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit combined
Criminal Litigation pass rate, showing how their 2012/13, 2013/14 and 2014/15 pass
rates compare. Hence Provider 1 achieved an 89% pass rate in 2011/12, 86% in
2012/13. 87% in 2013/14, and 76% in 2014/15. Provider 8 was the only Provider to
record a higher combined pass rate for 2014/15 as against 2013/14 (1.2%), with
Provider 9 showing a significant decline of over 28%. Provider 1 has the best 4-year
average pass rate at over 84%, whilst the weakest, on average, over the same period
is Provider 10 with an average pass rate just over 56%. Note that results for that BPP
Manchester cohort are not reflected in this table as there is no data pre 2013/14, and
results for Kaplan have been excluded, as their last intake was in 2013/14.
2014/15
Pass rate %
Criminal Litigation First Sit post-scale

45.00
2011/12
40.00
2012/13
35.00
2013/14
30.00
2014/15
25.00
20.00
15.00
10.00
5.00
0.00
Outstanding
Very Competent
Competent
Not Competent
Provider
The three-year trend analysis shows a continued decline in the Very Competent with
a corresponding increase in Not Competent, but little change in the other categories.
11. 2014/15 first sit results in Civil Litigation: MCQs
Civil Litigation
rate
2014/15
2013/14
2012/13
2011/12
63.6%
60.9%
46.3%
67.8%
A slight improvement on the 2013/14 all-Provider pre-scale performance in the Civil

Litigation MCQ assessment.
11.2 MCQ pre-scale first sit histogram question by question 2014/15
32
Pass rate %
Civil Litigation First Sit 2014/15 Pre-scale

100
90
80
70
60
50
40
30
20
10
0
1
11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
MCQ item number
This table indicates the range of pre-scale pass rates for the Civil Litigation MCQs
across all Providers, showing four MCQs with a pre-scale pass rate of below 40%,
(there were seven in 2013/14) compared with two in the 2014/15 Criminal MCQ
assessment.
11.3 Final exam board review of the Civil Litigation MCQ results
Question
MCQ4
MCQ5
Intervention
applied
No
intervention
Remove the
question from
the
assessment
CEB rationale
MCQ 04 43.1 % all-Provider pass rate, and strong
from Providers and External Examiners, discussing the
point raised regarding the syllabus coverage of the
question. The Board consulted the syllabus and agreed
it was covered. It was recognised that this was a
challenging question, however it was noted that there
should be a mixture of difficult questions in the paper.
MCQ 05 - 39.9% all-Provider pass rate, and very good
discrimination. It was noted that although the correct
answer for the question was D, more candidates chose
answer C. The Board considered if distractor C might
also potentially be correct.
The Chief Examiner
highlighted that there was a small technical inaccuracy
with distractor C, however it was agreed that this might
not be the reason why candidates chose C as the correct
answer. The Board discussed at length the options
available, noting that in principle the rule for technically
deficient questions should apply i.e. remove the
question. Part of the rationale for removing an entire
question was that, because of the defect in the question,
33
Question
Intervention
applied
CEB rationale
there was no correct answer, hence no candidate lost
anything by virtue of its removal. To the extent it could
be argued that there was a correct answer the statistics
strongly indicated that those choosing it may well have
been guessing (i.e. weak correlation with strong
candidate scores).
MCQ 8
MCQ 17
MCQ 20
No
intervention
No
intervention
Disregard the
question
The Board decided that question 5 would be removed

from the paper and the pass rate adjusted accordingly.
MCQ 08 - 24.9% all-Provider pass rate and good
discrimination. The Board noted the pass rate was below
25% and the comments from Providers were considered.
It was agreed the question matter was covered in the
syllabus. The Board discussed if the question was too
difficult for candidates and if distractor B was unfair. It
was agreed that it was a challenging question and the
examination paper needed a mixture of different difficulty
level questions. The Independent Psychometrician
confirmed the statistics showed this was a good question
and no concerns were raised.
MCQ 17 - 25.7% all-Provider pass rate and very good
from Providers and External Examiners suggesting
problems with identification of the correct answer. The
question was considered to be a negatively worded
question and there appeared to be some confusion with
candidates between selecting answers A and C. The
Board agreed that distractor C remained incorrect and
there was no need to intervene on the question.
MCQ 20 - 88.3% all-Provider pass rate with fair to
adequate discrimination. The Board considered the
technical comments from the External Examiner. The
Board agreed the question was technically deficient
meaning none of the answer options would have been
correct and therefore to be consistent with MCQ 5 agreed
to remove the question, with pass rates recalculated
accordingly. As with MCQ 5, part of the rationale for
removing an entire question was that, because of the
defect in the question, there was no correct answer
hence no candidate lost anything by virtue of its removal.
To the extent it could be argued that there was a correct
answer the statistics strongly indicated that those
34
Question
Intervention
applied
CEB rationale
choosing it may well have been guessing (i.e. weak
correlation with strong candidate scores)
MCQ 34
MCQ 35
The Board decided that question 20 would be removed

from the paper and the pass rate adjusted accordingly.
MCQ 34 - 34.4% all-Provider pass rate, with relatively
poor discrimination. The Board noted the low pass rate
with 50% of candidates opting for distractor A when the
correct answer was C. The comments from Providers
Credit options
and External Examiners were considered and the Board
C and A
agreed this was a poorly worded question.
The Board therefore agreed to credit both answers A and
C with a mark.
MCQ 35 - 48.2% all-Provider pass rate with very strong
discrimination. Whilst the Board was content regarding
syllabus coverage, it accepted concerns that answer A
Credit options might not be unambiguously correct, resulting in
unfairness to candidates who opted for distractor C.
A and C
The Board therefore agreed to credit both answers A and
C with a mark.
There were no other global or generic issues relating to the MCQ assessment
Civil Litigation
2014/15
2013/14
2012/13
2011/12
71.3%
68.6%
73.2%
83.7%
MCQ Post-scale All-Provider pass rate
The post-scale MCQ first sit pass rate for 2014/15 of 71.3% shows the impact of the
CEB intervention in raising the pre-scale pass rate of 63.6% by 7.7%, in effect a 12.1%
increase as a proportion of the pre-scale pass rate.
35
11.5 MCQ pre and post-scale first sit pass rates all Providers 2014/15
Civil Litigation First Sit 2014/15

MCQ pass rates per Provider
Pass rate %
100
80
60
MCQ 2014/15 pre-scale
40
1
Provider
10
11
The graph shows the combined effect of the interventions in respect of MCQs 5, 20,
34 and 35 across all Providers. Providers are ranged in order of pass rates (high to
low) according to their pre-scale cohort performance. Hence Provider 1 had the
highest pre- and post-scale pass rate. The interventions had a differential impact
across the Providers in the sense that it was not connected with the level of cohort
performance. Hence, Provider 1 benefited the most (13.3%), followed by Provider 8
(13%), and Provider 2 the least (3.9%).
11.6 MCQ post-scale first sit pass rate by Provider 2011/12 to 2014/15
Civil Litigation MCQ Post-scale pass rates

First Sit 2011/12 to 2014/15
90.0
85.0
Pass rates %
80.0
75.0
70.0
65.0
60.0
55.0
50.0
10
Provider
Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit Civil Litigation
MCQ pass rate, showing how their 2012/13, 2013/14 and 2014/15 pass rates
compare. Hence Provider 1 achieved an 86.5% pass rate in 2011/12, 76% in 2012/13.
67% in 2013/14, and 62.8% in 2014/15. Note Provider 1 has a year-on-year decline
across all four first sit cycles (the only Provider reported here for whom this is the
case). Four Providers record a lower cohort pass rate for 2014/15 compared to
36
2013/14, with Provider 9 declining most significantly by over 17%. Provider 2, by

contrast, achieves an improvement of over 13%. Provider 4 has the best 4-year
average pass rate at over 76%, whilst the weakest, on average, over the same period
is Provider 10 with an average pass rate just over 64%. Note that results for the BPP
Manchester cohort are not reflected in this table as there is no data pre 2013/14, and
results for Kaplan have been excluded, as their last intake was in 2013/14.
12. 2014/15 first sit results in Civil Litigation: SAQs
12.1 SAQ all Provider pre- scale first sit pass rate 2011/12 to 2014/15
Civil Litigation
rate
2014/15
2013/14
2012/13
2011/12
49.0%
52.7%
52.1%
61.4%
The Civil Litigation all-Provider SAQ pre-scale pass rate drops marginally by 3% from
the previous 2 rounds of assessment, and is 15% lower than the corresponding prescale Criminal Litigation SAQ figure.
12.2 SAQ pre-scale 2014/15 first sit histogram question by question (showing subparts)
Civil Litigation First Sit 2014/15 Pre-scale

80
70
Pass rate %
60
50
40
30
20
10
0
This table indicates the range of pre-scale pass rates for the Civil Litigation SAQs
across all Providers, clearly showing the relatively poor cohort performance in respect
of SAQ 1(d), SAQ 2(c), SAQ 3 (a) and (b), SAQ4 (b) and (c), although there is no SAQ
where the overall pass rate dips below 40%. In total 6 SAQ sub-elements record an
all-Provider pass rate below 40%, compared with 8 for the 2013/14 first sit cycle of the
same assessment.
37
12.3 Final exam board review of the Civil Litigation SAQ results
Question Intervention applied
SAQ1d
SAQ1
SAQ2c
SAQ2d
CEB rationale
SAQ1(d) had 3 marks available and the allProvider pass rate for the component was
21.4%. The Board noted the low pass rates at
all Providers. The Provider comments were
considered. The Board agreed the question and
answers were technically correct but felt that the
mark scheme was not sufficiently flexible - there
should have been 1 mark instead of half mark
Credit 0.5 mark where there is
available in 1(d) (i) as it was a key point in the
headroom to do so
answer. As a consequence, the Board
recommended all candidates to be awarded an
additional 0.5 where there was headroom to do
so.
The Board decided to award an additional 0.5
mark to all candidates where there was
headroom to do so for 1(d).
SAQ 1 award additional 0.5 mark for SAQ1 as
a whole (due to generic narrowness of the mark
scheme).
headroom to do so
SAQ2(c) had 3 marks available and an allProvider pass rate of 21.9%. The Board
considered the relatively low pass mark for all
Providers for this question, with 4 Providers
achieving pass rates below 11%. The Board
agreed they were happy that the question had
appropriate syllabus coverage and the answer
was technically correct. In relation to 2(c)
however, the Board agreed to award all
candidates 1 additional mark where there was
headroom to do so
headroom to do so with reference to the
narrowness of the mark scheme in relation to
the second bullet point on mark scheme,
including reliability affected by situation re:
Henrys wife.
The Board decided an additional 1 mark would
be award to all candidates in relation to 2c,
where there was headroom to do so.
SAQ2(d) had 2 marks available and an allCredit 0.5 mark where there is Provider pass rate of 44.9%. The Board noted
headroom to do so
the weaker pass rates throughout the question.
The mark scheme was comprised entirely of half
38
marks, thus candidates would need 3 half marks

to pass. The Board agreed an additional 0.5
mark should be given to all candidates, where
there was headroom to do so, in respect of
bullet point 1 in the mark scheme, WRU can put
written questions about the experts report as
this point should have attracted one mark given
its centrality in the answer.
SAQ2
SAQ3a
SAQ3b
SAQ3e
The Board decided to award a further 0.5 mark

to all candidates, where there is headroom to do
so in relation to SAQ 2(d).
SAQ2 award an additional 0.5 mark for SAQ2
as a whole (due to narrowness of mark
headroom to do so
scheme).
SAQ3(a) had 2 marks available with an allProvider pass rate of 38.6%. The Board noted
appropriate syllabus and the revisions to the
mark scheme after the consideration of
feedback. On the basis that neither 3(a) or 3(b)
operated as effective gateways to enable
Credit 0.5 mark where there is candidates to demonstrate what they knew, the
Board agreed to award an additional 0.5 mark
headroom to do so
for all candidates in respect of 3(a) where there
was headroom to do so.
The Board decided to award an addition 0.5
mark to all candidates in respect of 3(a) where
there was headroom to do so.
SAQ3(b) had 2 marks available with an allProvider pass rate of 24.6%. The Board agreed
to award an additional 1 mark, in respect on
3(b), for all candidates where there was
Credit 1 mark where there is headroom to do so for the same reasons as
headroom to do so
stated above in SAQ3(a).
The Board decided to award an additional 1
mark to all candidates in respect of 3b where
there was headroom to do so
SAQ3(e) had 2 marks available with an allProvider pass rate of 42.7%. The Board
discussed the mark scheme and comments
from Providers. It was agreed that in respect of
3(e), an additional 0.5 mark would be awarded
headroom to do so
to all candidates, where there was headroom to
do so, for the point in the mark scheme relating
to point (i) Permission to make an additional
claim is not required provided. On reflection the
39
Board agreed that this should have attracted 1

mark given its centrality to the answer.
SAQ4a
SAQ4b
The Board decided that an additional 0.5 mark

would be awarded to all candidates where there
was headroom to do so.
SAQ4(a) had 2 marks available with an allProvider pass rate of 42.1%. The Board
discussed the comments from Providers and
the mark scheme for the question. It was agreed
that an additional 0.5 mark would be awarded to
all candidates where there was headroom to do
Credit 0.5 mark where there is so. The Board felt the inflation point should
have attracted one mark given its centrality to
headroom to do so
the answer. This intervention brought the pass
rate from 42.1% pre scale to 57.6% post scale.
No intervention
The Board decided to award an additional 0.5

mark to all candidates where there is headroom
to do so.
SAQ4(b) had 2 marks available with an allProvider pass rate of 36.7%. The Board
discussed at length the range of performance
across all Providers, noted that there were no
Provider comments for this question and agreed
the mark scheme was very generous in respect
of 4(b), concluding that no interventions were
warranted.
intervention.
SAQ4c had 1 mark available with an allProvider pass rate of 26.5%.
SAQ4c
No intervention
intervention.
There were no other global or generic issues relating to the SAQ assessment.
12.4 SAQ post-scale first sit pass rates all-Providers 2011/12 to 2014/15
Civil Litigation
SAQ post-scale All-Provider pass
rate
2014/15
2013/14
2012/13
2011/12
65.0%
67.8%
61.5%
73.5%
The post-scale SAQ first sit pass rate for 2014/15 of 65% shows the impact of the CEB
intervention in raising the pre-scale pass rate of 49% by 16%, in effect a 32% increase
as a proportion of the pre-scale pass rate.
40
12.5 SAQ pre and post-scale first sit pass rates across all Providers 2014/15

95
85
Pass rate %
75
65
55
45
SAQ 2014/15 pre-scale
35

25
1
10
11
Provider
The graph shows the effect of the nine SAQ interventions detailed above at 12.3.
Providers are ranged in order of pass rates (high to low) pre-scale. Hence Provider 1
had both the highest pre-scale SAQ pass rate, but Providers 9 to 11 benefitted more
from the effects of the interventions with pass rates for their cohorts rising by
approximately 20%. Scaling had only a modest effect on the range of pass rates
across Providers. It was 40% pre-scale and 35% post-scale.
Civil Litigation SAQ Post-scale

Pass rate for First Sit 2011/12 to 2014/15
90
% pass 2011/12
% pass 2012/13
80
Pass rate %
% pass 2013/14
% pass 2014/15
70
60
50
40
30
1
Provider
41
10
Providers are ranked 1 to 10 based on their 2011/12 post-scale first sit Civil Litigation
SAQ pass rates, showing how their 2012/13, 2013/14 and 2014/15 pass rates
compare. Hence Provider 1 achieved a pass rate of 81% in 2011/12, 53.7% in
2012/13, and 74.6% in 2013/14 and 77.2% in 2014/15. Provider 2, by contrast, has
seen a very worrying year-on-year decline in cohort performance from 90% in
20111/12, to 59.3% in 2014/15. Four Providers reported in this graph recorded cohort
pass rates for the Civil Litigation SAQs that were higher than their 2013/14 pass rates,
with Provider 6 showing a 13% rise. Provider 5 recorded the biggest decline in pass
rates from 2013/14, a drop of over 24%. Provider 1 has the best 4-year average pass
rate at over 71%, whilst the weakest, on average, over the same period is Provider 10
with an average pass rate just over 47%. Note that results for the BPP Manchester
cohort are not reflected in this table as there is no data pre 2013/14, and results for
Kaplan have been excluded, as their last intake was in 2013/14.
12.7 MCQ and SAQ first sit post-scale pass rates by Provider 2014/15 high to low

90
Pass rate %
80
70
60
50
40
1
10
11
Provider
This table compares 2014/15 Civil Litigation MCQ and SAQ all-Provider post-scale
pass rates. The order of Providers is determined by a Providers combined post-scale
pass rate for the MCQ and SAQs taken together (i.e. the percentage of candidates at
the Provider passing both sections). The data shows Provider 1 with the strongest
performances in both MCQs and SAQs. Provider 2 is the only Provider to have a
higher pass rate for the SAQ assessments compared with the MCQs (0.2% higher).
Elsewhere the pattern is for a much stronger showing by cohorts in the MCQs, notably
for Providers 5 and 6 where it is 17% and 13% respectively. In general the pattern of
SAQ performance tracks that for MCQs albeit at a lower level of pass rate.
42
13. 2014/15 first sit post-scale results in Civil Litigation etc.: combined MCQ and
SAQ results
13.1 Combined MCQ and SAQ all Provider first sit post-scale pass rates and grade
boundary distribution 2014/15

Combined (MCQ and SAQ) All Provider post-scale
results
Total number sat (All
Providers)
Pass Combined
Fail Combined
1595
925
670
58.0%
42.0%

Outstanding
207
Very Competent
504
Competent
214
Not Competent
670
13.0%
31.6%
13.4%
42.0%
of the relevant confirmed MCQ and SAQ marks, which may or may not have been
MCQ and SAQ marks are combined to produce the pass rates for a knowledge area
as a whole. Hence the combined knowledge area marks for cohorts and individual
13.2 Combined MCQ and SAQ first sit pre and post-scale pass rates by Provider
2014/15
Civil Litigation First Sit 2014/15 Combined

(MCQ and SAQ) Pre and Post scale pass rates by Provider
Pass rate %
100
Combined 2014/15 pre-scale

Combined 2014/15 post-scale
80
60
40
20
1
Provider
43
10
11
The graph shows the combined effect of the interventions in respect of both the MCQs
and SAQs across all Providers. Providers are ranged in order of pre-scale combined
pass rates (high to low). Hence Provider 1 had the highest pre-scale and post-scale
pass rates. The intervention typically resulted in an 15% uplift, but, somewhat counterintuitively, the greatest impact was on Provider 1 where the increase was 20%), with
the result that the interventions actually increased the range between the strongest
and weakest cohort pass rates from 37% pre-scale to 43% post-scale
13.3 Combined MCQ and SAQ post-scale first sit pass rates all Providers 2011/12
to 2014/15
Civil Litigation

rate
2014/15
2013/14
2012/13
2011/12
58.0%
57.4%
56.2%
68.0%
The post-scale combined first sit pass rate for 2014/15 of 58% is largely in line with
the 2012/13 outcome for first sit candidates, 4% below that for Criminal Litigation
2014/15.
% students per category

Post scale grade boundaries by Provider
100.0
90.0
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
1
10
11
Provider
Not Competent
Competent
Very Competent
Oustanding
Providers are ranked according to their combined 2014/15 post-scale first sit pass
rates in Civil Litigation. Hence Provider 1 has the highest post-scale first sit pass rate
but not the highest percentage of Outstanding candidates (see Provider 2 with
22.3%). The distribution of Not Competent candidates does however follow the order
of Providers in the above chart.
44
13.5 Combined MCQ and SAQ first sit post-scale pass rates trend analysis 2011/12
to 2014/15
Pass rate %
Civil Litigation First Sit Combined (MCQ and SAQ)

Post scale pass rates by Provider 2011/12 to 2014/15
80.0
%pass 2011/12
75.0
% pass 2012/13
70.0
% pass 2013/14
65.0
% pass 2014/15
60.0
55.0
50.0
45.0
40.0
35.0
1
10
Provider
Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit combined Civil
Litigation pass rate, showing how their 2012/13, 2013/14 and 2014/15 pass rates
compare. Hence Provider 1 achieved a 71% pass rate in 2011/12, 50% in 2012/13,
47% in 2013/14, and 54% in 2014/15. Five Providers recorded a higher combined
pass rate for 2014/15 as against 2013/14 (Provider 6 up by over 10%), whilst Provider
8 recorded the sharpest decline at over 13%. Provider 9 is the only Provider to record
a year-on-year decline in combined pass rates across all 4 cycles of first sit
assessment. Provider 2 has the best 4-year average pass rate at over 64%, whilst the
weakest, on average, over the same period is Provider 10 with an average pass rate
just over 45%. Note that results for the BPP Manchester cohort are not reflected in this
table as there is no data pre 2013/14, and results for Kaplan have been excluded, as
their last intake was in 2013/14.
45
2014/15
Civil Litigation First Sit Post scale

50.0
2011/12
2012/13
2013/14
2014/15
% students
40.0
30.0
20.0
10.0
0.0
Outstanding
Very Competent
Competent
Not Competent
Category
The four-year trend analysis shows a slight recovery in the level of Outstanding
classifications with a corresponding drop in the Competent grouping.
14 First sit modes of assessment and subject areas 2014/15
14.1 MCQ first sit post-scale pass rates for 3 CEB areas 2014/15 compared
First Sit 2014/15 MCQ post scale

Pass rate for knowledge areas by Provider
100
Pass rate %
90
80
70
Ethics MCQ 2014/15 post scale
60
Crime MCQ 2014/15 post scale

Civil MCQ 2014/15 post scale
50
1
10
11
Provider
This graph compares the post-scale first sit MCQ pass rates for the three knowledge
areas across all Providers 2014/15. Providers are ranked according to the average of
their post-scale MCQ pass rates across the three knowledge areas. Hence Provider 1
had the highest average cohort pass rate across all three MCQ assessments and
Provider 11 the lowest average across all three. All Providers recorded their highest
46
MCQ pass rates in respect of Professional Ethics and all recorded their lowest MCQ
pass rates in respect of Civil Litigation. The gap between a Providers pass rate in
Professional Ethics and its pass rate in Civil Litigation tends to be more marked in
respect of the weaker cohorts. For Provider 1 the gap is 9%, whilst for Providers 8 and
10 the gap is over 33%. Provider 2 slightly bucks this trend with a surprisingly poor
showing in Civil Litigation where the pass rate is 28% below that for Professional
Ethics. Provider 1 has an average MCQ pass rate of over 93%, compared with 69.8%
for Provider 11 showing a 23% range between the strongest MCQ Provider
performance and the weakest.
14.2 SAQ first sit post-scale pass rates for 3 CEB areas 2014/15 compared
First Sit 2014/15 SAQ post scale

90
80
Pass rate %
70
60
50
Ethics SAQ 2014/15 post scale
40
Crime SAQ 2014/15 post scale
30
Civil SAQ 2014/15 post scale
20
1
Provider
10
11
This graph compares the post-scale first sit SAQ pass rates for the three knowledge
areas across all Providers 2014/15. Providers are ranked according to the average of
their post-scale SAQ pass rates across the three knowledge areas. Hence Provider 1
had the highest average cohort pass rate across all three SAQ assessments and
Provider 11 the lowest average across all three. Unlike the MCQ data reported at 14.1
there is a more complex picture in terms of Provider performance in respect of the
SAQ assessments. For three Providers their strongest cohort performance was in
respect of the Professional Ethics SAQ, for five it was in respect of the Criminal
Litigation SAQ and for three the Civil Litigation SAQ. In terms of weakest SAQ
performance, for five Providers this was in respect of Professional Ethics, for two it
was in respect of Criminal Litigation, and for four it was in respect of Civil Litigation.
It is also instructive to look at the average SAQ pass rate in the two litigation modules
for each Provider and compare this with the Providers pass rate in respect of the
Professional Ethics SAQ. Provider 1s average pass rate across the SAQs in the two
litigation modules is 85.4%, compared with a Professional Ethics SAQ pass rate of
81.6%, hence a 3.8% negative variance in respect of the Professional Ethics SAQ.
Across all 11 Providers, applying the same methodology, there is an average negative
variance of 5.3% between the average of their litigation module SAQ pass rates and
47
their pass rates for the Professional Ethics SAQ. Significantly, however, there are
three Providers where the negative variance is very much higher: Provider 7 (-24.4%);
Provider 8 (-24.7%) and Provider 10 (-21.7%). If these three outlier Providers are
removed from the equation, the remaining eight Providers actually record a positive
variance of 1.5% in respect of their pass rate in Professional Ethics SAQ compared
with the average of their litigation module SAQ pass rates, suggesting the overall
picture is being skewed by a very poor Professional Ethics SAQ cohort performance
at three Providers. Again the range between a Providers best SAQ pass rate and their
worst is typically 15%. For Providers 7, 8 and 10 the range is between 22% and 26%
because of their poor performance in the Professional Ethics SAQs. In terms of the
range of Provider cohort performance in SAQs as a whole it is striking that there is a
34% gap between the average SAQ pass rate for Provider 1 and that for Provider 11.
14.3 Combined MCQ and SAQ 2014/15 first sit post-scale pass rates for 3 CEB areas
2014/15 compared
First Sit 2014/15 Combined post scale

Ethics Combined 2014/15 post
scale
Crime Combined 2014/15 post
scale
Civil Combined 2014/15 post scale
90
Pass rate %
80
70
60
50
40
30
1
10
11
Provider
This graph compares the post-scale first sit combined pass rates for the three
knowledge areas across all Providers for 2014/15. Providers are ranked according to
the average of their post-scale combined pass rates across the three knowledge
areas. Hence Provider 1 had the highest average pass rate across the three combined
assessments, with pass rates of around 82% for all three modules (the best cohort
across all three). By contrast Provider 11 has an average combined pass rate across
the three modules of 44.8%, some 37% below that of Provider 1. The graph shows a
reasonably clear trend of declining pass rates from Provider 1 to Provider 11, but the
significantly poor performances by Providers 7, 8 and 10 in respect of Professional
Ethics (effectively the SAQ element) is apparent from the graphs, as is the fact that
Providers 4, 6 and 9 recorded their highest combined pass rates in respect of
Professional Ethics.
48
14.4 Comparison of Provider first sit post-scale pass rates by type of assessment
2014/15
First Sit 2014/15 MCQ and SAQ post scale

100
90
Pass rate %
80
70
60
Ethics MCQ 2014/15 post scale
Ethics SAQ 2014/15 post scale
Crime MCQ 2014/15 post scale
Crime SAQ 2014/15 post scale
Civil MCQ 2014/15 post scale
Civil SAQ 2014/15 post scale
50
40
30
20
1
10
11
Provider
This graph compares the post-scale first sit MCQ and SAQ pass rates for the three
knowledge areas across all Providers 2014/15. Providers are ranked according to the
average of their post-scale combined pass rates across the three knowledge areas.
The data shows a very strong performance from the Provider 1 cohort, achieving the
highest pass rate in each of the 6 assessments. This represents an average of 88%
of candidates at Provider 1 passing each paper, but it should be noted that this figure
drops to 81.9% (see 14.3 above) when the non-aggregation rule is applied whereby
candidates have to achieve a minimum of 60% in both the MCQ and SAQ elements
of the assessment to achieve a pass in the module overall. The poorest cohort
performances were concentrated amongst the lowest ranked Providers, with Providers
9, 10 and 11 accounting between them for the lowest pass rates recorded across all
six assessments (four of them at Provider 11). The absolute weakest performance
from any cohort in any assessment was delivered by Provider 10 in the Professional
Ethics SAQs where only 37% of candidates passed.
14.5 Post-scale 2014-15 first sit grade boundaries across the three knowledge areas
compared
Outstanding
Very Competent
Competent
Not Competent
Professional
Ethics
8.8%
35.4%
12.5%
43.3%
Criminal Litigation
Civil Litigation
17.5%
33.6%
11.5%
37.5%
13.0%
31.6%
13.4%
42.0%
49
The data shows a remarkable level of consistency across all 3 modules in respect of
the Very Competent, Competent and Not Competent with a small gap between
the percentage of candidates achieving Outstanding in Professional Ethics
compared to the litigation modules. The 8.8% figure for Professional Ethics represents
a small increase from the 7.2% recorded in 2013/14, whilst the percentage achieving
Outstanding in Civil Litigation has almost doubled compared to 2013/14. The 10%
rise in candidates classified as Not Competent in Criminal Litigation is disappointing.
15. First sit modes of assessment and subject areas trends 2010/11 to 2014/15
15.1 Changes in all-Provider post-scale first sit knowledge area subject pass rates
2011/12 to 2014/15
First Sit post scale MCQ and SAQ

Pass rate trend analysis for 2011/12 to 2014/15
100.0
Pass rate %
90.0
80.0
70.0
60.0
50.0
2011/12
2012/13
2013/14
2014/15
Ethics MCQ
Ethics SAQ
Criminal MCQ
Criminal SAQ
Civil MCQ
Civil SAQ
The 2014/15 first sit assessments in the BPTC knowledge areas represent the fourth
cycle of first sit assessments under the CEB processes. Hence it is now possible to
look across these four first sit assessment points to explore whether any discernible
data trends are emerging. This table shows how pass rates have changed across both
assessment modes in all three knowledge areas from the first sit in 2011/12 to the first
sit in 2014/15. Across the four cycles Professional Ethics MCQ has had the highest
average pass rate at 89.9%, with Criminal Litigation MCQs at 86.3%, and Professional
Ethics SAQs at 75.4%. Civil Litigation SAQs have the lowest four-year average at
67%. Across the four-year cycle Criminal Litigation MCQs have the narrowest range
of pass rates, all within a 5% band. Most modules record a 15% variance band with
Professional Ethics showing the widest range at 30% (but see comments at 14.2
regarding the impact of specific Provider cohort performance on the 2014/15 data).
Looking across each year the range between the lowest and highest all-Provider pass
rate across the six modules varies considerably. For the 2011/12 and 2013/14 cycles
the range was approximately 20%, for each of the remaining cycles it was 33%. There
is no knowledge area assessment displaying a four cycle year-on-year trend of rising
or declining pass rates.
50
15.2 Change in Provider pass rates 2011/12 to 2014/15
First Sit changes in Provider pass rate 2013/14 vs 2014/15

30
Professional Ethics change

Criminal Litigation change
20
Civil Litigation change
% change
10
0
1
10
11
-10
-20
-30
Provider
This shows a mixed profile in terms of comparing Provider first sit pass rates for
2014/15 with 2013/14. In total, seven of the eleven Providers are able to demonstrate
an improvement in at least one knowledge areas cohort pass rate when compared to
2013/14. Provider 1 returns an excellent performance with significant improvements
in pass rates in all three modules, an average increase in pass rates of 18%. Three
Providers are able to record improvements in two module areas, with five of the
improved scores are in respect of Professional Ethics, and six in respect of Civil
Litigation. The biggest single increase in performance is by Provider 1 in respect of
Professional Ethics, with an improvement of 24.7%. The most significant reversal is
recorded by Provider 6 in respect of Criminal Litigation, where the decline is almost
29%. More generally, the decline across nine Providers in the performance in Criminal
Litigation is disappointing.
16. Post examination board issues
16.1 Administrative checks on data were conducted after the July 2015 final CEB
meeting as a further quality assurance process.
16.2 BPTC candidates seeking confirmation of their marks are required to comply with
the regulations and guidance available on the BSB website. In the first instance a
candidate is required to seek a clerical error check from the relevant Provider. As a
next step the candidate can seek an enhanced clerical error check from the BSB, for
which a fee is payable. If any errors are found in the computation or transcription of
marks, CEB Chairs action can be taken to confirm the rectification with consequent
endorsement at the relevant Provider institution.
16.3 The BSB has not put in place any process by which a candidate can ask for a
remark of an SAQ paper, and there is no provision for any BSB appeal process that
51
allows any challenge to the exercise of academic judgement on the part of the CEB.
A candidate can submit a request for review of a CEB decision on the basis that the
CEB has acted irrationally or unfairly in confirming the cohort marks for the centrally
assessed examinations. An application for review must be made in the required form,
within stipulated time limits and a fee is payable. The BSB will filter applications and
refer those establishing an arguable case to the CEB Review Panel (an independent
panel appointed by the BSB). If the Review Panel finds that there is evidence that
warrants referral back to the CEB, it will give reasons for its decision and invite the
CEB to reconsider the decision in question. The guidance accompanying the
regulations makes clear to candidates who have failed examinations and are required
to take resits that an application for review should not be seen as an alternative to
attempting the resit examinations, as any determination of the CEB Review panel and,
in turn the CEB, is likely to be delivered after the time for resit examinations has
elapsed.
17. Quality enhancement and policy issues
17.1 Improved scrutiny of draft assessments resulted in a reduction in typographical
errors, which is to be welcomed. Feedback from Providers and External Examiners
on assessments after they have been attempted continues to be an enormously
important reference point for the subject boards, although Providers are encouraged
to be more specific where possible as generalised comments are very hard for the
boards to act on.
17.2 From time to time concerns are expressed relating to the perceived disparity
between the pass rates confirmed by the CEB in respect of the six assessments within
its jurisdiction. Candidates should be assured that cohort results are subject to
statistical analysis to ensure that the assessments are operating effectively, and that
both Provider and External Examiner feedback, particularly in relation to SAQ
assessments and marking schemes, is given very serious consideration. Details of
the CEB interventions set out in this report evidence that. It will also be apparent from
this report that, within a set of module results, a particular Provider cohort, or a number
of Provider cohorts, achieve disappointing results when compared with the wider
group. Whilst the CEB can intervene by way of scaling to compensate for any factor
outside the control of a Provider that might have impacted on the conduct of the
assessment at a Provider (for example disruption during an examination), the CEB
does not intervene to compensate for any factors related to the general delivery of a
module by a Provider (e.g. quality of tuition, coverage of syllabus etc.). Other factors
can also have an effect on pass rates from year to year including changes in the
syllabus and a change in the range of Providers themselves.
17.3 For the moment the CEB is continuing to recommend to the BSB that Providers
are not identified when details of cohort performance in the centralised examinations
are published. An immediate consideration is that there are 10 BPTC modules
representing 120 credits in total. Three of these modules are the centrally assessed
knowledge areas, representing 25% of the course (30 credits). The remaining seven
modules represent 75% of the BPTC final award (90 credits). It would be wrong,
therefore, to imply that the performance of any given Provider in relation to the
knowledge areas should be taken as a reliable proxy for the performance of that
52
Provider in respect of the BPTC as a whole. The CEB preference is that identification
of Providers and pass rates should only occur when there is a statistically reliable trend
data that brings together Provider cohort performance in both the centrally assessed
knowledge areas and the locally assessed skills areas.
Professor Mike Molan
Chair of the Central Examinations Board
31 July 2015
53

Central Examination Board Report August 2015

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Central Examination Board Report August 2015

Загружено:

Авторское право:

Доступные форматы

THE BAR STANDARDS BOARD

CENTRAL EXAMINATIONS BOARD

First Sit 2014/15

BPP London, Manchester and Leeds

1.7 Terms used in this report:

All-Provider refers to the aggregated data bringing together cohort

2. The setting and conduct of assessments

Professional Ethics 23 March 2015

subject to similar to arrangements in 2.3) as the UK start time (regardless of time

Representations made by Providers in the MCQ/SAQ feedback pro-formas

Crediting more than one correct answer to an MCQ

Adjusting marks globally for an SAQ as a whole, an SAQ paper as a whole, or

5.2 MCQ pre-scale 2014/15 first sit histogram question by question

Professional Ethics 2014/15 First Sit Pre-Scale

MCQ item number

45.7% all Provider pass rate, very good discrimination

The Board decided there was no reason for intervention.

Professional Ethics First Sit 2014/15

MCQ 2014/15 Pass rate

Professional Ethics MCQ Post-Scale pass rates

MCQ 2012/13 post-scale

MCQ 2013/14 post-scale

MCQ 2014/15 post-scale

Following a significant decline in Professional Ethics SAQ pre-scale pass rates in

Professional Ethics 2014/15 First Sit Pre-scale

SAQ sub-parts and totals

SAQ1(b) had 4 marks available and the all-Provider pass

The Board considered comments from Providers and

SAQ3(a) had 5 marks available and all-Provider pass

SAQ3(b) had 5 marks available and the all-Provider pass

Professional Ethics First Sit 2014/15

Professional Ethics SAQ Post-scale

SAQ 2011/12 post-scale

SAQ 2012/13 post-scale

SAQ 2013/14 post-scale

SAQ 2014/15 post-scale

Professional Ethics First Sit 2014/15

SAQ 2014/15 post-scale

Combined Grade Boundaries

Professional Ethics First Sit 2014/15

7.4 First sit 2014/15 post-scale grade boundaries by Provider

Professional Ethics First Sit 2014/15

Providers are ranked according to their 2014/15 post-scale combined Professional

Professional Ethics First Sit Combined (MCQ & SAQ)

Professional Ethics First Sit post-scale

No significant movement in classifications between 2013/14 and 2014/15.

8.2 MCQ pre-scale first sit histogram question by question

Criminal Litigation 2014/15 First Sit Pre-scale

MCQ item number

45.9% all-Provider pass rate with very good

The Board decided there was no reason for

The Board decided there was no reason for

The Board decided to intervene and credit D as well as

MCQ Post-scale All-Provider pass

Criminal Litigation First Sit 2014/15

MCQ 2014/15 pre-scale

Criminal Litigation MCQ Post-scale pass rates

MCQ 2011/12 post-scale

MCQ 2012/13 post-scale

MCQ 2013/14 post-scale

MCQ 2014/15 post-scale

Criminal Litigation First Sit 2014/15 Pre-scale

SAQ sub-parts and totals

wanted, and this conclusion was reinforced by the