Remedial Exams and Student Achievement

'

The Eects of Remedial Exams on Student Achievement: Evidence from Upper Secondary Schools in Italy
(Erich Battistin, Ilaria Covizzi and Antonio Schizzerotto) University of Padova and IRVAPP

Brussels, 20th June 2011 JRC-CRELL Catch the Train: Skills, Education and Jobs

&

Remedial Exams and Student Achievement

'

Introduction
Until the 2006/07 school year students in higher secondary schools (colleges) in Italy who did not meet prociency expectations were given a debito formativo (educational debt), that is a nal mark signaling failure in one or more subjects. Such lack in achievement was to be recovered in the following years, with no clear deadline and with mild (or simply no) enforcement from schools. The absence of any regulation soon became one of the main suspects for the low performance at 2003 and 2006 PISA tests. Aiming to improve eectiveness in learning, the Minister of Education of the time decided to introduce remedial exams in Italy (that - curiously enough - had already been abolished from the school year 1992/93). &

Remedial Exams and Student Achievement

'

Introduction
Under the new progression rule (which is still in operation): low performing students are compelled to recover their educational lack before the beginning of the new school year by sitting a remedial exam. students who have not been able to recover their debt, or have been assigned too many debts to be recovered at the end of the school year, face grade retention. all schools, although with a certain degree of autonomy, have to implement remedial education programmes for low-achieving students, and prepare them for the exam. failure in one single subject will lead to the remedial exam. Thus, students are compelled to reach adequate standards in all subjects to avoid grade retention. &

Remedial Exams and Student Achievement

'

Introduction
The autonomous province of Trento did not comply with the new rules: (i) very good performance at PISA tests of local students, and (ii) already longstanding tradition in oering eective remedial courses to low-achieving students.

&

Remedial Exams and Student Achievement

'

Punchline
General idea: set up the comparison of areas with remedial exam vis-a `-vis other areas (i.e. educational debt). Works if: the two groups of areas had similar performance prior to the policy roll out.

&

Remedial Exams and Student Achievement

'

Punchline
Natural experiment resulting from the geographical discontinuity in the implementation of the reform. Use geographical variation to study the reduced form policy eect of being at risk of grade retention on short-run achievement gains, vis-` a-vis the regime operated until the school year 2007/08. (in progress) Build bridges between structural channels and policy eects by relating family and school inputs to output. (i.e. competences developed).

&

Remedial Exams and Student Achievement

'

Main Findings
Between group heterogeneity. Clear pattern of zero or nearly positive (average) eects of the remedial exam for students on academic tracks and negative (average) eects for students in vocational schools. The intervention exacerbates preexisting inequalities in achievement among students from dierent school tracks.

&

Remedial Exams and Student Achievement

'

Main Findings
Evidence that schools may have reacted dierently. Schools react to the reform by admitting to the next grade students who in the status quo would have been given an educational debt. This is more so in academic schools. In academic schools the introduction of a remedial exam comes with less stringent rules to pass students to the next grade. This implies zero eect on average. In vocational schools the eects of less stringent rules are largely oset by the negative eects on eort caused by the remedial exam. These in turn imply more heterogeneous eects across students and negative average eects. & %

Remedial Exams and Student Achievement

'

Evaluation Design
Exploits a matched pair comparison of students in schools either side of the border: schools in province of Trento were matched to similar schools in the surrounding provinces of Bolzano, Brescia and Vicenza.

&

Remedial Exams and Student Achievement

'

10

Evaluation Design
School selection controls for observable dimensions such as school track, school size (as measured by trends in enrollment), school resources, as well as unobservable dimensions (such as reputation of the school) gathered from general knowledge of the socioeconomic background in which they operate. We focused on students attending the second and the third year during the school year 2008/09, thus aged between 14 and 15. For each school we randomly selected two classes in the second year (i.e. for the cohort of students enrolled for the rst time in school year 2007/08) and two classes in the third year (i.e. for the cohort of students enrolled for the rst time in school year 2006/07). About 2000 students and 44 schools. &

Remedial Exams and Student Achievement

'

11

Evaluation Design
Ensures variability in the duration of enrollment at school across the dierent regimes introduced by the reform.

&

Remedial Exams and Student Achievement

'

12

Data
Main sources: 1. Specially commissioned standardized test prepared by the National Institute for the Evaluation of the Educational System. Can in fact be considered as a small scale PISA survey.a 2. Large set of demographics and (contemporaneous) family inputs collected through integrated data surveyed from students and their parents. 3. Transcript les from schools on all marks since enrollment as well as on the national exam at the end of lower secondary school (8th grade; not yet standardized nationwide at that time). &
from the PISA format, all students answered all questions in the assessment, so that multiple measurements are available for each student.
a Dierently

Remedial Exams and Student Achievement

'

13

Data
Additional sources: 1. Specially granted information from the 2003 and 2006 PISA surveys, where only schools in narrowly dened areas that match closely the evaluation design are considered.

&

Remedial Exams and Student Achievement

'

14

Testable Implications of the Evaluation Design

No pre-reform dierences in achievement for schools in the provinces considered using PISA 2006 data.

Provinces: narrowly dened areas as in the evaluation design. Regions: same regions as in the design.

&

Remedial Exams and Student Achievement

'

15

Testable Implications of the Evaluation Design

Full support with respect to the socio-demographic indicators across cohorts and school types. Good degree of homogeneity for students in schools either side of the province border.
Academic Schools
Older Cohort Younger Cohort

Vocational Schools
Older Cohort Younger Cohort

.2

.4

.6 Propensity Score

.8

.2

.4

.6 Propensity Score

.8

.2

.4 .6 Propensity Score

.8

.2

.4 .6 Propensity Score

.8

Propensity scores for students in academic schools with different policy regimes

Propensity scores for students in academic schools with different policy regimes

&

Remedial Exams and Student Achievement

'

16

Testable Implications of the Evaluation Design

No pre-reform dierences in retention rates.
Academic Matching Passed, with no debt Admitted with debt 0.0417 (0.0502) -0.0352 (0.0491) Standard errors clustered by class Regressors include gender, class size and its square OLS 0.0407 (0.0732) -0.0226 (0.0682) Vocational Matching 0.0236 (.0706) -0.0377 (0.0692) OLS 0.0629 (0.0487) -0.0778 (0.0591)

&

Remedial Exams and Student Achievement

'

17

Reduced Form Eects: Academic Schools

Mean Matching Reading Math Science 7.847 (14.704) -9.866 (12.990) 7.921 (11.922) OLS 7.774 (9.382) -2.791 (10.186) 12.687 (8.921) Weighting 11.275 (15.443) -3.995 (20.215) 16.139 (10.462) 0.25 -9.311 (12.937) 2.821 (14.356) -5.999 (12.954) Older Cohort (enrolled in 3rd year at interview) 12.224 (16.557) -11.087 (11.797) 4.894 (17.502) 8.670 (18.263) -6.190 (7.474) 24.127 (12.329) Quantiles 0.50 0.75

Younger Cohort (enrolled in 2nd year at interview) Reading Math Science 2.208 (15.567) 13.742 (12.538) 21.843 (12.344) 9.034 (15.034) 16.579 (13.422) 18.749 (9.072) 10.495 (15.949) 18.552 (23.405) 16.479 (16.799) 0.455 (10.856) 14.186 (9.760) 15.204 (10.286) 13.173 (12.361) 13.127 (12.692) 17.316 (12.556) -6.190 (15.967) 1.069 (10.624) 5.554 (14.342)

Standard errors clustered by class

&

Remedial Exams and Student Achievement

'

18

Reduced Form Eects: Vocational Schools

Mean Matching Older Cohort Reading -21.793 (15.398) Math -5.849 (16.514) Science -4.908 (14.577) Younger Cohort Reading -16.947 (12.176) Math Science -1.633 (16.902) -29.962 (13.480) -19.833 (8.982) -6.195 (15.512) -32.333 (11.019) -20.843 (14.937) -0.050 (27.105) -30.518 (17.383) -6.777 (14.084) -29.413 (20.393) -25.160 (18.650) -16.214 (13.420) -8.792 (11.148) -19.966 (16.450) -21.124 (16.320) 13.573 (10.678) -28.386 (15.007) -25.092 (9.621) -5.339 (10.376) -8.047 (9.882) -22.740 (15.125) -5.187 (22.581) -7.224 (16.465) 4.192 (16.316) -8.443 (16.309) 0.580 (10.781) -8.140 (9.502) 5.342 (6.407) -10.473 (11.813) -22.298 (9.538) -22.775 (9.487) -6.173 (15.983) OLS Weighting 0.25 Quantiles 0.50 0.75

Standard errors clustered by class

&

Remedial Exams and Student Achievement

'

19

A Twist of Heterogeneity
Academic Schools
Science
Younger Cohort
1

Vocational Schools
Science
Older Cohort
1

Younger Cohort

Older Cohort

cdf

cdf

.5

0 200

.5

200

400

600

800 200

400

600

800

400

600

800 200

400

600

800

PISA score remedial exam Weighted cumulated density functions debt

PISA score remedial exam Weighted cumulated density functions debt

&

Remedial Exams and Student Achievement

'

20

Reduced Form Eects: Retention Rates

Academic Matching Passed 0.1227 (0.0523) OLS 0.1366 (0.0443) Vocational DD 0.0801 (0.0413) Matching 0.0863 (0.0441) OLS 0.0782 (0.0449) DD 0.0227 (0.0420)

Standard errors clustered by class

Dierences in retention rates at the end of the second year (rst policy-on year) for the cohort of students enrolled in the school year 2006/07. The DD is calculated using results for the same cohort in the previous year.

&