09/07/11

1
Aligning the TEF (Test dvaluation de franais)
with different language frameworks
Dominique CASANOVA, Chambre de commerce et dindustrie de Paris (France)
Alexandra CRENDAL, Chambre de commerce et dindustrie de Paris (France)
Pr. Marc DEMEUSE, Universit de Mons (Belgium)
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

2
1. What is the TEF?
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

3
A High-stakes test
Immigration to French speaking countries
Studies in French speaking countries
Recognizedinternationally
Since J uly 2002 by Citizenship Immigration Canada
Since March 2004 by the French Ministry of Higher Education
Since October 2006 by the Ministre de lImmigration et des Communauts culturelles du
Quebec (adapted format)
Meeting international standards
Since 2005 : certified OSI 9001
Since 2009 : CCIP became an ALTE full member after auditing the TEF
TEF, a high-stakes test
09/07/11

4
Reading Comprehension
50 questions 60 minutes
Listening Comprehension
60 questions 40 minutes
Vocabulary and Language Stuctures
40 questions 30 minutes
Oral Production
2 parts 35 minutes
Written Production
2 parts 60 minutes
What does the TEF consist of?
A1
C2
C1
B2
B1
A2
09/07/11

5
2. The NCLC
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

6
2008-2010 : national consultation new version
NCLC & Canadian Language Benchmarks
1996: Canadian Language Benchmarks
09/07/11

7
NCLC & Canadian Language Benchmarks
a descriptive scale of communicative
proficiency in French as a Second Language
(FSL) expressed as 12 benchmarks or reference
points
a set of descriptive statements about
successive levels of achievement on the
continuum of FSL performance;
statements (descriptions) of communicative
competencies and performance tasks in
which the learner demonstrates application of
language knowledge (competence) and skill;
a framework of reference for learning,
teaching, programming and assessing adult
French as a Second Language in Canada, and
a national standard for planning second
language curricula for a variety of contexts, and
a common yardstick for assessing the
outcomes.
09/07/11

8
3. RelatingTEF to SLCs
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
2 / 4 points 2 points 1 point 0 point
SLC 1 SLC 2 SLC 3 SLC 4 SLC 5 SLC 6 SLC 7 SLC 8 SLC 9 SLC 10 SLC 11 SLC 12
09/07/11

9
7 levels
12 levels
4 cut-points
Specifications - 2002
Standardisation - 2004
RelatingTEF to SLC 2002
09/07/11

10
RelatingTEF to SLCs - Main stages
Stage 1. Analysis of SLC (per skill and per level summaries)
Stage 2. Differentiation of adjacent SLCs at cut points
Stage 3. Correspondence between TEF and SLC scales
Stage 4. Correspondence between TEF specifications and SLCs
Stage 5. Empirical correspondence between TEF items and SLCs
09/07/11

11
3. Correspondance between TEF and SLC scales
09/07/11

12
4. Correspondance between TEF specifications
and SLCs for receptive skills
09/07/11

13
2 / 4 points 2 points 1 point 0 point
5. Empirical correspondence between
TEF items and SLCs (receptive skills)
Step 1. Individual classification of item descriptions
Step 2. Consensus on items reference level regarding their
general description
Step 3. Individual classification of items
Step 4. Agreement on items classification regarding CIC cut-points
SLC 1 SLC 2 SLC 3 SLC 4 SLC 5 SLC 6 SLC 7 SLC 8 SLC 9 SLC 10 SLC 11 SLC 12
Experts judgements (3 experts), 2-days meeting
09/07/11

14
SLC 1
st
step:
individual classification of item descriptions
Units
types
(CE)
Units
types
(CO)
Units
types
(LS)
Units
types
(CE)
Units
types
(CO)
Units
types
(LS)
Units
types
(CE)
Units
types
(CO)
Units
types
(LS)
Expert 1 (room 1) Expert 2 (room 2) Expert 3 (room 3)
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 2
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 3
SLC 1
unclassifiable unclassifiable unclassifiable
SLC 1
SLC 12
09/07/11

15
Expert 1
Expert 2
Expert 3
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
Common
SLC 1
Common classification of reference
for TEF item descriptions
Sharing results and viewpoints :
> Problems faced during classification
> Rationales for classification
> Comparison of the classifications / levels
> Rationales for inclassifiable items descriptions
> Discussion, consensus
SLC 2
nd
step:
Consensus meeting on item descriptions
09/07/11

16
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Unclassifiable
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 2
SLC 1
Unclassifiable
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 3
SLC 1
Unclassifiable
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
Common
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
Common
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
Common
SLC 1
SLC 3
rd
step:
individual classification of items samples
Items
(CE)
Items
(CO)
Items
(LS)
Items
(CE)
Items
(CO)
Items
(LS)
Items
(CE)
Itemss
(CO)
Items
(LS)
Expert 1 (room 1) Expert 2 (room 2) Expert 3 (room 3)
294 items
09/07/11

17
Agreement Frequency % Cumulated
frequency
% cumulated
4 20 27,78 20 27,78
3 24 33,33 44 61,11
2 28 38,89 72 100,00
0 0 0 72 100,00
Agreement for Reading comprehension items
Classification in the 4 main categories
09/07/11

18
Agreement Frequency % Cumulated
frequency
% cumulated
4 32 27,35 32 27,35
3 48 41,03 80 68,38
2 37 31,62 117 100,00
0 0 0 117 100,00
Agreement for Listeningcomprehension items
09/07/11

19
Agreement for Language structures items
Agreement Frequency % Cumulated
frequency
% cumulated
4 28 26,67 28 26,67
3 36 34,29 64 60,96
2 40 38,10 104 99,05
0 1 0,95 105 100,00
09/07/11

20
Feedback
No standardisation of the familiarization phase.
No training for classification of items
But consensus step after the classification of item descriptions.
More than 3 judges would have been safer.
09/07/11

21
4. RelatingTEF to CECR
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
Specification
Standardisation
Empirical study
Coe (2003), Manual for relating Language Examinations to the Common European
Framework of Reference for Languages (CEFR), Pilot version
09/07/11

22
7 levels
6 levels
Specifications : 2001 - 2002
Specifications and
Standardisation : 2008
AligningTEF with CEFR
09/07/11

23
Correspondance between TEF
specifications and CEFR
Global description of the test and the subtests
For each type of item
Domains
Situations
Themes
Tasks / language activities
Strategies
Scales
Descriptors
Levels
09/07/11

24
Theoretical framework and TEF items definition Common European Framework of
Reference
Item
codifica-
tion
TEF
level
Itemdescription CEFR
levels
CEFR descriptors
CEAU1 TEF 0
+
Reconnatre les situations lmentaires de la vie
courante:
- identifier la nature et la fonction dun document trs
court de lenvironnement quotidien (pancartes,
tiquettes ) ;
- reprer des mots isols.
- 20 mots maximum.
A1
Peut comprendre des textes trs courts et trs
simples, phrase par phrase, en relevant des noms,
des mots familiers et des expressions trs
lmentaires et en relisant si ncessaire.
CEAU2 TEF 1
Comprendre les situations simples de la vie courante
(sans complication) :
- comprendre les informations essentielles dun
document court de lenvironnement quotidien;
- textes plutt injonctifs ou explicatifs (publicits,
annonces, faire-part, notices )
- 50 mots maximum.
A1
Peut comprendre des textes trs courts et trs
simples, phrase par phrase, en relevant des noms,
des mots familiers et des expressions trs
lmentaires et en relisant si ncessaire.
CEAU3 TEF 1
Comprendre les situations simples de la vie courante
(sans complication) :
- comprendre un document court de lenvironnement
quotidien;
- reprer les informations essentielles (textes plutt
injonctifs ou explicatifs: publicits, annonces, faire-part,
notices .).
- 50 mots maximum.
A1
Peut comprendre des textes trs courts et trs
simples, phrase par phrase, en relevant des noms,
des mots familiers et des expressions trs
lmentaires et en relisant si ncessaire.
Correspondence between TEF
specifications and CEFR
09/07/11

25
Standardisation
Selection of judges (15)
Selection of items (185)
Familiarization 1 day
Classification 4 x 2 hours
Consensus 1 day
Data analysis
Pre-tests Real tests
Item Bank
Empirical level /
specification level
CE CO LS
Same level 31 29 17
1 level of difference 23 24 13
More than 1 level of
difference
18 18 12
09/07/11

26
Familiarization
Presentation of the TEF, of the standardisation procedure
Presentation of the CEFR (main concepts)
Analysis of levels of reference, self-assessment of ones level in a
foreign language, sorting of CEFR scales descriptors
Individual classification of illustrative tasks (Coe CD-Rom),
discussions to reach a consensus
Individual classification of an illustrative sample of TEF items,
discussion to reach a consensus
09/07/11

27
7
0+
Units
types
(CE)
Units
types
(CO)
Units
types
(LS)
Units
types
(CE)
Units
types
(CO)
Units
types
(LS)
Units
types
(CE)
Units
types
(CO)
Units
types
(LS)
Judge 1 (room 1) Judge 2 (room 2) Judge (room )
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
Eval 1
SLC 1
Eval 1
CECR A2
Eval 1
CECR A1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 2
CECR A1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval
CECR A1
CECR A1
CECR C2
CEFR experiment - 1
st
step:
individual classification of item descriptions
Reading: 16 item descriptions
Listening: 20 item descriptions
Languagestructures: 14 item descriptions
09/07/11

28
Judge 1 (room 1) Judge 2 (room 2) Judge (room )
Units
relles
(CE)
Units
relles
(CO)
Units
relles
(LS)
Units
relles
(CE)
Units
relles
(CO)
Units
relles
(LS)
Units
relles
(CE)
Units
relles
(CO)
Units
relles
(LS)
7
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
Eval 1
SLC 1
Eval 1
CECR A2
Eval 1
CECR A1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 2
CECR A1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval
CECR A1
CECR A1
CECR C2
CEFR experiment - 2
nd
step:
individual classification of items samples
Reading: 72 items
Listening: 71 items
Languagestructures: 42 items
09/07/11

29
14 judges, 6 Itemdescriptions, 43 items
Final individual classification of corresponding items and item descriptions
Sharing results and viewpoints
(limitedto the most problematic elements)
Comparison of the classifications / levels
J ustification basedon the CEFR
CEFR experiment - 3
rd
step:
Consensus meeting

09/07/11

30
4. Results
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

31
Item descriptions
Items which empirical level
corresponds to the specification level
80% of agreement 71% of agreement
Panel decisions versus specification levels
09/07/11

32
Average agreement between individual classifications of item descriptions and items with
- the level granted by the panel (median)
- the level granted by the CCIP .
75%
65%
77%
70%
54%
77%
75%
61%
67%
64%
50%
63%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Reading Listening Language Structures
Group ID
CCIP ID
Group items
CCIP items
Average agreement
09/07/11

33
The 43 most problematic items (high disagreement)
Reading : 11 items
Listening : 20 items
Language structures : 12 items
1- New individual classification before discussion
2- Discussion
3- Final individual classification
Consensus meeting
09/07/11

34
Consensus meeting Enhancement of the agreement
Impact of consensus on judges agreement
(most problematic items)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
E02 E03 E04 E05 E06 E07 E08 E09 E10 E11 E12 E13 E14 E15
Before discussion
After discussion
Mean :
46% 63%
09/07/11

35
Classification of other items
Results of classifications for the different pools of items
54%
23%
58%
58%
68%
37%
57%
8%
47%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Individual / Panel Indifidual / Specifications individual / Empirical
Same level
Slightly different
Strongly difrent
09/07/11

36
Familiarization is an essential stage (common representation,
common understanding, training)
Consensus is also an essential stage (enhancement of the
inter-judges agreement)
A consensus after the first classification would harmonize even
more the representations before the classification of items
A large panel is safer providedthat you spend time enough on
building common representations.
Feedback
09/07/11

37
6. Perspectives
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

38
In prospect
Speaking and Writing
NCLC 2010
Alignement with other tests (BTC)
09/07/11

39
E-mail : dcasanova@ccip.fr
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
Lvaluation des comptences linguistiques des adultes en franais langue trangre
dans une perspective de multirfrentialisation
Demeuse, M., Desroches, F., Crendal, A., Renaud, F., Oster, P. et Leroux X. (2004)
In Actes du 17e colloque international de lAssociation pour le Dveloppement des Mthodologies
dvaluation en ducation (ADMEE-Europe). Lisbonne, Portugal, 18-20 novembre 2004.
Validation empirique dun test de franais langue trangre en regard du Cadre
europen commun de rfrence pour les langues
Casanova, D., Crendal, A., Demeuse, M., Desroches, F. et Holle, A. (2010),
In Actes du 22e colloque international de lAssociation pour le Dveloppement des Mthodologies
dvaluation en ducation (ADMEE-Europe), 881-902. Braga, Portugal, 14-16 janvier 2010.