International English Language Testing System Test Review University of Illinois at Chicago Rachel Coldewey

IELTS Test Review Abstract

This paper reviews the purpose, content, reliability and validity of the International English Language Testing System developed and produced by Cambridge, IELTS Australia, and the British Council. After reviewing resources provided by the IELTS website, IELTS is considered a highly reliable and generally valid test for nonnative English speakers seeking to work or study in an English speaking country.

IELTS Test Review The IELTS exam is an English proficiency test developed by the British Council, IELTS Australia, and Cambridge. The test examines test takers listening,

speaking, reading, and writing skills. Test takers are typically those looking to study abroad, work abroad, or immigrate to a country where English is the main language spoken. IELTS scores achieved by test takers are used in determining eligibility for these purposes. The theoretical framework for the test is not clearly stated in the IELTS Handbook, which is accessible to the public on the IELTS website. A study comparing the TOEFL and IELTS scores did highlight the difference between the frameworks behind the two tests. ETSs (2010) study found:

The TOEFL and IELTS tests, while they have the same number of sections measuring similar skills and underlying constructs, were built from different frameworks and different test blueprints. For example, the TOEFL iBT test is the latest version of the TOEFL test and is built on the framework of communicative competency based on current second language acquisition theory and practice. The TOEFL iBT test contains numerous innovative task types that can only be presented on a computer-based test. (p. 14)

After thorough research, a clearly defined theoretical framework for the IELTS exam could not be located. Test developers for the IELTS should include this in their handbook because the framework on which the test is developed will have

IELTS Test Review something to do with the kinds of tasks that the test requires of its takers. Test users may be more apt to use a test built on a communicative competency framework if ultimately test takers will be responsible for communicative tasks in real life. If test takers ultimately will be more likely to need academic reading and

writing skills, communicative tasks may not be as important and test users may not be as interested in those types of tests. The test may also face a face validity issue with test users who want to know that a test is backed by recent second language acquisition research and not just by the whims of the test developers. The IELTS consists of four sections: a 30 minute listening section, a 60 minute reading section, a 60 minute writing section, and an 11-14 minute speaking section, which occur in that order. Test takers choose from two tracks for the test: for those seeking entry into a university or study abroad program there is an academic test, and for those seeking to immigrate, train, or work abroad there is a general training test. All test takers take the same listening and speaking tests. There is a computer-based version of the test at some test centers and test takers are given the choice between the computer-based and paper-based version at those centers. As mentioned before, the same listening test is administered to all test takers whether they are taking the academic or general training test. The listening test is passage based (Carr, 20101) and consists of four recorded monologues or conversations (the input) which test takers are then required to respond to by answering questions. The questions are all selected response tasks (Carr, 2011)

IELTS Test Review including short answer, multiple choice, and matching questions. (See appendix, figure 1). The academic reading section is a passage-based task (Carr, 2011) that consists of texts from books, journals, magazines, and newspapers (input). The texts are not geared toward any particular field and test takers are required to select the appropriate response. The general training reading section consists of nonacademic general texts but also texts like ads where test takers are asked to identify information, complete

sentences, or give short answers to questions about the text (see appendix, figure 2). All items are selected response (Carr, 2011). The academic writing section consists of two tasks: the first is an extended production task (Carr, 2011) of 150 words that requires test takers to respond to a prompt (Carr, 2011) such as a diagram or chart and summarize its main points. (See appendix, figure 4). The second task is another extended production task requiring the test taker to write at least 250 words in response to a statement or question prompt. The general training writing section also consists of two extended production tasks. The first requires test takers to write a letter of at least 150 words The second task is similar to the academic writing task: test takers are required to write at least 250 words in response to a statement or question prompt. Finally, the speaking test consists of an in-person interview with an examiner. Test takers are given a spoken prompt (see appendix, figure 3) and are expected to give both limited production (short answer) (Carr, 2011) responses and

IELTS Test Review extended production (speak at length) responses, as well as interact with the examiner.

All task types and how they are rated is clearly described in the handbook, as well as on the website. Samples of each of the task types is available to test takers and test users on the IELTS website. Raters score the listening and reading sections of the test objectively with an answer key. The writing and speaking portions of the test are scored by human raters using band descriptor rubrics. (Rater training will be addressed in the reliability section of this paper). The listening and reading sections are given raw scores (out of 40) which are then converted to the overall band scores that are reported to test takers and their sponsoring institutions. The writing section of the test is rated based on coherence and cohesion, task response, lexical resource and grammatical range and accuracy. The speaking section is rated based on fluency and coherence, lexical resource, grammatical range and accuracy, and pronunciation. The IELTS exam is a criterion-referenced test (Carr, 2011), which means that the test takers are given a score that reflects the percentage of the test that they got correct, but does not compare with other test takers. Interpretation of the scores really depends on the institution sponsoring the test taker and the demands of whatever language, work, training, or study requirements it has. This is a very important note to make about the IELTS score and the handbook does cover this, as well as describe what each IELTS band score means as far as language proficiency.

IELTS Test Review No reliability evidence is in the IELTS handbook. However, the IELTS

website has an extensive amount of reliability evidence posted and accessible to test takers and test users. Overall, the internal reliability for the listening and reading sections of the test are high- .90, .91, and .92 for academic reading, listening, and general training reading, respectively (IELTS, 2011). The reliability for speaking and writing tests (which as mentioned before, are rated by human examiners) is lower, but still high at .83 and .81, respectively (IELTS, 2011). There are 16 versions (or parallel forms) of the listening, academic reading, and general training reading tests, all of which are also highly reliable, ranging from .89-.93 for listening, .88-.93 for academic reading, and .90-.94 for general training reading (IELTS, 2011). Inter-rater reliability for the IELTS exam is demonstrated by the clear and detailed rubric (posted on the website, but not in the handbook) that examiners must use to score test takers, raters rigorous training and certification/recertification process, and current coefficient of .83-.86 for speaking test scores and .81-.89 for writing test scores (IELTS, 2011). Another demonstration of reliability can be found in test developers adjustment for difficulty across versions of the tests. As described in the scoring section of this paper, the objectively scored sections of the test are given a raw score out of 40, which are then converted into overall band scores. In order to make all versions of the test more equal, different boundaries for band scores are set according to the difficulty of that particular version of the test. Evidence of test-retest reliability could not be located for the IELTS exam. While test developers might want to include some research on the same group of

IELTS Test Review

test takers taking the same test twice (perhaps using a split-half reliability test), the rest of the reliability evidence is convincing enough for the IELTS to be considered a reliable test. Typically, an alpha score of.70 or higher is considered acceptably reliable. IELTS consistently demonstrates a much higher reliability in all sections of the test, across all versions. In any case, the test developers would certainly want to put this reliability information in the handbook. Currently it is only accessible on the website and requires extensive research and effort to obtain. For a test as reliable as the IELTS consistently is, it would be a good idea to make this easier for a test user to find by putting it in the handbook. The fact that IELTS offers two different tracks (academic and general training) for its test takers is evidence of content validity. Because the test is used for different purposes (both entrance to academic institutions and workrelated/immigration-related purposes), the two tracks aim to make the test more specifically target those two different end goals. IELTS also (as mentioned in the scoring section of this paper) describes in its handbook (pg. 5) that the interpretation of scores should be based on the linguistic demands of whatever program/institution is interpreting them and gives a detailed chart to help test users determine how acceptable certain scores might be for certain programs/purposes. A study on construct-validity regarding the reading tasks on the IELTS exam was done (Moore, Morton, Price, 2007) and there seemed to be a face validity issue with instructors who believed that IELTS reading tasks were decontextualized, unlike readings in real classrooms, which were focused, and within context.

IELTS Test Review

Because a reading test on a standardized test cannot feasibly be contextualized, this is not seen as a real validity issue. A comparison study done by Educational Testing Services, (Educational Testing Services, 2010) compared the IELTS and TOEFL tests. TOEFL is a commercially produced, high stakes, proficiency test similar to IELTS, testing the same constructs as IELTS and is therefore a good test with which to correlate IELTS scores. Criterion validity evidence can be found in this study in the table on page 8. Overall, the total test score correlation between IELTS and TOEFL across sections of the test is .73, (ETS, 2010) which is generally considered acceptably valid. Overall, the IELTS exam is a reliable English language proficiency exam that tests listening, speaking, writing, and reading and is generally valid depending on how institutions interpreting the scores interpret them. While test developers do offer extensive resources for test takers and test users (samples of test tasks, research, etc.) on the website, it would be helpful to put more information on reliability and validity in the test handbook itself.

IELTS Test Review APPENDIX Figure 1. (IELTS, 2011)

Listening sample task Multiple choice (to be used with IELTS Listening Recording 2) Tapescript for IELTS Listening Recording 2 (A customer has been arranging with a shipping agent to send a large box overseas. This is the last part of the conversation.) A OK right. Now obviously insurance is an important thing to consider and our companies are able to offer very good rates in a number of different all-inclusive packages. B Sorry, could you explain a bit more? A Yes, sorry, um. Theres really three rates according to quality of insurance cover theres the highest comprehensive cover which is Premium rate, then theres standard rate and then theres economy rate. That one will only cover the cost of the contents second hand. B Oh Ive been stung before with economy insurance so Ill go for the highest. A Mhhm and can I just check would you want home delivery or to a local depot or would you want to pick it up at the nearest port? B The portd be fine Ive got transport that end. A Fine and will you be paying by credit card? B Can I pay by cheque? Listening sample task Multiple choice (to be used with IELTS Listening Recording 2) SECTION 1 Questions 9 and 10 Choose the correct letter, A, B or C. 9 Type of insurance chosen A Economy B Standard C Premium 10 Customer wants goods delivered to A port B home C depot

Figure 2. (IELTS, 2011)

General Training Reading sample task Matching information SECTION 1 Questions 1 8 Look at the five advertisements, A-E. Which advertisement mentions the following? Write the correct letter, A-E, in boxes 1-8 on your answer sheet. NB You may use any letter more than once. 1 2 3 4 5 6 7 8 up-to-date teaching systems that the institution has been established for a significant time examination classes that arrangements can be made for activities outside class the availability of courses for school students language teaching for special purposes a wide variety of language choices evening classes

IELTS Test Review

10

Figure 3. (IELTS, 2011)

Speaking sample task Part 1 Part 1 Introduction and interview [This part of the test begins with the examiner introducing himself or herself and checking the candidates identification. It then continues as an interview.] Lets talk about your hometown or village. What kind of place is it? Whats the most interesting part of your town/village? What kind of jobs do the people in your town/village do? Would you say its a good place to live? (Why?)

IELTS Test Review REFERENCES

11

Carr, N. (2011). Designing and analyzing language tests. Oxford: Oxford University Press. Educational Testing Services, (2010, December 17). Linking toefl ibt scores to ielts scores- a research report. Retrieved from http://www.ets.org/s/toefl/pdf/linking_toefl_ibt_scores_to_ielts_scores.pdf IELTS. (2007). Handbook. Retrieved from www.ielts.org. IELTS. (2011,). Test performance 2011. Retrieved from www.ielts.org Moore, T., Morton, J., Price, S. (2007). Construct validity in the ielts academic reading test: A comparison of reading requirements in ielts test items and in university study. Retrieved from www.ielts.org