Knowledge Application Assessment of Foreign Language Listening Proficiency The history of language proficiency testing in the U.S.

government started in the Department of State in the mid-50s, when the a requirement was identified for an objective system to assess the speaking abilities of government employees (Herzog, 2014). For about the last 60 years, the U.S. government has used this system, which developed into the Interagency Language Roundtable Skill Level Descriptions (ILR-SLD), to describe proficiency in foreign language in order to determine requirements for language-related positions, to qualify employees for such positions, and to award bonus pay based on language proficiency. The ILR-SLDs use broadened until it became the official scale of the U.S. government, referenced by the Office of Personnel Management when describing positions requiring language. In addition to speaking, Skill Level Descriptions for reading, listening, and writing were eventually developed, but these newer scales show the evidence of influence by the speaking scale (Interagency Language Roundtable, 2014). The Speaking SLDs were based on testing in an interview-type testing situation, where speaking skills as described in the SLDs could be elicited directly. Each SLD describes a particular level of structural accuracy, breadth of vocabulary, and ability to accomplish certain functions in speaking. The functions demonstrate the influence of the use of the scales in the State Department context of living and working abroad in a professional context; for instance, at the lower levels the SLDs describe satisfy minimum courtesy requirements and accommodation needs, while at the professional level the SLDs state that the examine can participate effectively in most formal and informal conversation in practical social and professional topics.

A review of the Listening SLDs, which were written after the Speaking SLDs, show a clear parallelism with the Speaking SLDs, as well as some influence from the Reading SLDs. For instance, the Speaking SLDs at the 0+ level states that the examinee can satisfy immediate needs using rehearsed utterances, and likewise the Listening SLDs state that at the 0+ level the examinee can understand a number of memorized utterances in areas of immediate needs. The Listening SLDs describe the listening ability of someone participating in a conversation, which while difficult to assess is possible in an interview exam. The SLDs also describe nonparticipatory listening, such as listening to radio broadcasts and public addresses, examples of which appear at the higher levels. Similarities with the Reading SLDs appear in the description of the types of texts that are comprehensible at each of level of listening. The assessment challenge arises when translating the ILR listening SLDs into an indirectas opposed to interview-typetests of listening. The most widely-used test of listening proficiency is the Defense Language Proficiency Test (DLPT) listening test, a multiple choice test composed of authentic listening passages and dialogs. Test construction occurs in a somewhat circular manner: First, audio texts are selected and assigned a SLD level. According to documents in the in-house item writers manual, those selecting audio texts may also refer to the Reading SLDs, which provide more thorough descriptions of texts than do the Listening SLDs. In any case, this level assignment requires some interpretation, since the SLDs describe characteristics of the listener, not the text. Then multiple choice items are prepared for the selected audio texts. Examinees are then assigned an ILR-based score in accordance with the highest level of the texts for which they can consistently provide correct answers. These scores then indicate a general ability level. For instance, according to the ILR-SLDs, a score of 2 indicates Limited Working Proficiency, and a 3 indicates General Professional Proficiency.

Thus, the ILR Scale provides a broad description of the proficiency of the examinee; and indeed, positions in the government requiring language usually require a particular level of proficiency, generally indicated as a 2 or 3. However, this reporting of a single score on the ILR scale may not provide potential employers specific information about the pattern of strengths and weaknesses of a potential audio translator. For instance, an individual achieving an ILR level 2 in listening could be an intermediate learner of the language, or someone who learned the language in the home but was not educated in the language, or it could be a highly-educated native speaker of the language who had difficulty understanding the multiple choice questions in English. Each of the those profiles might be suitable for different types of language-required jobs; however, information about the examinees pattern of strengths cannot be determined by the ILR-based score. The skill of foreign language listening is one of the most important to government linguists. The translation of audio texts is an important skill which is in great demand. However, of the foreign language skills that are commonly assessed, foreign language listening is one of the least researched and least well understood (Buck, 2001), particularly when compared to the other receptive language skill of reading. There is evidence that in addition to the more obvious linguistic skills that contribute to listening proficiency, such as vocabulary and structure knowledge of the foreign language (Buck and Tatsuoka, 1998; Goh, 2000), additional cognitive skills, such as the use of metacognitive strategies (Vandergrift, 2003; Vandergrift, Goh, Mareschal, & Tafaghodtari, 2006), the ability to control anxiety (Mills, Pajares and Herron, 2006), and working memory capacity (Conway, Cowan & Bunting, 2001), may play important roles as well. These higher level skills may in fact distinguish the novice from the proficient learner.

In order to provide more insight into the meaning of listening proficiency test scores, I am currently undertaking a pilot study of the factors, whether features such as linguistic difficulty or higher level skills, that contribute to the difficulty of listening test items. In this study, I will be analyzing English listening items from a retired multiple choice English listening test. Using the Linear Logistic Test Model (LLTM) (Fischer, 1973), I will be able to incorporate terms representing cognitive complexity into the Item Response Theory model in order to give weights to those difficulty factors. As demonstrated by Dimitrov (2007), the Least Squares Distance Method of LLTM can provide information on cognitive attributes required for correct responses to binary test items using item parameters, which is useful when item-level data may not be available. In this research, I will undertake in-depth studies of the literature in test validity theory, linear logistic test model and comparisons with other cognitive diagnostic models, educational psychology theories of listening ability (in both first and second language) and expertise literature as it may apply to second language proficiency. This research has implications for test construction, instruction in listening skills, and for contributing to research on the definition of the listening construct, particularly important in assessments of the validity of listening tests. I hope that this research may become part of a broader discussion of the description of listening ability in scales such as the ILR.

References Buck, G. (2001). Assessing listening. Cambridge; New York: Cambridge University Press. Buck, G., & Tatsuoka, K. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15, 119 157. Conway, A. R. ., Cowan, N., & Bunting, M. R. (2001). The cocktail party phenomenon revisited: The importance of working memory capacity. Psyhonomic Bulletin & Review, 8(2), 331335. Dimitrov, D. M. (2007). Least squares distance method of cognitive validation and analysis for binary items using their item response theory parameters. Applied Psychological Measurement, 31(5), 367-387. Fischer, G. H. (1973). Linear logistic model as an instrument in educational research. Acta Psychologica, 37, 359374. Goh, C. M. C. (2000). A cognitive perspective on language learners listening comprehension problems. System, 28(1), 5575. Herzog, M. (accessed March 26, 2014). An overview of the history of the ILR language proficiency skill level descriptions and scale. http://www.govtilr.org/Skills/IRL%20Scale%20History.htm Interagency Language Roundtable. ILR skill level descriptions. Accessed March 26, 2014. http://www.govtilr.org/ Mills, N. A., Pajares, F., & Herron, C. (2006). A re-evaluation of the role of anxiety: Selfefficacy, anxiety, and their relation to reading and listening proficiency. Foreign Language Annals, 39(2), 276295.

Vandergrift, L. (2003). Orchestrating strategy use: toward a model of the skilled second language listener. Language Learning, 53(3), 463496. Vandergrift, L., Goh, C. M. C., Mareschal, C. J., & Tafaghodtari, M. H. (2006). The Metacognitive Awareness Listening Questionnaire: Development and validation. Language Learning, 56(3), 431462.