2009 Second International Symposium on Knowledge Acquisition and Modeling
WordNet-based Way to Identify Chinglish in Automated Essay Scoring Systems
Wen Zhuge 1, Jingyu Hua 2
1 Foreign Language College, Hangzhou Dianzi University, Hangzhou, China
2 College of Information Engineering, Zhejiang University of Technology, Hangzhou, China
e-mail: chirs3@163.com, eehjy@163.com
AbstractRecent years have witnessed the success of
Automated Essay Scoring (AES) in western countries.
However, its further promotion in China is limited by the
negative transfer of non-native learners. Therefore, this
paper proposed a new way to identify Chinglish, which
obstructs the development of AES in China. This WordNet-
based method starts with dealing with semantic relations
between English verbs, calculates semantic distances
between subjects and objects, and then realizes the
identification of Chinglish by threshold. Experiments
conducted in one university show that the proposed way
performs well in identifying Chinglish in college students
English essays.
Keywords-Automated essay scoring; Identification of
Chinglish; WordNet; Semantic calculation
1. INTRODUCTION
Writing has been considered as an effective method to
measure a language learners language proficiency. It
accounts for a large proportion of classroom teaching as
well as proficiency tests. However, the traditional way to
score an essay not only costs a great deal of labor and
material resources, but also is greatly affected by scorers
language ability and personal preferences, therefore
reducing the credibility of the test [1]. However,
automated essay scoring (AES) based on corpus and
artificial intelligence technology could perform this task
with high efficiency and impersonality [2]. With labor and
resources saved, language testing is entering into an era in
need of AES systems
Currently, there already exist an ocean of AES systems
overseas, such as PEG, IEA, and Intelli-Metric, which are
based on shallow parsing, latent semantic analysis and the
combination of artificial intelligence and statistics,
respectively. They have already achieved a relatively
accurate scoring task [3-7]. However, these systems are
English speakers oriented, thus they neglect the problems
caused by the negative transfer of non-native learners. As
far as Chinese learners are concerned, Chinglish
sentences, i.e. the hybird of Chinese way of thinking and
English syntax, should be given priority in the research of
AES technologies.
Seeing that Chinglish sentences accord with English
syntax, it is difficult for the common syntactic analyzers
employed by AES system to identify these special
linguistic phenomena. Utilizing the semantic web put
forward by WordNet, the researchers start with the
semantic relations between English verbs, deals with
semantic relations between English verbs, calculates
semantic distances between subjects and objects, and then
realizes the identification of Chinglish by threshold in this
paper.
978-0-7695-3888-4/09 $26.00
$25.00 2009 IEEE
DOI 10.1109/KAM.2009.322
The paper is organized as follows: Section 2 clarifies
the definition of Chinglish, as well as the obstacle it
causes in AES systems. Subsequently, a semantic structure
fit for identifying Chinglish is presented in Section 3.
Finally, experimental results and conclusion are made in
Section 4.
2. CHINGLISH IN ESSAYS
2.1 Definition of Chinglish
Transfer is a term used by psychologists in their
accounts of the way in which present learning is affected
by past learning[8]. Odlin has suggested that transfer is
the influence resulting from the similarities and
differences between the target language and any other
language that has been previously (and perhaps
imperfectly) acquired [9]. When learning a foreign
language, an individual already knows his mother tongue,
so the knowledge and skill of the native language will be
transferred to the foreign language unconsciously. Since
the way one thinks determines the way one speaks, the
differences between the native and the foreign language in
linguistic structure, social culture and logic thought,
predestine the existence of negative transfer in second
language learning.
The Han nationality puts more emphasis on the whole
when they think, which, in turn results in an analytical
language --- Chinese, while British and Americans, who
weigh reason and analysis more, create a synthetic
language --- English. This difference in way of thinking
directly leads to the appearance of Chinglish.
Generally speaking, Chinglish appears in two forms:
vocabulary and theme [10]. The former is caused by the
unequivalence of the concept aroused by certain words in
Chinese and English, whose meaning are simply taken for
granted in one culture. Take the sentence We played very
happily in the party last night for example. In Chinese, the
concept of play covers almost every interment in our life;
yet, the counterpart of this concept in English varies. They
play computer games, have fun in a party, amuse
themselves with books, or enjoy the movie very much. The
latter results from the confusion of a subject and a theme.
A subject in an English sentence is usually a person or a
thing, represented by a noun, pronoun, or noun phrase. But
subject can take any form in a Chinese sentence. Any
language elements can appear in the place of a subject.
Such as Our home has a dog.
2.2 Obstacles Caused in AES
As mentioned above, Chinglish sentences are correct
in terms of syntax, thus AES systems mainly based on
syntax analysis will make wrong judgment about these
sentences.
233
229
 Burstein has made a comparison experiment with E- 3.2 Semantic Description of The New Method
Rater in [11], in which he collected essays from both There is always a semantic database behind an AES
native learners and nonnative learners and had these system, the scheme of which directly affects the credibility
essays scored by teachers and E-Rater respectively. His of the system. To add collocation information, along with
research results can be shown in Table I. the description of semantic features possessed by the
TABLE I. COMPARISON OF ESSAY SCORES GIVEN BY TEACHERS
subject and object of a certain verb on the platform set up
AND E-RATER by WordNet will be beneficial to the identification of
Chinglish. For instance, to tag one semantic entry of the
Teachers E-Rater verb play (participate in games or sport), the semantic
Mother
Standard Standard
tongue Average
Deviation
Average
Deviation
feature of its subject (on the tree of (living_thing)) and its
object of ({game} {device}) should be included.
Arabic 3.83 0.973 3.67 0.947
According to Case Grammar, arguments covering
Chinese 4.09 0.884 4.12 1 agent, patient, location, and instrument / benefactive etc.
are all attached to verbs. For most verbs, subject, object
Spanish 3.96 0.986 3.7 0.915
and benefactive are what researchers care most. Thus, we
American 4.96 0.624 4.93 0.814 exclude the other semantic information in our scheme and
depict a web between nouns and verbs in Table II.
From Table I, we can clearly see that only essays from
Chinese learners receive higher marks from E-Rater than TABLE II. RELATION BETWEEN NOUNS AND VERBS
teachers. To account for it, Ge pointed that Chinese
learners large vocabulary shadowed their misuse of Part of Semantic
Vocabulary Agent Patient
sentence structure, thus cheating the AES system and speech Category
affecting its accuracy [12].
House n. location human none
3. WORDNET-BASED WAY TO IDENTIFY CHINGLISH Food,
Living
dog n. animal living
thing
3.1 Semantic Relations in WordNet thing

In December 2006, WordNet launched its latest father n. human entity entity
edition, WordNet 3.0 for Unix systems. The WordNet wash v. change
Living
object
project was launched by a group of psycho-lexicologists thing
Living
and linguists in Princeton University since 1985. Being an have v. possesion
thing
entity
electronic dictionary based on psychological research, Living
WordNet is an effective combination of both traditional walk v. change location
thing
lexicographical knowledge and modern computer
technology. It employs the spelling system, which is Take walk for example, it is marked as a change verb
familiar to anyone with some knowledge of English, to in our scheme, whose agent should be human beings or
symbolize word form and introduces synsets to stand for animals, who are capable of walking. Besides, walk in
word sense. English is an intransitive verb, so a patient other than
The most ambitious feature of WordNet, however, is location cannot follow it. Thus, I walked into the dog is
its attempt to organize lexical information in terms of incorrect while I walked into the house is acceptable. This
word senses, rather than word forms. In that respect, scheme is quite effective in identifying typical Chinglish
WordNet resembles a thesaurus more than a dictionary. Its sentences.
intricate yet quiet clear representation of hyponymy, Because of the limited space of this paper, the way to
which is realized by pointers, chains and lists in its construct such a semantic database will not be mentioned.
database, created a hierarchical semantic system, or an All the analysis below will be performed with the database
inheritance system for words. Thus, one can easily trace constructed by the semantic dictionary of English verbs
the hyperym, hyponym, co-hyponym and even holonym based on WordNet and FrameNet [15].
and meronym of certain word easily through WordNet 3.3 Indentification of Chinglish
browser.
With the help of the noun categories, as well as the
All the nouns in WordNet form a single thematic
semantic web of hypernym and hyponym provided by
hierarchical tree, whose root is entity, because inheritance
WordNet, this new method starts with semantic relations
underlines their semantic relations. Adjectives and adverbs
between verbs. By calculating semantic distances between
assemble satellite synsets of antonyms, since the bipolar subjects and objects, the identification of Chinglish can be
nature is their unique and outstanding feature. Altogether, realized by threshold. Take the two sentences mentioned
there are 25 categories in WordNet, which can be in the precious section as examples, we can employ the
furthered grouped into 11 classes: entity, abstraction, method to make a judgment.
psychofeatures, natural phenomenon, activity, event,
Our home has a dog.
group, location, possession, shape and state. Similarly,
verbs, which scattered in 15 semantic domains, weave a The syntactic analyzer tells us that has is the verb in
network of entailment, for this network covers almost all this sentence. Matching it with its stem have in the
the semantic relations among verbs [13-14]. database, we got the semantic restriction of its agent and
patient, being living thing and entity respectively. Then,
the method casts back the semantic categories of the agent

230
234
and patient in the current sentence, i.e. home and dog.
WordNet returns the research as:
08559508 15 n 02 home 4 place 6 005 @
08558963 n 0000 + 02537960 v 0201 + 00477661
a 0102 + 02005347 v 0101 ~ 08559766 n 0000 |
where you live at a particular time; "deliver the
package to my home"; "he doesn't have a home to
go to"; "your place or mine?"
As weve mentioned in the previous section that @->
means is the hyponym of, we can see that synset {home,
place} has a hypernym, numbered 08558963, i.e. synset
{residence, abode }. As long as @-> does not point to ,
the search keeps going. Ultimately, weve got:
home, place @-> residence, abode
@-> address
@-> geographic point, geographical
point
@-> point
@-> location
@-> object, physical object
@-> physical entity
@-> entity
The subject locates under the root nod of {physical
entity}, while according to the semantic description of the
subject features of have stored in the database, its subject
should be on the tree under the nod {living thing}. Both
{physical entity} and {living-thing} are on the second layer
of a semantic tree. According to ontology, the higher two
nods are, the remoter they will become. So the subject
home and the verb have do not match semantically. That is
to say, although this sentence is grammatically correct, it
is improper semantically.
We played very happily in the party last night.
Because play is used as an intransitive verb here, we
can make use of the place noun party to perform the
calculation. After calculation, we get:
party @-> affair, occasion, social occasion, function,
social function
@-> social event
@-> event
@-> psychological feature
@-> abstraction
@-> abstract entity
@-> entity
The comparison between the semantic categories where
party and the object of play fall into, i.e. {abstract entity},
and {game} {device}, tells us that these two nods also
belong to two different tress, and therefore, Chinglish can
be found here. Thus it can be seen that, utilizing the
semantic categories and the semantic relations provided by
WordNet is an efficient way to identify Chinglish in an
essay. We can analyze the structure of sentences by syntax
analyzer, and then match the semantic description of the
collocation information of the verb with its subjects or
objects, calculate the semantic distances to make a
judgment. The whole process can be summarized in
Figure
231
235
1.
4. CONCLUSION
This paper proposed a new way, which is based on
WordNet and semantic information description, to identify
Chinglish. As shown in the examples, this method can
identify structurally correct Chinglish effectively. We also
made experiments with college students in Zhejiang
province in China. Of the 300 randomly collected sample
essays written by freshman who majored in
communication technologies, there are 147 Chinglish
sentences identified by English teachers. And our method
successfully identified 93 of them. This new method can
be used as a supplement to the existing AES systems.
Merging the structure of FrameNet, and matching its
semantic concept with HowNet, the power of this method
could be strengthened and help improve the accuracy of
existing AES systems in the future.
ACKNOWLEDGMENT
This paper is sponsored by Zhejiang Provincial
Science Program (2007C21026) and Zhejiang Natural
Science Fund (Y1090645).
REFERENCES
[1] J. F. Wen, A reliable scoring system for norm-referenced tests of
English writing, Journal of Guangzhou University(Social Science
Edition), vol.2, no.11, Nov. 2003, pp. 84-87, doi: cnki:ISSN:1671-
394X.0.2003-11-020
[2] L. Streeter, J. Pstoka, D. Laham, and D. MacCuish, The credible
grading machine: Automated essay scoring in the dod, http:
//www. k2a2t. com /papers/CredGrad2ing2002. pdf, 2003 /2006 -
03 20
[3] S. L. Ge and X. X. Chen, An Overview of Current Automated
Essay Scoring Techniques, Media in Foreign Language
Instruction, no.117, Oct. 2007, pp. 25-29 , doi:
CNKI:SUN:WYDH.0.2007-05-006.
[4] M. Hearst, K. Kukich, L. Hirschman, E. Breck, M. Light, J. Burge,
L. Ferro, T. K. Landauer, D. Laham, and P. W. Foltz, The debate
on automated essay grading, IEEE Intelligent Systems, vol.15,
no. 5, Sept./Oct. 2000, pp. 22-37.
[5] S. Elliot, IntelliMetric: from here to validity, Automated essay
scoring: a cross disciplinary perspective, Lawrence Erlbaum
Associates, 2003, pp. 71-86.
[6] T. Landauer, D. Laham, and P. Foltz, Automated Essay Scoring
and Annotation of Essays with the Intelligent Essay Assessor,
Automated essay scoring: a cross disciplinary perspective,
Lawrence Erlbaum Associates, 2003, pp. 87-112.
[7] S. Valenti, F. Neri, and A. Cucchiarelli, An overview of current
research on automated essay grading, Journal of Information
Technology Education, vol. 2, 2003, pp. 319-330.
[8] J. Miao, and X. G. Ni, A Theoretical Study of Language
Transfer, Journal of Inner MongoliaCollege of Education, vol. 13,
no.4, Dec. 2000, pp. 12-17, doi: cnki:ISSN:1008-7451.0.2000-04-
004.
[9] T. Odlin, Language Transfer:Cross-Linguistic influence in
language Learning, Shanghai Foreign Language Education Press.
Shanghai, Oct. 2001.
[10] Y. S. Zhao, N. N. Liu, Error Analysis about College English
Writing based on the Difference between Chinese Thinking and
English Thinking, Journal of Weifang Educational
College,vol.21, no. 2, 2008, doi: CNKI:SUN:WFJY.0.2008-02-
042.
[11] J. Burstein, and M. Chodorow, M. Automated essay scoring for
nonnative English speakers, Joint Symposium of the Association
of Computational Linguistics and the International Association of
Language Learning Technologies, Workshop on Computer-
 Mediated Language Assessment and Evaluation of Natural [14] C. Fellbaum, WordNet: an electronic lexical database,
Language Processing, College Park, Maryland, June 1999. Cambridge,MA, USA: MIT Press, 1998.
[12] S. L. Ge and X. X. Chen, Automatic Scoring for Chinese EFL [15] W. Zhuge, The Construction of Semantic Dictionary of English
learners, Proceedings of the 3rd Students' Workshop on Verbs Based on WordNet and FrameNet, unpublished
Computational Linguistic, 2006.
[13] Q. X. Chen, WordNet: an on-line Thesaurus, Applied
Linguistics, no.2, 1998, pp.93-99.

Figure 1. The process of identifying Chinglish

232
236