Вы находитесь на странице: 1из 19

Building Lexicon for Sentiment

Analysis from Massive Collection of


HTML Documents
Kaji, N. & Kitsuregawa, M.
2007 EMNLP-CoNLL Conference

Tyler Thornblade
Introduction
Creation of high-quality polar phrase lexicon
Fully automatic approach
Sacrifice recall for precision
Make up for low recall by using an enormous
corpus
General Approach
Start with a large
HTML corpus
Find polar
sentences
Extract polar
phrases from polar
sentences
Analyze polar
phrases and add to
lexicon
Sentence extraction: Syntactic Clues

Manually created list of cue phrases (single


underline)
Automatic detection of polar sentences
(double underline) based on syntactic
constraints
Sentence extraction: Layout Structure

Heuristics for extracting polar data from


itemized lists <UL>, <OL>
Heuristics for extracting polar data from
various kinds of tabular data
Evaluation of polar sentence corpus
500 sentences selected
Two annotators evaluated whether
sentences were polar or non-polar
Annotator A precision 91.4%
Annotator B precision 92.0%
Inter-annotator agreement 93.5% Kappa 0.90
Most errors due to lack of context
E.g. “There is much information” marked as
positive
Polar phrase extraction
Extract phrase candidates from polar
sentences using structural clues
Count occurrences in positive and negative
sentences
Uses known cue phrase list (with modifiers for
negations)
Drop counts of phrases not in the main clause
 E.g. “Although the price is high, the shape is
beautiful”
Drop counts of phrases that appear less than
three times in total in both positive and
negative sentences
Polar phrase evaluation, Chi-square
First create a table of frequencies

Evaluate using Chi-square


Polar phrase evaluation, PMI
Reuse table of frequencies

Evaluate using PMI


Polar phrase evaluation, finish
Finally, for both Chi-square and PMI, use
configurable threshold
PV > theta, positive phrase
PV < -theta, negative phrase
By adjusting theta, we can balance recall vs.
precision
Evaluation of lexicon
Pulled a list of 500 adjective phrases
randomly selected from Web
After removing parse errors and duplicates,
405 unique phrases
No overlap with development set
Balance: 158 positive, 150 negative, 97
neutral
 Based on human annotation
 Two annotators, Kappa 0.73
Baseline: Turney 2002, co-occurrence in a
window
Turney used “excellent” and “poor”, they use
最高 “ best” and 最低 “ worst”
Evaluation of lexicon
Evaluation of lexicon
Direct analysis of lexicon
Human analysis of 200 items from lexicon
Two annotators, average precision 71.3%
Kappa 0.66
Error analysis
Turney method had trouble with neutral
sentences (37 out of 48 errors)
Good performance on colloquial phrases
(e.g. dasai) not commonly found in
dictionary/thesaurus
Lexicon captured a lot of non-adjectival data
of interest
影響を受け難い It is hard to receive the effect
グラフィックが綺麗だ The graphics are pretty
手入れが楽だ It is easy to maintain
Subjective responses
Sentence level analysis
Only analyzed 0.1% of sentence corpus
“Most” errors due to context, so by a looser
standard precision may be significantly higher
than 92%

Phrase level analysis


Rigorous
How hard did they try with Turney? Bad
results.
Picked an easy target (adjectival phrases)
Human analysis only analyzed 2% of lexicon
Closing thoughts
Best ideas
Very high precision, low recall, and crunch a
lot of data
Page layout cues for extracting data
Domain independent
Applicable to other langauges (See Cem’s
Kanayama et al discussion)
Method captured a lot of nouns and verbs,
even though they didn’t evaluate this aspect
Related works
Contrasting works
 Words with Attitude, Kamps, J. and Marx, M. (Danielle, LEX)
[Thesaurus]

Seed word based approaches


 Automatic Seed Word Selection for Unsupervised Sentiment
Classification of Chinese Text, Zagibalov, T. & Carroll, J. (Tyler,
Clues & Class)
 Identifying and Analyzing Judgment Opinions, Kim, S. & Hovy, E.
(Matt OES)
 Turney, P. Thumbs Up or Thumbs Down? Semantic Orientation
Applied to Unsupervised Classification of Reviews (Michael,
Docclass)

Supervised approach to polar phrase identification


 Recognizing Contextual Polarity in Phrase-Level Sentiment
Analysis, Wilson, T., Wiebe, J., Hoffmann, P (Michael, Clues &
Class)
 Extracting Aspect-Evaluation,Kobayashi, N., Inui, K. &
Related works (continued)

Syntactic patterns
 Detection of Users' Wants and Needs from Opinions,
Kanayama, H. & Nasukawa, T. (Cem, Apps)
 Deeper Sentiment Analysis Using Machine
Translation Technology, Hiroshi Kanayama and
Tetsuya Nasukawa and Hideo Watanabe (Shilpa,
Multi)
Takes advantage of syntactic patterns but uses a very different
(MT) approach. See also Kanayama and Nasukawa 2006 for
enhancements to Turney’s window-based approach to co-
occurrence detection.

Layout patterns
 Minqing Hu and Bing Liu. 2004. Mining and
summarizing customer reviews (Not read, in list)
 Kim, S. & Hovy, E. 2006. Automatic Identification of
Pro and Con Reasons in Online Reviews (Yaw, OD)