Вы находитесь на странице: 1из 44

Associative Tag Recommendation Exploiting Multiple Textual Features

Fabiano Muniz Belm


fmuniz@dcc.ufmg.br

Eder Martins Jussara Almeida

Tatiana Pontes Marcos Gonalves


1

DCC - UFMG

Tags

Keywords assigned by users to a content

Madonna

Tags

Tags assigned to Madonna in LastFM


2

Tags

Present in many Web 2.0 applications

Video Focus on rich media content

Image

Audio

Rich Media Content


Challenges to current multimedia Information Retrieval (IR) methods
Scalability Poor quality of user generated material (usually) [Boll, 2007]

Tags

Good alternative for content organization, propagation and retrieval


Main data source for several IR services
Classification [Figueiredo et al., 2009]

Searching [Li et al., 2008] Content Recommendation [Guy et al., 2010]

Tag Recommendation Goals

Suggest terms related to the content of a target object Improve tag quality
completeness and accuracy noise
Ex.: misspellings, unrelated words

Improve effectiveness of tag-based IR services

Web 2.0 Objects and Textual Features


object

Web 2.0 Objects and Textual Features


title object

description

comments tags
8

Tag Recommendation Strategies: State-of-the-Art


Sum+
[Sigurbjornsson & Zwol, 2008]

Exploits tag co-occurrence and a metric of relevance

Lazy Associative Tag Recommender (LATRE) [Menezes et al., 2010]


Generates, on demand, more complex tag co-occurrence patterns

Co-occurrence and Text Based Tag Recommender (CTTR)


adapted from [Lipczak et al., 2009]

Exploits multiple textual features


Do not exploit tags previously assigned to the target object
9

Tag Recommendation Strategies: State-of-the-Art


Exploit at most 2 of these dimensions:
Term co-occurrence with tags pre-assigned to the target object

Multiple textual features


Metrics of relevance

Our Goal:
Exploit the 3 dimensions jointly
10

Problem Statement

Develop a function to estimate the relevance of a candidate term as a recommendation for the target object
Relevant term:

Describes well the objects content

Distinguishes an object from others

1st term

relevance

ranking

2nd term 3rd term

...

relevance
11

Problem Statement
Input
Io: set of tags previously assigned to the object

Fo = {Fo1, Fo2, ..., Fon}: set of textual features in object o Textual features exploited here: title and description

Output
Co : list of ranked candidate terms for the object o

Co Io =

12

Goals and Contributions


Propose new tag recommendation strategies
8 heuristic recommendation functions
Extensions of state-of-the-art co-ocurrence based methods

New sources of tags

2 Learning-to-Rank (L2R) based strategies


RankSVM Genetic Programming

Extensive Experimental Evaluation


3 popular Web 2.0 applications: YouTube, YahooVideo, LastFM
Various scenarios
13

Relevance Metrics for Tag Recommendation


Tag Co-occurrence Metrics
Descriptive Power Heuristic Metrics

Discriminative Power Heuristic Metrics

14

Tag Co-occurrence Metrics


Estimate the relevance of terms which co-occur with previously assigned tags

Sum [Sigurbjornsson et al., 2008]


Based on association rules: (set of tags X) (candidate term c)
Importance of an association rule: confidence () Given: candidate term c, initial set of tags I

: confidence of the rule X c : Set of selected association rules : size limit for X

15

Descriptive Power Heuristic Metrics


Estimate how well a term describes the target objects content

We exploit 4 heuristic metrics TF (Term Frequency) TS (Term Spread)


wTF (Weighted TF)

wTS (Weighted TS)

our proposal

16

Descriptive Power: Term Frequency (TF)


TF (c, o) = Number of occurrences of the term candidate c in textual features of object o
Considers all textual features as a single list of terms Previously exploited for tag recommendation
[Wang et al., 2009; Cao et al., 2009]

Traditionally used in IR

17

Descriptive Power: Term Spread (TS)

TS (c, o) = #textual features in object o which contain term c Previously employed for quality assessment of textual features
[Figueiredo et al., 2009]

TS(Madonna, o) = 2

TS(political, o) = 1

18

Descriptive Power: wTF and wTS


Built from TF and TS
Different textual features may present different descriptive capacities

wTF and wTS weight a term based on the average descriptive power of its containing textual feature Average descriptive power of Fi : Average Feature Spread AFS(Fi)
[Figueiredo et al., 2009]

Feature Instance Spread: FIS(Foi)

19

Discriminative Power
Capacity of a term to distinguish an object from others

Inverse Feature Frequency (IFF): adaptation of IDF

N = size of the training set


f(c, tag) = #objects containing c as a tag

Rationale: very frequent terms are not discriminative


E.g.: term video in YouTube
20

Discriminative Power: Stability and Entropy


Stability [Sigurbjornsson et al., 2008]
Reduces the importance of very common as well as very rare terms

Entropy [Heymann et al., 2008]


Estimates the predictability of a term as a tag

Term that occurs consistently with certain tags


low entropy, high discriminative power

Term that occurs indiscriminately with many other tags


high entropy, low discriminative power
21

Tag Recommendation Strategies: Our New Heuristics


Extensions built from state-of-the-art methods Sum+ and LATRE Inclusion of a descriptive power metric
TS, TF, wTS or wTF

Candidate terms
Generated from co-occurrences with pre-assigned tags
Extracted from multiple textual features

Let DP be the selected metric, i.e., DP = TS, TF, wTS or wTF


Sum+DP = Sum+ + (1 - ) DP

LATRE+DP = LATRE + (1 - ) DP
where parameter 0 1 is a weighting factor
22

Tag Recommendation Strategies: Our New Heuristics 8 heuristics Sum+ or LATRE Sum+TS Sum+TF Sum+wTS Sum+wTF

TF, TS, wTF or wTS

LATRE+TS LATRE+TF LATRE+wTS LATRE+wTF

23

Learning-to-Rank Based Strategies


Idea: learn a function which estimates the relevance of candidate terms based on a list L of metrics of relevance

12 metrics
Sum (with =1 and =3) Vote, Vote+ and Sum+: alternatives to Sum TS, TF, wTS, wTF IFF, Stability, Entropy

Co-occurrence

Descriptive Power

Discriminative Power

24

Learning-to-Rank Based Strategies


For each candidate term c for each object o, we assign:
Mc: vector containing the values of the metrics related to c

Yc: binary label indicating if c is relevant (Yc=1) or not (Yc=0)

Exploited techniques
RankSVM

Genetic Programming (GP)

25

Experimental Evaluation: Data Sets


9 million videos

160,000 videos

2.8 million artists


26

Evaluation Methodology
Automatic Evaluation
Tags in an object

input Evaluation Metrics

expected answer

Precision on top k positions of the rank (P@k) Recall

Average Precision (AP)


27

Evaluation Methodology
2 evaluation scenarios
Data divided in 3 subsets according to the number of tags by object

Smallest, Medium, Largest intervals


Mixed scenario: union of the 3 subsets Number of tags by object LastFM YahooVideo Smallest 2-4 2-6 Medium 5-6 7-11 Largest 7-507 12-52 Mixed 2-507 2-52

YouTube

2-5

6-9

10-77

2-77

Parameterization of the methods


Experiments performed in a validation set: described in the paper
28

Representative Results
State-of-the-Art Mixed Scenario. Average P@5 and 95% Confidence Intervals
Strategy Sum+ LastFM 0.411 0.001 YahooVideo 0.484 0.003 YouTube 0.245 0.002

LATRE
CTTR

0.405 0.001
0.260 0.001

0.608 0.003
0.465 0.004

0.285 0.004
0.376 0.002

LATRE outperforms Sum+ in most of the cases Up to 25% in P@5 CTTR: good alternative in some cases
29

Representative Results
State-of-the-Art Methods VS. Our New Heuristics Mixed Scenario Average P@5 and 95% Confidence Intervals
Strategy
Sum+ LATRE CTTR

LastFM
0.411 0.001 0.405 0.001 0.260 0.001

YahooVideo
0.484 0.003 0.608 0.003 0.465 0.004

YouTube
0.245 0.002 0.285 0.004 0.376 0.002

Sum+TF
Sum+TS Sum+wTF Sum+wTS

0.417 0.001
0.418 0.002 0.417 0.001

0.643 0.003
0.674 0.003 0.666 0.002

0.462 0.001
0.475 0.002 0.490 0.002

Gains over the best state-of-the-art method 40% in P@5 32% in Recall 62% in AP

LATRE+TF LATRE+TS LATRE+wTF LATRE+wTS

TF < TS < wTF < wTS for LATRE+DP and 0.417 0.002 0.707 0.002 0.502 0.003 Sum+DP 0.408 0.001 0.698 0.002 0.472 0.002
0.411 0.001

Most promising 0.408 0.001 0.729 0.002 0.503 0.003 heuristic: 0.411 0.001 0.733 0.003 0.489 0.003 LATRE+wTS
30

0.716 0.003

0.467 0.003

Representative Results: Average P@5 and 95% Confidence Intervals


LastFM

Strategy
LATRE+wTS GP-based RankSVM-based Strategy LATRE+wTS

Mixed
0.411 0.001 0.433 0.003 0.411 0.002 Mixed 0.733 0.003

Smallest
0.486 0.005 0.500 0.005 0.499 0.007 YahooVideo Smallest 0.818 0.005

Medium
0.465 0.005 0.476 0.002 0.450 0.005 Medium 0.729 0.003

Largest
0.388 0.003 0.434 0.003 0.393 0.003 Largest 0.780 0.007

GP-based RankSVM-based Strategy


LATRE+wTS GP-based RankSVM-based

0.743 0.007 0.752 0.002 Mixed


0.489 0.003 0.507 0.001 0.512 0.002

0.822 0.004 0.826 0.003


YouTube

0.734 0.003 0.725 0.003 Medium


0.471 0.005 0.477 0.002 0.484 0.005

0.789 0.006 0.800 0.005 Largest


0.450 0.007 0.462 0.003 0.468 0.003
31

Smallest
0.593 0.003 0.595 0.003 0.601 0.005

Conclusions and Future Work


New tag recommendation strategies
Heuristics: gains of up to 40% in P@5, 32% in Recall, 62% in AP
L2R based strategies
Modest extra gains
Promising and flexible approach

Largest gains due to the use of descriptive power metric (e.g., wTS)

Future Work
Manual assessment of relevance, usefulness, diversity

New metrics and techniques: content and user features


Personalized Recommendation Evaluation with no pre-assigned tags
32

Thank You

33

Associative Tag Recommendation Exploiting Multiple Textual Features


Fabiano Muniz Belm
fmuniz@dcc.ufmg.br

Eder Martins Jussara Almeida

Tatiana Pontes Marcos Gonalves


34

DCC - UFMG

Prototype
GreenMeter: Demo in SIGIR 2011

35

Outline
1. Motivation 2. Contextualization 3. State-of-the-Art 4. Goals and Contributions 5. Relevance Metrics for Tag Recommendation

6. Tag Recommendation Strategies 7. Experimental Evaluation 8. Conclusions and Future Work


36

LastFM Strategy Sum+ LATRE CTTR Sum+TF Sum+TS Mixed 0.411 0.001 0.405 0.001 0.260 0.001 0.417 0.001 0.418 0.002 Smallest 0.454 0.003 0.465 0.004 0.285 0.003 0.470 0.004 Medium 0.433 0.004 0.457 0.005 0.267 0.002 0.436 0.004 Largest 0.391 0.001 0.384 0.004 0.217 0.002 0.395 0.001 0.396 0.001

0.472 0.005 0.439 0.004

Sum+wTF Sum+wTS LATRE+TF


LATRE+TS LATRE+wTF LATRE+wTS

0.417 0.001 0.417 0.002 0.408 0.001


0.411 0.001 0.408 0.001 0.411 0.001

0.470 0.005 0.470 0.005 0.479 0.004


0.487 0.005 0.479 0.004 0.486 0.005

0.436 0.004 0.438 0.004 0.459 0.005


0.466 0.005 0.459 0.005 0.465 0.005

0.395 0.001 0.396 0.001 0.385 0.004


0.388 0.003 0.385 0.004 0.388 0.003

YahooVideo
Strategy Sum+ Mixed 0.484 0.003 Smallest 0.453 0.007 Medium Largest 0.511 0.004 0.615 0.004

LATRE
CTTR Sum+TF Sum+TS Sum+wTF Sum+wTS LATRE+TF LATRE+TS LATRE+wTF

0.608 0.003
0.465 0.004 0.643 0.003 0.674 0.003 0.666 0.002 0.707 0.002 0.698 0.002 0.716 0.003 0.729 0.002

0.525 0.006
0.649 0.005 0.755 0.002 0.764 0.003 0.784 0.003 0.795 0.004 0.781 0.002 0.785 0.001 0.821 0.004

0.619 0.004 0.731 0.006


0.407 0.004 0.394 0.003 0.597 0.003 0.663 0.005 0.631 0.003 0.714 0.004 0.612 0.003 0.673 0.004 0.660 0.003 0.721 0.004 0.673 0.004 0.745 0.007 0.710 0.003 0.778 0.007 0.706 0.003 0.754 0.006

LATRE+wTS

0.733 0.003

0.818 0.005

0.729 0.003 0.780 0.007

YouTube Strategy Mixed Smallest Medium Largest

Sum+
LATRE CTTR Sum+TF

0.245 0.002
0.285 0.004 0.376 0.002 0.462 0.001

0.211 0.006
0.219 0.005 0.463 0.005 0.552 0.004

0.212 0.003
0.242 0.005 0.337 0.004 0.421 0.006

0.282 0.004
0.326 0.006 0.269 0.002 0.403 0.003

Sum+TS
Sum+wTF Sum+wTS LATRE+TF

0.475 0.002
0.490 0.002 0.502 0.003 0.472 0.002

0.560 0.004
0.583 0.003 0.593 0.003 0.557 0.004

0.433 0.005
0.451 0.005 0.461 0.005 0.436 0.004

0.419 0.003
0.416 0.004 0.431 0.004 0.425 0.006

LATRE+TS

0.467 0.003

0.561 0.004
0.596 0.004 0.593 0.003

0.445 0.004
0.464 0.004 0.471 0.005

0.441 0.007
0.439 0.005 0.450 0.007

LATRE+wTF 0.503 0.003 LATRE+wTS 0.489 0.003

Representative Results: Average P@5


LastFM Strategy LATRE+wTS Mixed 0.411 0.001 Smallest 0.486 0.005 Medium 0.465 0.005 Largest 0.388 0.003

GP-based
RankSVM-based

0.433 0.003
0.411 0.002

0.500 0.005
0.499 0.007

0.476 0.002
0.450 0.005

0.434 0.003
0.393 0.003

YahooVideo Strategy
LATRE+wTS GP-based RankSVM-based

Mixed
0.733 0.003 0.743 0.007 0.752 0.002

Smallest
0.818 0.005 0.822 0.004 0.826 0.003 YouTube

Medium
0.729 0.003 0.734 0.003 0.725 0.003

Largest
0.780 0.007 0.789 0.006 0.800 0.005

Strategy LATRE+wTS

Mixed 0.489 0.003

Smallest 0.593 0.003

Medium 0.471 0.005

Largest 0.450 0.007

GP-based RankSVM-based

0.507 0.001 0.512 0.002

0.595 0.003 0.601 0.005

0.477 0.002 0.484 0.005

0.462 0.003 0.468 0.003

Strategy
Sum+ LATRE CTTR

LastFM
0.411 0.001 0.405 0.001 0.260 0.001

YahooVideo
0.484 0.003 0.608 0.003 0.465 0.004

YouTube
0.245 0.002 0.285 0.004 0.376 0.002

Sum+TF
Sum+TS

0.417 0.001
0.418 0.002

0.643 0.003
0.674 0.003

0.462 0.001
0.475 0.002

Sum+wTF
Sum+wTS LATRE+TF LATRE+TS LATRE+wTF

0.417 0.001
0.417 0.002 0.408 0.001 0.411 0.001 0.408 0.001

0.666 0.002
0.707 0.002 0.698 0.002 0.716 0.003 0.729 0.002

0.490 0.002
0.502 0.003 0.472 0.002 0.467 0.003 0.503 0.003

LATRE+wTS

0.411 0.001

0.733 0.003

0.489 0.003

Representative Results: Average P@5


State-of-the-Art Methods
LastFM Strategy Sum+ LATRE CTTR Strategy Sum+ LATRE CTTR Strategy Sum+ Mixed 0.41 0.41 0.26 Mixed 0.48 0.61 0.47 Mixed 0.25 Smallest 0.45 0.47 0.29 YahooVideo Smallest 0.45 0.52 0.65 YouTube Smallest 0.21 Medium 0.21 Largest 0.28 Medium 0.51 0.62 0.41 Largest 0.62 0.73 0.40 Medium 0.43 0.46 0.27 Largest 0.39 0.38 0.22

LATRE outperforms Sum+ in most of the cases


Up to 25% in P@5

CTTR: good alternative in some cases

LATRE
CTTR

0.29
0.38

0.22
0.46

0.24
0.34

0.33
0.27
42

Representative Results: Average P@5


New Heuristics
Strategy Best baseline LastFM Mixed Smallest Medium 0.41 0.47 0.46 Largest 0.39

Sum+wTS
LATRE+wTS Strategy

0.42
0.41

0.47
0.49

0.44
0.46

0.40
0.39 Largest

Gains over the best stateof-the-art method 40% in P@5 32% in Recall 62% in AP
TF < TS < wTF < wTS for LATRE+DP and Sum+DP Most promising heuristic: LATRE+wTS
43

YahooVideo Mixed Smallest Medium 0.61


0.71

Best baseline
Sum+wTS

0.65
0.80

0.62
0.66

0.73
0.72

LATRE+wTS Strategy
Best baseline

0.74

0.82

0.73

0.78 Largest
0.28

YouTube Mixed Smallest Medium 0.25 0.21 0.21

Sum+wTS
LATRE+wTS

0.50
0.49

0.59
0.59

0.46
0.47

0.43
0.45

Representative Results: Average P@5


L2R Based Strategies
LastFM Strategy LATRE+wTS GP based RankSVM Strategy Mixed 0.41 Smallest 0.49 Medium 0.46 Largest 0.39

Modest gains over our best heuristic


Up to 12% in P@5 Up to 10% in AP Same Recall

0.43
0.41 Mixed

0.50
0.50

0.48
0.45 Medium

0.43
0.39 Largest

YahooVideo
Smallest

LATRE+wTS GP based RankSVM Strategy


LATRE+wTS GP based RankSVM

0.71
0.74

0.80
0.82

0.66
0.73

0.78
0.79

The best L2R based strategy depends on the dataset

0.75 Mixed
0.49

0.83
YouTube

0.73 Medium
0.47

0.80 Largest
0.45

Smallest
0.59

0.51
0.51

0.60
0.60

0.48
0.48

0.46
0.47
44

Вам также может понравиться