Вы находитесь на странице: 1из 20

CSE 564

Pattern Recognition
Lecture 3

Bayesian Classifier
Syed Fawad Hussain

Bayesian Classifier
 Have two dice h1 and h2
 The probability of rolling an i given die h1 is denoted P(i|h1).

This is a conditional probability


 Pick a die at random with probability P(hj), j=1 or 2. The

probability for picking die hj and rolling an i with it is called


joint probability and is P(i, hj)=P(hj)P(i| hj).
 For any events X and Y, P(X,Y)=P(X|Y)P(Y)
 If we know P(X,Y), then the so-called marginal probability P(X)

can be computed as

P( X ) = P( X , Y )
Y

Syed Fawad Hussain

GIK Institute

Basics of Probability
 Quiz: We have two six-sided dice. When they are tolled, it could end up

with the following occurance: (A) dice 1 lands on side 3, (B) dice 2
lands on side 1, and (C) Two dice sum to eight. Answer the following
questions:

1) P( A ) = ?
2) P(B) = ?
3) P(C) = ?
4) P( A | B) = ?
5) P(C | A ) = ?
6) P( A , B) = ?
7) P( A , C ) = ?
8) Is P( A , C ) equals P(A) P(C) ?
Syed Fawad Hussain

GIK Institute

Basics of Probability

Prior, conditional and joint probability for random


variables

Prior probability: P(X )

Conditional probability: P( X1 |X2 ), P(X2 | X1 )

Joint probability: X = ( X1 , X2 ), P( X ) = P(X1 ,X2 )

Relationship: P(X1 ,X2 ) = P( X2 | X1 )P( X1 ) = P( X1 | X2 )P( X2 )

Independence: P( X2 | X1 ) = P( X2 ), P( X1 | X2 ) = P( X1 ), P(X1 ,X2 ) = P(

Bayesian Rule

P( X |C )P(C )
Likelihood Prior
P(C |X ) =
Posterior =
P( X )
Evidence
Syed Fawad Hussain

GIK Institute

Bayes Rule
P ( d | h) P ( h)
p(h | d ) =
P(d )

Who is who in Bayes rule

P(h) :
P(d | h) :

Understanding Bayes' rule


d = data
h = hypothesis (model)
- rearranging
p ( h | d ) P ( d ) = P ( d | h) P ( h )
P ( d , h) = P ( d , h)
the same joint probability
on both sides

prior belief (probability of hypothesish before seeingany data)


likelihood(probability of the data if the hypothesish is true)

P(d ) = P(d | h) P(h) : data evidence(marginalprobability of the data)


h

P(h | d ) :
Syed Fawad Hussain

posterior (probability of hypothesish after having seen the data d )


GIK Institute

Syed Fawad Hussain

GIK Institute

Syed Fawad Hussain

GIK Institute

Syed Fawad Hussain

GIK Institute

Syed Fawad Hussain

GIK Institute

Syed Fawad Hussain

GIK Institute

Probabilistic Classification

MAP classification rule

MAP: Maximum A Posterior


Assign x to c* if
P(C = c * |X = x ) > P(C = c |X = x ) c c * , c = c1 , , c L

Generative classification with the MAP rule

Apply Bayesian rule to convert them into posterior


probabilities
P( X = x |C = ci )P(C = ci )
P(C = ci |X = x) =
P( X = x)
P( X = x |C = ci )P(C = ci )
for i = 1,2 , , L

Then apply the MAP rule

Syed Fawad Hussain

GIK Institute

Bayesian Classification

Bayes classification
P(C |X ) P( X |C )P(C ) = P( X1 , , Xn |C )P(C )


Difficulty: learning the joint probability

Nave Bayes classification

Assumption that all input attributes are conditionally


independent! P(X1 , X2 , , Xn |C) = P(X1 |X2 , , Xn ; C)P(X2 , , Xn |C)
= P(X1 |C)P(X2 , , Xn |C)
= P(X1 |C)P(X2 |C) P(Xn |C)

MAP classification rule: for

[ P( x1 |c * ) P( xn |c * )]P(c * ) > [ P( x1 |c) P( xn |c)]P(c), c c * , c = c1 , , c L


Syed Fawad Hussain

GIK Institute

Bayesian Classification

Nave Bayes Algorithm (for discrete input attributes)

Learning Phase: Given a training set S,

For each target value of c i (c i = c 1 , , c L )


P ( C = c i ) estimate P ( C = c i ) with examples in S ;
For every attribute value x jk of each attribute X j ( j = 1 , , n ; k = 1 , , N j )
P ( X j = x jk | C = c i ) estimate P ( X j = x jk | C = c i ) with examples in S ;


Output: conditional probability tables; for Xj , Nj L elements

Test Phase: Given an unknown instance X = ( a1 , , an,)

Look up tables to assign the label c* to X if

[ P( a1 |c * ) P ( an |c * )]P (c * ) > [ P ( a1 |c ) P ( an |c )]P (c ), c c * , c = c1 , , c L


Syed Fawad Hussain

GIK Institute

NB Example

Suppose we have a collection


of balls Green and Red

Total balls = 60

Green = 40, Red = 20

Suppose we now introduce a


new ball (white) and would
like to know its color.

The color of a ball is likely to


be similar to those of its
vicinity
Syed Fawad Hussain

GIK Institute

NB Example

Syed Fawad Hussain

GIK Institute

NB Example

Syed Fawad Hussain

GIK Institute

Syed Fawad Hussain

GIK Institute

Example
 Learning Phase
Outlook

Play=Yes

Play=No

Temperature

Play=Yes

Play=No

Sunny

2/9
4/9
3/9

3/5
0/5
2/5

Hot

2/9
4/9
3/9

2/5
2/5
1/5

Overcast
Rain

Humidity
High
Normal

Mild
Cool

Play=Yes Play=No

3/9
6/9

4/5
1/5

P(Play=Yes) = 9/14
Syed Fawad Hussain

Wind

Play=Yes

Play=No

Strong

3/9
6/9

3/5
2/5

Weak

P(Play=No) = 5/14
GIK Institute

Example

Test Phase

Given a new instance,

x=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong)


Look up tables

P(Outlook=Sunny|Play=Yes) = 2/9

P(Outlook=Sunny|Play=No) = 3/5

P(Temperature=Cool|Play=Yes) = 3/9

P(Temperature=Cool|Play==No) = 1/5

P(Huminity=High|Play=Yes) = 3/9

P(Huminity=High|Play=No) = 4/5

P(Wind=Strong|Play=Yes) = 3/9

P(Wind=Strong|Play=No) = 3/5

P(Play=Yes) = 9/14

P(Play=No) = 5/14

MAP rule

P(Yes|x): [P(Sunny|Yes)P(Cool|Yes)P(High|Yes)P(Strong|Yes)]P(Play=Yes) = 0.0053


P(No|x): [P(Sunny|No) P(Cool|No)P(High|No)P(Strong|No)]P(Play=No) = 0.0206
Syed Fawad
Hussainthe fact P(Yes|x) < P(No|x), we label x to be No.GIK Institute
Given

Вам также может понравиться