March 2006 Alon Slapak 1 of 1

Bayes Classification
A practical approach
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 2 of 2
Discriminant function
Definition: a discriminant function is an
n-dimensional hypersurface which divides
the n-dimensional feature space into two
separate areas contain separate classes.
A 2-dimwnsinal
discriminant
function
A 1-dimwnsinal
discriminant
function
2-dimensional
feature space
1-dimensional
feature space
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 3 of 3
Discriminant function
Let h(x) be a discriminant function. A two-category
classifier uses the following rule:
Decide e
1
if h(x) > 0 and e
2
if h(x) < 0
If h(x) = 0 x is assigned to either class.
h(x)=x-6.25
h(x) < 0
h(x) < 0
1 2
( ) 1.02 2.5 h x x ~ x
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 4 of 4
Thomas Bayes
At the time of his death, Rev. Thomas
Bayes (1702 1761) left behind two
unpublished essays attempting to
determine the probabilities of causes
from observed effects. Forwarded to the
British Royal Society, the essays had
little impact and were soon forgotten.

When several years later, the French
mathematician Laplace independently
rediscovered a very similar concept, the
English scientists quickly reclaimed the
ownership of what is now known as the
Bayes Theorem.
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 5 of 5
Conditional Probability
Definition: Let A and B be events with P(B) > 0.
The conditional probability of A given B, denoted
by P(A|B), is defined as:

P(A|B) = P(A B)/P(B)

A
B
10 20 5
Venn Diagram
Given: N(A) = 30
N(A B) = 10
P(B | A) = N(A B)/N(A) = 10/30 = 1/3
Example:
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 6 of 6
Bayes theorem
Since P(A | B) = P(A B)/P(B),

we have: P(A | B)P(B) = P(A B)

Symmetrically we have: P(B | A)P(A) = P(B A) = P(A B)

Therefore: P(A | B)P(B) = P(B | A)P(A)

And:
( )
( ) ( )
( )
Bayes' theorem
P A B P B
P B A
P A
=
where
P(A | B) is the conditional probability,
P(A) , P(B) are the prior probabilities,
P(B | A) is the posterior probability

Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 7 of 7
Bayes theorem in a pattern recognition notation
Given classes e
i
and a pattern x,
( | ) ( )
( | )
( )
( ) ( | ) ( )
j j
j
j j
j
P P
P
P
where
P P P
e e
e
e e
=
=

x
x
x
x x
prior
probability
The prior probability reflects knowledge of the relative frequency of
instances of a class
likelihood
The likelihood is a measure of the probability that
a measurement value occurs in a class.
evidence
The evidence is a scaling term
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 8 of 8
Bayes classifier
The following phrase classify each pattern x to one
of two classes:
2
1
2
1
1 2
1 1 2 2
( | ) ( | )
( | ) ( ) ( | ) ( )
( ) ( )
P P
P P P P
P P
e
e
e
e
e e
e e e e
<
>
<

>
x x
x x
x x
or (since P(x) is common to both sides):
( )
2
1
2
1
1 1 2 2
1 2
2 1
( | ) ( ) ( | ) ( )
( | ) ( )
( | ) ( )
P P P P
P P
l
P P
e
e
e
e
e e e e
e e
e e
<
>
<
=
>
x x
x
x
x
Means, decide e
1
if
P(e
1
|x) > P(e
2
|x)
Likelihood ratio
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 9 of 9
Bayes Discriminant function
Since a ratio of probabilities may yield very small
values, it is common to use the log of the
likelihood ratio:
and the derived Bayes discriminant function is:
( ) ( ) ( )
1
1 2
2
( )
ln ( | ) ln ( | ) ln
( )
P
h P P
P
e
e e
e
| |
= +
|
\ .
x x x
( ) ( )
2
1
1 2
2 1
( | ) ( )
ln ln ln
( | ) ( )
P P
l
P P
e
e
e e
e e
< | | | |
=
| |
>
\ . \ .
x
x
x
0 < ) x ( h if 2 e and 0 > ) x ( h if 1 e Decide Remember:
If h(x) = 0 x is assigned to either class.
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 10 of 10
Example - Gaussian Distributions
A multi dimensional Gaussian distribution is:
( )
( )
( )
( )
2
1
2
/ 2
1/ 2
2 1
th
th
1
2 | |
where:
( ) ( )
is the feature vector
is the mean vector of the class
is the covariance matrix of the class
feature space dimension
i
i
d
i
n
i
T
i i i
i
i
P e
d
i
i
n
e
t

=
E
= E

x
x
x x m x m
x
m
120 130 140 150 160 170 180 190
30
35
40
45
50
55
60
65
70
75
80
height [cm]
w
e
i
g
h
t

[
k
g
]
Females
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 11 of 11
Example - Gaussian Distributions
A multi dimensional Gausian distribution is:
( )
( )
( )
( )
( )
( ) ( ) ( ) ( ) ( ) ( )
( ) ( )
2 2
1 2
1 1
1
2 2
/ 2 / 2
1/ 2 1/ 2
2
1 2
2 2
1 1 1
1 1 2 2 2 2 2 2
2
2 2
1 1 1
1 2 2
2 2
( ) 1 1
ln ln ln
( )
2 | | 2 | |
( ) 1 1
ln 2 ln | | ln 2 ln | | ln
2 2 ( )
| | ( ) 1 1
ln ln
2 2 | | ( )
d d
n n
n n
P
e e
P
P
d d
P
d
h
P
d
P
e
e
t t
e
t t
e
e
e

| | | |
| |
= + = | |
|
| |
E E
\ .
\ . \ .
| |
= + E + E =
|
\ .
| | | E
= +
|
E
\ .
x x
x x
x
x
x
1 1
1 1 1
1 1 1 2 2 2 2
2 2
| | ( ) 1 1
( ) ( ) ( ) ( ) ln ln
2 2 | | ( )
T T
P
P
e
e

| | | | E
= E E
|
=

+
| |
E
\ \ .
|
\ .
.
x m x m x m x m
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 12 of 12
Example - Gaussian Distributions
Assume two Gaussian distributed classes with
And
clear all
N1 = 150; N2 = 150;
E1 = [50 40; 40 50]; E2 = [50 40; 40 50];
M1 = [30,55]'; M2 = [60,40]';
%-------------------------------------------------------------------------
% Classes drawing
%-------------------------------------------------------------------------
[P1,A1] = eig(E1); [P2,A2] = eig(E2);
y1=randn(2,N1); y2=randn(2,N2);
for i=1:N1,
x1(:,i) =P1*sqrt(A1)* y1(:,i)+M1;
end;
for i=1:N2,
x2(:,i) =P2*sqrt(A2)* y2(:,i)+M2;
end;
figure;
plot(x1(1,:),x1(2,:),'^',x2(1,:),x2(2,:),'or');
axis([0 100 0 100]);
xlabel('x1')
ylabel('x2')
1 2
E = E
1 2
( ) ( ) P P e e =
( ) ( )
( )
2 1 1 1 2 2
1
2
T
T T
h = + x m m x m m m m
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 13 of 13
Example - Gaussian Distributions
%-------------------------------------------------------------------------
% Classifier drawing
%-------------------------------------------------------------------------
ep=1.2;
k=1;
for i=0:k:100,
for j=0:k:100,
x=([i;j]);
h=0.5*(x-M1)'*inv(E1)*(x-M1)-0.5*(x-M2)'*inv(E2)*(x-M2)+0.5*log(det(E1)/det(E2));
if (abs(h)<ep),
hold on; plot(i,j,'*k'); hold off;
end;
end;
end;

h(x) > 0
h(x) < 0
( ) ( ) ( )
2 1 1 1 2 2
1
0
2
T
T T
h = + = x m m x m m m m
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 14 of 14
Example - Gaussian Distributions
Assume two Gaussian distributed classes with
And
clear all
N1 = 150; N2 = 150;
E1 = [50 40; 40 50]; E2 = [50 -40; -50 50];
M1 = [30,55]'; M2 = [60,40]';
%-------------------------------------------------------------------------
% Classes drawing
%-------------------------------------------------------------------------
[P1,A1] = eig(E1); [P2,A2] = eig(E2);
y1=randn(2,N1); y2=randn(2,N2);
for i=1:N1,
x1(:,i) =P1*sqrt(A1)* y1(:,i)+M1;
end;
for i=1:N2,
x2(:,i) =P2*sqrt(A2)* y2(:,i)+M2;
end;
figure;
plot(x1(1,:),x1(2,:),'^',x2(1,:),x2(2,:),'or');
axis([0 100 0 100]);
xlabel('x1')
ylabel('x2')
1 2
E = E
1 2
( ) ( ) P P e e =
( )
1 1
1 1
1 1 1 2 2 2 2
2
| | 1 1
( ) ( ) ( ) ( ) ln
2 2 | |
T T
h

| | E
= E E +
|
E
\ .
x x m x m x m x m
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 15 of 15
Example - Gaussian Distributions
%-------------------------------------------------------------------------
% Classifier drawing
%-------------------------------------------------------------------------
ep=1;
k=1;
for i=0:k:100,
for j=0:k:100,
x=([i;j]);
h=0.5*(x-M1)'*inv(E1)*(x-M1)-0.5*(x-M2)'*inv(E2)*(x-M2)+0.5*log(det(E1)/det(E2));
if (abs(h)<ep),
hold on; plot(i,j,'*k'); hold off;
end;
end;
end;

h(x) > 0
h(x) < 0
h(x) < 0
1 1 1
1 1 1 2 2 2
2
1 1 1
( ) ( ) ( ) ( ) ( ) ln 0
2 2 2
T T
h

E
= E E + =
E
x x m x m x m x m
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 16 of 16
Exercise
Synthesize two classes with different a-
priory probabilities. Show how the
probabilities influence the discriminant
function.
Synthesize three classes and plot the
discriminant functions.
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 17 of 17
Summary
Steps for Building a Bayesian Classifier

Collect class exemplars
Estimate class a priori probabilities
Estimate class means
Form covariance matrices, find the inverse and determinant for each
Form the discriminant function for each class
Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography
March 2006 Alon Slapak 18 of 18
Bibliography
1. K. Fukunaga, Introduction to Statistical Pattern
Recognition, 2
nd
ed., Academic Press, San Diego,
1990.
2. L. I. Kuncheva, J. C. Bezdek amd R. P.W. Duin,
Decision Templates for Multiple Classier Fusion: An
Experimental Comparison, Pattern Recognition, 34,
(2), pp. 299-314, 2001.
3. R. O. Duda, P. E. Hart and D. G. Stork, Pattern
Classification (2nd ed), John Wiley & Sons, 2000.


Example

Discriminant
function


Bayes
theorem

Bayes
discriminant
function
Bibliography