Bayesian Decision Making

Discriminant functions, Normal

Distribution

Classifiers, Discriminant Functions

and Decision Surfaces

The multi-category case

Set of discriminant functions gi(x), i = 1,, c
The classifier assigns a feature vector x to
class i
if:
gi(x) > gj(x) j i

Generic view of a
pattern classifier

Discriminant Functions
Let gi(x) = -R(i | x)
(max. discriminant corresponds to min. risk!)
For the minimum error rate, we take
gi(x) = P(i | x)
(max. discrimination corresponds to max.
posterior!)
gi(x) p(x | i) P(i)
gi(x) = ln p(x | i) + ln P(i)
(ln: natural
logarithm!)

Discriminant functions
Discriminant functions do not change the
decision, when scaled by some positive
constant k.
The decision is not affected when a
constant is added to all
discriminant
functions.

Discriminant Functions
Feature space divided into c decision regions
if gi(x) > gj(x) j i then x is in Ri
(Ri means assign x to i)
The two-category case
A classifier is a dichotomizer that has two
discriminant functions g1 and g2
Let g(x) g1(x) g2(x)
Decide 1 if g(x) > 0 ; Otherwise decide 2

Dichotomizer

gi(x) = ln p(x | i) + ln P(i)

(ln: natural logarithm!)

The computation of g(x) for dichotomizer

or

Normal /Gaussian Distribution

10

Multivariate Gaussian Distribution

Where

and

d x d Covariance matrix

11

12

Multivariate Gaussian Distribution

13

Covariance matrix property

If w is any d dimensional vector, the
T
variance of w x can not be negative.
T

This leads to the quadratic form w w to

be non-negative
positive semi-definite
nature of
Eigen values of are non-negative.

14

Multivariate Gaussian Distribution

Reminder: the covariance matrix is symmetric and positive semidefinite.

Entropy - the measure of uncertainty

Normal distribution has the maximum entropy over all distributions with a
given mean and variance.

2D Gaussian

15

16

Multivariate Gaussian Distribution

17

Mahalanobis Distance