Bayesian Decision Making

Risk minimization framework

Let {1, 2,, C} be the set of C states
of nature (or categories)
Let {1, 2,, a} be the set of possible
a actions
Let (i | j) be the loss incurred for taking
action i when the state of nature is j

Risk minimization framework

The expected loss:

Given an observation with vector x, the conditional risk is:

At every x, a decision is made: (x), by minimizing the expected

loss.
Our final goal is to minimize the total risk over all x.

Risk minimization framework

Two-category classification
1 : deciding 1
2 : deciding 2
ij = (i | j) is loss incurred for deciding i
when the true state of nature is j

Conditional risk:
R(1 | x) = 11P(1 | x) + 12P(2 | x)
R(2 | x) = 21P(1 | x) + 22P(2 | x)

Risk minimization framework

Our rule is the following:
if R(1 | x) < R(2 | x)
action 1: decide 1 is taken
This results in the equivalent rule :
Decide 1 if:
(21- 11) p(x | 1) P(1) >
(12- 22) p(x | 2) P(2)
and decide 2 otherwise

Likelihood ratio

The preceding rule is equivalent to the following

rule:

p (x | 1 ) 12 22 P(2 )
if
>
.
p(x | 2 ) 21 11 P(1 )
Then take action 1 (decide 1)
Otherwise take action 2 (decide 2)

Likelihood ratio

Regions of decision and zero-one loss function,

therefore:

12 22 P(2 )
Let
.
=
21 11 P(1 )
p(x | 1 )
then decide 1 if :
>
p(x | 2 )

Likelihood ratio

Likelihood ratio

Likelihood ratio

10

Optimal decision property

If the likelihood ratio exceeds a threshold
value independent of the input pattern x, we
can take optimal actions

11

Zero-one loss function

Actions are decisions on classes
If action i is taken and the true state of nature is j
then: the decision is correct if i = j and in error if i j

Seek a decision rule that minimizes the probability of

error which is the error rate

Zero-one loss function

12

Introduction of the zero-one loss function:

0 i = j
( i | j ) =
1 i j
Therefore, the conditional risk is: All errors are equally costly.
j =c

R( i | x) = ( i | j ) P( j | x)
j =1

= P( j | x) = 1 P(i | x)
j i

The risk corresponding to this loss function is the average probability

error

Zero-one loss function

13

Minimize the risk requires maximizing P(i | x)

(since R(i | x) = 1 P(i | x))

For Minimum error rate

Decide i if P (i | x) > P(j | x) j i