Pattern Recognition 2011

Lab Sheet 2
Solution
Exercise 1 Total probability and Bayes Theorem 5
Suppose that we have three coloured boxes r (red), b (blue), and g (green). Box r contains
3 apples (a), 4 oranges (o), and 3 limes (), box b contains 1 apple, 1 orange, and 0 limes,
and box g contains 3 apples, 3 oranges, and 4 limes. A box is chosen at random with
probabilities p (r) = 0.2, p (b) = 0.2, p (g) = 0.6 and a piece of fruit is removed from the
box with equal probability of selecting any of the items in the box.
a. What is the probability of selecting an apple? 2
b. If we observe that the selected fruit is in fact an orange, what is the probability
that it came from the green box? 3
________________________________________________
a. p (a) = p (a | r) p (r) +p (a | b) p (b) +p (a | g) p (g) =
3
10
0.2+
1
2
0.2+
3
10
0.6 = 34%
b. p (g | o) =
p(o | g)p(g)

=r,b,g
p(o | )
= 50%
Exercise 2 Probability density and likelyhood function 5
Suppose two one-dimensional likelyhoods with equal prior probabilities are of the form
p (x |
i
) e

|xa
i
|
b
i
for i = 1, 2 and 0 b
i
.
a. Write an expression for each density. 2
b. Determine the likelyhood ratio
p(x|
1
)
p(x|
2
)
. 1
1
c. Plot the likelyhood ration for a
1
= 0, b
1
=1, a
2
=1, and b
2
= 2. 2
________________________________________________
a. p (x |
i
) =
1
2b
i
e

|xa
i
|
b
i
b.
b
2
b
1
exp
_
|xa
2
|
b
2

|xa
1
|
b
1
_
c.
10 5 0 5 10
0
1
2
3
x
p
(
x
|
w
1
)

/

p
(
x
|
w
2
)
Exercise 3 Minimum-error-rate classication 14
(For this exercise you might require to read section 2.3 (Minimum error rate classica-
tion) which will be dealt with in the lecture on April 20th!)
Generalize the minimax decision rule in order to classify patterns from three categories
having triangle densities as follows:
p(x|
i
) = T(
i
,
i
) =
_
_
_
(
i
|x
i
|)

2
i
for |x
i
| <
i
0 otherwise
,
where
i
> 0 is the half-width of the distribution (i = 1, 2, 3). Assume that
1
<
2
<
3
and
i
,
i+1

i+1

i

i
+
i+1
for i = 1, 2.
a. Explain if and how the assumptions constrain the generality of the problem. 3
b. In terms of the priors P (
i
), means
i
, and half-widths
i
, nd the optimal decision
points x

1
and x

2
under a zero-one (categorization) loss. 4
2
c. However, assume that the priors P (
i
) are unknown. Find two decision points
x

1
and x

2
so that the error rate is independent of the priors for such triangular
distributions. Why are these decision points called the minimax decision points?
(Consult section 2.3.1 in the textbook if you dont know the answer.) 4
d. Let {
i
,
i
} = {0, 1} , {0.5, 0.5} , and {1, 1}. Find the minimax decision rule (i.e.,
x

1
and x

2
) for this case. What is the minimax risk R? 3
________________________________________________
a. Assumption
1
<
2
<
3
: excludes the special case that two or all of the three
means are equal. No further loss of generality, only simplies notation.
Assumption
i
,
j

i+1

i

i
+
i+1
for i = 1, 2: Ensures that there are exactly
two decision points and the probabilities density functions do not spread beyond
center of neighboring probability density functions.
b. x

i
=
P(
i
)
2
i+1
(
i
+
i
)+P(
i+1
)
2
i
(
i+1

i+1
)
P(
i
)
2
i+i
+P(
i+1
)
2
i
for i = 1, 2
c.
R = a

R
1
+R
2
P (
3
) p
3
(x) dx +

R
1
+R
3
P (
2
) p
2
(x) dx +

R
2
+R
3
P (
1
)
. .
1P(
2
)P(
3
)
p
1
(x) dx
=

R
2
+R
3
p
1
(x) dx + P (
2
)
_

R
1
+R
3
p
2
(x) dx
. .
1

R
2
p
2
(x) dx

R
2
+R
3
p
1
(x) dx
. .

R
1
p
1
(x) dx1
_
+P (
3
)
_

R
1
+R
2
p
3
(x) dx
. .
1

R
3
p
3
(x) dx

R
2
+R
3
p
1
(x) dx
. .

R
1
p
1
(x) dx1
_
=

R
2
+R
3
p
1
(x) dx + P (
2
)
_

R
1
p
1
(x) dx

R
2
p
2
(x) dx
_
. .
=0 for minimax
+P (
3
)
_

R
1
p
1
(x) dx

R
3
p
3
(x) dx
_
. .
=0 for minimax
3
Condition for minimax solution:

R
1
p
1
(x) dx =

R
2
p
2
(x) dx =

R
3
p
3
(x) dx
1

R
1
p
1
(x) dx = 1

R
2
p
2
(x) dx = 1

R
3
p
3
(x) dx
(
1
+
1
x

1
)
2

2
1
=
(x

1
(
2

2
))
2
+ (
2
+
2
x

2
)
2

2
2
=
(x

2
(
3

3
))
2

2
3
Note that you can the rst and the last term imply

1
+
1
x

1
=
x

2
(
3

3
)

3
.
d. The above condition holds even if constraint

i
,
j

i+1

i

i
+
i+1
is replaced by

1

2

2

3

3
,
1
+
1

2
+
2

3
.
For the specic values, we obtain: 1 x

1
= x

2
and 4x
2
1
+ 4
_
1 x
2
2
. .
x

!
_
= x
2
2
,
hence (x

1
, x

2
) =
_
1
1+

8
,

8
1+

8
_
(0.26, 0.74).
R = 1

R
1
p
1
(x) dx =
(
1
+
1
x

1
)
2
2
2
1
=
_
1
1
1+

8
_
2
2
=
4
(1+

8)
2
0.27
Exercise 4 Descriptive statistics / Matlab 11
Exercise has been shifted from Lab Sheet 1. Please familiarize yourself with Matlab in
case it is new for you!!
Download the le pattern1.mat from the course website and load its content into your
MATLAB session using the load command. In your MATLAB workspace, two variables
must appear: patterns and targets. The rst variable contains 580 patterns coded as
2D feature vectors. The second variable is a vector of 580 labels assigning the patterns
to one of two classes, w
0
or w
1
. Write a MATLAB script to accomplish the following
tasks:
a. Display the patterns in a 2D plot with the help of the scatter command. Use
dierent colors for the classes 0 and 1. 2
b. Use the hist command, to visualize the 1D distributions of the x
1
and x
2
com-
ponents and of the class variable w. 2
4
c. Usually, we are interested in the class-conditional distributions, such as x
1
|w
0
and
x
1
|w
1
. Use the output of the hist command to compute the class-conditional
distributions for both components of the feature vector: that is x
1
|w
0
, x
1
|w
1
and
x
2
|w
0
, x
2
|w
1
. Normalize the histograms to range from 0 to 1 and display the 4
class-conditional distributions in a single gure using the subplot command. Is it
possible to use only one of the two components for the purpose of classication? 3
d. Write your own script (i.e., do not use the mean or cov commands) to compute
the class- conditional mean vector the covariance matrix and the correlation co-
ecients for the two classes. Try to utilize matrix notation avoiding explicit for
loops whenever possible. 4
________________________________________________
Total Score 35
5