Exercises

Exercises
E7.1 Consider the prototype patterns given to the left.

p1 p2 i. Are p 1 and p 2 orthogonal?

ii. Use the Hebb rule to design an autoassociator network for these
patterns.
iii. Test the operation of the network using the test input pattern p t
shown to the left. Does the network perform as you expected? Ex-
pt plain.

E7.2 Repeat Exercise E7.1 using the pseudoinverse rule.

E7.3 Use the Hebb rule to determine the weight matrix for a perceptron network
(shown in Figure E7.1) to recognize the patterns shown to the left.
p1 p2
Inputs Sym. Hard Limit Layer

p n a
6x1
W 1x1 1x1
1x6

6 1
7
a = hardlims (Wp)

Figure E7.1 Perceptron Network for Exercise E7.3

E7.4 In Problem P7.7 we demonstrated how networks can be trained using the
Hebb rule when the prototype vectors are given in binary (as opposed to bi-
polar) form. Repeat Exercise E7.1 using the binary representation for the
prototype vectors. Show that the response of this binary network is equiv-
alent to the response of the original bipolar network.

E7.5 Show that an autoassociator network will continue to perform if we zero

the diagonal elements of a weight matrix that has been determined by the
Hebb rule. In other words, suppose that the weight matrix is determined
from:
T
W = PP – QI ,

7-31
 7 Supervised Hebbian Learning

where Q is the number of prototype vectors. (Hint: show that the prototype
vectors continue to be eigenvectors of the new weight matrix.)

E7.6 We have three input/output prototype vector pairs:

 1  t = 1   p = 1  t = – 1   p = 0  t = 1  .
 p1 = 1   2 2   3 3 
 0   1   1 

i. Show that this problem cannot be solved unless the network uses a
bias.
ii. Use the pseudoinverse rule to design a network for these prototype
vectors. Verify that the network correctly transforms the prototype
vectors.

E7.7 Consider the reference patterns and targets given below. We want to use
these data to train a linear associator network.

 2 t =   4 t =   –2  t = 
 p1 = 1 26   p2 = 2 26   p3 = 3 – 26 
 4   2   –2 

i. Use the Hebb rule to find the weights of the network.

ii. Find and sketch the decision boundary for the network with the
Hebb rule weights.
iii. Use the pseudo-inverse rule to find the weights of the network. Be-
cause the number, R, of rows of P is less than the number of col-
umns, Q, of P , the pseudoinverse can be computed by
+ T T –1
P = P  PP  .

iv. Find and sketch the decision boundary for the network with the
pseudo-inverse rule weights.
v. Compare (discuss) the decision boundaries and weights for each of
the methods (Hebb and pseudo-inverse).

E7.8 Consider the three prototype patterns shown in Figure E7.2.

i. Are these patterns orthogonal? Demonstrate.
ii. Use the Hebb rule to determine the weight matrix for a linear au-
toassociator to recognize these patterns.
iii. Draw the network diagram.

7-32
 Exercises

iv. Find the eigenvalues and eigenvectors of the weight matrix. (Do not
solve the equation W – I = 0 . Use an analysis of the Hebb
rule.)

-1

p1 p2 p3 1

Figure E7.2 Prototype Patterns for Exercise E7.8

E7.9 Suppose that we have the following three reference patterns and their tar-
gets.

 3 t =   6 t =   –6  t = 
 p1 = 1 75   p2 = 2 75   p3 = 3 – 75 
 6   3   3 

i. Draw the network diagram for a linear associator network that

could be trained on these patterns.
ii. Use the Hebb rule to find the weights of the network.
iii. Find and sketch the decision boundary for the network with the
Hebb rule weights. Does the boundary separate the patterns? Dem-
onstrate.
iv. Use the pseudo-inverse rule to find the weights of the network. De-
scribe the difference between this boundary and the Hebb rule
boundary.
7
E7.10 We have the following input/output pairs:

 1 t =   1 t = 
 p1 = 1 1   p2 = 2 –1 
 1   –1 

i. Use the Hebb rule to determine the weight matrix for the percep-
tron network shown in Figure E7.3.
ii. Plot the resulting decision boundary. Is this a “good” decision
boundary? Explain.
iii. Repeat part i. using the Pseudoinverse rule.
iv. Will there be any difference in the operation of the network if the
Pseudoinverse weight matrix is used? Explain.

7-33
 7 Supervised Hebbian Learning

Inputs Sym. Hard Limit Layer

a
1x1
p n
2x1
W 1x1
1x2

2 1

a = hardlims(Wp)

Figure E7.3 Network for Exercise E7.10

»2+2
ans = E7.11 One question we might ask about the Hebb and pseudoinverse rules is:
4 How many prototype patterns can be stored in one weight matrix? Test this
experimentally using the digit recognition problem that was discussed on
page 7-10. Begin with the digits “0” and “1”. Add one digit at a time up to
“6”, and test how often the correct digit is reconstructed after randomly
changing 2, 4 and 6 pixels.
i. First use the Hebb rule to create the weight matrix for the digits “0”
and “1”. Then randomly change 2 pixels of each digit and apply the
noisy digits to the network. Repeat this process 10 times, and record
the percentage of times in which the correct pattern (without noise)
is produced at the output of the network. Repeat as 4 and 6 pixels of
each digit are modified. The entire process is then repeated when
the digits “0”, “1” and “2” are used. This continues, one digit at a
time, until you test the network when all of the digits “0” through
“6” are used. When you have completed all of the tests, you will be
able to plot three curves showing percentage error versus number of
digits stored, one curve each for 2, 4 and 6 pixel errors.
ii. Repeat part (i) using the pseudoinverse rule, and compare the re-
sults of the two rules.

7-34
 10 Widrow-Hoff Learning

Exercises
E10.1 An adaptive filter ADALINE is shown in Figure E10.1. Suppose that the
weights of the network are given by

w 1 1 = 1 , w 1 2 = – 4 , w 1 3 = 2 ,

and the input to the filter is

 y  k   =    0 0 0 1 1 2 0 0   .

Find the response  a  k   of the filter.

Inputs ADALINE

y(k)
w1,1
D w1,2 n(k) a(k)
Σ
SxR

D w1,3

a(k) = purelin (Wp(k) + b)

Figure E10.1 Adaptive Filter ADALINE for Exercise E10.1

E10.2 In Figure E10.2 two classes of patterns are given.

i. Use the LMS algorithm to train an ADALINE network to distin-
guish between class I and class II patterns (we want the network to
identify horizontal and vertical lines).
ii. Can you explain why the ADALINE network might have difficulty
with this problem?

Class I Class II

Figure E10.2 Pattern Classification Problem for Exercise E10.2

10-42
 Exercises

E10.3 Suppose that we have the following two reference patterns and their tar-
gets:
 1  t = 1  ,  p = 1  t = –1  .
 p1 = 1   2 2 
 1   –1 

In Problem P10.3 these input vectors to an ADALINE were assumed to oc-

cur with equal probability. Now suppose that the probability of vector p 1
is 0.75 and that the probability of vector p 2 is 0.25. Does this change of
probabilities change the mean square error surface? If yes, what does the
surface look like now? What is the maximum stable learning rate?

E10.4 In this exercise we will modify the reference pattern p 2 from Problem
P10.3:
 1  t = 1  ,  p = –1  t = –1  .
 p1 = 1   2 2 
 1   –1 

i. Assume that the patterns occur with equal probability. Find the
mean square error and sketch the contour plot.
ii. Find the maximum stable learning rate.
»2+2
ans = iii. Write a MATLAB M-file to implement the LMS algorithm for this
4 problem. Take 40 steps of the algorithm for a stable learning rate.
Use the zero vector as the initial guess. Sketch the trajectory on the
contour plot.
iv. Take 40 steps of the algorithm after setting the initial values of both
parameters to 1. Sketch the final decision boundary.
v. Compare the final parameters from parts (iii) and (iv). Explain your
results.

E10.5 We again use the reference patterns and targets from Problem P10.3, and
assume that they occur with equal probability. This time we want to train
an ADALINE network with a bias. We now have three parameters to find:
w 1 1 , w 1 2 and b .
i. Find the mean square error and the maximum stable learning rate.
»2+2
ans = ii. Write a MATLAB M-file to implement the LMS algorithm for this
4 problem. Take 40 steps of the algorithm for a stable learning rate.
Use the zero vector as the initial guess. Sketch the final decision
boundary.
iii. Take 40 steps of the algorithm after setting the initial values of all 10
parameters to 1. Sketch the final decision boundary.
iv. Compare the final parameters and the decision boundaries from
parts (iii) and (iv). Explain your results.

10-43
 10 Widrow-Hoff Learning

E10.6 We have two categories of vectors. Category I consists of

 1 –1 
  .
 1 2 

Category II consists of

 0 
  –4  .
 –1 1 

We want to train a single-neuron ADALINE network without a bias to rec-

ognize these categories (t = 1 for Category I and t = -1 for Category II). As-
sume that each pattern occurs with equal probability.
i. Draw the network diagram.
ii. Take four steps of the LMS algorithm, using the zero vector as the
initial guess. (one pass through the four vectors above - present each
vector once). Use a learning rate of 0.1.
iii. What are the optimal weights?
iv. Sketch the optimal decision boundary.
v. How do you think the boundary would change if the network were
allowed to have a bias? If the boundary would change, indicate the
approximate new position on your sketch of part iv. You do not need
to perform any calculations here - just explain your reasoning.

E10.7 Suppose that we have the following three reference patterns and their tar-
gets:

 3 t =   6 t =   –6  t = 
 p1 = 1 75  ,  p2 = 2 75  ,  p3 = 3 – 75  .
 6   3   3 

Each pattern is equally likely.

i. Draw the network diagram for an ADALINE network with no bias
that could be trained on these patterns.
ii. We want to train the ADALINE network with no bias using these
patterns. Sketch the contour plot of the mean square error perfor-
mance index.
iii. Find the maximum stable learning rate for the LMS algorithm.

10-44
 Exercises

iv. Sketch the trajectory of the LMS algorithm on your contour plot. As-
sume a very small learning rate, and start with all weights equal to
zero. This does not require any calculations.

E10.8 Suppose that we have the following two reference patterns and their tar-
gets:

 1 t =   –2  t = 
 p1 = 1 –1  ,  p2 = 2 1 .
 2   1 

The probability of vector p 1 is 0.5 and the probability of vector p 2 is 0.5.We

want to train an ADALINE network without a bias on this data set.
i. Sketch the contour plot of the mean square error performance in-
dex.
ii. Sketch the optimal decision boundary.
iii. Find the maximum stable learning rate.
iv. Sketch the trajectory of the LMS algorithm on your contour plot. As-
sume a very small learning rate, and start with initial weights
W0 = 0 1 .

E10.9 We have the following input/target pairs:

 4  t = 5  ,  p = 2  t = –2  ,  p = –4  t = 9  .
 p1 = 1   2 2   3 3 
 2   –4   4 

The first two pair each occurs with probability of 0.25, and the third pair
occurs with probability 0.5. We want to train a single-neuron ADALINE
network without a bias to perform the desired mapping.
i. Draw the network diagram.
ii. What is the maximum stable learning rate?
iii. Perform one iteration of the LMS algorithm. Apply the input p 1 and
use a learning
T
rate of  = 0.1 . Start from the initial weights
x0 = 0 0 .
10

10-45
 10 Widrow-Hoff Learning

E10.10 Repeat E10.9 for the following input/target pairs:

 2  t = 1  ,  p = –4  t = –1  ,  p = 4  t = 1  .
 p1 = 1   2 2   3 3 
 –4   4   2 

The first two pair each occurs with probability of 0.25, and the third pair
occurs with probability 0.5. We want to train a single-neuron ADALINE
network without a bias to perform the desired mapping.

E10.11 We want to train a single-neuron ADALINE network without a bias, using

the following training set, which categorizes vectors into two classes. Each
pattern occurs with equal probability.

 –1  t = –1   p = 2  t = –1   p = 0  t = 1   p = –1  t = 1 
 p1 = 1  2 2  3 3  4 4 
 2  –1  –1  0 

i. Draw the network diagram.

ii. Take one step of the LMS algorithm (present p 1 only) starting from
the initial weight W  0  = 0 0 . Use a learning rate of 0.1.

iii. What are the optimal weights? Show all calculations.

iv. Sketch the optimal decision boundary.
v. How do you think the boundary would change if the network were
allowed to have a bias? Indicate the approximate new position on
your sketch of part iv.
vi. What is the maximum stable learning rate for the LMS algorithm?
vii. Sketch the contour plot of the mean square error performance sur-
face.
viii. On your contour plot of part vii, sketch the path of the LMS algo-
rithm for a very small learning rate (e.g., 0.001) starting from the
initial condition W  0  = 2 0 . This does not require any calcula-
tions, but explain how you obtained your answer.

E10.12 Suppose that we have the following three reference patterns and their tar-
gets:

 2 t =   4 t =   –2  t = 
 p1 = 1 26  ,  p 2 = 2 26  ,  p 3 = 3 – 26  .
 4   2   –2 

10-46
 Exercises

The probability of vector p 1 is 0.25, the probability of vector p 2 is 0.25 and

the probability of vector p 3 is 0.5.
i. Draw the network diagram for an ADALINE network with no bias
that could be trained on these patterns.
ii. Sketch the contour plot of the mean square error performance in-
dex.
iii. Show the optimal decision boundary (for the weights that minimize
mean square error) and verify that it separates the patterns into the
appropriate categories.
iv. Find the maximum stable learning rate for the LMS algorithm. If
the target values are changed from 26 and -26 to 2 and -2, how
would this change the maximum stable learning rate?
v. Perform one iteration of the LMS algorithm, starting with all
weights equal to zero, and presenting input vector p 1 . Use a learn-
ing rate of  = 0.5 .
vi. Sketch the trajectory of the LMS algorithm on your contour plot. As-
sume a very small learning rate, and start with all weights equal to
zero.

E10.13 Consider the adaptive predictor in Figure E10.3.

Inputs ADALINE

t(k) = y(k)
y(k) D w1,1
n(k) a(k)
+

D w1,2 Σ
SxR

- e(k)

a(k) = w1,1 y(k - 1) + w1,2 y(k - 2)

Figure E10.3 Adaptive Predictor for Exercise E10.13

Assume that y  k  is a stationary process with autocorrelation function

Cy  n  = E  y  k   y  k + n    .
10
i. Write an expression for the mean square error in terms of C y  n  .
ii. Give a specific expression for the mean square error when

10-47
 10 Widrow-Hoff Learning

y  k  = sin  ------ .
k
 5

iii. Find the eigenvalues and eigenvectors of the Hessian matrix for the
mean square error. Locate the minimum point and sketch a rough
contour plot.
iv. Find the maximum stable learning rate for the LMS algorithm.
v. Take three steps of the LMS algorithm by hand, using a stable
learning rate. Use the zero vector as the initial guess.
»2+2
ans = vi. Write a MATLAB M-file to implement the LMS algorithm for this
4 problem. Take 40 steps of the algorithm for a stable learning rate
and sketch the trajectory on the contour plot. Use the zero vector as
the initial guess. Verify that the algorithm is converging to the op-
timal point.
vii. Verify experimentally that the algorithm is unstable for learning
rates greater than that found in part (iv).

E10.14 Repeat Problem P10.9, but use the numerals “1”, “2” and “4”, instead of the
letters “T”, “G” and “F”. Test the trained network on each reference pattern
and on noisy patterns. Discuss the sensitivity of the network. (Use the Neu-
ral Network Design Demonstration Linear Pattern Classification (nnd10lc).)

10-48