Pattern Recognition Problem

Introduction to Statistical Pattern Recognition
Pattern Recognition Problem
R.P.W. Duin
Pattern Recognition Group
Delft University of Technology
The Netherlands
prcourse@prtools.org
What is this? What occasion? Where are the faces? Who is who?
18 May 2008
Pattern Recognition Problems
18 May 2008
Pattern Recognition Problems
A
B
C
To which class
belongs an image
To which class (segment)

belongs every pixel?
Which group?
18 May 2008
Pattern Recognition Books
18 May 2008
Where is an object of
interest (detection);
What is it (classification)?
Pattern Recognition: OCR
Fukunaga, K., Introduction to statistical pattern recognition, second edition, Academic Press, 1990.
Kohonen, T., Self-organizing maps, Springer Series in Information Sciences, Volume 30, Berlin, 1995.
Ripley, B.D., Pattern Recognition and Neural Networks, Cambridge University Press, 1996.
Devroye, L., Gyorfi, L., and Lugosi, G., A probabilistic theory of pattern recognition, Springer, 1996.
Schurmann, J. Pattern classification, a unified view of statistical and neural approaches, Wiley, 1996
Kurzweil Reading Edge
Gose, E., Johnsonbaugh, R., and Jost, S., Pattern Recognition and Image Analysis, Prentice-Hall, 1996.
Automatic text reading machine

with speech synthesizer
Vapnik, V.N., Statistical Learning Theory, Wiley, New York, 1998.

Duda, R.O., Hart, P.E., and Stork, D.G. Pattern Classification, 2d Edition Wiley, New York, 2001.
Hastie, T., Tibishirani, R., Friedman, J., The Elements of Statistical Learning, Springer, Berlin, 2001.
Webb, A. Statistical Pattern Recognition Wiley, 2002.
F. van der Heiden, R.P.W. Duin, D. de Ridder, and D.M.J. Tax, Classification, Parameter Estimation,
State Estimation: An Engineering Approach Using MatLab, Wiley, New York, 2004, 1-440.
Bishop, C.M., Pattern Recognition and Machine Learning, Springer 2006
Theodoridis, S. and Koutroumbas, K. Pattern Recognition, 4th ed., Academic Press, New York, 2008.
Statsoft, Electronic Textbook, http://www.statsoft.com/textbook/stathome.html

18 May 2008
18 May 2008
Pattern Recognition: Traffic
Pattern Recognition: Traffic
License Plates
Road sign detection

Road sign recognition
18 May 2008
Pattern Recognition: Images
18 May 2008
Pattern Recognition: Faces
100 Objects
Is he in the database?
18 May 2008
Pattern Recognition: Speech
18 May 2008
Yes!
10
Pattern Recognition: Pathology
MRI Brain Image
18 May 2008
11
18 May 2008
12
Pattern Recognition: Pathology
Pattern Recognition: Seismics
3000
2500
2000
Flow cytometry
Earthquakes
1500
1000
500
18 May 2008
50
100
150
200
250
300
13
Pattern Recognition: Shape Recognition
18 May 2008
14
Pattern Recognition: Shapes
Pattern Recognition is very often Shape Recognition:

Images: B/W, grey value, color, 2D, 3D, 4D
Time Signals
Spectra
Examples of objects for different classes
Object of unknown class to be classified
A ? B
18 May 2008
15
Pattern Recognition System

Sensor
Representation
18 May 2008

Generalization
Sensor
Representation
Generalization
area
pixel_2
area
16
perimeter
Feature Representation
18 May 2008
Pixel Representation
perimeter
17
18 May 2008
pixel_1
18

Sensor

Generalization
Representation
Generalization
Representation
Sensor
D(x,xB1)
Classifier_2
B
A
B
Dissimilarity Representation
D(x,xA1)
19
Applications
size
Set of Examples
Representation
21
Possible Object Representations

Measurement samples
f(t)
t
Feature vector
area
Classification
error
Minimize desired set of examples

Minimize amount of explicit knowledge
Minimize complexity of representation
Minimize cost of recognition
Minimize probability of classification error
18 May 2008
Sensor
Test object
classified as A
Outline samples (shape)
Symbolic structures
A
F
E
Dissimilarities
Statistics needed to
solve class overlap
Representation
Generalization
Feature Space
Classification
x2
A -> A||B
B -> B1B2 B3..Bn
d(y,x2 )
x1
d(y,x1 )
22
perimeter
Sets of segments or primitives
18 May 2008
Representation (Model)
Pattern Recognition
System
cost
(speed)
no strong models available
Physical Knowledge
complexity
Notes: not always, but very often image based
(Attributed) graphs
20
18 May 2008
Requirements and Goals
Biomedical: EEG, ECG, Rntgen, Nuclear, Tomography, Tissues,

Cells, Chromosomes, Bio-informatics
Speech Recognition, Speaker Identification.
Character Recognition
Signature Verification
Remote Sensing,
Meteorology
Industrial Inspection
Robot Vision
Digital Microscopy
18 May 2008
Classifier_1
Combining Classifiers
18 May 2008
23
18 May 2008
24
Bayes decision rule, formal

Bayes:
Decisions based on densities
p(A|x)
>
p(B|x)
A else B
p(x|A) p(A)
p(x)
>
p(x|B) p(B)
p(x)
A else B
> p(x|B) p(B)
A else B
p(x|A) p(A)
2-class problems: S(x) = p(x|A) p(A) - p(x|B) p(B) > 0 A else B
length
What is the gender of somebody with this length?
n-class problems: Class(x) = argmax(p(x|) p())
Bayes:
25
18 May 2008
p(length | male)
p(length | female)
Decisions based on densities
18 May 2008
p(female | length) = p(length | female) p(female) / p(length)

p(male | length) = p(length | male) p(male) / p(length)
26
Bayes decision rule
p(length | female) p(female) p(length | male) p(male)
p(female | length) > p(male | length) female else male

Bayes:
p(female) = 0.3
p(male) = 0.7
1.0
p(length | female) p(female)

p(length)
>
p(length | male) p(male)

p(length)
p(length | female) p(female) > p(length | male) p(male) female else male
length
What is the gender of somebody with this length?
Bayes:
p(female | length) = p(length | female) p(female) / p(length)
pdf estimated from training set
p(male | length) = p(length | male) p(male) / p(length)

27
18 May 2008
18 May 2008
What are Good Features?
class prior probabilities

known, guessed or estimated
28
Compactness
Representations of real world similar objects are close.
There is no ground for any generalization (induction) on representations
that do not obey this demand.
Good features are discriminative.

Good features are have to be defined by experts
(A.G. Arkedev and E.M. Braverman, Computers and Pattern Recognition, 1966.)
Sometimes to be found by statistics

e.g. Fisher Criterion:
A B
A
18 May 2008
The compactness hypothesis is not

sufficient for perfect classification
as dissimilar objects may be close.
class overlap
probabilities
x2
A B
J F( A, B) = -------------------------2
2
A + B
(area)
(perimeter) x1
29
18 May 2008
30
Distances and Densities
Distances: Scaling Problem

A
A
x2
? to be classified as
B because it is most
close to an object A
area
(area)
area
x1
perimeter
Before scaling: D(X,A) < D(X,B)
31
18 May 2008
How to Scale in Case of no Natural Features
After scaling: D(X,A) > D(X,B)
32
Density Estimation:
A
x2
(area)
color'
color
perimeter
(perimeter)
18 May 2008
A because the local

density of B is larger.
B
?
?
perimeter
perimeter
(perimeter)
color
var(color)
perimeter
perimeter'=
var(perimeter)
color'=
Make variances equal:
18 May 2008
x1
What is the probability of finding an object

of class A (B) on this place in the 2D space?
33
The Gaussian distribution (3)
18 May 2008
(perimeter)
x1
What is the probability of finding an object

of class A (B) on this place in the 1D space?
34
Multivariate Gaussians
0.5
Normal distribution =
Gaussian distribution
0.4
0.2
95% of data between

[ - 2, + 2 ] (in 1D!)
0.1
0 1-dimensional density:
-5
18 May 2008
Standard normal
distribution:
= 0, 2 = 1
0.3
1 ( x )2
p( x) =
exp
2
2
2
2
G=
3 1
1 2
k - dimensional density:
p( x) =
35
18 May 2008
1
2 k det(G )
1
exp ( x ) T G 1 ( x )
36
Multivariate Gaussians (2)
Density estimation (1)

G= 3 0
0 1
G= 1 0
0 1
The density is defined on the whole feature space.

Around object x, the density is defined as:
p( x) =
G=
G= 1 0
0 3
37
18 May 2008
3 -1
-1 1
Density estimation (2)
dP ( x) fraction of objects
=
volume
dx
Given n measured objects, e.g. persons height (m)

how can we estimate p(x)?
38
18 May 2008
Parametric estimation
Assume Gaussian model
Estimate mean and covariance from data
Parametric estimation:
Assume a parameterized model, e.g. Gaussian
Estimate parameters from data
Resulting density is of the assumed form
1
n
G=
i =1
1
n 1
( x )
i =1
( xi )
1.5
Non parametric estimation:

Assume no formal structure/model, choose approach
Estimate density with chosen approach
Resulting density has no formal form
0.5
-0.5
-1
39
18 May 2008
-1.5
-1.5
Nonparametric estimation
Histogram method:
1. Divide feature space in
N2 bins
2. Count the number of
objects in each bin
3. Normalize:
p ( x ) =
nij
18 May 2008
0.5
K (r ) > 0
K (r) dr = 1
For test object z sum all bins
p( z ) =
i , j =1
-0.5
1.5
2.5
Fix volume of bin, vary positions of bins, add contribution of each bin
Define bin-shape (kernel):
dy
ij
-1
Parzen density estimation (1)
n dxdy
40
18 May 2008
dx
1
z xi
K
hn i h
-1
-2
-3
-2
41
18 May 2008
-1
42
Parzen density estimation (2)

With Gaussian kernel:
K ( x) =
1
h 2
exp
Parametric vs. Nonparametric
( )
2
x
2h2
Parametric estimation, based on some model:

Model parameters to estimate
More samples required than parameters
Model assumption could be incorrect resulting in
erroneous conclusions
Non parametric estimation, hangs on data directly:
Assume no formal structure/model
Almost no parameters to estimate
Erroneous estimates are less likely
Parzen:
p ( x )
x
18 May 2008
43
18 May 2008
44

Pattern Recognition Problem

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Pattern Recognition Problem

Загружено:

Авторское право:

Доступные форматы

Introduction to Statistical Pattern Recognition