Unit - 3

BAYESIAN DECISION THEORY

INTRODUCTION:
In order to make inference from data, which comes from a process
that is not completely known (Partially known world;not explored
fully),we apply probability as the means of predicting the outcome
of the random process.
Eg-Tossing a coin, buying lottery ticket.
Unobservable variables:-
The variables, which are hidden or cannot be
observed, though they predict a processs
outcomes deterministically (if known prior with
precision),are called as unobservable variables

[ FOOD FOR THOUGHT: composition of coin ,force,

directions are UNOBESERVABLE VARIABLES with
which we can deterministically predict the
outcome of the coin toss. ANYWAYS THEY ARE
HINDERED. ]

Observable variables:
The variables (or attributes), which are visible and
can be observed. These represent the knowledge
which we have direct access.
A deterministic outcome is predicted from the unknown variables,
by applying those variables to a function

f(.) is the deterministic outcome from the unobservable piece of

knowledge.
Since we cannot model the process in the above way, we define
the outcome as a random variable drawn from the probabilistic
distribution P(X=x), that specifies the process.
Tossing a coin is BERNOULLI DISTRIBUTION(only two outcomes)
IF X is a random variable, the variable takes one of the two
possibilities(in the case of BERNOULLI), heads or tails, which
assumes the value 1 for head and 0 for tails.

p0 is the probability that the outcome is head.

If P(X) is not known, we collect a sample of the observables .

We try to build a predictor ,from the given set of examples or
SAMPLE X.
In tossing a coin, a sample X is taken which contains the
outcomes of the last N tosses. Then using the sample X, we can
predict the .
If X={1,1,1,0,1,0,0,1,1}
And estimate is

CLASSIFICATION:
It is the method of labelling an object with class label,which
is based on the particular attributes. Labelling a test
instance under a particular category, based on the existing
knowledge about the class forms tha classification.

For example consider a bank, who wants to provide loan to a

customer. In the process of decision making, we would like
to classify the customer under two categories,namely, low
risk customer and high risk customer,
based on which the bank will sanction the loan.
Let us consider the list of observable variables as income
and savings.
These two variables are taken as the basis, upon which the
bank will sanction the loan.

Applying the bernoullis randaom variable C, to indicate the

credibility of a particular customer,which is conditioned on
the variables x=[x1,x2].
C=1 ,high risk customer
C=0, low risk customer.
Prior Probability:-
The value of C that a particular instance takes,before
actually looking at the observables x, satisfying
P(C=0)+P(C=1)=1
Here P(C=1) is the prior probability that C takes the value
1,which indicates the probability of high-risk customer,
regardless of x.

Class likelihood:-

p(x|C) is called the class likelihood and is the conditional

probabilityclass likelihood that an event belonging to C has
the associated observation value x. In our case, p(x1, x2|
C= 1) is the probability that a high-risk customer has his or
her X1 = x1 and X2 = x2

Evidence:-
p(x), the evidence, is the marginal probability that an
observation x is evidence seen, regardless of whether it is
a positive or negative example.

Posterior probability:-
The probability of happening of an event, given the
evidence and prior probability, the posterior
probability is the probability that the event will take
place after the observation.
 Posterior= prior X likelihood
Evidence

In general case, we have K mutually exclusive and exhaustive

classes i.e., Ci , i=1,2,.K.
We have prior probabilities satisfying,

P(x|Ci) is the probability of seeing x as the input,when it is

actually known to belong to class Ci . The posterior probability of
class Ci is calculated as,

And for minimum error ,the bayes classifier chooses the class
with highest posterior probability., i.e.,

Choose Ci if

LOSSES AND RISKS:-

All the decisions are not equally good or costly. The losses and
risks involved in a decision should also be taken in to account
which making decisions and predictions.
Let a i be the decision to assign input to class Ci and ik as the
loss incurred for taking a i when the input actually belongs to Ck.
Then the expected action a i is

And we choose the action with minimum risk.

In some applications, wrong decisionsnamely, misclassifications
may have very high cost, and it is generally required that a more
complex
for example, manualdecision is made if the automatic system has
low
certainty of its decision. For example, if we are using an optical digit
recognizer to read postal codes on envelopes, wrongly recognizing the
code
causes the envelope to be sent to a wrong destination.