Probability Theory Review

CS221: Introduction to Artificial Intelligence Naran Bayanbat 10/14/2011

Slides used material from CME106 course reader and CS229 handouts

Topics
Axioms of Probability Product and chain rules Bayes Theorem Random variables PDFs and CDFs Expected value and variance

Introduction
Sample space - set of all possible outcomes of a random experiment
Dice roll: {1, 2, 3, 4, 5, 6} Coin toss: {Tails, Heads}

Event space sample space

- subsets of elements in a

Dice roll: {1, 2, 3} or {2, 4, 6} Coin toss: {Tails}

Introduction
: Axioms of Probability
0 1, for all () = 1

lim

Set operations
Let = {1, 2, 3} and B = {2, 4, 6}
= 2 and = 1, 2, 3, 4, 6 = 1, 3

Properties:
min , + = 1 () If then ()

Conditional Probability
| probability of A given B

Conditional Probability
| probability of A given B

Conditional Probability
| =

A and B are independent if

| = ()

A and B are conditionally independent given C if

, ) = ) )

Conditional Probability
Joint probability:
1 2 , , ) =
1 ,, 2 ,,

Product rule:
1 , , = 1 2 , , ) 2 , ,

Chain rule of probability:

1 , , =
=1

1 , 2 , 1 )

Conditional Probability
| probability of A given B Example:
() probability that school is closed probability that it snows | probability of school closing if it snows | probability of snowing if school is closed

Conditional Probability
| probability of A given B Example:
() probability that school is closed probability that it snows
P(A, B) 0.005 | =

P(B) P(A|B)

0.02 0.25

Bayes Theorem
We can relate P(A|B) and P(B|A) through Bayes rule:
)() = ()

Bayes Theorem
We can relate P(A|B) and P(B|A) through Bayes rule:
)() = ()

Posterior Probability

Likelihood

Normalizing Constant

Prior Probability

Bayes Theorem
We can relate P(A|B) and P(B|A) through Bayes rule:
)() = ()

can be eliminated with law of total probability:

=

( )

Random Variables
Random variable X is a function s.t.

Do ya feel lucky, punk?

Examples:
Russian roulette: X = 1 if gun fires and X = 0 otherwise

= 1 = and = 0 =
6

5 6

X = # of heads in 10 coin tosses

Cumulative Distribution Functions

Defined as 0, 1 such that = ( ) Properties:
0 1 lim = 0 lim = 1

Probability Density Functions

For discrete random variables, defined as:
= 1 < = ( )

Relates to CDFs:
=
( )

( ) = 1

Probability Density Functions

For continuous random variables, defined as:
< +

Relates to CDFs:
= =

Probability Density Functions

Example:
Let X be the angular position of freely spinning pointer on a circle ()

Probability Density Functions

Example:
Let X be the angular position of freely spinning pointer on a circle f(X) = 2
1

1/2 X 2

Probability Density Functions

Example:
Let X be the angular position of freely spinning pointer on a circle f(X) = 2
2
1

1/2 X

Probability Density Functions

Example:
Let X be the angular position of freely spinning pointer on a circle f(x) = 2 =
1 2 1

1/2

for 0 2
2 F(x) 1

0 /3

x 2

Probability Density Functions

Example:
Let X be the angular position of freely spinning pointer on a circle f(x) = 2 =
1 2 1

1/2

for 0 2
1 6

x 2 F(x) 1 x 2

0 /3 =

Expectation
Given a discrete r.v. X and a function ,
=
()

()

For a continuous r.v. X and a function ,

()

Expectation
Expectation of a r.v. X is known as the first moment, or the mean value, of that variable -> X = Properties:
= for any constant () = [ ] for any constant + () = () + ()

Variance
For a random variable X, define: = Another common form: = 2 Properties:
2

= 0 for any constant () = 2 [ ] for any constant

Gaussian Distributions
Let ~ , 2 , then 1 1 2 2 = 2 2 where represents the mean and 2 the variance Occurs naturally in many phenomena Noise/error Central limit theorem

Questions?