Lec 01

Statistics vs.
Probability Probability Triple Construction of Probability Triple Conditional Probability and Independence
STAT 3001/7301: Mathematical Statistics

Week 1 - Lecture 1
Fred Roosta
School of Mathematics and Physics

University of Queensland
fred.roosta@uq.edu.au
Statistics vs. Probability Probability Triple Construction of Probability Triple Conditional Probability and Independence
Info
Instructor (1st Half): Fred Roosta

Email: fred.roosta@uq.edu.au
Office: 67-548
Consultation :
Mondays 3-4pm
Wednesdays 1-2pm
Email for appointment
Or just drop by and knock! ,
Background Quiz! /
We will follow the book closely, but if we progress fast enough,

before I hand over to Alan, we can have additional 1–3 lectures on:
High-Dimensional Statistics, and/or
Theory of Machine Learning, and/or
Bayesian vs. Frequentist.
This would be to your advantage as you will learn something

additional and will also not be examined on them!...So help me out
get there!
Chapter 1: Probability Models

Statistics vs. Probability
What do probability and statistics have in common?
They both deal with random experiments.

A random experiment is an experiment whose outcome cannot
be determined in advance.
Examples of random experiments:

Selecting a ball from an urn containing balls numbered 1 to
50 and note the number of the ball
Tossing a coin three times and note the sequence of heads
and tails
An Airplane’s both engines failing mid-flight
Donald Trump tweeting something silly!

Difference between probability and statistics?


Probabilist (by making assumptions about model) would ask:
50 and note the number of the ball.
Q: What is the probability that the number is less than 10
Tossing a coin 3 times and noting the sequence of H and T
Q: On average, how many tosses before seeing HHT
Q: Out of 1000 sorties of an airline, what is the probability
than 1 such incident happen?

Q: If he types randomly, expected time of the first appearance
of the word COVFEFE!!!


Statistician (by observing the data) would ask

50 and note the number of the ball.
Q: Are 2’s more common than 3’s in the urn?
Tossing a coin 3 times and noting the sequence of H and T
Q: Is the coin fair?
Q: Is it safer to fly with “AeroMaybe” or “US Scareways”?
Q: Was he typing randomly or COVFEFE is the nulear lunch
code?
Probability Triple
Probability Triple (Ω, F, P):
Ω: Sample Space
F: Collection of Events
P: Probability Measure
Probability Triple: (Ω, F, P)
Definition (Ω: Sample Space)

Set of all possible outcomes in some random experiment
Example: With tossing a coin twice, Ω = {HH, TT , HT , TH}
Ω could be
Finite, e.g., tossing a coin twice ⇒ Ω = {HH, TT , HT , TH}
Countably infinite, e.g., the number of the trial at which the
first head appears ⇒ Ω = {1, 2, 3, . . .}
Uncountablly infinite, e.g., choosing a number at random from
the [0, 1] interval ⇒ Ω = [0, 1]
Definition (Event)
Any subset A of Ω, i.e., A ⊆ Ω, including the empty set ∅ and the
entire sample space Ω is called an event.
Example: With tossing a coin twice, Ω = {HH, TT , HT , TH},

so event of having at least a head is {HH, HT , TH} ⊂ Ω
Events
Events are sets, so we can apply the usual set operations to them,
i.e,
A ∪ B, A ∩ B, A ⊂ B, A \ B, and Ac
Theorem (De Morgan’s Laws)

For any collection of sets {Ai }i∈I , where I is some, possibly
uncountable, indexing set, we have
!c
[ \
Ai = Aci
i∈I i∈I
!c
\ [
Ai = Aci
i∈I i∈I
Events
Proof of De Morgan’s Laws.

!c
[ [
a∈ Ai ⇔a∈
/ Ai
i∈I i∈I
⇔a∈
/ Ai , ∀i ∈ I
⇔ a ∈ Aci , ∀i ∈ I
\
⇔a∈ Aci .
i∈I
The other one is proved by taking the complement of both

sides.
Definition (F: Collection of Events, i.e., σ−Field or σ−Algebra)

A collection of subsets of Ω that is closed under complements and
countable unions.
Closed under complements:
A ∈ F ⇒ Ac ∈ F
Closed under countable unions:
A1 , A2 , A3 , . . . ∈ F ⇒ A1 ∪ A2 ∪ A3 . . . ∈ F
If Ω is finite or countably infinite, we can have

F = {all subsets of Ω, including Ω itself}, i.e., power set of
Ω.
For Ω = [0, 1], then F contains all the intervals

[a, b], 0 ≤ a ≤ b ≤ 1, but it also contains many more subsets
(not all of the form interval) of [0,1]. It may exclude a few
subsets as well.
Definition (P: Probability Measure)

Probability measure is any function P : F → [0, 1] for which we
have
P(Ω) = 1
Countable additivity, i.e., given a countable collection of
disjoint events, i.e., Ai ∈ F, i = 1, 2, . . . , and
Ai ∩ Aj = ∅, i 6= j, we have
!
[ X
P Ai = P(Ai ).
i i
Theorem (Properties of P)
1 P(∅) = 0.
2 P(Ac ) = 1 − P(A).
3 Monotonicity: A1 ⊆ A2 =⇒ P(A1 ) ≤ P(A2 ).
4 Principle of Inclusion-Exclusion:
P(A1 ∪ A2 ) = P(A1 ) + P(A2 ) − P(A1 ∩ A2 ).
5 Countable Sub-additivity: Given a countable collection of
events {Ai } ∈ F (whether disjoint or not), we have
!
[ X
P Ai ≤ P(Ai ).
i i
Proof of Countable Sub-additivity.

Items 1–4 are proved in the book. For 5, we have
P(A1 ∪ A2 ∪ A3 ∪ . . .) = P(A1 ∪ (A2 \ A1 ) ∪ (A3 \ A2 ∪ A1 ) ∪ . . .)

= P(A1 ) + P(A2 \ A1 ) + P(A3 \ A2 ∪ A1 ) + . . .
≤ P(A1 ) + P(A2 ) + P(A3 ) + . . . .
Construction of Probability Triple
Definition (Uniform or Equally Likely: Finite Ω)

Let Ω be a finite sample space consisting of N outcomes, i.e.,
Ω = {ω1 , ω2 , . . . , ωN }. We say that the outcomes are equally likely
if P(ω1 ) = P(ω2 ) = . . . = P(ωN ) = 1/N.
For any A ⊆ Ω containing n outcomes, we get P(A) = n/N, i.e., it

boils down to counting!
Example
Q: Suppose there are 5 pairs of shoes in a closet and 4 shoes are
taken out at random. What is the probability that among the 4
that are taken out, there is at least one complete pair?
A:
10

1 There are 10 shoes and we pick 4...so |Ω| = 4 = 210
2 A = {two pairs, only one pair}
# two pairs = 52 = 10

3
8
4 # only one pair = 51 2 − 4
1 = 120
5 P(A) = (10 + 120)/210 ≈ 0.62
Definition (Non-Uniform: Finite Ω)

Let Ω be a finite sample space consisting of N outcomes, i.e.,
Ω = {ω1 , ω2 , . . . , ωN }. Given any sequence {pi }N
i=1 such that
0 ≤p1 , p2 , . . . , pN ≤ 1,
p1 + p2 + . . . + pN = 1,
we can define non-uniform probability on outcome ωi as

P(ωi ) = pi .
For any A ⊆ Ω containing n outcomes, we get

X
P(A) = pi .
i:ωi ∈A
Theorem (More general case of countable (finite or infinite) Ω)

Let Ω be a finite or countable non-empty set. PLet
p(ω) : Ω → [0, 1] be any function satisfying ω∈Ω p(ω) = 1. Then
there is a valid probability triple (Ω, F, P) where F is the collection
of all subsets of ω, i.e., power set of Ω, and for A ∈ F, we have
X
P(A) = p(ω).
ω∈A
Proof.
Verify for yourself that the statement is correct.
However, for uncountable Ω, e.g., Ω = [0, 1], things are a bit more
involved...
Definition (Uniform: Ω = [0,1] )

Let Ω = [0, 1]. For any 0 ≤ a ≤ b ≤ 1, let P([a, b]) = b − a. Then
P is called the uniform probability distribution on [0, 1].
This completely defines the measure P. In particular, for any

A ⊆ Ω, we get
P(A) = Length(A).
Conditional Probability
Conditional Probability and Independence



Definition (Conditional Probability)

The conditional probability of an Event A given B is defined as
P(A ∩ B)
P(A | B) = ,
P(B)
when P(B) 6= 0.
Aside: When P(B) = 0, the above is not well-defined. In such cases, one uses measure-theoretic approach to define
conditional probability.
From conditional probability, we get P(A ∩ B) = P(A)P(B | A).

We can generalize..
Theorem (Product Rule)
Let {Ai }ni=1 ⊂ F be a sequence of events s.t. P(A1 ∩ A2 . . . ∩ An−1 ) > 0.
Then
P(A1 ∩ A2 . . . ∩ An−1 ∩ An )
= P(A1 ) P(A2 | A1 ) P(A3 | A1 ∩ A2 ) . . . P(An | A1 ∩ . . . ∩ An−1 ).
Theorem (Law of Total Probability)

n
Sn A ∈ F be an event {Bi }i=1 ⊂ F be a partition of Ω, i.e.,
Let
i=1 Bi = Ω and Bi ∩ Bj = ∅, i 6= j. Then we have
n
X
P(A) = P(A | Bi )P(Bi ).
i=1
Proof.
Write A = A ∩ Ω = A ∩ ( ni=1 Bi ) = ni=1 (A ∩ Bi ). Now use
S S
countable additivity and product rule.
Combining law of total probability with the definition of

conditional probability gives Bayes’ Rule:
Theorem (Bayes Rule)
Let A ∈ F be an event such n
Sn that P(A) > 0 and let {Bi }i=1 ⊂ F
be a partition of Ω, i.e., i=1 Bi = Ω and Bi ∩ Bj = ∅, i 6= j.
Then we have
P(A | Bj )P(Bj )
P(Bj | A) = Pn .
i=1 P(A | Bi )P(Bi )
Independence
Two events are independent if P(A | B) = P(A) or equivalently

P(B | A) = P(B) or equivalently P(A ∩ B) = P(A)P(B).
More generally:
Definition (Independence)
A (possibly infinite) collection of events {Ai }i∈I ∈ F is said to be
independent, if for any k ∈ N and any distinct choice of indices
i1 , i2 , ik ∈ I, we have
P(Ai1 ∩ Ai1 ∩ . . . ∩ Aik ) = P(Ai1 ) P(Ai2 ) . . . P(Aik ).
In the special case, where this property only holds for k = 2, they
are called pairwise-independent.

Lec 01

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Lec 01

Загружено:

Авторское право:

Доступные форматы

Statistics vs.

STAT 3001/7301: Mathematical Statistics

School of Mathematics and Physics

Instructor (1st Half): Fred Roosta

We will follow the book closely, but if we progress fast enough,

High-Dimensional Statistics, and/or

Theory of Machine Learning, and/or

Bayesian vs. Frequentist.

This would be to your advantage as you will learn something

Chapter 1: Probability Models

Statistics vs. Probability

What do probability and statistics have in common?

They both deal with random experiments.

Statistics vs. Probability

Examples of random experiments:

Donald Trump tweeting something silly!

Statistics vs. Probability

Difference between probability and statistics?

Statistics vs. Probability

Donald Trump tweeting something silly!

of the word COVFEFE!!!

Statistics vs. Probability

Selecting a ball from an urn containing balls numbered 1 to

Donald Trump tweeting something silly!

Q: Was he typing randomly or COVFEFE is the nulear lunch

Probability Triple (Ω, F, P):

Probability Triple: (Ω, F, P)

Definition (Ω: Sample Space)

Example: With tossing a coin twice, Ω = {HH, TT , HT , TH}

Probability Triple: (Ω, F, P)

Example: With tossing a coin twice, Ω = {HH, TT , HT , TH},

Theorem (De Morgan’s Laws)

Proof of De Morgan’s Laws.

The other one is proved by taking the complement of both

Probability Triple: (Ω, F, P)

Definition (F: Collection of Events, i.e., σ−Field or σ−Algebra)

Closed under complements:

Closed under countable unions:

Probability Triple: (Ω, F, P)

If Ω is finite or countably infinite, we can have

For Ω = [0, 1], then F contains all the intervals

Probability Triple: (Ω, F, P)

Definition (P: Probability Measure)

Probability Triple: (Ω, F, P)

Probability Triple: (Ω, F, P)

Proof of Countable Sub-additivity.

P(A1 ∪ A2 ∪ A3 ∪ . . .) = P(A1 ∪ (A2 \ A1 ) ∪ (A3 \ A2 ∪ A1 ) ∪ . . .)

Construction of Probability Triple

Definition (Uniform or Equally Likely: Finite Ω)

For any A ⊆ Ω containing n outcomes, we get P(A) = n/N, i.e., it

Construction of Probability Triple

Construction of Probability Triple

Definition (Non-Uniform: Finite Ω)

we can define non-uniform probability on outcome ωi as

For any A ⊆ Ω containing n outcomes, we get

Construction of Probability Triple

Theorem (More general case of countable (finite or infinite) Ω)

Construction of Probability Triple

Construction of Probability Triple

Definition (Uniform: Ω = [0,1] )

This completely defines the measure P. In particular, for any

Conditional Probability and Independence

Conditional Probability and Independence