BST 401 Probability Theory

Xing Qiu Ha Youn Lee

Department of Biostatistics and Computational Biology

University of Rochester

September 14, 2009

Qiu, Lee BST 401

Outline

1 Measures

Qiu, Lee BST 401

Countable additivity of a set function
Let µ(·) be an extended real-valued set function defined on a
σ-field F , i.e.,
It takes sets A in F as input variables.
It maps these sets into extended real numbers, e.g., real
numbers plus ∞.
It is said to be
finitely additive if: ForSfinitecollection of disjoint sets
N
A1 , . . . , AN in F , µ
P
n An = n µ(An ).
Countably additive (σ-additive) if: For countably infinite
collection of disjoint sets A1 , . . . , An , . . . in F ,
∞
!
[ X
µ An = µ(An ). (1)
n n

Qiu, Lee BST 401

Countable additivity of a set function
Let µ(·) be an extended real-valued set function defined on a
σ-field F , i.e.,
It takes sets A in F as input variables.
It maps these sets into extended real numbers, e.g., real
numbers plus ∞.
It is said to be
finitely additive if: ForSfinitecollection of disjoint sets
N
A1 , . . . , AN in F , µ
P
n An = n µ(An ).
Countably additive (σ-additive) if: For countably infinite
collection of disjoint sets A1 , . . . , An , . . . in F ,
∞
!
[ X
µ An = µ(An ). (1)
n n

Qiu, Lee BST 401

Countable additivity of a set function
Let µ(·) be an extended real-valued set function defined on a
σ-field F , i.e.,
It takes sets A in F as input variables.
It maps these sets into extended real numbers, e.g., real
numbers plus ∞.
It is said to be
finitely additive if: ForSfinitecollection of disjoint sets
N
A1 , . . . , AN in F , µ
P
n An = n µ(An ).
Countably additive (σ-additive) if: For countably infinite
collection of disjoint sets A1 , . . . , An , . . . in F ,
∞
!
[ X
µ An = µ(An ). (1)
n n

Qiu, Lee BST 401

Countable additivity of a set function
Let µ(·) be an extended real-valued set function defined on a
σ-field F , i.e.,
It takes sets A in F as input variables.
It maps these sets into extended real numbers, e.g., real
numbers plus ∞.
It is said to be
finitely additive if: ForSfinitecollection of disjoint sets
N
A1 , . . . , AN in F , µ
P
n An = n µ(An ).
Countably additive (σ-additive) if: For countably infinite
collection of disjoint sets A1 , . . . , An , . . . in F ,
∞
!
[ X
µ An = µ(An ). (1)
n n

Qiu, Lee BST 401

Basic definition

A measure on a σ-field F is a function µ(A) such that

It is non-negative: µ(A) ≥ 0.
Measure of an empty set is always zero: µ(φ) = 0.
(Remark: The reverse is not true.)
It is countably additive.
Exercise: finite/countable additivity are about commuting union
and addition of disjoint sets. What happens if A1 , A2 , . . . are not
disjoint?

Qiu, Lee BST 401

Basic definition

A measure on a σ-field F is a function µ(A) such that

It is non-negative: µ(A) ≥ 0.
Measure of an empty set is always zero: µ(φ) = 0.
(Remark: The reverse is not true.)
It is countably additive.
Exercise: finite/countable additivity are about commuting union
and addition of disjoint sets. What happens if A1 , A2 , . . . are not
disjoint?

Qiu, Lee BST 401

Basic definition

A measure on a σ-field F is a function µ(A) such that

It is non-negative: µ(A) ≥ 0.
Measure of an empty set is always zero: µ(φ) = 0.
(Remark: The reverse is not true.)
It is countably additive.
Exercise: finite/countable additivity are about commuting union
and addition of disjoint sets. What happens if A1 , A2 , . . . are not
disjoint?

Qiu, Lee BST 401

Finite/Probability Measure

If µ(Ω) is finite, µ is called a finite measure.

Let µ2 (A) = Cµ1 (A). µ1 , µ2 share a lot of mathematical
properties.
As a special case, for every finite measure µ, we can
1
definite µ∗ (A) = µ(Ω) µ(A) so that µ∗ (Ω) = 1.
If µ(Ω) = 1, µ is called a probability measure.
In other words, essentially every finite measure is
equivalent to a probability measure.
By convention, we define a measure/probability space to be a
triple (Ω, F , µ). Where Ω is the whole space, F is the σ-field
with Ω as its whole space, and µ is a measure defined on F .

Qiu, Lee BST 401

Finite/Probability Measure

If µ(Ω) is finite, µ is called a finite measure.

Let µ2 (A) = Cµ1 (A). µ1 , µ2 share a lot of mathematical
properties.
As a special case, for every finite measure µ, we can
1
definite µ∗ (A) = µ(Ω) µ(A) so that µ∗ (Ω) = 1.
If µ(Ω) = 1, µ is called a probability measure.
In other words, essentially every finite measure is
equivalent to a probability measure.
By convention, we define a measure/probability space to be a
triple (Ω, F , µ). Where Ω is the whole space, F is the σ-field
with Ω as its whole space, and µ is a measure defined on F .

Qiu, Lee BST 401

Finite/Probability Measure

If µ(Ω) is finite, µ is called a finite measure.

Let µ2 (A) = Cµ1 (A). µ1 , µ2 share a lot of mathematical
properties.
As a special case, for every finite measure µ, we can
1
definite µ∗ (A) = µ(Ω) µ(A) so that µ∗ (Ω) = 1.
If µ(Ω) = 1, µ is called a probability measure.
In other words, essentially every finite measure is
equivalent to a probability measure.
By convention, we define a measure/probability space to be a
triple (Ω, F , µ). Where Ω is the whole space, F is the σ-field
with Ω as its whole space, and µ is a measure defined on F .

Qiu, Lee BST 401

Finite/Probability Measure

If µ(Ω) is finite, µ is called a finite measure.

Let µ2 (A) = Cµ1 (A). µ1 , µ2 share a lot of mathematical
properties.
As a special case, for every finite measure µ, we can
1
definite µ∗ (A) = µ(Ω) µ(A) so that µ∗ (Ω) = 1.
If µ(Ω) = 1, µ is called a probability measure.
In other words, essentially every finite measure is
equivalent to a probability measure.
By convention, we define a measure/probability space to be a
triple (Ω, F , µ). Where Ω is the whole space, F is the σ-field
with Ω as its whole space, and µ is a measure defined on F .

Qiu, Lee BST 401

Finite/Probability Measure

If µ(Ω) is finite, µ is called a finite measure.

Let µ2 (A) = Cµ1 (A). µ1 , µ2 share a lot of mathematical
properties.
As a special case, for every finite measure µ, we can
1
definite µ∗ (A) = µ(Ω) µ(A) so that µ∗ (Ω) = 1.
If µ(Ω) = 1, µ is called a probability measure.
In other words, essentially every finite measure is
equivalent to a probability measure.
By convention, we define a measure/probability space to be a
triple (Ω, F , µ). Where Ω is the whole space, F is the σ-field
with Ω as its whole space, and µ is a measure defined on F .

Qiu, Lee BST 401

About Ω

The Ω term of a probability space is a set of all outcomes,

and are not necessarily numbers.
For example, we sometimes denote the set of two possible
outcomes of a Bernoulli distribution as Ω = {H, T },
represents Head and Tail.
In fact you can have Ω defined as the set of humans such
as“all students who are taking BST401”, a σ-algebra
containing all subsets of you, and define a probability to
quantify the probability that one of you guys in such a
subset will one day win the Nobel prize.

Qiu, Lee BST 401

About Ω

The Ω term of a probability space is a set of all outcomes,

and are not necessarily numbers.
For example, we sometimes denote the set of two possible
outcomes of a Bernoulli distribution as Ω = {H, T },
represents Head and Tail.
In fact you can have Ω defined as the set of humans such
as“all students who are taking BST401”, a σ-algebra
containing all subsets of you, and define a probability to
quantify the probability that one of you guys in such a
subset will one day win the Nobel prize.

Qiu, Lee BST 401

About Ω

The Ω term of a probability space is a set of all outcomes,

and are not necessarily numbers.
For example, we sometimes denote the set of two possible
outcomes of a Bernoulli distribution as Ω = {H, T },
represents Head and Tail.
In fact you can have Ω defined as the set of humans such
as“all students who are taking BST401”, a σ-algebra
containing all subsets of you, and define a probability to
quantify the probability that one of you guys in such a
subset will one day win the Nobel prize.

Qiu, Lee BST 401

Examples

General definition of a counting measure: for any Ω, µ is

well defined on the largest σ-field 2Ω .
Counting measures on a finite set, Z, and Q.
Formally prove that every counting measure is a valid
measure.
Discrete probabilities: K unique outcomes:
, . . . , ωK }, p1 , . . . , pK , Kk=1 pk = 1,
P
Ω = {ω1P
µ(A) = ωk ∈A pk Here K can be countably infinite.
Bernoulli distribution. (fair coin/non-fair coin).
Binomial
distribution. Ω = {0, 1, . . . , N}, F = 2Ω
pk = Nk pk (1 − p)N−k .
+
Poisson distribution. Ω = Z+ = {0, 1, 2, . . .}, F = 2Z ,
k
pk = e−λ λk ! .

Qiu, Lee BST 401

Examples

General definition of a counting measure: for any Ω, µ is

well defined on the largest σ-field 2Ω .
Counting measures on a finite set, Z, and Q.
Formally prove that every counting measure is a valid
measure.
Discrete probabilities: K unique outcomes:
, . . . , ωK }, p1 , . . . , pK , Kk=1 pk = 1,
P
Ω = {ω1P
µ(A) = ωk ∈A pk Here K can be countably infinite.
Bernoulli distribution. (fair coin/non-fair coin).
Binomial
distribution. Ω = {0, 1, . . . , N}, F = 2Ω
pk = Nk pk (1 − p)N−k .
+
Poisson distribution. Ω = Z+ = {0, 1, 2, . . .}, F = 2Z ,
k
pk = e−λ λk ! .

Qiu, Lee BST 401

Examples

General definition of a counting measure: for any Ω, µ is

well defined on the largest σ-field 2Ω .
Counting measures on a finite set, Z, and Q.
Formally prove that every counting measure is a valid
measure.
Discrete probabilities: K unique outcomes:
, . . . , ωK }, p1 , . . . , pK , Kk=1 pk = 1,
P
Ω = {ω1P
µ(A) = ωk ∈A pk Here K can be countably infinite.
Bernoulli distribution. (fair coin/non-fair coin).
Binomial
distribution. Ω = {0, 1, . . . , N}, F = 2Ω
pk = Nk pk (1 − p)N−k .
+
Poisson distribution. Ω = Z+ = {0, 1, 2, . . .}, F = 2Z ,
k
pk = e−λ λk ! .

Qiu, Lee BST 401

Examples

General definition of a counting measure: for any Ω, µ is

well defined on the largest σ-field 2Ω .
Counting measures on a finite set, Z, and Q.
Formally prove that every counting measure is a valid
measure.
Discrete probabilities: K unique outcomes:
, . . . , ωK }, p1 , . . . , pK , Kk=1 pk = 1,
P
Ω = {ω1P
µ(A) = ωk ∈A pk Here K can be countably infinite.
Bernoulli distribution. (fair coin/non-fair coin).
Binomial
distribution. Ω = {0, 1, . . . , N}, F = 2Ω
pk = Nk pk (1 − p)N−k .
+
Poisson distribution. Ω = Z+ = {0, 1, 2, . . .}, F = 2Z ,
k
pk = e−λ λk ! .

Qiu, Lee BST 401

Examples

General definition of a counting measure: for any Ω, µ is

well defined on the largest σ-field 2Ω .
Counting measures on a finite set, Z, and Q.
Formally prove that every counting measure is a valid
measure.
Discrete probabilities: K unique outcomes:
, . . . , ωK }, p1 , . . . , pK , Kk=1 pk = 1,
P
Ω = {ω1P
µ(A) = ωk ∈A pk Here K can be countably infinite.
Bernoulli distribution. (fair coin/non-fair coin).
Binomial
distribution. Ω = {0, 1, . . . , N}, F = 2Ω
pk = Nk pk (1 − p)N−k .
+
Poisson distribution. Ω = Z+ = {0, 1, 2, . . .}, F = 2Z ,
k
pk = e−λ λk ! .

Qiu, Lee BST 401

Examples

General definition of a counting measure: for any Ω, µ is

well defined on the largest σ-field 2Ω .
Counting measures on a finite set, Z, and Q.
Formally prove that every counting measure is a valid
measure.
Discrete probabilities: K unique outcomes:
, . . . , ωK }, p1 , . . . , pK , Kk=1 pk = 1,
P
Ω = {ω1P
µ(A) = ωk ∈A pk Here K can be countably infinite.
Bernoulli distribution. (fair coin/non-fair coin).
Binomial
distribution. Ω = {0, 1, . . . , N}, F = 2Ω
pk = Nk pk (1 − p)N−k .
+
Poisson distribution. Ω = Z+ = {0, 1, 2, . . .}, F = 2Z ,
k
pk = e−λ λk ! .

Qiu, Lee BST 401

Examples

General definition of a counting measure: for any Ω, µ is

well defined on the largest σ-field 2Ω .
Counting measures on a finite set, Z, and Q.
Formally prove that every counting measure is a valid
measure.
Discrete probabilities: K unique outcomes:
, . . . , ωK }, p1 , . . . , pK , Kk=1 pk = 1,
P
Ω = {ω1P
µ(A) = ωk ∈A pk Here K can be countably infinite.
Bernoulli distribution. (fair coin/non-fair coin).
Binomial
distribution. Ω = {0, 1, . . . , N}, F = 2Ω
pk = Nk pk (1 − p)N−k .
+
Poisson distribution. Ω = Z+ = {0, 1, 2, . . .}, F = 2Z ,
k
pk = e−λ λk ! .

Qiu, Lee BST 401

Basic Properties of a Measure

Monotonicity. If A ⊆ B, then µ(A) 6 µ(B).

P
Subadditivity. If A ⊆ ∪i A
Pi , then µ(A) 6 i µ(Ai ). As a
special case, µ(∪i Ai ) 6 i µ(Ai ).
Continuity from below. Ai ↑ A =⇒ µ(Ai ) ↑ µ(A).
Continuity from above. Ai ↓ A =⇒ µ(Ai ) ↓ µ(A).

Qiu, Lee BST 401

Basic Properties of a Measure

Monotonicity. If A ⊆ B, then µ(A) 6 µ(B).

P
Subadditivity. If A ⊆ ∪i A
Pi , then µ(A) 6 i µ(Ai ). As a
special case, µ(∪i Ai ) 6 i µ(Ai ).
Continuity from below. Ai ↑ A =⇒ µ(Ai ) ↑ µ(A).
Continuity from above. Ai ↓ A =⇒ µ(Ai ) ↓ µ(A).

Qiu, Lee BST 401

Basic Properties of a Measure

Monotonicity. If A ⊆ B, then µ(A) 6 µ(B).

P
Subadditivity. If A ⊆ ∪i A
Pi , then µ(A) 6 i µ(Ai ). As a
special case, µ(∪i Ai ) 6 i µ(Ai ).
Continuity from below. Ai ↑ A =⇒ µ(Ai ) ↑ µ(A).
Continuity from above. Ai ↓ A =⇒ µ(Ai ) ↓ µ(A).

Qiu, Lee BST 401

Basic Properties of a Measure

Monotonicity. If A ⊆ B, then µ(A) 6 µ(B).

P
Subadditivity. If A ⊆ ∪i A
Pi , then µ(A) 6 i µ(Ai ). As a
special case, µ(∪i Ai ) 6 i µ(Ai ).
Continuity from below. Ai ↑ A =⇒ µ(Ai ) ↑ µ(A).
Continuity from above. Ai ↓ A =⇒ µ(Ai ) ↓ µ(A).

Qiu, Lee BST 401

The Borel σ-fields

Borel measure on R: Ω = R, B, µ(A) = the length of A.

Where the Borel σ-field is defined to be the smallest σ-field
containing all open intervals (a, b). (or all closed intervals
[a, b], or all intervals of form (a, b].)
First step: open sets. More details will be discussed later.
Terminology: the smallest σ-field containing a collection of
sets S is called a) the minimal σ-field over S ; b) the
σ-field generated by S .
B is a σ-field generated by the collection of open/closed
intervals/sets.
Not every subset of R is a Borel set! Checkout the Vitali set
from Wikipedia.

Qiu, Lee BST 401

The Borel σ-fields

Borel measure on R: Ω = R, B, µ(A) = the length of A.

Where the Borel σ-field is defined to be the smallest σ-field
containing all open intervals (a, b). (or all closed intervals
[a, b], or all intervals of form (a, b].)
First step: open sets. More details will be discussed later.
Terminology: the smallest σ-field containing a collection of
sets S is called a) the minimal σ-field over S ; b) the
σ-field generated by S .
B is a σ-field generated by the collection of open/closed
intervals/sets.
Not every subset of R is a Borel set! Checkout the Vitali set
from Wikipedia.

Qiu, Lee BST 401

The Borel σ-fields

Borel measure on R: Ω = R, B, µ(A) = the length of A.

Where the Borel σ-field is defined to be the smallest σ-field
containing all open intervals (a, b). (or all closed intervals
[a, b], or all intervals of form (a, b].)
First step: open sets. More details will be discussed later.
Terminology: the smallest σ-field containing a collection of
sets S is called a) the minimal σ-field over S ; b) the
σ-field generated by S .
B is a σ-field generated by the collection of open/closed
intervals/sets.
Not every subset of R is a Borel set! Checkout the Vitali set
from Wikipedia.

Qiu, Lee BST 401

The Borel σ-fields

Borel measure on R: Ω = R, B, µ(A) = the length of A.

Where the Borel σ-field is defined to be the smallest σ-field
containing all open intervals (a, b). (or all closed intervals
[a, b], or all intervals of form (a, b].)
First step: open sets. More details will be discussed later.
Terminology: the smallest σ-field containing a collection of
sets S is called a) the minimal σ-field over S ; b) the
σ-field generated by S .
B is a σ-field generated by the collection of open/closed
intervals/sets.
Not every subset of R is a Borel set! Checkout the Vitali set
from Wikipedia.

Qiu, Lee BST 401

The Borel σ-fields

Borel measure on R: Ω = R, B, µ(A) = the length of A.

Where the Borel σ-field is defined to be the smallest σ-field
containing all open intervals (a, b). (or all closed intervals
[a, b], or all intervals of form (a, b].)
First step: open sets. More details will be discussed later.
Terminology: the smallest σ-field containing a collection of
sets S is called a) the minimal σ-field over S ; b) the
σ-field generated by S .
B is a σ-field generated by the collection of open/closed
intervals/sets.
Not every subset of R is a Borel set! Checkout the Vitali set
from Wikipedia.

Qiu, Lee BST 401

The Borel σ-fields

Borel measure on R: Ω = R, B, µ(A) = the length of A.

Where the Borel σ-field is defined to be the smallest σ-field
containing all open intervals (a, b). (or all closed intervals
[a, b], or all intervals of form (a, b].)
First step: open sets. More details will be discussed later.
Terminology: the smallest σ-field containing a collection of
sets S is called a) the minimal σ-field over S ; b) the
σ-field generated by S .
B is a σ-field generated by the collection of open/closed
intervals/sets.
Not every subset of R is a Borel set! Checkout the Vitali set
from Wikipedia.

Qiu, Lee BST 401