LECTURE 9

LECTURE OUTLINE
Min-Max Problems
Saddle Points
Min Common/Max Crossing for Min-Max

Given : X Z  , where X n , Z m
consider
minimize sup (x, z)
zZ

subject to x X
and
maximize inf (x, z)
xX

subject to z Z.
Minimax inequality (holds always)
sup inf (x, z) inf sup (x, z)

zZ xX

xX zZ

SADDLE POINTS
Denition: (x , z ) is called a saddle point of if
(x , z) (x , z ) (x, z ),

x X, z Z

Proposition: (x , z ) is a saddle point if and only

if the minimax equality holds and
x arg min sup (x, z), z arg max inf (x, z) (*)
xX zZ

zZ xX

Proof: If (x , z ) is a saddle point, then

inf sup (x, z) sup (x , z) = (x , z )

xX zZ

zZ

= inf (x, z ) sup inf (x, z)

xX

zZ xX

By the minimax inequality, the above holds as an

equality holds throughout, so the minimax equality
and Eq. (*) hold.
Conversely, if Eq. (*) holds, then
sup inf (x, z) = inf (x, z ) (x , z )

zZ xX

xX

sup (x , z) = inf sup (x, z)

zZ

xX zZ

Using the minimax equ., (x , z ) is a saddle point.

VISUALIZATION

(x,z)
Curve of maxima
^ )
(x,z(x)

Saddle point
(x*,z*)

Curve of minima
^
)
(x(z),z

The curve of maxima (x, z(x)) lies above the

curve of minima (
x(z), z), where
z(x) = arg max (x, z),
z

x
(z) = arg min (x, z).
x

Saddle points correspond to points where these

two curves meet.

MIN COMMON/MAX CROSSING FRAMEWORK

Introduce perturbation function p : m  [, ]

p(u) = inf sup (x, z)

xX zZ

u z

u m

Apply the min common/max crossing framework

with the set M equal to the epigraph of p.
Application of a more general idea: To evaluate a quantity of interest w , introduce a suitable
perturbation u and function p, with p(0) = w .
Note that w = inf sup . We will show that:
Convexity in x implies that M is a convex set.
Concavity in z implies that q = sup inf .
w

w
infx supz (x,z)
= min common value w*

M = epi(p)

M = epi(p)

(,1)

(,1)
infx supz (x,z)
= min common value w*
supz infx (x,z)
0
q()

u
supz infx (x,z)
= max crossing value q*

= max crossing value q*

0
q()

IMPLICATIONS OF CONVEXITY IN X
Lemma 1: Assume that X is convex and that
for each z Z, the function (, z) : X   is
convex. Then p is a convex function.
Proof: Let

F (x, u) =

supzZ (x, z)

u z

if x X,
if x
/ X.

Since (, z) is convex, and taking pointwise supremum preserves convexity, F is convex. Since
p(u) = infn F (x, u),
x

and partial minimization preserves convexity, the

convexity of p follows from the convexity of F .
Q.E.D.

THE MAX CROSSING PROBLEM

The max crossing problem is to maximize q()
over n , where
q() =

inf

(u,w)epi(p)

{w +
u} =

= infm p(u) + u

u

{w +
u}

inf

{(u,w)|p(u)w}

Using p(u) = inf xX supzZ (x, z)

obtain

q() = infm inf sup (x, z) +

u

xX zZ

u (

u z

, we


z)

By setting z = in the right-hand side,

inf (x, ) q(),

xX

Z.

Hence, using also weak duality (q w ),

sup inf (x, z) sup q() = q

zZ xX

m

w = p(0) = inf sup (x, z)

xX zZ

IMPLICATIONS OF CONCAVITY IN Z
Lemma 2: Assume that for each x X, the
function rx : m  (, ] dened by

(x, z) if z Z,
rx (z) =

otherwise,
is closed and convex. Then

inf xX (x, ) if Z,
q() =

if
/ Z.
Proof: (Outline) From the preceding slide,
inf (x, ) q(),

xX

Z.

We show that q() inf xX (x, ) for all

Z and q() = for all
/ Z, by considering
separately the two cases where Z and
/ Z.
First assume that  Z. Fix x X, and for
 > 0, consider the point , rx () , which does
not belong to epi(rx ). Since epi(rx ) does not contain any vertical lines, there exists a nonvertical
strictly separating hyperplane ...

MINIMAX THEOREM I
Assume that:
(1) X and Z are convex.
(2) p(0) = inf xX supzZ (x, z) < .
(3) For each z Z, the function (, z) is convex.
(4) For each x X, the function (x, ) : Z 
 is closed and convex.
Then, the minimax equality holds if and only if the
function p is lower semicontinuous at u = 0.
Proof: The convexity/concavity assumptions guarantee that the minimax equality is equivalent to
q = w in the min common/max crossing framework. Furthermore, w < by assumption, and
the set M [equal to M and epi(p)] is convex.
By the 1st Min Common/Max Crossing The = q iff for every sequence
orem,
we
have
w



(uk , wk ) M with uk 0, there holds w
lim inf k wk . This is equivalent to the lower
semicontinuity assumption on p:
p(0) lim inf p(uk ), for all {uk } with uk 0
k

MINIMAX THEOREM II
Assume that:
(1) X and Z are convex.
(2) p(0) = inf xX supzZ (x, z) > .
(3) For each z Z, the function (, z) is convex.
(4) For each x X, the function (x, ) : Z 
 is closed and convex.
(5) 0 lies in the relative interior of dom(p).
Then, the minimax equality holds and the supremum in supzZ inf xX (x, z) is attained by some
z Z. [Also the set of z where the sup is attained
is compact if 0 is in the interior of dom(f ).]
Proof: Apply the 2nd Min Common/Max Crossing Theorem.

EXAMPLE I


Let X = (x1 , x2 ) | x 0 and Z = {z  |
z 0}, and let

(x, z) = e x1 x2 + zx1 ,
which satisfy the convexity and closedness assumptions. For all z 0,

x x

1 2 + zx
inf e
1 = 0,
x0

so supz0 inf x0 (x, z) = 0. Also, for all x 0,

sup

e x1 x2

+ zx1 =

z0

if x1 = 0,
if x1 > 0,

so inf x0 supz0 (x, z) = 1.

p(u)

x1 x2

p(u) = inf sup e

x0 z0

epi(p)
1

=
u

1
0

if u < 0,
if u = 0,
if u > 0,

+ z(x1 u)

EXAMPLE II
Let X = , Z = {z  | z 0}, and let
(x, z) = x + zx2 ,
which satisfy the convexity and closedness assumptions. For all z 0,

1/(4z) if z > 0,
inf {x + zx2 } =

if z = 0,
x
so supz0 inf x (x, z) = 0. Also, for all x ,

sup {x + zx2 } =
z0

if x = 0,
otherwise,

so inf x supz0 (x, z) = 0. However, the sup is

not attained.
p(u)

p(u) = inf sup{x + zx2 uz}

x z0

epi(p)

=
u

if u 0,
if u < 0.

CONDITIONS FOR ATTAINING THE MIN

Dene

rx (z) =

tz (x) =

(x, z) if z Z,

if z
/ Z,
(x, z) if x X,

if x
/ X,

r(z) = sup rx (z)

xX

t(x) = sup tz (x)

zZ

Assume that:
(1) X and Z are convex, and t is proper, i.e.,
inf sup (x, z) < .

xX zZ

(2) For each x X, rx () is closed and convex, and for each z Z, tz () is closed and
convex.
(3) All the level sets {x | t(x) } are compact.
Then, the minimax equality holds, and the set of
points attaining the inf in inf xX supzZ (x, z) is
nonempty and compact.
Note: Condition (3) can be replaced by more
general directions of recession conditions.

PROOF
Note that p is obtained by the partial minimization
p(u) = infn F (x, u),
x

where

F (x, u) =

supzZ (x, z)

u z

if x X,
if x
/ X.

We have
t(x) = F (x, 0),
so the compactness assumption on the level sets
of t can be translated to the compactness assumption of the partial minimization theorem. It follows
from that theorem that p is closed and proper.
By the Minimax Theorem I, using the closedness of p, it follows that the minimax equality holds.
The inmum over X in the right-hand side of
the minimax equality is attained at the set of minimizing points of the function t, which is nonempty
and compact since t is proper and has compact
level sets.

SADDLE POINT THEOREM

Dene

rx (z) =

tz (x) =

(x, z) if z Z,

if z
/ Z,
(x, z) if x X,

if x
/ X,

r(z) = sup rx (z)

xX

t(x) = sup tz (x)

zZ

Assume that:
(1) X and Z are convex and
either < sup inf (x, z), or inf sup (x, z) < .
zZ xX

xX zZ

(2) For each x X, rx () is closed and convex, and for each z Z, tz () is closed and
convex.
(3) All the level sets {x | t(x) } and {z |
r(z) } are compact.
Then, the minimax equality holds, and the set of
saddle points of is nonempty and compact.
Proof: Apply the preceding theorem. Q.E.D.

SADDLE POINT COROLLARY

Assume that X and Z are convex, tz and rx are
closed and convex for all z Z and x X, respectively, and any one of the following holds:
(1) X and Z are compact.
(2) Z is compact and there exists a vector z
Z
and a scalar such that the level set x
X | (x, z) is nonempty and compact.
(3) X is compact and there exists a vector x
X
and a scalar such that the level set z
Z | (x, z) is nonempty and compact.
(4) There exist vectors x X and z Z, and a
scalar such that the level sets

x X | (x, z) ,

z Z | (x, z) ,

are nonempty and compact.

Then, the minimax equality holds, and the set of
saddle points of is nonempty and compact.