Lagranges theorem

Theorem
Assume that
0
is dierentiable and let
k
k = 1, 2, . . . , p be continuously
dierentiable and assume that
0
has a local minimum at x
+
on the set
X $ x [
k
(x) = 0, k = 1, . . . , p $ x [ F (x) = 0 .
Then there are multipliers
l = (
0
,
1
, . . . ,
p
) R
p+1
such that
1
l ,= 0,
2
L
/
x
(l , x
+
) =

p
k=0

k

/
k
(x
+
) = 0.
If
/
i
(x
+
) i = 1, . . . , p are linearly independent then
0
= 1 is possible.
Medvegyev (CEU) Presession 2013 424 / 437
Lagranges theorem
Example
Constraint qualication is important.
Consider the problem
3x + 2y min
x
2
+ y
2
= 0
The "Lagrangian" is L = 3x + 2y +
_
x
2
+ y
2
_
.
L
x
= 3 + 2x = 0,
L
y
= 2 + 2y = 0.
2/2y = = 3/2x
which has no feasible solution, althought there is an optimal solution
x
+
= y
+
= 0. At x
+
= y
+
= 0 grad( (x
+
, y
+
)) =
_
2x
+
2y
+
_
= 0.
Medvegyev (CEU) Presession 2013 425 / 437
Lagranges theorem
First of all observe that if the gradients of the constraints at x
+
are not
linearly independent then the theorem is obvious as one can choose
0
$ 0
and by the dependence of the row vector gradients
/
k
(x
+
) there are
constants
k
, k = 1, . . . , p not all of them zero such that
p

k=1

/
k
(x
+
) = 0.
Hence L
/
x
=

p
k=0

k

/
k
(x
+
) = 0. If the gradients are independent then
one should prove that the row vector
/
0
(x
+
) is a linear combination of the
row vectors
/
k
(x
+
) , k = 1, . . . , p. If p = n then it is trivial as in this case
the matrix formed from the gradients is invertible. Otherwise we can use
the implicit function theorem.
Medvegyev (CEU) Presession 2013 426 / 437
Lagranges theorem
Lagrange theorem is just a combination of Fermats principlem chain rule
and the Implict Function Theorem. The basic idea in optimization theory
is that if a point x
0
is a local minimum then for any h "variation" x
0
+ th
has a minimum at t = 0. Then one should calculate the derivarive using
the chain rule. With IFT one eliminates the constrains. Let
0
(x
1
, x
2
) be
optimal over
W X = W F (x
1
, x
2
) = 0 .
Medvegyev (CEU) Presession 2013 427 / 437
Lagranges theorem
By IFT there is a dierentiable function x
1
(x
2
) and a neighborhood V of
x
+
2
such that F (x
1
(x
2
) , x
2
) = 0 and
(x
1
(x
2
) , x
2
) [ x
2
V _ W
where V _ R
np
is also open. Observe that x
2
has n p components and
x
2
(x
1
) is a function having n p variables and p values. The function
(x
2
) $
0
(x
1
(x
2
) , x
2
)
is a real valued function of n p variables. By Fermats principle its
gradient is zero at x
+
2
.

x
2
(x
+
2
) $

x
2

0
(x
1
(x
+
2
) , x
+
2
) =

x
2

0
(x
+
1
, x
+
2
) =
=

x
2

0
(x
+
) = 0.
Medvegyev (CEU) Presession 2013 428 / 437
Lagranges theorem
By the chain rule

x
2

0
(x
+
) =

x
1

0
(x
1
(x
+
2
) , x
+
2
)
x
1
x
2
+

x
2

0
(x
1
(x
+
2
) , x
+
2
)
x
2
x
2
=
=

x
1

0
(x
+
)
x
1
x
2
+

x
2

0
(x
+
) .
Observe that as x
2
has n p components and x
1
has p components the
dimensions are
1 (n p) = (1 p) (p (n p)) + 1 (n p)
Medvegyev (CEU) Presession 2013 429 / 437
Lagranges theorem
By the implicit dierentiation rule
x
1
x
2
=
_
F
x
1
_
1
F
x
2
where by denition F (x
1
, x
2
) = 0 is the equation formed by the constrains

k
, (k = 1, 2, . . . , m)
That is
0 =

x
1

0
(x
+
)
x
1
x
2
+

x
2

0
(x
+
) I =
=

x
1

0
(x
+
)
_
F
x
1
_
1
F
x
2
+

x
2

0
(x
+
) .
Medvegyev (CEU) Presession 2013 430 / 437
Lagranges theorem
Reordering and with simple substitution
u
T
=

x
1

0
(x
+
)
_
F
x
1
_
1

x
2

0
(x
+
) =

x
1

0
(x
+
)
_
F
x
1
_
1
F
x
2
$
$ u
T
F
x
2
Obviously
u
T
F
x
1
$

x
1

0
(x
+
)
_
F
x
1
_
1
F
x
1
=

x
1

0
(x
+
) .
Medvegyev (CEU) Presession 2013 431 / 437
Lagranges theorem
That is combining the two equations

/
0
(x
+
) =
_

x
1

0
(x
+
) ,

x
2

0
(x
+
)
_
= u
T
_
F
x
1
,
F
x
2
_
=
= u
T
F
/
(x
+
) = u
T
_
_
_
_
_

/
1
(x
+
)

/
2
(x
+
)
.
.
.

/
2
(x
+
)
_
_
_
_
_
==
p

k=1
u
k

/
k
(x
+
) .
Medvegyev (CEU) Presession 2013 432 / 437
Lagranges theorem
Lagrange theorem is just an application of the implicite function theory
and Fermat principle on
(x
2
) $
0
(x
1
(x
2
) , x
2
) or on (t) $
0
(x
1
(x
2
+ th) , x
2
+ th)
Hence using another IFT one can also prove the next generalization.
Theorem
Assume that
k
k = 0, 1, . . . , p are dierentiable (One can just assume
that
k
, k = 0, 1, . . . , p are dierentiable just at x
+
and
k
,
k = 1, 2, . . . , p are continuous. ) and assume that
0
has a local
minimum at x
+
on the set
X $ x [
k
(x) = 0, k = 1, . . . , p $ x [ F (x) = 0 .
Then there are multipliers
l = (
0
,
1
, . . . ,
p
) R
p+1
such that
1
l ,= 0,
2
L
/
x
(l , x
+
) =

p
k=0

k

/
k
(x
+
) = 0.
If
/
i
(x
+
) i = 1, . . . , p are linearly independent then
0
= 1 is possible.
Medvegyev (CEU) Presession 2013 433 / 437