Lagrange duality
总结自 section note 5: Convex Optimization Overview, Part II
0. IntroductionPermalink
In the study of convex optimization, the mathematical optimization problems follow the form below:
where
is a vector known as the optimization variable is a convex function that we want to minimize is a convex set describing the set of feasible (capable of being done or carried out) solutions
A necessary and sufficient condition for
In the more general setting of convex optimization problem with constraints, however, this simple optimality condition does not work. One primary goal of duality theory is to characterize the optimal points of convex programs in a mathematically rigorous way.
Here we propose a form of generic differentiable convex optimization problems, as:
where
is the optimization variable and are differentiable convex functions are affine functions- An affine function is a function of the form
for some ,
- An affine function is a function of the form
is one of the convex inequality constraints is one of the affine equality constraints
1. The LagrangianPermalink
Given a convex constrained minimization problem of the form
where
is the primal variables of the Lagrangian and are the dual variables of the Lagrangian, a.k.a Lagrange multipliers
Intuitively, the Lagrangian can be thought of as a modified version of the objective function to the original convex optimization problem
For any convex optimization problem, there always exist settings of the dual variables such that the unconstrained minimum of the Lagrangian with respect to the primal variables (keeping the dual variables fixed) coincides with the solution of the original constrained minimization problem.
We formalize this intuition when we describe the KKT conditions in Section 6.
2. Primal and dual problemsPermalink
To show the relationship between the Lagrangian and the original convex optimization problem
The primal problemPermalink
The primal objective function
Then the unconstrained minimization problem
is known as the primal problem.
- Generally, we say that a point
is primal feasible if , and , - We typically use the vector
to denote the solution of , and we let denote the optimal value of the primal objective.
The dual problemPermalink
By switching the order of the minimization and maximization above, we obtain an entirely different optimization problem.
The dual objective function
Then the constrained maximization problem
is known as the dual problem.
- Generally, we say that
are dual feasible if , - We typically use the vector
to denote the solution of , and we let denote the optimal value of the dual objective.
3. Interpreting the primal problemPermalink
当
当
考虑到我们最原始的问题
4. Interpreting the dual problemPermalink
The dual objective,
Lemma 1: If
Proof:
Lemma 2 (Weak Duality): For any pair of primal and dual problems,
This is obvious following Lemma 1.
Lemma 3 (Strong Duality): For any pair of primal and dual problems which satisfy certain technical conditions called constraint qualifications, then
Slater’s condition: a primal/dual problem pair satisfy Slater’s condition if there exists some feasible primal solution
注意 Slater’s condition 并不能实现 Strong Duality,书上只是拿它举个例子说明啥叫 “technical conditions”。要实现 Strong Duality,还得靠下文的 KTT conditions。
注意,Strong Duality 的意义并不是简单地得到
- 求 dual problem
- 等价于
- 求 primal problem
- 等价于
- 求最原始的 optimization 问题
5. Complementary slacknessPermalink
在我们讲 KTT conditions 之前,我们先来说一下 Complementary slackness。
Lemma 4 (Complementary Slackness): If strong duality holds, then
证明比较简单。之前 Lemma 2 有
又因为
这个
6. The KKT (Karush-Kuhn-Tucker) conditionsPermalink
Theorem 1: Suppose that
- (Primal feasibility)
, and , , - (Dual feasibility)
, , - (Complementary slackness)
, , and - (Lagrangian stationarity)
.
Then
These conditions are known as the Karush-Kuhn-Tucker (KKT) conditions.
Comments