Convex optimization
Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard. Convex optimization has applications in a wide range of disciplines, such as automatic control systems, estimation and signal processing, communications and networks, electronic circuit design, data analysis and modeling, finance, statistics (optimal experimental design), and structural optimization, where the approximation concept has proven to be efficient. With recent advancements in computing and optimization algorithms, convex programming is nearly as straightforward as linear programming. Definition

A convex optimization problem is an optimization problem in which the objective function is a convex function and the feasible set is a convex set. A function   mapping some subset of into  is convex if its domain is convex and for all  and all  in its domain, the following condition holds: . A set S is convex if for all members  and all  , we have that . Concretely, a convex optimization problem is the problem of finding some  attaining
,
where the objective function  is convex, as is the feasible set . If such a point exists, it is referred to as an optimal point or solution; the set of all optimal points is called the optimal set. If  is unbounded below over  or the infimum is not attained, then the optimization problem is said to be unbounded. Otherwise, if  is the empty set, then the problem is said to be infeasible. Standard form
A convex optimization problem is in standard form if it is written as

where:

  is the optimization variable;
 The objective function  is a convex function; 
 The inequality constraint functions , , are convex functions;
 The equality constraint functions , , are affine transformations, that is, of the form: , where  is a vector and  is a scalar. This notation describes the problem of finding  that minimizes  among all  satisfying ,  and , . The function  is the objective function of the problem, and the functions  and  are the constraint functions. The feasible set  of the optimization problem consists of all points  satisfying the constraints. This set is convex because  is convex, the sublevel sets of convex functions are convex, affine sets are convex, and the intersection of convex sets is convex. A solution to a convex optimization problem is any point  attaining . In general, a convex optimization problem may have zero, one, or many solutions. Many optimization problems can be equivalently formulated in this standard form. For example, the problem of maximizing a concave function  can be re-formulated equivalently as the problem of minimizing the convex function . The problem of maximizing a concave function over a convex set is commonly called a convex optimization problem. Properties
The following are useful properties of convex optimization problems:

 every local minimum is a global minimum;
 the optimal set is convex;
 if the objective function is strictly convex, then the problem has at most one optimal point. These results are used by the theory of convex minimization along with geometric notions from functional analysis (in Hilbert spaces) such as the Hilbert projection theorem, the separating hyperplane theorem, and Farkas' lemma. Applications

The following problem classes are all convex optimization problems, or can be reduced to convex optimization problems via simple transformations:
 

Least squares
Linear programming
 Convex quadratic minimization with linear constraints
Quadratic minimization with convex quadratic constraints
Conic optimization
Geometric programming
Second order cone programming
Semidefinite programming
Entropy maximization with appropriate constraints
Convex optimization has practical applications for the following. Portfolio optimization. Worst-case risk analysis. Optimal advertising. Variations of statistical regression (including regularization and quantile regression). Model fitting (particularly multiclass classification). Electricity generation optimization. Combinatorial optimization. Non-probabilistic modelling of uncertainty. Localization using wireless signals

Lagrange multipliers

Consider a convex minimization problem given in standard form by a cost function  and inequality constraints  for . Then the domain  is:

The Lagrangian function for the problem is

For each point  in  that minimizes  over , there exist real numbers  called Lagrange multipliers, that satisfy these conditions simultaneously:

  minimizes  over all 
  with at least one 
  (complementary slackness). If there exists a "strictly feasible point", that is, a point  satisfying

then the statement above can be strengthened to require that . Conversely, if some  in  satisfies (1)–(3) for scalars  with  then  is certain to minimize  over . Algorithms
Unconstrained convex optimization can be easily solved with gradient descent (a special case of steepest descent) or Newton's method, combined with line search for an appropriate step size; these can be mathematically proven to converge quickly, especially the latter method. Convex optimization with linear equality constraints can also be solved using KKT matrix techniques if the objective function is a quadratic function (which generalizes to a variation of Newton's method, which works even if the point of initialization does not satisfy the constraints), but can also generally be solved by eliminating the equality constraints with linear algebra or solving the dual problem. Finally, convex optimization with both linear equality constraints and convex inequality constraints can be solved by applying an unconstrained convex optimization technique to the objective function plus logarithmic barrier terms. (When the starting point is not feasible - that is, satisfying the constraints - this is preceded by so-called phase I methods, which either find a feasible point or show that none exist. Phase I methods generally consist of reducing the search in question to yet another convex optimization problem.) Convex optimization problems can also be solved by the following contemporary methods:
 Bundle methods (Wolfe, Lemaréchal, Kiwiel), and
 Subgradient projection methods (Polyak),
 Interior-point methods, which make use of self-concordant barrier functions  and self-regular barrier functions. Cutting-plane methods
Ellipsoid method
Subgradient method
Dual subgradients and the drift-plus-penalty method
Subgradient methods can be implemented simply and so are widely used. Dual subgradient methods are subgradient methods applied to a dual problem. The drift-plus-penalty method is similar to the dual subgradient method, but takes a time average of the primal variables. Software 
There is a large software ecosystem for convex optimization. This ecosystem has two main categories: solvers on the one hand and modeling tools (or interfaces) on the other hand. Solvers implement the algorithms themselves and are usually written in C. They require users to specify optimization problems in very specific formats which may not be natural from a modeling perspective. Modeling tools are separate pieces of software that let the user specify an optimization in higher-level syntax. They manage all transformations to and from the user's high-level model and the solver's input/output format. The table below shows a mix of modeling tools (such as CVXPY and Convex.jl) and solvers (such as CVXOPT and MOSEK). This table is by no means exhaustive. Extensions

Extensions of convex optimization include the optimization of biconvex, pseudo-convex, and quasiconvex functions. Extensions of the theory of convex analysis and iterative methods for approximately solving non-convex minimization problems occur in the field of generalized convexity, also known as abstract convex analysis. See also
 Duality
 Karush–Kuhn–Tucker conditions
 Optimization problem
 Proximal gradient method

Notes

References
  
  
 
 
 

 Hiriart-Urruty, Jean-Baptiste, and Lemaréchal, Claude. (2004). Fundamentals of Convex analysis. Berlin: Springer. Nesterov, Yurii. (2004). Introductory Lectures on Convex Optimization,  Kluwer Academic Publishers
 

 
 Schmit, L.A.; Fleury, C. 1980: Structural synthesis by combining approximation concepts and dual methods. J. Amer. Inst. Aeronaut. Astronaut 18, 1252-1260

External links

EE364a: Convex Optimization I and EE364b: Convex Optimization II, Stanford course homepages
6.253: Convex Analysis and Optimization, an MIT OCW course homepage
 Brian Borchers, An overview of software for convex optimization
Convex Optimization Book by Lieven Vandenberghe and Stephen P. Boyd

Convex analysis
 
Mathematical optimization