Lagrange Multipliers

In science and mathematics it is very common for the need to determine the local extrema of a function, $f(x,y,z,\cdots)$ subject to some constraint. If the constraint (function), which depends on same set of independent variables, can be expressed as $h(x,y,z,\cdots) = 0$, then one can rewrite a new constrained function as:

$$ g\left(x,y,z,\cdots,\lambda \right) = f\left(x,y,z,\cdots\right) + \lambda h\left(x,y,z,\cdots\right).$$

The variable $\lambda$ is known as the Lagrange multiplier. The condition for an extrema (i.e., minima or maxima) occurs when the partial derivatives are zero, more explicitly:

$$\begin{align}\frac{\partial g}{\partial x} &= \frac{\partial f}{\partial x} - \lambda \frac{\partial h}{\partial x} = 0 \\\frac{\partial g}{\partial y} &= \frac{\partial f}{\partial y} - \lambda \frac{\partial h}{\partial y} = 0  \\ \vdots \\\frac{\partial g}{\partial \lambda} &= \frac{\partial f}{\partial \lambda} - \lambda \frac{\partial h}{\partial \lambda} = 0,\end{align}$$

with the last term $\frac{\partial g}{\partial \lambda} = h(x,y,z) =  0$. This sets up our system of equations that can be solved algebraically. To provide some additional context, this approach is used in areas such as thermodynamics and machine learning (more commonly referred to as regularization). The more general form of the function $g$ when several constraints are used is given by the Lagrangian function:

$$ \mathcal{L}\left(x,y,z,\cdots,\{\lambda\}\right) = f\left(x,y,z,\cdots\right) + \sum_i \lambda_i \, h_i\left(x,y,z,\cdots\right) $$

Lets see the use of Langrange multiplier in action with a simple example, a parabola constrained by a circle. Our function, $g(x,y) = 10x^2 - 5y$, and our constraint is $h(x,y) = x^2 + y^2 -1$.  Writing the new function $h(x,y,\lambda)$,

$$ h\left(x,y,\lambda\right) = 10x^2 - 5y + \lambda(x^2+y^2-1) $$

Now working through the partial derivatives of $h(x,y,\lambda)$:

$$\frac{\partial g}{\partial x} = 10x+\lambda(2x) = 0$$
$$\frac{\partial g}{\partial y} = -5 +\lambda(2y) = 0$$
$$\frac{\partial g}{\partial \lambda} = x^2+y^2-1 = 0$$

We now find the Lagrange multiplier, $\lambda$, using the first equation,

$$\begin{align}
2x(10+\lambda) &= 0 \\
\lambda &= -10
\end{align}
$$

we can calculate $y$, with the second equation,

$$\begin{align}
-5-10*(2y) &= 0 \\
y &= -0.25
\end{align}
$$

and with the constraint equation we can find $x = 0.9862$. This example is very simple and does not demosntrate the full power of Lagrange multipliers, which is extremely powerful in multivariate calculus. The quote for this blog post is by Joseph-Louis Lagrange himself:

"I regard as quite useless the reading of large treatises of pure analysis: too large a number of methods pass at once before the eyes. It is in the works of applications that one must study them; one judges their ability there and one apprises the manner of making use of them"
- Joseph-Louis Lagrange

References & Additional Reading

[1] Advanced Engineering Mathematics, Wiley, 8th ed., E. Kreyszig.

[2] The Mathematics Companion, IoP, 1st ed., A.C. Fischer-Cripps.

[3] Thermodynamics in Materials Science, Robert DeHoff, 2nd ed.

Reuse and Attribution

Partial Derivatives

Partial derivatives occur when taking the derivative of a function of two or more variables, e.g., $f(x,y)$.  If we want to determine the derivative of the function $f(x,y)$ with respect to one of the independent variables, we do this while holding the other independent variable(s) constant. This process is partial differentiation and is given in calculus terms as:

$$ \frac{\partial f(x,y)}{\partial x} = \lim_{x\rightarrow x_{o}} \frac{f\left(x,y_{c}\right)-f\left(x_{o},y_{c}\right)}{x-x_{o}}.$$

As mentioned above, we keep $y\rightarrow y_{c}$ constant while taking the derivative of the function with respect to $x$. Higher order partial derivatives are similar to normal derivatives and are commutative, for example:

$$\frac{\partial}{\partial x} \frac{\partial f(x,y)}{\partial y} = \frac{\partial}{\partial y} \frac{\partial f(x,y)}{\partial x}.$$

In cases where we are taking partial derivatives of a function dependent on another function(s), for example:

$$ f(g,h)=f(g(x,y),h(x,y),$$

then the partial derivatives of $f$ with respect to $x$ and $y$ are given by the chain rule, so that:

$$ \frac{\partial f(x,y)}{\partial x} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial x} + \frac{\partial f}{\partial h} \frac{\partial h}{\partial x}$$

and in similar fashion for the partial derivative with respect $y$. A common application of partial derivatives is found in vector calculus with the grad, div, and curl operators. Mathematically they are written as:

$$ \text{grad}\, f(x,y,z) = \nabla f = \frac{\partial f}{\partial x} \mathbf{a_x} + \frac{\partial f}{\partial y} \mathbf{a_y} + \frac{\partial f}{\partial z} \mathbf{a_z} $$

$$ \text{div}\, \mathbf{f}(x,y,z) = \nabla \cdot \mathbf{f} = \frac{\partial f_{x}}{\partial x} + \frac{\partial f_{y}}{\partial y}  + \frac{\partial f_{z}}{\partial z} $$

$$ \text{curl}\, \mathbf{f}(x,y,z) = \nabla \times \mathbf{f} = \frac{\partial f_{z}}{\partial y}-\frac{\partial f_{y}}{\partial z} \mathbf{a_x}+\frac{\partial f_{x}}{\partial z}-\frac{\partial f_{z}}{\partial x} \mathbf{a_y}+\frac{\partial f_{y}}{\partial x}-\frac{\partial f_{x}}{\partial y} \mathbf{a_z}$$

where $\mathbf{a_{i}}$ is the unit vector in Euclidean space. There is not much more to performing partial derivatives, but they are ubiquitous in partial differential equations and vector calculus which are essential mathematics for most engineers and scientists.

The selected quote comes from Alan Turing, the father of modern computer science.

"Science is a differential equation. Religion is a boundary condition."
-Alan Turing, 1954

References  & Additional Reading

[1] Advanced Engineering Mathematics, Wiley, 8th ed., E. Kreyszig.

[2] The Mathematics Companion, IoP, 1st ed., A.C. Fischer-Cripps.

Reuse and Attribution