Dual Numbers

A while back when I was doing some exploration of writing a simple NN code to improve my understanding of neural networks and deep learning in general, I came across dual numbers. They're a type of number that generalizes the concept of real and complex numbers. But what makes them so interesting is that they can encode both a function value and its derivative in a single number. This means that we can use them to simplify the calculation of derivatives and solve complex problems efficiently.

So how does one think of dual numbers? What's the difference between a dual number and a complex number? One way to think about dual numbers is that they consist of two parts: a scalar part and a skew part. The scalar part is just a regular real number, while the skew part is a multiple of a new number, often denoted as $\epsilon$, that satisfies the property $\epsilon^2=0$. This means that every dual number can be written as $a+b\epsilon$, where $a$ and $b$ are real numbers.

What's most interesting is that the skew part of a dual number is that it provides an approximation of the first derivative of a function evaluated at a particular point. By using the dual number representation of the function at that point, one can calculate both the function value and its derivative in one shot.  One reason dual numbers have applications in deep learning is that algebra on dual numbers provides the chain rule for calculus, therefore they can be used to compute derivatives of complicated functions involving multiple variables and interdependencies.

As an example, say I want to evaluate the function $f(x)=x^2+2x$ at $x=3$. The dual number representation of $f(3)$ is $f(3+\epsilon)=f(3)+f'(3)\epsilon$, where $f'(x)=\frac{df(x)}{dx}$. We can compute $f(3)$ directly as $f(3)=3^2+2\cdot3=9+6=15$. To compute $f'(3)$, we can take the derivative of $f$ with respect to $x$: $f'(x)=2x+2$. Evaluating this at $x=3$, we get $f'(3)=2\cdot3+2=8$. Therefore, the dual number representation of $f$ at $x=3$ is $15+8\epsilon$.

One of the benefits of dual numbers is the derivative of the composition of two functions, $f(g(x))$ requires only the derivatives of the individual functions. Specifically, if $f(x)$ and $g(x)$ are two functions, then the dual number representation of their composition $f(g(x))$ is $(f(g(x)), f'(g(x))g'(x))$. This is especially useful when dealing with complex functions involving multiple variables and complicated interdependencies.

Dual numbers are actually a useful mathematical concept because they have practical applications in a wide range of fields. It's pretty cool that one can encode function values and derivatives in a single number, which makes it possible to simplify the calculation of derivatives and solve complex problems efficiently. On my computational blog, I have an example using dual numbers to calculate the derivative of an interatomic potential, Dual Numbers Pluto blog.

Reuse and Attribution

Exact Differential Equations

The general form of a first-order differential equation is given by the following:

$$ M(x,y)dx + N(x,y)dy = 0$$

our differential equation is said to be exact if it satisfies the following exactness test:

$$\frac{\partial M\left(x,y\right)}{\partial y} = \frac{\partial N\left(x,y\right)}{\partial x}$$

The goal is to determine a function $f(x,y)$ that satisfies the following:

$$df = M(x,y)dx + N(x,y)dy$$

$$ \frac{\partial f\left(x,y\right)}{\partial x} = M\left(x,y\right)$$

$$ \frac{\partial f\left(x,y\right)}{\partial y} = N\left(x,y\right)$$

Let us look at the following example differential equation:

$$\left(y^{2}-2x\right)dx + \left(2xy+1\right)dy=0$$

Taking the partial derivatives of the functions corresponding to $M\left(x,y\right)$ and $N\left(x,y\right)$, we get:

$$ \frac{\partial M}{\partial y} = 2y $$

$$ \frac{\partial N}{\partial x} = 2y $$

So our differential equation is indeed exact and we now can find the total function, $f(x,y)$, whose derivative is equal to our differential equation. This is done by integrating  the functions $M\left(x,y\right)$ and $N\left(x,y\right)$,

$$M\left(x,y\right) = \frac{\partial f\left(x,y\right)}{\partial x}$$

$$f = \int{\left(y^{2}-2x\right) dx} = xy^{2}-x^{2} $$

similarly for $N\left(x,y\right)$,

$$N\left(x,y\right) = \frac{\partial f\left(x,y\right)}{\partial y}$$
$$f = \int{\left(2xy+1\right)dy} = xy^{2}+y $$

In both cases, we ignore the constant of integration. We now can identify unique terms and construct the function, $f(x,y)$, by summing these terms:

$$f\left(x,y\right) = xy^{2}-x^{2}+y=\text{constant}$$

So we have identified a function, $f\left(x,y\right)$, that is a solution to our exact differential equation.

Now for our quote:

I became an atheist because, as a graduate student studying quantum physics, life seemed to be reducible to second-order differential equations. It thus became apparent to me that mathematics, physics, and chemistry had it all and I didn't see any need to go beyond that.
-Attributed to Francis Collins but unconfirmed.

Reuse and Attribution

Partial Derivatives

Partial derivatives occur when taking the derivative of a function of two or more variables, e.g., $f(x,y)$.  If we want to determine the derivative of the function $f(x,y)$ with respect to one of the independent variables, we do this while holding the other independent variable(s) constant. This process is partial differentiation and is given in calculus terms as:

$$ \frac{\partial f(x,y)}{\partial x} = \lim_{x\rightarrow x_{o}} \frac{f\left(x,y_{c}\right)-f\left(x_{o},y_{c}\right)}{x-x_{o}}.$$

As mentioned above, we keep $y\rightarrow y_{c}$ constant while taking the derivative of the function with respect to $x$. Higher order partial derivatives are similar to normal derivatives and are commutative, for example:

$$\frac{\partial}{\partial x} \frac{\partial f(x,y)}{\partial y} = \frac{\partial}{\partial y} \frac{\partial f(x,y)}{\partial x}.$$

In cases where we are taking partial derivatives of a function dependent on another function(s), for example:

$$ f(g,h)=f(g(x,y),h(x,y),$$

then the partial derivatives of $f$ with respect to $x$ and $y$ are given by the chain rule, so that:

$$ \frac{\partial f(x,y)}{\partial x} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial x} + \frac{\partial f}{\partial h} \frac{\partial h}{\partial x}$$

and in similar fashion for the partial derivative with respect $y$. A common application of partial derivatives is found in vector calculus with the grad, div, and curl operators. Mathematically they are written as:

$$ \text{grad}\, f(x,y,z) = \nabla f = \frac{\partial f}{\partial x} \mathbf{a_x} + \frac{\partial f}{\partial y} \mathbf{a_y} + \frac{\partial f}{\partial z} \mathbf{a_z} $$

$$ \text{div}\, \mathbf{f}(x,y,z) = \nabla \cdot \mathbf{f} = \frac{\partial f_{x}}{\partial x} + \frac{\partial f_{y}}{\partial y}  + \frac{\partial f_{z}}{\partial z} $$

$$ \text{curl}\, \mathbf{f}(x,y,z) = \nabla \times \mathbf{f} = \frac{\partial f_{z}}{\partial y}-\frac{\partial f_{y}}{\partial z} \mathbf{a_x}+\frac{\partial f_{x}}{\partial z}-\frac{\partial f_{z}}{\partial x} \mathbf{a_y}+\frac{\partial f_{y}}{\partial x}-\frac{\partial f_{x}}{\partial y} \mathbf{a_z}$$

where $\mathbf{a_{i}}$ is the unit vector in Euclidean space. There is not much more to performing partial derivatives, but they are ubiquitous in partial differential equations and vector calculus which are essential mathematics for most engineers and scientists.

The selected quote comes from Alan Turing, the father of modern computer science.

"Science is a differential equation. Religion is a boundary condition."
-Alan Turing, 1954

References  & Additional Reading

[1] Advanced Engineering Mathematics, Wiley, 8th ed., E. Kreyszig.

[2] The Mathematics Companion, IoP, 1st ed., A.C. Fischer-Cripps.

Reuse and Attribution