What is dot product?Dot productExample - computing the dot product between two vectorsUsing dot product to compute the angle between two vectorsExample - computing the angle between two vectorsProperties of dot productsDot product of two perpendicular vectorsCommutative property of dot productPosition of scalars can be changedDot Product Distributes Over Vector AdditionRelationship between magnitude and dot productDot product of a vector with itself is equal to the square of the vector's magnitudeExpressing dot product of two vectors using magnitude (2)
hubGraph View (new) smart_toyML Guides casinoProb and Stats Guides
memoryML models
Simple linear regression Logistic regression k-means clustering Hierarchical clustering DBSCAN Naive bayes Decision tree k-nearest neighbors Perceptrons
settings_suggestFeature engineering
Feature scaling Log transformation Feature encoding Grid search Random search Principal component analysis
trending_downOptimization
Gradient descent
assignment_turned_inModel evaluation
Confusion matrix ROC curves Cross validation Mean squared error Mean absolute error Root mean squared error R-squared
gradePySpark
Getting started with PySpark Resilient distributed dataset (RDD) Getting started with PySpark on Databricks Caching RDDs and DataFrames Using SQL on DataFrames User-defined functions
functionsMathematics
Odds and odds ratio Orthogonal projections
Comprehensive Guide on on Dot Product in Linear Algebra
schedule Jan 8, 2023
local_offer Linear Algebra
What is dot product?
When we are dealing with real numbers, there is only one interpretation of product - multiplication. However, in the world of vectors, there are two types of products - dot product and cross product. The dot product appears everywhere in machine learning and is therefore a must-know topic. In this guide, we will explore the dot product in depth!
Suppose we have the following two vectors $\boldsymbol{v}$ and $\boldsymbol{w}$:
$$\boldsymbol{v}=\begin{pmatrix} v_1\\ v_2\\ v_3\\ \end{pmatrix},\;\;\;\; \boldsymbol{w}=\begin{pmatrix} w_1\\ w_2\\ w_3\\ \end{pmatrix}$$
The dot product is defined as:
$$\boldsymbol{v}\cdot{\boldsymbol{w}}= v_1w_1+ v_2w_2+ v_3w_3$$
Note that $\boldsymbol{v}$ and $\boldsymbol{w}$ are in $\mathbb{R}^3$, but the dot product returns a scalar. The dot product is also sometimes referred to as the scalar product or the inner product.
Computing the dot product between two vectors
Compute the dot product $\boldsymbol{v}\cdot{\boldsymbol{w}}$ where $\boldsymbol{v}$ and $\boldsymbol{w}$ are defined as:
$$\boldsymbol{v}= \begin{pmatrix} 1\\2 \end{pmatrix},\;\;\;\; \boldsymbol{w}= \begin{pmatrix} 3\\4 \end{pmatrix}$$
Solution. The dot product $\boldsymbol{v}\cdot\boldsymbol{w}$ is:
$$\begin{align*} \boldsymbol{v}\cdot{\boldsymbol{w}}&= (1)(3)+(2)(4)\\ &=11 \end{align*}$$
Theorem.
Using dot product to compute the angle between two vectors
Let the angle between $\boldsymbol{v}$ and $\boldsymbol{w}$ be $\theta$. The dot product of $\boldsymbol{v}$ and $\boldsymbol{w}$ is:
$$\boldsymbol{v}\cdot{\boldsymbol{w}}= \Vert{\boldsymbol{v}}\Vert\Vert{\boldsymbol{w}}\Vert \cos{(\theta)}$$
Here, $ \Vert{\boldsymbol{v}}\Vert$ is the magnitude of $\boldsymbol{v}$.
Proof. Suppose we have the following:
Here, the vector $\boldsymbol{w}-\boldsymbol{v}$ points from $A$ to $B$ because of the geometric interpretation of vector additionlink.
The lengths of the sides are:
Using the cosine rule from elementary trigonometry, we have that:
$$\begin{equation}\label{eq:xF6GwOYW3PvcsraNGkP} \Vert{\boldsymbol{w}-\boldsymbol{v}}\Vert^2 =\Vert\boldsymbol{v}\Vert^2+ \Vert\boldsymbol{w}\Vert^2-2\Vert{\boldsymbol{v}}\Vert \Vert{\boldsymbol{w}}\Vert\cos(\theta) \end{equation}$$
Let's assume $\boldsymbol{w}$ and $\boldsymbol{v}$ are both in $\mathbb{R}^3$, though the proof can easily be generalized to $\mathbb{R}^n$. The vector $\boldsymbol{w}-\boldsymbol{v}$ is:
$$\boldsymbol{w}-\boldsymbol{v} =\begin{pmatrix} w_1-v_1\\ w_2-v_2\\ w_3-v_3\\ \end{pmatrix}$$
The square of the magnitude of $\boldsymbol{w}-\boldsymbol{v}$ is:
$$\begin{align*} \Vert{\boldsymbol{w}-\boldsymbol{v}}\Vert^2 &= \Big(\;\sqrt{(w_1-v_1)^2+ (w_2-v_2)^2+ (w_3-v_3)^2}\;\Big)^2\\ &= (w_1-v_1)^2+ (w_2-v_2)^2+ (w_3-v_3)^2 \end{align*}$$
Similarly, $\Vert{\boldsymbol{v}}\Vert^2$ and $\Vert{\boldsymbol{w}}\Vert^2$ are:
$$\begin{align*} \Vert{\boldsymbol{v}}\Vert^2 &= v_1^2+v^2_2+v^3_3\\ \Vert{\boldsymbol{w}}\Vert^2 &= w_1^2+w^2_2+w^3_3 \end{align*}$$
Substituting these squared magnitudes into \eqref{eq:xF6GwOYW3PvcsraNGkP} gives:
$$\begin{align*} (w_1-v_1)^2+ (w_2-v_2)^2+ (w_3-v_3)^2&= v^2_1+v^2_2+v^2_3+ w^2_1+w^2_2+w^2_3 -2 \Vert{\boldsymbol{v}}\Vert\Vert{\boldsymbol{w}}\Vert\cos{(\theta)} \\ v_1w_1+v_2w_2+v_3w_3&=\Vert{\boldsymbol{v}}\Vert\Vert{\boldsymbol{w}}\Vert\cos{(\theta)}\\ \boldsymbol{v}\cdot{\boldsymbol{w}}&=\Vert{\boldsymbol{v}}\Vert\Vert{\boldsymbol{w}}\Vert\cos{(\theta)} \end{align*}$$
This completes the proof.
Computing the angle between two vectors
Compute the angle between the following two vectors:
$$\boldsymbol{v}= \begin{pmatrix} 2\\ 3 \end{pmatrix},\;\;\;\; \boldsymbol{w}= \begin{pmatrix} 4\\ 1 \end{pmatrix}$$
Solution. We know from theoremlink that:
Rearranging the above to make $\theta$ the subject:
$$\begin{align*} \theta&= \arccos{\Big(\frac{\boldsymbol{v}\cdot{\boldsymbol{w}}} {\Vert{\boldsymbol{v}}\Vert\Vert{\boldsymbol{w}}\Vert}\Big)}\\ &=\mathrm{arccos}{\Big(\frac{(2)(4)+(3)(1)} {(2^2+3^2)^{1/2}(4^2+1^2)^{1/2}}\Big)}\\ &\approx42.5 \end{align*}$$
Therefore, the angle between vectors $\boldsymbol{v}$ and $\boldsymbol{w}$ is around $42.5$ degrees. This is confirmed visually below:
Properties of dot products
Dot product of two perpendicular vectors
If $\boldsymbol{v}$ and $\boldsymbol{w}$ are vectors that are perpendicular to each other, then:
$$\boldsymbol{v}\cdot{\boldsymbol{w}}=0$$
Proof. If $\boldsymbol{v}$ and $\boldsymbol{w}$ are perpendicular, then we have the following scenario:
From theoremlink, we have that:
$$\begin{align*} \boldsymbol{v}\cdot{\boldsymbol{w}}&= \Vert{\boldsymbol{v}}\Vert\Vert{\boldsymbol{w}}\Vert \cos{(\theta)}\\ &= \Vert{\boldsymbol{v}}\Vert\Vert{\boldsymbol{w}}\Vert \cos{(90)}\\ &= 0 \end{align*}$$
Commutative property of dot product
The commutative property of the dot product states that the ordering of the dot product between two vectors does not matter:
$$\boldsymbol{v}\cdot\boldsymbol{w}= \boldsymbol{w}\cdot\boldsymbol{v}$$
Proof. We will prove the case for $\mathbb{R}^3$ here, but the proof can easily be generalized for $\mathbb{R}^n$. Let vectors $\boldsymbol{v}$ and $\boldsymbol{w}$ be:
$$\boldsymbol{v}= \begin{pmatrix} v_1\\v_2\\v_3\\ \end{pmatrix},\;\;\;\; \boldsymbol{w}= \begin{pmatrix} w_1\\w_2\\w_3\\ \end{pmatrix}$$
Starting from the left-hand side:
$$\begin{align*} \boldsymbol{v}\cdot\boldsymbol{w}&= v_1w_1+v_2w_2+v_3w_3\\ &=w_1v_1+w_2v_2+w_3v_3\\ &=\boldsymbol{w}\cdot\boldsymbol{v} \end{align*}$$
Position of scalars can be changed
If $\boldsymbol{a}$ and $\boldsymbol{b}$ are vectors in $\mathbb{R}^n$ and $\lambda$ is some constant scalar, then:
$$(\lambda{\boldsymbol{v}})\cdot{\boldsymbol{w}}= \boldsymbol{v}\cdot(\lambda{\boldsymbol{w}}) =\lambda(\boldsymbol{v}\cdot\boldsymbol{w})$$
$$\begin{align*} (\lambda\boldsymbol{v})\cdot{\boldsymbol{w}}&= \begin{pmatrix} \lambda{v_1}\\\lambda{v_2}\\\lambda{v_3}\\ \end{pmatrix}\cdot \begin{pmatrix} w_1\\w_2\\w_3\\ \end{pmatrix}\\ &=\lambda{v_1w_1}+\lambda{v_2w_2}+\lambda{v_3w_3} \end{align*}$$
To prove the first equality:
$$\begin{align*} (\lambda\boldsymbol{v})\cdot{\boldsymbol{w}} &=\lambda{v_1w_1}+\lambda{v_2w_2}+\lambda{v_3w_3}\\ &=v_1(\lambda{w_1})+v_2(\lambda{w_2})+v_3(\lambda{w_3})\\ &=\boldsymbol{v}\cdot(\lambda{\boldsymbol{w}}) \end{align*}$$
To prove the second equality:
$$\begin{align*} (\lambda\boldsymbol{v})\cdot{\boldsymbol{w}} &=\lambda{v_1w_1}+\lambda{v_2w_2}+\lambda{v_3w_3}\\ &=\lambda(v_1w_1)+\lambda(v_2w_2)+\lambda(v_3w_3)\\ &=\lambda(\boldsymbol{v}\cdot\boldsymbol{w}) \end{align*}$$
Dot Product Distributes Over Vector Addition
If $\boldsymbol{u}$, $\boldsymbol{v}$ and $\boldsymbol{w}$ are vectors in $\mathbb{R}^n$, then:
$$\begin{align*} (\boldsymbol{u}+\boldsymbol{v})\cdot\boldsymbol{w}= \boldsymbol{u}\cdot\boldsymbol{w}+\boldsymbol{v}\cdot\boldsymbol{w} \end{align*}$$
Proof. We will prove the case for $\mathbb{R}^3$ here, but the proof can easily be generalized for $\mathbb{R}^n$. Let vectors $\boldsymbol{u}$, $\boldsymbol{v}$ and $\boldsymbol{w}$ be as follows:
$$\boldsymbol{u}= \begin{pmatrix} u_1\\u_2\\u_3\\ \end{pmatrix},\;\;\;\; \boldsymbol{v}= \begin{pmatrix} v_1\\v_2\\v_3\\ \end{pmatrix},\;\;\;\; \boldsymbol{w}= \begin{pmatrix} w_1\\w_2\\w_3\\ \end{pmatrix}$$
The vector $\boldsymbol{u}+\boldsymbol{v}$ is:
$$\boldsymbol{u}+\boldsymbol{v}= \begin{pmatrix} u_1+v_1\\u_2+v_2\\u_3+v_3\\ \end{pmatrix}$$
The left-hand side $(\boldsymbol{u}+\boldsymbol{v})\cdot\boldsymbol{w}$ is therefore:
$$\begin{align*} (\boldsymbol{u}+\boldsymbol{v})\cdot\boldsymbol{w}&= \begin{pmatrix} u_1+v_1\\u_2+v_2\\u_3+v_3\\ \end{pmatrix}\cdot \begin{pmatrix} w_1\\w_2\\w_3 \end{pmatrix}\\&= w_1u_1+w_1v_1+ w_2u_2+w_2v_2+ w_3u_3+w_3v_3 \\&=(w_1u_1+w_2u_2+w_3u_3)+(w_1v_1+w_2v_2+w_3v_3) \\&=\boldsymbol{u}\cdot{\boldsymbol{w}}+\boldsymbol{v}\cdot\boldsymbol{w} \end{align*}$$
Relationship between magnitude and dot product
The following relationships between vector magnitude and dot product is useful for future proofs.
Dot product of a vector with itself is equal to the square of the vector's magnitude
The magnitude of a vector $\boldsymbol{v}$ can be expressed as dot product like so:
$$\Vert{\boldsymbol{v}}\Vert =(\boldsymbol{v}\cdot\boldsymbol{v})^{1/2}$$
Taking the square on both sides gives:
$$\Vert{\boldsymbol{v}}\Vert^2 =\boldsymbol{v}\cdot\boldsymbol{v}$$
Proof. Consider the following vector:
$$\boldsymbol{v}=\begin{pmatrix} v_1\\ v_2\\ \vdots\\ v_n\\ \end{pmatrix}$$
By definition, the magnitude of $\boldsymbol{v}$ is:
$$\begin{align*} \Vert\boldsymbol{v}\Vert &=(v_1^2+v_2^2+\cdots+v_n^2)^{1/2}\\ &=\Big((v_1)(v_1)+(v_2)(v_2)+\cdots+(v_n)(v_n)\Big)^{1/2}\\ &=(\boldsymbol{v}\cdot\boldsymbol{v})^{(1/2)} \end{align*}$$
Expressing dot product of two vectors using magnitude (2)
If $\boldsymbol{v}$ and $\boldsymbol{w}$ are vectors, then:
$$\boldsymbol{v}\cdot{\boldsymbol{w}}= \frac{1}{4}\Vert\boldsymbol{v}+\boldsymbol{w}\Vert^2- \frac{1}{4}\Vert\boldsymbol{v}-\boldsymbol{w}\Vert^2$$
Proof. Let's start from the right-hand side:
$$\begin{align*} \Vert\boldsymbol{v}+\boldsymbol{w}\Vert^2&= (\boldsymbol{v}+\boldsymbol{w})\cdot(\boldsymbol{v}+\boldsymbol{w})\\ &= (\boldsymbol{v}\cdot\boldsymbol{v})+ 2(\boldsymbol{v}\cdot\boldsymbol{w})+(\boldsymbol{w}\cdot\boldsymbol{w})\\ &= \Vert\boldsymbol{v}\Vert^2+ 2(\boldsymbol{v}\cdot\boldsymbol{w})+ \Vert\boldsymbol{w}\Vert^2\\ \end{align*}$$
Note that the first step follows from theoremlink.
Similarly, we have that:
$$\begin{align*} \Vert\boldsymbol{v}-\boldsymbol{w}\Vert^2&= (\boldsymbol{v}-\boldsymbol{w})\cdot(\boldsymbol{v}-\boldsymbol{w})\\ &= (\boldsymbol{v}\cdot\boldsymbol{v})- 2(\boldsymbol{v}\cdot\boldsymbol{w})+(\boldsymbol{w}\cdot\boldsymbol{w})\\ &= \Vert\boldsymbol{v}\Vert^2- 2(\boldsymbol{v}\cdot\boldsymbol{w})+ \Vert\boldsymbol{w}\Vert^2\\ \end{align*}$$
$$\begin{align*} \frac{1}{4}\Vert\boldsymbol{v}+\boldsymbol{w}\Vert^2- \frac{1}{4}\Vert\boldsymbol{v}-\boldsymbol{w}\Vert^2&= \frac{1}{4}( \Vert\boldsymbol{v}\Vert^2+ 2(\boldsymbol{v}\cdot\boldsymbol{w})+ \Vert\boldsymbol{w}\Vert^2 )-\frac{1}{4} (\Vert\boldsymbol{v}\Vert^2- 2(\boldsymbol{v}\cdot\boldsymbol{w})+ \Vert\boldsymbol{w}\Vert^2)\\ &=\frac{1}{2}(\boldsymbol{v}\cdot\boldsymbol{w})+ \frac{1}{2}(\boldsymbol{v}\cdot\boldsymbol{w})\\ &=\boldsymbol{v}\cdot\boldsymbol{w} \end{align*}$$