Inner Products

Definition.

An inner product on an $F$-vector space $V$ is a function that takes each ordered pair $(u,v)$ of vectors of $V$ to a number $⟨u,v⟩\in F$ and has the following properties.

1. Positivity: $⟨v,v⟩\ge 0$ for all $v\in V$.
2. Definiteness $⟨v,v⟩=0$ if and only if $v=0$.
3. Additivity in the first slot: $⟨u+v,w⟩=⟨u,w⟩+⟨v,w⟩$ for all $u,v,w\in V$.
4. Homogeneity in the first slot: $⟨\lambda u,v⟩$ for all $\lambda \in F$ and all $u,v\in V$.
5. Conjugate symmetry: $⟨u,v⟩=\overline{⟨v,u⟩}$ for all $u,v\in V$.

Every inner product $⟨\cdot ,\cdot ⟩$ induces a norm $||\cdot ||$ on $V$. This induced norm is defined as such: for an inner product space $V$ and $v\in V$, the induced norm of $v$ is$||v||=\sqrt{⟨v,v⟩}.$

Two vectors $u,v\in V$ are orthogonal if $⟨u,v⟩=0$.

Remark(s).

Note that we need not specify additivity or homogeneity in the second slot as the additivity and homogeneity in the first slot and conjugate symmetry are enough to deduce additivity and homogeneity in the second slot. In general, mathematicians tend to opt for simpler and more bare-bones definitions when they can. This can be thought of both as a aesthetic endeavor and an application of Occam’s razor as simpler definitions will likely be the most flexible.

The most straightforward inner product is of course the dot product. In the space of real random variables or random vectors, the expected value of the product of two random variables is an inner product: $⟨X,Y⟩=\mathbb{E}[XY]$. This implies that the covariance, defined as $\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])]$ is an inner product for the space of real random variables with mean $0$. And for this covariance inner product space, variance is the induced norm. Thus, we can apply linear algebraic results to probability theory and statistics to gain a deeper understanding and perhaps some justification for certain definitions.

Inner products and norms introduce geometry into our vector spaces. The norm generalizes the notion of distance, the inner product generalizes the notion of an angle, and orthogonality generalizes the notion of perpendicularity to more abstract spaces. Intuitively, vectors with a larger inner product are more similar, while orthogonal vectors represent completely different things.

---

Math people use the word orthogonal a lot when they talk about general ideas or concepts. Two concepts are orthogonal if they are completely unrelated or belong to different categories. It’s a useful word to have on hand.

From LADR, Axler.

Resources: The Vector Space of Random Variables