The Inner Product

The inner product (or ``dot product'', or ``scalar product'')
is an operation on two vectors which produces a scalar. Defining an
inner product for a Banach space specializes it to a Hilbert
space (or ``inner product space''). There are many examples of
Hilbert spaces, but we will only need
for this
book (complex length vectors, and complex scalars).

It is straightforward to show that properties 1 and 3 of a norm hold
(see §5.8.2). Property 2 follows easily from the Schwarz
Inequality which is derived in the following subsection.
Alternatively, we can simply observe that the inner product induces
the well known norm on .

Cauchy-Schwarz Inequality

The Cauchy-Schwarz Inequality (or ``Schwarz Inequality'')
states that for all
and
, we have

We can quickly show this for real vectors
,
, as
follows: If either
or
is zero, the inequality holds (as
equality). Assuming both are nonzero, let's scale them to unit-length
by defining the normalized vectors
,
, which are
unit-length vectors lying on the ``unit ball'' in (a hypersphere
of radius ). We have

which implies

or, removing the normalization,

The same derivation holds if
is replaced by
yielding

The last two equations imply

In the complex case, let
, and define
. Then
is real and equal to
. By the same derivation as above,

The triangle inequality states that the length of any side of a
triangle is less than or equal to the sum of the lengths of the other two
sides, with equality occurring only when the triangle degenerates to a
line. In , this becomes

Vector Cosine

In the case of real vectors
, we can always find a real number which satisfies

We thus interpret as the angle between two vectors in .

Orthogonality

The vectors (signals) and
5.11are said to be orthogonal if
, denoted .
That is to say

Note that if and are real and orthogonal, the cosine of the angle
between them is zero. In plane geometry (), the angle between two
perpendicular lines is , and
, as expected. More
generally, orthogonality corresponds to the fact that two vectors in
-space intersect at a right angle and are thus perpendicular
geometrically.

Note that the converse is not true in . That is,
does not imply
in . For a counterexample, consider ,
, in which case

while
.

For real vectors
, the Pythagorean theorem Eq.(5.1)
holds if and only if the vectors are orthogonal. To see this, note
that, from Eq.(5.2), when the Pythagorean theorem holds, either
or is zero, or
is zero or purely imaginary,
by property 1 of norms (see §5.8.2). If the inner product
cannot be imaginary, it must be zero.

Note that we also have an alternate version of the Pythagorean
theorem:

Projection

The orthogonal projection (or simply ``projection'') of
onto
is defined by

The complex scalar is called the
coefficient of projection. When projecting onto a unit
length vector , the coefficient of projection is simply the inner
product of with .

Motivation: The basic idea of orthogonal projection of onto
is to ``drop a perpendicular'' from onto to define a new
vector along which we call the ``projection'' of onto .
This is illustrated for in Fig.5.9 for and
, in which case

Figure:
Projection of
onto
in 2D space.

Derivation: (1) Since any projection onto must lie along the
line collinear with , write the projection as
. (2) Since by definition the projection error
is orthogonal to , we must have