%
%  revised at Feb. 26th. the first part and the second part
%  is greatly changed
%  revised at July 27th, 2009:
%  just move the representation discussion after the operator
%  and Hilbert space

\chapter{Representation}\label{REPRESENTATION:1}
%
% here in this chapter, we introduce the representation, which is
% the eigen states for the selected "CSCCO". from this group of
% eigen states, we can actually get everything we wish to get.
%
% 1  definition of the representation for operator and wave function
% 2  the transformation between different representations
%
% later, we hope to add the discussion related to the coordinate
% and momentum.
%
So far we have gotten general discussion related to the Hilbert
space and operators. the Hilbert space is used to character some
quantum state for a system, and the operator is corresponding to
physical quantity in quantum mechanics.

In the section \ref{LIV_in_Hilbert}, we have brought up the relation
that there are many different ways to represent the same Hilbert
space, and they are identical with each other. Now in this section,
we will further investigate the details around this issue.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Representation by matrix}\label{REPRESENTATION:2}
%
% 1  definition of the representation, what's the meaning; and how
%    to get it
% 2  express an arbitrary wave function by representation
% 3  express the operator
% 4  the operation between operator, how to express them
%    in we express the operator into the matrix form
% 5  how to get eigen values
% 6  how to express the expectation value
%
In an arbitrary Hilbert space, suggest that we have a complete set of
$\ket{i}$ ($i=0, 1, 2\cdots n$, here we only use its label to
distinguish them; and suppose that the complete basis sets are in
number of $n$). Thus for any $\Psi$, it can be expressed as:
\begin{equation}\label{REPRESENTATIONeq:1}
\ket{\Psi} = \sum_{i}a_{i}\ket{i}
\end{equation}

Here this set of $\ket{i}$ ($i=0, 1, 2\cdots n$) is called a kind of
``representation'', since we can use it to express any arbitrary wave
functions related to this system. This complete sets is the eigen
states for some physical quantity, usually we use the eigen states for
the Hamiltonian operator so that they are the energy
representation. in the following content, we can see that if such
representation has been selected, then all the information related to
the system (including the operator as well as the wave function to
express the states of the system) is fixed. Thus, this set of basis
functions in the Hilbert space determines everything we wish to get.
Such conclusion has very prominent meaning that only one sets of
representation is enough to express the quantum system, the others are
identical to the selected one.

By the way, the representation can also use continuous basis
functions, such as the plane wave functions for free particle. From
Fourier transformation, we can know that any square-integrable wave
function can be viewed as ``wave packets'' of plane wave functions
(see \ref{sec:HTEAAWFFFP_in_basic} and \ref{sec:PWF_in_Hilbert}).
Compared the continuous complete sets, what we need to change in the
discrete basis functions is only convert the summation into the
integral. Hence throughout the whole chapter, we will use the
discrete basis functions for simplicity.

Now let's go to prove this general idea. Before that, let's go to
see how to express an arbitrary wave function in matrix form. If we
have gotten the complete sets to describe the quantum system; the
only thing to express an arbitrary state is to determine the
coefficients (weight) of $a_{i}$ in (\ref{REPRESENTATIONeq:1}). Here
for a complected sets, we have:
\begin{equation}\label{}
a_{i} = \langle i|\Psi\rangle
\end{equation}
The vector forming by the $a_{i}$ here is called ``expression'' in a
certain representation, by using matrix form we can express them in
another way:
\begin{equation}\label{REPRESENTATIONeq:2}
\ket{\Psi} \Leftrightarrow
\begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\end{equation}
Here we called the $\ket{\Psi}$ is ``$a_{i}$'' expression.
Furthermore, the vector of $a_{i}$ is always set to the column.
Later we use $A$ to denote this column vector.

and naturally the \brat{\Psi} as:
\begin{equation}\label{REPRESENTATIONeq:3}
\bra{\Psi} \Leftrightarrow
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} &  \cdots   & a_{n}^{*} \\
\end{bmatrix}
\end{equation}
Here it's clear that for the arbitrary $\Psi$ each of $a_{i}$
represents the ``weight'' of total wave function on the
representation of $\ket{i}$ or $\bra{i}$. Furthermore, the bra can
be seen as the conjugated vector of ket:
\begin{equation}\label{}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} &  \cdots   & a_{n}^{*} \\
\end{bmatrix}
= A^{+}
\end{equation}

So here the inner product can be written as:
\begin{align}\label{REPRESENTATIONeq:4}
\langle\Psi|\Psi\rangle &= \begin{bmatrix}
                            a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
                          \end{bmatrix}
                          \begin{bmatrix}
                          a_{1} \\
                          a_{2} \\
                          \cdots   \\
                          a_{n} \\
                          \end{bmatrix} \nonumber \\
                       &= \sum_{i}a_{i}a_{i}^{*}
\end{align}

We can see that this expression is same with the internal product of
$\langle\Psi|\Psi\rangle$,  but in a much simpler format. Actually
the expression in the (\ref{REPRESENTATIONeq:2}) and
(\ref{REPRESENTATIONeq:3}) manifests the ``dimensional'' character
of wave function in a more clearly way.

Moreover, the operator can be expressed in the same way:
\begin{align}\label{REPRESENTATIONeq:5}
\ket{\Phi} &= \hat{A}\ket{\Psi} \rightarrow \nonumber \\
\sum_{i}a_{i}\ket{i} &= \hat{A}\sum_{j}b_{j}\ket{j}
\: \underrightarrow{multiply \, with \, \bra{i} }\nonumber \\
a_{i} &= \sum_{j}b_{j}\langle i|\hat{A}|j\rangle \rightarrow \nonumber \\
\begin{bmatrix}
  a_{1} \\
  a_{2} \\
  \cdots \\
  a_{n} \\
\end{bmatrix}
&= \begin{bmatrix}
     \bra{1}\hat{A}\ket{1} & \bra{1}\hat{A}\ket{2} & \cdots & \bra{1}\hat{A}\ket{n} \\
     \bra{2}\hat{A}\ket{1} & \bra{2}\hat{A}\ket{2} & \cdots & \bra{2}\hat{A}\ket{n} \\
     \cdots                &                \cdots & \cdots & \cdots                \\
     \bra{n}\hat{A}\ket{1} & \bra{n}\hat{A}\ket{2} & \cdots & \bra{n}\hat{A}\ket{n} \\
   \end{bmatrix}
\begin{bmatrix}
  b_{1} \\
  b_{2} \\
  \cdots \\
  b_{n} \\
\end{bmatrix} \nonumber \\
&= \begin{bmatrix}
     A_{11} & A_{12} & \cdots & A_{1n} \\
     A_{21} & A_{22} & \cdots & A_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     A_{n1} & A_{n2} & \cdots & A_{nn} \\
   \end{bmatrix}
\begin{bmatrix}
  b_{1} \\
  b_{2} \\
  \cdots \\
  b_{n} \\
\end{bmatrix}
\end{align}

Here we can see that if the $\hat{A}$ is a hermite operator, we
have:
\begin{equation}\label{REPRESENTATIONeq:6}
A_{ij} = \bra{i}\hat{A}\ket{j} = \bra{i}\hat{A}^{+}\ket{j} =
\bra{j}\hat{A}\ket{i}^{*} = A_{ji}^{*}
\end{equation}
that means the matrix formed is also a hermitian matrix. For the
diagonal element of $A_{ii}$, obviously they are real and they are
the mean value for the operator of $\hat{A}$ in the state of $i$. If
the representation is the eigen states for the operator of
$\hat{A}$, then this matrix is in diagonal form.

The non-diagonal element of $A_{ij}$ for $i\neq j$, usually related
to some transition state between two states of $i$ and $j$. For
example, if the $\ket{i}$ is a set of approximated wave functions
for an given molecule by CI procedure (therefore the $\ket{i}$ is
some linear combination of configurations), suppose the $\hat{A}$ is
the Dipole moment operator:
\begin{equation}\label{}
\hat{A} = \sum_{i}e_{i}\overrightarrow{x}_{i} +
\sum_{i}e_{i}\overrightarrow{y}_{i} +
\sum_{i}e_{i}\overrightarrow{z}_{i}
\end{equation}
Here the $e_{i}$ is the electron charge on ith particle (however if
that's the electron wave function the $e_{i} = 1$), and $x_{i}$ etc.
represents the ith particle's cartesian coordinate. therefore, the
non-diagonal element of $A_{ij}$ describes the ``transition'' state
from state $\ket{i}$ to state $\ket{j}$. They are important in
describing the vibration, dipole moment and the external field etc.
phenomenon.

Now we can see that if the $\ket{i} (i=1,2, \cdots n)$ is given to
be certain, by the matrix formed by the $A_{ij}$ we can directly get
$\hat{A}\ket{\Psi}$. Here, if the complete set of $\ket{i}$ has been
chosen to be certain, and the operator of $\hat{A}$ is made sure;
the matrix determined in the above expression is totally fixed. In
the matrix expression, the matrix formed by $A_{ij}$ is equivalent
to the operator itself. Next, we will see this through several
examples.

For the addition between operator $\hat{A}$ and $\hat{B}$, for the
$\Psi=\sum_{i}a_{i}\ket{i}$ similarly we can see that:
\begin{align}\label{REPRESENTATIONeq:7}
(\hat{A}+\hat{B})\ket{\Psi} &= \begin{bmatrix}
     A_{11} & A_{12} & \cdots & A_{1n} \\
     A_{21} & A_{22} & \cdots & A_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     A_{n1} & A_{n2} & \cdots & A_{nn} \\
   \end{bmatrix}
\begin{bmatrix}
  b_{1} \\
  b_{2} \\
  \cdots \\
  b_{n} \\
\end{bmatrix} +
\begin{bmatrix}
     B_{11} & B_{12} & \cdots & B_{1n} \\
     B_{21} & B_{22} & \cdots & B_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     B_{n1} & B_{n2} & \cdots & B_{nn} \\
   \end{bmatrix}
\begin{bmatrix}
  b_{1} \\
  b_{2} \\
  \cdots \\
  b_{n} \\
\end{bmatrix} \nonumber \\
& = \begin{bmatrix}
     A_{11} + B_{11} & A_{12}+B_{12} & \cdots & A_{1n}+B_{1n} \\
     A_{21} + B_{21} & A_{22}+B_{22} & \cdots & A_{2n}+B_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     A_{n1} + B_{n1} & A_{n2}+B_{n2} & \cdots & A_{nn}+B_{nn} \\
   \end{bmatrix}
\begin{bmatrix}
  b_{1} \\
  b_{2} \\
  \cdots \\
  b_{n} \\
\end{bmatrix}
\end{align}

It's natural to see that the addition between two operators is
equivalent to the addition between two matrix. Similarly, the
substraction, multiplication operation between $\hat{A}$ and
$\hat{B}$ are also equivalent to the same operations between the
corresponding matrix. If two operators are commuted with each other,
their corresponding matrix can also commuted with each other.

Next we can show that if two operators of $\hat{A}$ and $\hat{B}$
are commuted with each other, then they share the same eigen states
of $\ket{i}$. The commutation relation between $\hat{A}$ and
$\hat{B}$ can be expressed as:
\begin{equation}\label{}
\sum_{j}A_{ij}B_{jk} = \sum_{j}B_{ij}A_{jk}
\end{equation}
If we choose the $\ket{i}$ as the eigen states for $A$ (that's not
hurting the generality since any other sets can be also expressed as
linear combination based on A's eigen states), then $A_{ij} = 0$ for
$i\neq j$. therefore the above equation can be:
\begin{equation}\label{}
A_{ii}B_{ik} = B_{ik}A_{kk} \Rightarrow B_{ik}(A_{kk} - A_{ii}) = 0
\end{equation}
then if $A_{kk} \neq A_{ii}$ as $i \neq k$; then $B_{ik} = 0$ so
that B is diagonal matrix, too.

the above proof requires that the eigen states should be
non-degenerate, however; if the eigen states are degenerate; then
they forms some subspace. In this subspace, any linear combination
of the eigen states are still the eigen states for the $\hat{A}$,
from the discussion in (\ref{SE:1}) we can know that there's always
some method to find a group of linear combination vectors based on
the original $\ket{i}$ to make $B_{ik} = 0$. Therefore, the
conclusion is correct for any eigen state sets.

Furthermore, for the transposing operation and the conjugating
operation with operators, it can easily show that they are
corresponded to the matrix transposition and conjugation. Thus, the
hermitian operation is equal to the hermitian operation to the
corresponding matrix.

Next let's show how to solve the eigen functions under the matrix
representation.

Suggest that we expanded the $\Psi$ to a set of wave functions
$\ket{i}$, which are the eigen states for the Hamiltonian operator:
\begin{equation}\label{}
\Psi = \sum_{i}a_{i}\ket{i}
\end{equation}
For an arbitrary eigen function of $\hat{f}\Psi = f\Psi$, we have:
\begin{align}\label{REPRESENTATIONeq:8}
\hat{f}\Psi &= f\sum_{i}a_{i}\ket{i} \nonumber \\
\begin{bmatrix}
     f_{11} & f_{12} & \cdots & f_{1n} \\
     f_{21} & f_{22} & \cdots & f_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     f_{n1} & f_{n2} & \cdots & f_{nn} \\
   \end{bmatrix}
\begin{bmatrix}
  a_{1} \\
  a_{2} \\
  \cdots \\
  a_{n} \\
\end{bmatrix}
&= \begin{bmatrix}
     f & 0 & \cdots & 0 \\
     0 & f & \cdots & 0 \\
     \cdots & \cdots & \cdots & \cdots \\
     0 & 0 & \cdots & f \\
   \end{bmatrix}
\begin{bmatrix}
  a_{1} \\
  a_{2} \\
  \cdots \\
  a_{n} \\
\end{bmatrix} \Rightarrow \nonumber \\
\begin{vmatrix}
     f_{11} -f & f_{12} & \cdots & f_{1n} \\
     f_{21} & f_{22} -f & \cdots & f_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     f_{n1} & f_{n2} & \cdots & f_{nn} -f \\
   \end{vmatrix}
&= 0
\end{align}
Obviously this is the matrix eigen function. We can see that the
eigen function for the operator is equivalent to the eigen function
of matrix. The eigen values are the diagonal element in the matrix.
Once this matrix equation is solved, then we can get $n$ eigen
values of $f$ which corresponding to $n$ eigen states; which can be
in turn solved as the linear combination of the coefficients of
$a_{i}$.

By the way, we can also express the Schrodinger equation in the
matrix form. Suggest the Schrodinger equation is:
\begin{equation}\label{}
i\hbar\frac{\partial}{\partial t}\ket{\Psi(\bm{r},t)} =
\hat{H}\ket{\Psi(\bm{r},t)}
\end{equation}
Then we can express the $\ket{\Psi(\bm{r},t)}$ into some
representation of $\ket{i}$:
\begin{equation}\label{}
\ket{\Psi(\bm{r},t)} = \sum_{i}a_{i}(t)\ket{i}
\end{equation}
Here the $\ket{i}$ is a series of stationary states, which does not
contain time $t$. Then bring into back to the Schrodinger equation,
we can have:
\begin{equation}\label{}
i\hbar\sum_{i}\frac{\partial a_{i}(t)}{\partial t}\ket{i}
=\sum_{i}a_{i}(t)\hat{H}\ket{i}
\end{equation}
Then by multiplying with $\bra{j}$ to the left equation and
integration; we can get:
\begin{equation}\label{}
i\hbar\frac{\partial a_{j}(t)}{\partial t}
=\sum_{i}a_{i}(t)\bra{j}\hat{H}\ket{i}
\end{equation}
That is equivalent to the matrix form equation:
\begin{equation}\label{REPRESENTATIONeq:14}
\begin{bmatrix}
  i\hbar\frac{\partial a_{1}(t)}{\partial t} \\
  i\hbar\frac{\partial a_{2}(t)}{\partial t} \\
  \cdots \\
  i\hbar\frac{\partial a_{n}(t)}{\partial t} \\
\end{bmatrix}
=
\begin{bmatrix}
     \bra{1}\hat{H}\ket{1} & \bra{2}\hat{H}\ket{1} & \cdots & \bra{n}\hat{H}\ket{1} \\
     \bra{1}\hat{H}\ket{2} & \bra{2}\hat{H}\ket{2} & \cdots & \bra{n}\hat{H}\ket{2} \\
     \cdots                & \cdots                & \cdots &                \cdots \\
     \bra{1}\hat{H}\ket{n} & \bra{2}\hat{H}\ket{n} & \cdots & \bra{n}\hat{H}\ket{n} \\
   \end{bmatrix}
   \begin{bmatrix}
  a_{1} \\
  a_{2} \\
  \cdots \\
  a_{n} \\
\end{bmatrix}
\end{equation}
That's the Schrodinger equation for an arbitrary representation.
Here if it's the energy representation, then $\ket{i}$ are the eigen
states for the Hamiltonian; so the H matrix above should be
diagonal; if in the other representation, for example; the eigen
states for the momentum, then this matrix is non-diagonal.

Moreover, if the Schrodinger equation is time independent, then the
Schrodinger equation is:
\begin{equation}\label{}
\hat{H}\Psi = E\Psi
\end{equation}
By the similar procedure, we can get its matrix form as:
\begin{equation}\label{REPRESENTATIONeq:15}
\begin{bmatrix}
     \bra{1}\hat{H}\ket{1} & \bra{2}\hat{H}\ket{1} & \cdots & \bra{n}\hat{H}\ket{1} \\
     \bra{1}\hat{H}\ket{2} & \bra{2}\hat{H}\ket{2} & \cdots & \bra{n}\hat{H}\ket{2} \\
     \cdots                & \cdots                & \cdots &                \cdots \\
     \bra{1}\hat{H}\ket{n} & \bra{2}\hat{H}\ket{n} & \cdots & \bra{n}\hat{H}\ket{n} \\
   \end{bmatrix}
   \begin{bmatrix}
  a_{1} \\
  a_{2} \\
  \cdots \\
  a_{n} \\
\end{bmatrix}
=
\begin{bmatrix}
     E & 0 & \cdots & 0 \\
     0 & E & \cdots & 0 \\
     \cdots & \cdots & \cdots & \cdots \\
     0 & 0 & \cdots & E \\
   \end{bmatrix}
   \begin{bmatrix}
  a_{1} \\
  a_{2} \\
  \cdots \\
  a_{n} \\
\end{bmatrix}
\end{equation}
That's the matrix form of time independent Schrodinger equation.

Here it's worthy to compare the (\ref{REPRESENTATIONeq:15}) with the
linear variation process, which is equivalent to the Schrodinger
equation (we have proved this point later in the \ref{SE:2}). In the
linear variation process, the wave function is approximated as the
combination between the trial functions of $\Theta(\bm{r})$:
\begin{equation}\label{}
\Psi(\bm{r}) = \sum_{i}a_{i}\Theta(\bm{r})
\end{equation}
Usually in the quantum chemistry, the $\Theta(\bm{r})$ is composed
as Slater determinants from the molecular orbitals. In the following
chapter (in the discussion of CI algorithm), it can shown that we
can get the same form of equation with (\ref{REPRESENTATIONeq:15});
where the representation of $a_{i}$ is replaced by the space of
trial functions of $\Theta(\bm{r})$.

To some extent, the trial functions of $\Theta(\bm{r})$; is also
some space from which we can create the eigen states for the
Hamiltonian. This process is mathematically same with that to create
the eigen states for the Hamiltonian from the other representations
(such as the representation for the momentum) in the
(\ref{REPRESENTATIONeq:14}). Hence, the linear variation process can
be also ascribed to matrix representation.

Finally, for the expectation value for the operator $\hat{A}$, we
can express it in the matrix form as:
\begin{align}\label{REPRESENTATIONeq:9}
\bra{\Psi}\hat{A}\ket{\Psi} &=
\sum_{i}\sum_{j}a^{*}_{i}a_{j}\bra{i}\hat{A}\ket{j} \nonumber
\\
&=
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix}
\begin{bmatrix}
     A_{11} & A_{12} & \cdots & A_{1n} \\
     A_{21} & A_{22} & \cdots & A_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     A_{n1} & A_{n2} & \cdots & A_{nn} \\
   \end{bmatrix}
\begin{bmatrix}
  a_{1} \\
  a_{2} \\
  \cdots \\
  a_{n} \\
\end{bmatrix}
\end{align}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Density operator in matrix form}
%
% discussion of the density operator in matrix form, definition and
% its properties
%
In the above content, for an arbitrary quantum state expressed by
some specific representation, it can be generally as:
\begin{equation}
\ket{\Psi} \Leftrightarrow
                           \begin{bmatrix}
                             a_{1} \\
                             a_{2} \\
                             \cdots   \\
                             a_{n} \\
                           \end{bmatrix}
\end{equation}

\begin{equation}\label{}
\bra{\Psi} \Leftrightarrow \begin{bmatrix}
                             a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
                           \end{bmatrix}
\end{equation}
Here $\langle\Psi|\Psi\rangle$ gives its internal product. However,
we can reverse the internal product as $\ket{\Psi}\bra{\Psi}$, from
its matrix form we can see that it's correspond to some kind of
operator; that is:
\begin{align}\label{REPRESENTATIONeq:10}
\hat{D} = |\Psi\rangle\langle\Psi| &=
\begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix} \nonumber \\
&=
\begin{bmatrix}
a_{1}a_{1}^{*} & a_{1}a_{2}^{*} & \cdots & a_{1}a_{n}^{*} \\
a_{2}a_{1}^{*} & a_{2}a_{2}^{*} & \cdots & a_{2}a_{n}^{*} \\
\cdots         & \cdots         & \cdots & \cdots         \\
a_{n}a_{1}^{*} & a_{n}a_{2}^{*} & \cdots & a_{n}a_{n}^{*} \\
\end{bmatrix}
\end{align}
This kind of matrix is just the density operator we have defined in
the previous chapter (\ref{OPERATOR:1}), now here let's go to see
how to use matrix to re-express its characters.

\begin{theorem}\label{}
$\hat{D}$ is hermitian.
\end{theorem}

\begin{proof}
\begin{equation}\label{}
\begin{split}
  \hat{D}^{+} &=
\left(  \begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix} \right)^{+}\\
    &= \begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix} \\
&= \hat{D}^{+}
\end{split}
\end{equation}
\qedhere
\end{proof}

\begin{theorem}\label{}
For any $\Phi$, $\langle\Phi|\hat{D}|\Phi\rangle \geq 0$.
\end{theorem}

\begin{proof}
Suggest that the $\ket{\Phi}$ can be expressed as:
\begin{equation}\label{}
\ket{\Phi} \Leftrightarrow
\begin{bmatrix}
b_{1} \\
b_{2} \\
\cdots   \\
b_{n} \\
\end{bmatrix}
\end{equation}
Then we have:
\begin{equation}\label{}
\begin{split}
  \langle\Phi|\hat{D}|\Phi\rangle &=
  \begin{bmatrix}
b_{1}^{*} & b_{2}^{*} & \cdots & b_{n}^{*} \\
\end{bmatrix}
  \begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix}
  \begin{bmatrix}
b_{1} \\
b_{2} \\
\cdots   \\
b_{n} \\
\end{bmatrix} \\
    &=(\sum_{i=1}^{n}b_{i}^{*}a_{i})(\sum_{i=1}^{n}a_{i}^{*}b_{i})
    \\
    &=|\sum_{i=1}^{n}b_{i}^{*}a_{i}|^{2} \geq 0
\end{split}
\end{equation}
\qedhere
\end{proof}

\begin{theorem}\label{}
$\hat{D}^{2} = \hat{D}$.
\end{theorem}

\begin{proof}
\begin{equation}\label{}
\begin{split}
  \hat{D}^{2} &= \begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix}
\begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix}  \\
    &= \begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix}  \quad (\langle\Psi|\Psi\rangle = 1)\\
&= \hat{D}
\end{split}
\end{equation}
So $\hat{D}$ is idempotent. \qedhere
\end{proof}

\begin{theorem}\label{}
$\hat{D}\hat{B} = \hat{B}\hat{D}$. If $\hat{B}$ is hermitian.
\end{theorem}

\begin{proof}
Suggest that $\hat{B}$ is represented by some $n\times n$ matrix
with element of $B_{ij}$, then we have:
\begin{equation}\label{}
\begin{split}
  \hat{D}\hat{B} &=
\begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix}
\begin{bmatrix}
     B_{11} & B_{12} & \cdots & B_{1n} \\
     B_{21} & B_{22} & \cdots & B_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     B_{n1} & B_{n2} & \cdots & B_{nn} \\
   \end{bmatrix} \\
    &=
    \begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
\sum_{k=1}^{n}a_{k}^{*}B_{k1} & \sum_{k=1}^{n}a_{k}^{*}B_{k2}
 & \sum_{k=1}^{n}a_{k}^{*}B_{kn} \\
\end{bmatrix}
\end{split}
\end{equation}
Hence we can get that $(\hat{D}\hat{B})_{ij} =
a_{i}\sum_{k=1}^{n}a_{k}^{*}B_{kj}$.

On the other hand, since that $\hat{B}$ and $\hat{D}$ are both
hermitian, then we have:
\begin{equation}\label{}
\begin{split}
  (\hat{D}\hat{B})_{ij} &= (\hat{B}\hat{D})^{*}_{ji} \\
    &= a^{*}_{j}\sum_{k=1}^{n}a_{k}B^{*}_{jk} \Rightarrow \\
   \hat{D}\hat{B} &=
\begin{bmatrix}
     B_{11} & B_{12} & \cdots & B_{1n} \\
     B_{21} & B_{22} & \cdots & B_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     B_{n1} & B_{n2} & \cdots & B_{nn} \\
   \end{bmatrix}
\begin{bmatrix}
a_{1} \\
a_{2} \\
\cdots   \\
a_{n} \\
\end{bmatrix}
\begin{bmatrix}
a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix}
\end{split}
\end{equation}
 \qedhere
\end{proof}

Now let's consider the project operator:
\begin{equation}\label{REPRESENTATIONeq:18}
\hat{P}_{i} = \ket{i}\bra{i}
\end{equation}
For each vector of $\ket{i}$, we can express it into the matrix as:
\begin{equation}\label{REPRESENTATIONeq:16}
\ket{i} = \begin{bmatrix}
            0 \\
            0 \\
            \cdots \\
            1 \\
            \cdots \\
            0 \\
          \end{bmatrix}
\end{equation}
Here in the column vector the $1$ is in the ith row position.
Actually that's the ith unit vector. Based on the
(\ref{REPRESENTATIONeq:16}), we can express the
(\ref{REPRESENTATIONeq:18}) as:
\begin{align}\label{REPRESENTATIONeq:19}
\hat{P}_{i} &=
\begin{bmatrix}
0 \\
0 \\
\cdots \\
1 \\
\cdots \\
0 \\
\end{bmatrix}
\begin{bmatrix}
  0 & 0 & \cdots & 1 & \cdots & 0 \\
\end{bmatrix} \nonumber \\
&=
\begin{bmatrix}
  0      & 0      & \cdots & 0      & \cdots & 0    \\
  0      & 0      & \cdots & 0      & \cdots & 0     \\
  \cdots & \cdots & \cdots & \cdots & \cdots & \cdots \\
  0      & 0      & \cdots & 1      & \cdots & 0      \\
  \cdots & \cdots & \cdots & \cdots & \cdots & \cdots \\
  0      & 0      & \cdots & 0      & \cdots & 0      \\
\end{bmatrix}
\end{align}
Here in the (\ref{REPRESENTATIONeq:19}) the matrix simply has only
one element of $1$ in the ith diagonal position. It's easy to see
that for an arbitrary state of $\ket{\Psi}$, where it can be
expressed as:
\begin{equation}\label{}
\ket{\Psi} = \sum_{i}a_{i}\ket{i}
\end{equation}
The projector operator on the $\ket{\Psi}$ is:
\begin{equation}\label{}
\hat{P}_{i}\ket{\Psi} = a_{i}
\end{equation}
Where the ith weight in the $\ket{\Psi}$ has been projected out.

Based on the $\hat{\hat{P}_{i}}$ we can also express the closure
relation as:
\begin{equation}\label{}
\hat{P} = \sum_{i=1}^{n}\hat{P}_{i} = \hat{I}
\end{equation}
Now through the matrix form let's demonstrate this point.

For each $\hat{P}_{i}$, it's expressed by some $n\times n$ matrix
with only one element of $1$ in row $i$ column $i$. Hence the adding
of all the $\hat{P}_{i}$ leads to some matrix that the diagonal line
is filled with $1$, and other elements are all equal to $0$. That's
the unit matrix of $\hat{I}$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Representation and its transformation}
\label{transformation_in_representation}
%
% 1  why we have different choice of representations?
% 2  prove that any representations are identical with
%    each other
%
%
In the above content, we can see that if a set of basis functions
has been chosen, any other vectors and the operators can be
thoroughly expressed from it.

This set of basis functions are certainly determined by it's CSCO.
Just take hydrogen atom as an example, $(\hat{H}, \hat{l}^{2},
\hat{l}_{z})$ and $(\hat{H}, \hat{l}^{2}, \hat{l}_{x})$ will
definitely yield different set of wave functions (this is easy to
understand. Since that $[\hat{l}_{x}, \hat{l}_{z}] \neq 0$, their
eigen states must be different with each other).

Because that for a given system, we may have different choice of
CSCO which lead to different set of basis functions; so some
question arise: Are different representations are physically
identical with each other? For a specific operator whether they can
yield the same measurement result? That's we are going to discuss in
this section.

Suggest that we have two representations for the system: the first
set is ${\ket{\phi_{1}}, \ket{\phi_{2}}, \cdots, \ket{\phi_{n}}}$,
the other is ${\ket{\varphi_{1}}, \ket{\varphi_{2}}, \cdots,
\ket{\varphi_{n}}}$.

Thus for any arbitrary wave function of $\Psi$, we have:
\begin{align}\label{}
\ket{\Psi} &= \sum_{i}a_{i}\ket{\phi_{i}} \nonumber \\
\ket{\Psi} &= \sum_{j}b_{j}\ket{\varphi_{j}}
\end{align}

So $\sum_{i}a_{i}\ket{\phi_{i}} = \sum_{j}b_{j}\ket{\varphi_{j}}$,
by multiplying with $\bra{\phi_{i}}$ and make integration; we have:
\begin{align}\label{REPRESENTATIONeq:12}
a_{i} &= \sum_{j}\langle\phi_{i}|\varphi_{j}\rangle b_{j}
\Rightarrow \nonumber \\
   \begin{bmatrix}
     a_{1} \\
     a_{2} \\
     \cdots \\
     a_{n} \\
   \end{bmatrix} &=
\begin{bmatrix}
     s_{11} & s_{12} & \cdots & s_{1n} \\
     s_{21} & s_{22} & \cdots & s_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     s_{n1} & s_{n2} & \cdots & s_{nn} \\
   \end{bmatrix}
      \begin{bmatrix}
     b_{1} \\
     b_{2} \\
     \cdots \\
     b_{n} \\
   \end{bmatrix}
\end{align}
Here the $s_{ij} = \langle\phi_{i}|\varphi_{j}\rangle$, and
obviously the $S$ matrix is some unitary matrix, it's easy to prove
that $S^{+}S = SS^{+} = I$.

The matrix of $S$ is carried out to transform the representation of
$\ket{\varphi}$ to $\ket{\phi}$. Because such transformation is
unitary, Therefore, it can be expected that the transformation
between different representations will not change the observable
measurement values for the dynamic operator.

For an arbitrary operator $\hat{A}$, it's expectation value under
representation under $\ket{\phi}$ representation is:
\begin{equation}\label{REPRESENTATIONeq:13}
\bra{\Phi}\hat{A}\ket{\Phi} =
\begin{bmatrix}
  a_{1}^{*} & a_{2}^{*} & \cdots & a_{n}^{*} \\
\end{bmatrix}
   \begin{bmatrix}
     A_{11} & A_{12} & \cdots & A_{1n} \\
     A_{21} & A_{22} & \cdots & A_{2n} \\
     \cdots & \cdots & \cdots & \cdots \\
     A_{n1} & A_{n2} & \cdots & A_{nn} \\
   \end{bmatrix}
   \begin{bmatrix}
     a_{1} \\
     a_{2} \\
     \cdots \\
     a_{n} \\
   \end{bmatrix}
\end{equation}

If we insert the (\ref{REPRESENTATIONeq:12}) into the above
expression, it can be transformed into:
\begin{align}\label{REPRESENTATIONeq:17}
\bra{\Phi}\hat{A}\ket{\Phi}_{(\phi)} &=
\begin{bmatrix}
  b_{1}^{*} & b_{2}^{*} & \cdots & b_{n}^{*} \\
\end{bmatrix}
S^{+}AS
   \begin{bmatrix}
     b_{1} \\
     b_{2} \\
     \cdots \\
     b_{n} \\
   \end{bmatrix} \nonumber \\
   &=\bra{\Phi}\hat{A}\ket{\Phi}_{(\varphi)}
\end{align}
Here the $S$ is the unitary transformation matrix between
$\ket{\varphi}$ and $\ket{\phi}$. Thus different representation will
not alter the expectation value of an operator.

All in all, representations are physically identical with each
other. So we can choose a proper one which is suitable for
investigation and extract all the quantum information needed from
it.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../../main"
%%% End:


We investigated two practical architectures for the co-located integrated information and energy receiver. Both designs are equipped with individual ID and EH receivers. The former is a conventional baseband demodulator while the latter can be implemented with the proposed rectifier structure in Section \ref{sec:rectenna-design}.

\subsection{Time Switching}\label{sec:time-switching}
A \textit{Time Switching} (TS) receiver (Fig. \ref{fig:ts-receiver}) operates as either an information decoder or an energy harvester at a fixed time. In the design, the transmitter divides the transmission block into orthogonal power and data slots with length ratios $\alpha $ and $1 - \alpha $ respectively. It then optimizes the waveform for WPT or WIT individually. On the other hand, the receiver periodically switches between ID and EH receivers in the corresponding slots. Perfect synchronization between transmitter and receiver is required for precise mode control. It can achieve different rate-energy tradeoffs by adjusting the slot length ratio $\alpha $ jointly with the transmit signals. Since the information decoder and energy harvester may work in different power ranges, TS can be combined with a "near-far" scheduling \cite{Zhang2013} to benefit the system efficiency.

\begin{figure}[ht]
  \centering
    \includegraphics[width=\textwidth]{ts_receiver}
  \caption{Structure of TS receiver \cite{Clerckx2019}}
  \label{fig:ts-receiver}
\end{figure}

\subsection{Power Splitting}\label{sec:power-splitting}
In \textit{Power Splitting} (PS) receiver (Fig. \ref{fig:ps-receiver}), we introduce a PS ratio $\rho $ to split the received signal into individual power stream (with proportion $\rho $) and information stream (with proportion $1 - \rho $). At the transmitter, the signal is jointly optimized for information and power transmission according to CSIT. When perfectly matched, the EH and ID receivers are with input voltage $\sqrt {\rho {R_{{\text{ant}}}}} y(t)$ and $\sqrt {(1 - \rho ){R_{{\text{ant}}}}} y(t)$ respectively. Different rate-energy pairs can be obtained by varying the PS ratio $\rho $. It is argued in \cite{Zhang2013} that PS is the best strategy for linear harvester model with negligible RF-to-baseband noise, but \cite{Clerckx2016} demonstrated that the PS-only transmission is suboptimal when considering rectifier nonlinearity.

\begin{figure}[ht]
  \centering
    \includegraphics[width=0.8\textwidth]{ps_receiver}
  \caption{Structure of PS receiver \cite{Clerckx2019}}
  \label{fig:ps-receiver}
\end{figure}


\section{Planning}
\frame{\tableofcontents[currentsection, hideothersubsections]}

\begin{frame}
\frametitle{How to control a robot from START to +1 (GOAL)?}
\begin{figure}
    \centering
    \includegraphics[scale=0.25]{grid_world_only}
\end{figure}
\pause

\begin{itemize}
  \item Wall following? right or left? who decides? \pause
  \item What if START can be in any grid? \pause
  \item What if actions are uncertain?
\end{itemize}
\end{frame}

\begin{frame}
\frametitle{How to control a robot from START to +1 (GOAL)?}
\begin{figure}
    \centering
    \includegraphics[scale=0.25]{grid_world_with_transition}
\end{figure}
\pause

\begin{itemize}
  \item Given environment model, one possible solution: \\
  \textbf{MDP planning}: sequential decision making under action uncertainty
\end{itemize}
\end{frame}

\begin{frame}
\frametitle{MDP: Markov Decision Process}
\begin{figure}
    \centering
    \includegraphics[scale=0.25]{agent_environment_rl_intro_p60}
\end{figure}
\pause
\begin{columns}
  \column{0.5\textwidth}
    $MDP = (S, A, T, R, \gamma)$ ... the model \pause
    \begin{itemize}
      \item $S$ is a set of states
      \item $A$ is a set of actions \pause
      \item $T(s,a,s') = P(s'| s, a)$ \\ is a transition function \pause
      \item $r:(s,a,s') \mapsto \mathbb{R}$ \\ is a reward function \pause
      \item $\gamma \in [0,1]$ is a discount factor
    \end{itemize}
    \pause
  \column{0.5\textwidth}
    \textbf{Agent's goal in MDP planning:}\\
    to maximize the return, $\mathcal{R}$,\\
    (i.e. the expected cumulative reward) \\
    \begin{align*}
      \mathcal{R}_t & = r_{t+1} + \gamma~r_{t+2} + \gamma^2~r_{t+3} + \ldots  \\
      & = \sum_{k=0}^{\infty} \gamma^k~r_{t+k+1}
    \end{align*}
\end{columns}
\end{frame}

\begin{frame}
\frametitle{MDP Planning: Value Function and Policy (Plan)}
\textbf{(State) value function:} the goodness of a state
\begin{align*}
V(s) & = \mathbb{E}[\mathcal{R}_t | S_t = s] \\
& = \mathbb{E} [r_{t+1} + \gamma~r_{t+2} + \gamma^2~r_{t+3} + \ldots | S_t = s] \\
& = \mathbb{E} [r_{t+1} + \gamma~ (r_{t+2} + \gamma~r_{t+3} + \ldots | S_t = s)] \\
& = \mathbb{E} [r_{t+1} + \gamma~ (\mathcal{R}_{t+1} | S_t = s)] \\
& = \mathbb{E} [r_{t+1} + \gamma~V(S_{t+1}| s)]
\end{align*}
\pause

\textbf{A policy (plan)} maps every state to an action, $\pi: s \mapsto a$, \\
generally, $\pi (a|s) = P[A_t = a|S_t = s]$
\end{frame}

\begin{frame}
\frametitle{MDP Planning: From Value Function To Policy}
$\pi = greedy(V)$ \\
Recall: the agents' goal is to maximize the expected cumulative return

\begin{columns}
  \column{0.5\textwidth}
    \begin{figure}
        \centering
        \includegraphics[scale=0.25]{value_function_example_silver_course}
        \caption{a (state) value function}
    \end{figure}

  \column{0.5\textwidth}
    \begin{figure}
        \centering
        \includegraphics[scale=0.20]{policy_example_silver_course}
        \caption{a policy (plan)}
    \end{figure}
\end{columns}
\end{frame}

\begin{frame}
\frametitle{MDP Planning: Value Iteration (Dynamic Programming)}
\begin{figure}
    \centering
    \includegraphics[scale=0.4]{value_iter_rl_intro_p92}
\end{figure}
\end{frame}

\begin{frame}
\frametitle{MDP Planning: Value Iteration Example}
\begin{figure}
    \centering
    \includegraphics[scale=0.3]{value_iter_shortest_path_silver_course_2}
\end{figure}
\end{frame}

\begin{frame}
\frametitle{MDP Planning}
Iterations toward an optimal policy (plan)
\begin{figure}
    \centering
    \includegraphics[scale=0.35]{gpi_rl_intro_p96}
\end{figure}
\end{frame}

\begin{frame}
\frametitle{MDP Planning}
\begin{figure}
    \centering
    \includegraphics[scale=0.35]{policy_grid_world_aima_p832}
\end{figure}
\end{frame}


\documentclass{memoir}
\usepackage{notestemplate}

% \begin{figure}[ht]
%     \centering
%     \incfig{riemmans-theorem}
%     \caption{Riemmans theorem}
%     \label{fig:riemmans-theorem}
% \end{figure}

\begin{document}

\section{Group Actions}

By this point, the reader should have a solid foundation in the basic ideas of group theory. There are richer details to follow, but some of them rely on more advanced ideas in algebra. In particular, group actions are fundamentally connected to permutation groups and group presentations. More details on these topics can be found elsewhere, but one can also get a basic understanding of group actions without this prerequisite knowledge.

\begin{defn}[(Left) Action]
	A (left) \textbf{action} of a group \(G\) on a set \(\Omega\) is a function \(\mu:G \times \Omega  \to \Omega\) with the following two properties:
	\begin{itemize}
		\item \(\mu(g_1,\mu(g_2,x)) = \mu(g_1g_2,x)\) for all \(x \in \Omega\), \(g_1,g_2 \in G\).
		\item \(\mu(e,x) = x\) for all \(x \in \Omega\).
	\end{itemize}
\end{defn}
It immediately follows that \(\mu(g^{-1},\mu(g,x)) = \mu(g,\mu(g^{-1},x)) = x\) for all \(x \in \Omega, g \in G\).\\

While \(\mu (e,x) = x\), it doesn't have to be the only element which does so. The \textbf{kernel} of the action is the set of elements of \(G\) that act trivially on \(\Omega \):
\begin{align*}
	\textrm{Ker}\mu  = \left\{g \in G \mid g \cdot x = x \quad \forall x \in \Omega  \right\}
\end{align*}
If this set is trivial, then we say the action is \textbf{faithful}. Regardless, the kernel forms a normal subgroup, as we will see later.

\begin{prop}
	\begin{itemize}
		\item For any \(g \in G\), the map \(\sigma _g:\Omega\to \Omega\) defined by \(\sigma _g x = \mu(g,x)\) is a permutation.
		\item The map \(\theta:G\to S_n\) defined by \(\theta (g) = \sigma _g\) is a homomorphism (where \(S_n\) is the set of permutations of \(\Omega\), so \(n = \left| \Omega \right| \) )
		\item Conversely, given a homomorphism \(\theta:G\to S_n\), there is an action \(\mu\) of \(G\) on \(\Omega\) given by \(\mu(g,x) = \theta (g)x\).
	\end{itemize}
\end{prop}

Viewing group actions via permutations is a good way to get a grasp of group actions. One can view group actions of \(G\) on \(\Omega \) as each element \(g \in G\) permuting the set \(\Omega \). We call the homomorphism \(G\to S_{\Omega }\) given by \(\theta (g) = \sigma_g\) to be the \textbf{permutation representation} associated to the given action.

\begin{exmp}
	\begin{itemize}
		\item Let \(H\) be a subgroup of \(G\). Let \(\Omega\) be the set of all left cosets of \(H\) in \(G\) (written as \(gH\) for some \(g \in G\)). Define an action by \(\mu(g,g'H) = (gg')H\). This is the action of \textbf{left multiplication}.
		\item Define an action of \(G\) on itself (\(\Omega = G\) ) by the rule \(\mu(g,x) = gxg^{-1}\). This is the action of \textbf{conjugation}.
		\item Let \(\Omega\) be the set of all subgroups of \(G\). Then \(G\) acts on \(\Omega\) by \textbf{conjugation}: \(\mu(g,H) = gHg^{-1}\).
	\end{itemize}
\end{exmp}
An equivalence relation on \(\Omega\) is formed by a group action via the rule that \(x \sim y\) if there exists \(g \in G\) with \(\mu(g,x) = y\). The equivalence classes are called \textbf{orbits}. The set \(\Omega\) decomposes into a disjoint union of orbits.
\begin{defn}[Transitivity]
	We say that an action is \textbf{transitive} if there is just one orbit, and \textbf{intransitive} otherwise.
\end{defn}
Left multiplication is transitive, but conjugation is in general not. The orbits for conjugation of \(G\) onto itself are the \textbf{conjugacy classes} of \(G\).
\begin{defn}[Stabilizer]
	The \textbf{stabilizer} of an element \(x \in \Omega\) is the set
	\begin{align*}
		\left\{g \in G \mid \mu(g,x) = x \right\}
	\end{align*}
	of elements of \(G\) for which the corresponding permutation fixes \(x\). It is denoted \(G_x\).
\end{defn}
Notice that the union of all stabilizers on \(\Omega \) is exactly the kernel of the action.

\begin{exmp}
	Let \(A\) be a subset of \(G\). Consider the action of \(G\) on \(A\) by conjugation, i.e. \(\mu (g,a) = gag^{-1}\). Then the stabilizer of an element \(a \in A\) is called the \textbf{centralizer of \(a\)} denoted by \(C_G(a)\). Considering the entire subset \(A\), we define
	\begin{align*}
		C_G(A) = \left\{g \in G \mid gag^{-1} = a \forall a \right\}
	\end{align*}
	to be the \textbf{centralizer of \(A\)}.\\

	It turns out that \(C_G(A) \leq G\) is a subgroup.
\end{exmp}
One will find that abelian groups prove not to be good examples for centralizers, as one will quickly see that in an abelian group \(G\), \(C_G(A) = G\) for all subsets \(A\). However, one can check that
\begin{align*}
	C_{Q_8}(i) = \left\{ \pm 1, \pm i \right\} .
\end{align*}
There are some similar subgroups that will be of interest soon.\\

Recall the center subgroup of \(G\) denoted by
	\begin{align*}
		Z(G) = \left\{g \in G \mid gx = xg \; x \in G \right\} .
	\end{align*}
The center plays an important role with these new ideas, as one can see that \(Z(G) = C_G(G)\) (and hence we already know that \(Z(G)\) is a subgroup).\\

One might recall that we also defined conjugation by subgroups earlier. This also corresponds to a variation of a centralizer.

\begin{defn}[Normalizer]
	Let \(A\) be a subset of \(G\), and consider the action of \(G\) on \(A\) by coset conjugation, i.e. \(\mu (g,A) = gAg^{-1} = \left\{gag^{-1}  \mid a \in A \right\} \). Then the set of elements which fix \(A\) is called the \textbf{normalizer} of \(A\) in \(G\), denoted by
	\begin{align*}
		N_G(A) = \left\{g \in G \mid gAg^{-1} = A \right\} .
	\end{align*}
\end{defn}
This is actually a larger subgroup containing the centralizer-- if \(g \in C_G(A)\), then \(gag^{-1} = a \in A\), and so \(C_G(A) \leq N_G(A)\). One can show that \(N_G(A)\) is a subgroup as well.

\begin{prop}
	Let \(N\leq G\) be a subgroup of \(G\). Then \(N \triangleleft G\) is a normal subgroup of \(G\) if and only if \(N_G(N) = G\).
\end{prop}

There are some other nice properties of subgroups that come about from the normalizer.

\begin{prop}
	If \(H,K \leq G\) are subgroups of \(G\), and \(H\leq N_G(K)\), then \(HK \leq G\).\\

	Thus, if \(K \triangleleft G\) then \(HK \leq G\) for any \(H\leq G\).
\end{prop}

\begin{defn}
	If \(A\) is any subset of \(N_G(K)\), we say \(A\) \textbf{normalizes} \(K\).
\end{defn}

\begin{exmp}
	Consider \(G = S_4\) and \(H = D_8\). Let \(K = \langle (123) \rangle \). We can view \(D_8\) as a subgroup of \(S_4\) by identifying each symmetry by its permutation on the four vertices. Lagrange's Theorem tells us that \(H\cap K = 1\) (as their orders are relatively prime), and hence \(\left| HK \right| =24\) and hence \(HK = S_4\).\\

	Furthermore, \(H\) nor \(K\) normalizes the other.
\end{exmp}
It is worthwhile to notice that if we take \(S = \mathcal{P}(G)\), then the action of \(G\) on \(S\) by coset conjugation admits a stabilizer on its elements \(A\) by \(N_G(A)\).\\

Finally, we can also let \(N_G(A)\) act on \(A\) by conjugation \(a\mapsto gag^{-1}\). Then the centralizer \(C_G(A)\) is precisely the kernel of this action, and so we have now conceptualized these main definitions via group actions.\\

Now we will state some theorems that formalize these ideas.

\begin{thm}[Second Isomorphism Theorem]
	Let \(G\) be a group with \(A,B \leq G\). Assume \(A \leq N_G(B)\). Then
\begin{itemize}
	\item \(AB \leq G\)
	\item \(B \triangleleft AB\)
	\item \(A\cap B \triangleleft A\)
	\item \(AB / B \cong A / (A\cap B)\)
\end{itemize}
\end{thm}
Note that \(AB / A\) is not necessarily a group, i.e. \(A\) is not necessarily normal in \(AB\).

\begin{thm}[Orbit-Stabilizer Theorem]
	Given an action of \(G\) onto \(\Omega\), and \(x \in \Omega\), the stabilizers \(G_x\) form a subgroup of \(G\). Furthermore, there is a bijection between the orbit of \(x\) and the set of left cosets of \(G_x\) in \(G\) given by
	\begin{align*}
		\mu^{i}x \mapsto \mu^{i}G_x.
	\end{align*}
\end{thm}
If \(G\) is finite, the size of the orbit of \(x\) is equal to \( \left| G:G_x \right| = \left| G \right| / \left| G_x \right| \).

\begin{cor}
	Every transitive action is isomorphic to an action by left multiplication on the left cosets of a subgroup. Furthermore, the actions on the left cosets of two subgroups \(H,K\) are isomorphic if and only if \(H,K\) are conjugate.
\end{cor}

Note: Let \( \textrm{fix}(g)\) denote the number of elements in \(\Omega\) that are mapped to themselves when \(g\) is applied to them as an action.

\begin{thm}[Orbit-Counting Lemma]
	The number of orbits of \(G\) on \(\Omega\) is given by
	\begin{align*}
	\frac{1}{\left| G \right| } \sum_{g\in G} \textrm{fix}(g).
\end{align*}
\end{thm}

\begin{lemma}[Burnside's Lemma]
	The number of orbits is equal to the average number of fixed points. We can write this by
	\begin{align*}
		\left| G \right| \cdot \text{(number of orbits)} = \sum_{g\in G} \left| S^{g} \right|
	\end{align*}
\end{lemma}

We can get some interesting results by considering the action of \(G\) on itself, or subsets of itself. This corresponds to the notion of permutations of a group.

\begin{thm}
	Let \(H\leq G\) be a subgroup and let \(G\) act by left multiplication on the set \(A\) of left cosets of \(H\) in \(G\). We denote this action by \(\sigma_H\), the associated permutation representation. Then
	\begin{itemize}
		\item \(G\) acts transitively on \(A\)
		\item The stabilizer in \(G\) of \(eH \in A\) is the subgroup \(H\)
		\item The kernel of \(\sigma_H\) is \(\bigcap_{x \in G} xHx ^{-1}\), and \(\textrm{Ker}\sigma_H\) is the largest normal subgroup of \(G\) contained in \(H\)
	\end{itemize}
\end{thm}

\begin{cor}[Jordan's Theorem]
\begin{itemize}
	\item Let \(G\) act transitively on the finite set \(\Omega\), where \(\left| \Omega \right| >1\). Then there is an element of \(G\) which fixes no point of \(\Omega\).
	\item Let \(H\) be a proper subgroup of a finite group \(G\). Then
		\begin{align*}
			\bigcup_{g \in G} g^{-1}Hg \neq G.
		\end{align*}
\end{itemize}
\end{cor}

\begin{thm}[Cayley's Theorem]
	Every group of size \(n\) is isomorphic to a subgroup of \(S_n\).
\end{thm}
\begin{proof}
	Let \(S_n\) be all permutations of the group \(G = \left\{ e,g_2,\ldots,g_n \right\} \) and define \(\varphi:G\to S_n\) by \(g \mapsto {g_i}\choose{gg_i}\). In other words, we assign to \(g\in G\) the permutation of \(G\) onto itself; we multiply every \(g_i\) by the given \(g\) from the left. One can check that \(\varphi\) is an injective homomorphism, and so \(G \cong \textrm{Image}(\varphi)\leq S_n\).
\end{proof}

Cayley's theorem can also be seen as a consequence of the above theorem, as one can simply take \(H\) to be trivial to get the result.
\begin{prop}
	If \(G\) is a finite group with \(\left| G \right| =n\), and \(p\) is the smallest prime that satisfies \(p\mid n\), then any subgroup of index \(p\) is normal.
\end{prop}
This can be proven via Cayley's Theorem.

\subsection{Conjugacy Classes}
\label{sub:conjugacy_classes}

Earlier, we obtained interesting results such as Cayley's theorem by having \(G\) act on itself via left multiplication. We can also get some interesting results by letting \(G\) act on itself by conjugation:
\begin{align*}
	g \cdot a = gag^{-1} \quad \forall g,a \in G.
\end{align*}

\begin{defn}[Conjugates]
	Two elements \(a,b \in G\) are said to be \textbf{conjugate in \(G\)} if there is some \(g \in G\) such that \(b = gag^{-1}\). The orbits of \(G\) acting on itself by conjugation are called the \textbf{conjugacy classes of \(G\)}.
\end{defn}
One can clearly see that \(a,b\) are conjugate in \(G\) if they are contained in the same orbit.

\begin{exmp}
	Observe that if \(G\) is abelian, then the action of \(G\) on itself by conjugation is trivial, and hence not interesting.\\

If \(G\) is non-trivial, then \(G\) will never act transitively on itself by conjugation. This is because the identity will always have its own orbit.\\

As an example, the conjugacy classes of \(S_3\) are
\begin{align*}
	\left\{ 1 \right\} , \left\{ (12),(13),(23) \right\} , \left\{ (123),(132) \right\}
\end{align*}
\end{exmp}

\(G\) can act on subsets of itself by conjugation, as well. We define this for \(S\subset G\) by:
\begin{align*}
	gSg^{-1} = \left\{gsg^{-1} \mid s \in S \right\} .
\end{align*}
We can even use this to define a group action on a higher level-- that is, we can define an action of \(G\) on \(\mathcal{P}(G)\) by \(g\cdot S\).

\begin{defn}[Set Conjugates]
	Two subsets \(S,T \subset G\) are said to be \textbf{conjugate in \(G\)} if there is some \(g \in G\) such that \(T = gSg^{-1}\), i.e. they are in the same orbit of \(G\) acting on its subsets by conjugation.
\end{defn}

Our previous propositions give us the index of these conjugates within the group:
\begin{prop}
	The number of conjugates of \(S\subset G\) is the index of the normalizer of \(S\)
	\begin{align*}
		\left| G : N_G(S) \right| .
	\end{align*}
	The number of conjugates of an element \(s\) of \(G\) is hence
	\begin{align*}
		\left| G:C_G(s) \right| .
	\end{align*}
\end{prop}

\begin{thm}
	Let \(G\) be a finite group and let \(g_1,\ldots,g_n\) be representatives of the distinct conjugacy classes of \(G\) not contained in the center \(Z(G)\leq G\). Then
	\begin{align*}
		\left| Z(G) \right| + \sum_{i=1}^{n} \left| G : C_G(g_i) \right| = \left| G \right| .
	\end{align*}
\end{thm}
This essentially partitions the order of \(G\) into its abelian and non-abelian parts.

This has a lot of useful consequences naturally. For example, we can use this to help classify groups of prime power order.

\begin{cor}
	Let \(G\) be a group with \(\left| G \right| = p^{\alpha }\) for some \(p\) prime, \(\alpha \geq 1\). Then \(G\) has a nontrivial center.
\end{cor}
\begin{proof}
	The class equation tells us that
	\begin{align*}
		\left| G \right|  = \left| Z(G) \right| + \sum_{i=1}^{n} \left| G:C_G(g_i) \right|
	\end{align*}
	Because \(C_G(g_i)\) are non-central conjugacy classes they cannot be the full group \(G\). Thus, \(p\mid \left| G:C_G(g_i) \right| \), and hence divides the sum. Thus \(p\mid \left| Z(G) \right| \), proving the result desired.
\end{proof}

\begin{cor}
	If \(\left| G \right| = p^2\) for some prime \(p\), then \(G\) is abelian and isomorphic to \(\Z_{p^2}\) or \(\Z_p \times \Z_p\).
\end{cor}

Now we will look at the specific case of \(S_n\).

\begin{prop}
	Let \(\sigma ,\tau \) be elements of the symmetric group \(S_n\) and suppose \(\sigma \) has cycle decomposition
	\begin{align*}
		(a_1a_2\ldots a_{k_1})(b_1b_2\ldots b_{k_2})\ldots
	\end{align*}
	Then \(\tau \sigma \tau ^{-1}\) has cycle decomposition
	\begin{align*}
		(\tau (a_1)\tau (a_2)\ldots \tau (a_{k_1})) \ldots (\tau (b_1) \tau (b_2) \ldots \tau (b_{k_2})).
	\end{align*}
\end{prop}
This provides an easy means of computing conjugation in \(S_n\) of course, but moreover it tells us that conjugation doesn't change the general structure of cycles.

\begin{defn}[Cycle lengths and partitions]
	If \(\sigma  \in S_n\) is the product of disjoint cycles of length \(n_1,n_2,\ldots,n_r\) with
	\begin{align*}
		n_1\leq n_2\leq \ldots\leq n_r,
	\end{align*}
	then the integers \(\left\{ n_i \right\}_{i=1}^{r}\) are called the \textbf{cycle type of \(\sigma \)}.
\end{defn}
	If \(n \in \Z_+\), a \textbf{partition of \(n\)} is any nondecreasing sequence of positive integers whose sum is \(n\).\\

	The previous proposition tells us that the cycle type of a permutation is unique.
	\begin{prop}
		Two elements of \(S_n\) are conjugate in \(S_n\) if and only if they have the same cycle type.\\

		The number of conjugacy classes of \(S_n\) is the number of partitions of \(n\).
	\end{prop}

	We can use these results to prove important ideas-- for example, one can give a combinatorial proof that \(A_5\) is a simple group.

\end{document}

Abstract Algebra - Introductory/Algebra I/Notes/source/2020-04-28-GroupAction.tex


% !TEX root = index.tex
\section{Topological Preliminaries}
\setlength{\epigraphwidth}{0.41\textwidth}
\epigraph{Humpty Dumpty sat on a wall,\\
	Humpty Dumpty had a great fall.\\
	All the king's horses and all the king's men\\
	Could not put Humpty together again.}{}

The first step is to break a space up into \emph{simpler spaces} and try to \emph{glue} the pieces back. Simpler spaces will mean contractible spaces and gluing back will be done using locally constant functions.
\begin{align*}
 \xymatrix@R-2pc{
 X \ar@{~>}[r] & \U \\
 \mbox{Topological Space} & \mbox{Good cover of } X
 }
\end{align*}
\begin{definition}
	A topological space\footnote{For us a topological space is simply a subset of $ \R^2 $ or $ \R^3$.} $ X$ is said to be \textbf{contractible} if there exists a point $ x_0 \in X$ and a continuous map
	\begin{align*}
		\Phi: X \times [0,1] \rightarrow X
	\end{align*}
	such that
	\begin{align*}
		\Phi(x,0) & = x   \\
		\Phi(x,1) & = x_0
	\end{align*}
	for all $ x \in X$ i.e. there are ``continuously varying paths'' connecting each point in $ X$ to $ x_0$.
\end{definition}

\begin{figure}[H]
	\centering
	\includegraphics[width=6cm]{contractible}
	\caption{The interior of a triangle is a contractible space.}
\end{figure}

In some sense, a contractible space is as simple a space as is topologically possible. Most algebro-topological invariants cannot distinguish a contractible space from a point. This makes contractible spaces very useful as one may need infinitely many points to \emph{construct} a space but only finitely many contractible spaces to do so, as we'll see below.\\

\begin{ques}
	A subset $ X$ of $ \R^n$ is said to be \textbf{star-shaped} if there exists a point $x_0$ such that for any other point $x \in X$ the segment connecting $x$ to $x_0$ lies entirely in $ X$. Prove that star-shaped subsets of $\R^n$ are contractible.
\end{ques}
\begin{remark}
	This proves that the sets $\R^n$, $ [0,1]$ , $(0,1) $, union of $ x$ and $ y$ axis in $ \R^2$, interior of a convex polygon in $\R^2$, interior of a convex polyhedron in $\R^3$ are all contractible.
\end{remark}
	Not every space is contractible, otherwise topology would have been a very boring subject (which it isn't). The first application of Cech Cohomology will be proving that spaces like the circle $ S^1$, the spheres $ S^2$, $ S^n$, torus, projective space, other higher genus surfaces are {not} contractible.\\

	\begin{ques} Let $X$, $Y$ be topological spaces.
		\begin{enumerate}
			\item Show that if $X$, $Y$ are contractible then so is $X \times Y$. In particular, if $ X$ is contractible then so is $ X \times [0,1]$.
			\item Show that if $ X \times Y$ is contractible, then so is $ X$.
			\item* A subspace $ A \subseteq X$ is said to be a \textbf{retract} of $ X$ if there exists a continuous map $$ r: X \rightarrow A$$ such that $ r(a) = a$ for all $ a \in A$. Show that if $ X$ is contractible and $ A $ is a retract of $ X$ then $ A$ is also contractible.
			\item Is it true that every subset of a contractible space is contractible?
		\end{enumerate}
	\end{ques}

	\begin{definition}
		We say that $X$ is \textbf{connected}\footnote{Technical point: This should really be called \emph{path-connected} but we will only be dealing with spaces where the two notions coincide.} if for any two points $ x_0,x_1 \in X$ there exists a continuous map $ c:[0,1] \rightarrow X$ such that $ c(0) = x_0$ and $ c(1) = x_1$. Define a relation on the set $X$ as $x_0 \sim_{conn} x_1$ if there exists a path in $X$ connecting $x_0$ to $x_1$.
	\end{definition}
	\begin{ques} Let $X$ be a topological space.
		\begin{enumerate}
			\item Show that if $ X$ is contractible then $ X$ is connected.
			\item Show that $\sim_{conn}$ is an equivalence relation.
		\end{enumerate}
	\end{ques}
	\begin{definition}
		Define the \textbf{connected components} of $X$, denoted $\pi_0(X)$, to be the equivalence classes of $X$ under $\sim_{conn}$.
	\end{definition}
	\begin{remark}
		All our spaces will have finitely many connected components.
	\end{remark}

\newpage
\subsection{Covers}
\begin{definition}
	A finite collection of open subsets\footnote{We'll sometimes be lazy and use simplices instead of open subsets.} $ \U = \{ U_1, U_2, \dots, U_n\}$ of a topological space $ X$ is said to be a \textbf{cover} of $ X$ if $X = U_1 \cup U_2 \cup \dots \cup U_n$.
\end{definition}

\begin{example}$ $
	\begin{enumerate}
		\item $ \U = \{ X \}$ is always a cover of $ X$ of any space $X$.
		\item If $ X$ is a triangle, then $ \U = \{ U_1, U_2, U_3\} $ with $ U_i$ being a ``side'' of the triangle is a cover of $ X$.
	\end{enumerate}
\end{example}
\noindent Not all covers are equal. We need the covers to satisfy the following extra condition.
\begin{definition}Let $ [n] = \{ 1, 2, \cdots, n \}$.
	To every non-empty subset $ I \subseteq [n]$ we can associate a subset $U_I$ of $ X$ defined as
	\begin{align*}
		U_I := \bigcap_{i \in I} U_i
	\end{align*}
	The cover $ \U = \{ U_1, U_2, \dots, U_n\}$ is said to be a \textbf{good cover} if for every non-empty subset $ I \subseteq [n]$ every connected component of the space $ U_I$ is contractible.
\end{definition}
\begin{remark}
	In the above definition, $U_I$ can be empty as every connected component of an empty set is contractible.
\end{remark}
\begin{definition}
	The \textbf{dimension} of a cover $ \U $ is the largest $k$ such that there exists some $I \subseteq [n]$ with $|I| = k+1$ and $U_I$ non-empty.
\end{definition}
\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.6]
		%% vertices
		\draw[fill=black] (0,0) circle (3pt);
		\draw[fill=black] (4,0) circle (3pt);
		\draw[fill=black] (2,3) circle (3pt);
		%% vertex labels
		\node at (2,-0.5) {$U_1$};
		\node at (0.5,1.75) {$U_2$};
		\node at (3.5,1.75) {$U_3$};
		\node at (2,-0.5) {$U_1$};
		\node at (-1,0) {$U_{\{1,2\}}$};
		\node at (5,0) {$U_{\{1,3\}}$};
		\node at (2,3.5) {$U_{\{2,3\}}$};
		%%% edges
		\draw[thick] (0,0) -- (4,0) -- (2,3) -- (0,0);
	\end{tikzpicture}
	\caption{The set of ``sides'' $\U = \{U_1, U_2, U_3\}$ is a good cover of the triangle. In this case $U_{\{1,2\}} = U_1 \cap U_2$, $U_{\{2,3\}} = U_2 \cap U_3$, $U_{\{1,3\}} = U_1 \cap U_3$ are the vertices and $U_{\{1,2,3\}} = U_1 \cap U_2 \cap U_3$ is empty.}
\end{figure}
\newpage
\begin{remark}
	For the following problems, construct good covers with the least possible dimension; the higher the dimension of the cover, the harder the cohomology computations.
\end{remark}
\begin{ques}
	Construct good covers of the following spaces. It's enough (and recommended) to simply draw pictures.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $ \R^n$
			\item $ S^1$ (= the circle)
			\item A Tree
			\item The bipartite graph $ K_{2,3}$
			\item $ \R^2 \setminus \{(0,0) \}$
			\item $ \R^2 \setminus \{(0,0), (1,0), \dots, (k,0) \}$ for some positive integer $ k$
			\item $ S^2$ (= the sphere)
			\item $ S^2 $ minus a point
			\item $ S^2 $ minus 2 points
		\end{enumerate}
	\end{multicols}
\end{ques}

\begin{ques} Find good covers of
	\begin{enumerate}
		\item $S^1 \vee S^1$ = two circles glued at a point (pronounced 'S one wedge S one')
		\item Bouquet of $n$-circles\\
		\begin{tikzpicture}[scale=0.6]
			\begin{polaraxis}[grid=none, axis lines=none]
				\addplot[mark=none,domain=0:360,samples=300] { abs(cos(8*x/2))};
			\end{polaraxis}
		\end{tikzpicture}
		\item $S^1 \vee S^2$ = a circle and a sphere glued at a point (pronounced 'S one wedge S two')
	\end{enumerate}
\end{ques}

\begin{ques}*
	Let $X$, $X'$ be topological spaces with good covers $\U$, $\U'$ respectively. Find a good cover of $X \times X'$. Find a good cover of $S^1 \times S^1$ (= torus). Draw a picture.
\end{ques}

\begin{ques}*
	Find a good cover of the solid torus (which is homeomorphic to $[0,1] \times [0,1] \times S^1$).
\end{ques}

\begin{ques}**
	Find a good cover of the $g$-holed torus.
\end{ques}
It is not true that all spaces admit good covers. However, the spaces that do not admit one are very exotic in nature. For example, every smooth manifold admits a good cover (this is called the Nerve Lemma) however this is false if we drop the adjective smooth. For this class we'll assume that all the spaces under consideration admit a good cover.


\documentclass[10pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{hyperref}

\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{tikz}

\usetikzlibrary{arrows,shapes,automata,positioning,calc}

\usepackage{caption}
\usepackage{subcaption}

\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=cyan,
    pdftitle={Overleaf Example},
    pdfpagemode=FullScreen,
    }

\title{Chapter 3 - Finite Markov Decision Processes}
\author{Stéphane Liem NGUYEN}
\begin{document}
\maketitle

Some keywords, citations and formulas.

\paragraph{What are value functions ?}
Value functions tell us \textit{how good} it is for the agent to be in a given state or how good it is for the agent to take an action from a given state. "How good" is related to the expected return, the expected cumulative future reward when the agent follows a particular behaviour, a policy.

For example, the optimal action-value function $q_*(s, a)$ is telling us what is the expected return if we start in state $s$, take action $a$ then follows the optimal policy/strategy $\pi_*$ afterwards.

Here are the formal definitions of the value functions under policy $\pi$; the first one is for the state-value function and the second is the action-value function.
\begin{equation}
v_\pi(s) \doteq \mathbb{E}_\pi[G_t \lvert S_t = s]
\end{equation}

\begin{equation}
q_\pi(s, a) \doteq \mathbb{E}_\pi[G_t \lvert S_t = s, A_t = a]
\end{equation}

with $G_t = R_{t+1} + \gamma G_{t+1}, \forall t < T$ and $G_T = 0$. The discount factor/rate $\gamma$ is a scalar value in $[0, 1]$ that determines how farsighted is the agent. Is the agent taking into account more than the immediate reward for deciding which action to pick in a given state ?

The undiscounted case ($\gamma=1$) is used most of the time for episodic tasks where there's a notion of a terminal state or final time step $T$ (random variable). It can be used for continuing tasks but it's not covered in the Chapter so we can just keep in mind that the discounted case ($0 \leq \gamma < 1$) is used for continuing tasks to keep the return finite.

The value of the \textit{terminal state} (or \textit{absorbing state}. There's only one single terminal state with different possible rewards for different outcomes.) for the \textit{episodic tasks} is $0$.

\paragraph{Action-value function and Model-free} When we do not want to select actions based on the knowledge of the environment dynamics, action-value functions can be used because they "cache the results of all
one-step-ahead searches." (Upper part of the page $65$). In other words, action-value functions are used in the \textit{model-free} case where methods select actions without creating a model of the environment--without estimating the transition probabilities as well as the expected rewards based on real trajectories.

\paragraph{Bellman (expectation) equations for the four value functions}

Bellman equation for the state-value function for policy $\pi$
\begin{equation}
v_\pi(s) = \sum_{a} \pi(a \lvert s) \sum_{s', r} p(s', r \lvert s, a) \left[r + \gamma v_\pi(s')\right]
\end{equation}

Bellman equation for the optimal state-value function (Bellman optimal equation for $v_*$)
\begin{equation}
v_*(s) = \max_{a} \sum_{s', r} p(s', r \lvert s, a) \left[r + \gamma v_*(s')\right]
\end{equation}

Bellman equation for the action-value function for policy $\pi$
\begin{equation}
q_\pi(s, a) = \sum_{s', r} p(s', r \lvert s, a) \left[r + \gamma \sum_{a'} \pi(a' \lvert s') q_\pi(s', a')\right]
\end{equation}

Bellman equation for the optimal action-value function (Bellman optimal equation for $q_*$)
\begin{equation}
q_*(s, a) = \sum_{s', r} p(s', r \lvert s, a) \left[r + \gamma \max_{a'} q_*(s', a')\right]
\end{equation}

where $a \in \mathcal{A}(s)$, $s'$ and $s \in \mathcal{S}$ and $r \in \mathcal{R}$

\section{Citations of some parts of the book}

\begin{enumerate}
\item p. $49$: "In a Markov decision process, the probabilities given by $p$ completely characterize the \textbf{environment's dynamics}. [...] The state must include \textit{information about all aspects of the past} agent–environment interaction that make a \textit{difference for the future}. If it
does, then the state is said to have the \textbf{Markov property}. We will assume the Markov
property throughout this book, though starting in Part II we will consider \textbf{approximation
methods} that do not rely on it, and in Chapter 17 we consider how a \textbf{Markov state} can
be efficiently \textbf{learned} and \textbf{constructed from non-Markov observations}."

\item p. $50$: "In general, \textit{actions can be any decisions we want to learn how to make}, and
states can be anything we can know that might be useful in making them."
\item p. $50$: "The general rule we follow is that \textbf{anything that cannot be changed arbitrarily} by
the agent is considered to be outside of it and thus \textbf{part of its environment}."
\item p. $50$: "The agent-environment \textbf{boundary} represents the limit of the
agent's \textbf{absolute control, not of its knowledge}."
\item p. $53$, \textit{reward hypothesis} : "That all of what we mean by \textbf{goals} and purposes can be well thought of as
the \textbf{maximization of the expected value} of the cumulative \textbf{sum} of a received
scalar signal (called \textbf{reward})."

\item p. $68$: "The \textbf{online nature} of reinforcement learning makes it possible to \textbf{approximate optimal policies} in ways that put more effort into learning to make \textit{good decisions} for
\textbf{frequently encountered states}, at the expense of less effort for infrequently encountered
states. This is one key property that distinguishes reinforcement learning from other
approaches to approximately solving MDPs."
\end{enumerate}

\end{document}


\chapter{Genetic Algorithms}

The previous three chapters showed you how to write Unicon programs with
many kinds of Internet and system capabilities that are expected of
most modern applications. Unicon is great for generic computing tasks,
but it really excels when its advanced features are applied in
application areas where the development of the algorithms is complex.
This chapter describes how to use Unicon to build an entire, somewhat
complex application with reusable parts. The field of \index{genetic
algorithms}\textit{genetic algorithms} (GAs) is an exciting application
domain with lots of opportunities for exploratory programming. When
you're finished with this chapter, you will
\begin{itemize}
\item Understand the basics of genetic algorithms.
\item See how to build genetic algorithm engine in Unicon.
\item Use that GA engine to build programs for your own projects.
\end{itemize}

\section{What are Genetic Algorithms?}

The broad field of evolutionary computation has been an area of active
research since the 1950s. Initially, it was led by computer scientists
that believed evolution could be used as an optimization tool for
engineering problems. Genetic Programming (GP) focuses on evolving
computer programs to perform various tasks. On the other hand, GAs
focus on the simpler task of evolving data that is used to solve a
problem. The GAs that we'll study have a binary
representation. Increasingly there has been a shift towards non-binary
representations such as floating-point numbers in GA-related projects.
That field has typically been given the more general name of
evolutionary algorithms. GAs have one of the most well-defined
mathematical foundations in all of evolutionary computation, and are a
good place to start exploring.

John Holland invented the first GAs in the 1960s. His goal was to study
adaptation as it occurs in nature and then to create computer systems
to model the adaptive process. Holland combined four elements that are
common to all GAs:

\begin{itemize}
\item A population of individuals
\item Selection based on fitness
\item Mating of individuals
\item Random mutation
\end{itemize}
Consider the very simple problem of finding the largest number encoding
in binary with six digits. Assume the GA knows nothing about binary
encoding.

While making use of a population might seem to be a necessary element
for any evolutionary computation, it is not. Instead, you could focus
all your efforts on improving one individual. In this case, fitness is
exactly the numerical value of an individual. For example, you could
examine the fitness of this one individual with an exhaustive search:

\iconcode{
\>   best := 0 \\
\>   every i := 0 to 2\^{}6 do \\
\>   \ \ \ best {\textless}:= i
}

Suppose you only make use of the elements of a population and selection
based on fitness. You could have a population of six individuals,
randomly initialized, and you could attempt to improve the overall
fitness of the six by replacing the lowest fitness individual with a
random one. The code below shows how to implement this idea:

\iconcode{
\>   maxi := 2\^{}6 \\
\>   population := [?maxi, ?maxi, ?maxi, ?maxi, ?maxi, ?maxi] \\
\>   every i := 1 to 100 do \{ \\
\>   \ \ \ worst := 1 \\
\>   \ \ \ every i := 1 to 5 do \\
\>   \ \ \ \ \ \ if population[worst] {\textless}= population[i] then \\
\>   \ \ \ \ \ \ \ \ \ worst := i \\
\>   \ \ \ population[i] := ?maxi \\
\>   \ \ \ \}
}

Before modeling mating and mutation, you'll have to
create a more detailed representation of the internals of an
individual.

\section{Operations: Fitness, Crossover, and Mutation}

An individual is represented by a string from a binary alphabet.
Incidentally, natural evolution of DNA is based on a quaternary
alphabet, but the size of the alphabet is unimportant for a computer
model. In Unicon, you could represent these individuals with lists of
integers. However, strings of \texttt{"1"}
and \texttt{"0"} characters provide a
representation that is easier to use. So, here is a more explicit
representation of a population:

\iconcode{
\>   population := ["010111", "000101", "111101", "111011", "111110", "010110"]
}

\subsection*{Fitness}

The fitness can be computed by converting the string representation into
an integer as follows: \texttt{integer("2r"
{\textbar}{\textbar} population[1])}. The \texttt{2r} means that this
is a literal representation of an integer in base two form. There are
many different possible selection schemes used in GAs. This chapter
uses one that has proven to be very robust in a large number of
different GA applications, called \textit{tournament selection}. The
general idea is to group the individuals and have them compete
head-to-head. The winners of the tournaments are selected to live in
the next generation; their bits are copied into an element in the new
population. Tournaments of size two work well. All you must do is
randomly pair up the individuals, and move the one with the higher
fitness to the next generation. Because you generally will want the
population size to remain constant, you'll have to do
this pairing twice. Here is a tournament selection on the above
population:

\iconcode{
\>   population[1] := "010111"
\# 23 winner \\
\>   population[6] := "010110"
\# 22 \\
\>   population[3] := "111101"
\# 61 winner \\
\>   population[4] := "111011"
\# 59 \\
\>   population[5] := "111110"
\# 62 winner \\
\>   population[2] := "000101"
\# 5
}

The second round of selections is listed here:

\iconcode{
\>   population[5] := "111110"
\# 62 winner \\
\>   population[6] := "010110"
\# 22 \\
\>   population[2] := "000101"
\# 5 \\
\>   population[3] := "111101"
\# 61 winner \\
\>   population[4] := "111011"
\# 59 winner \\
\>   population[1] := "010111"
\# 23
}

The end result of tournament selection is listed here:

\iconcode{
\>   next\_gen[1] := population[1] :=
"010111" \# 23 \\
\>   next\_gen[2] := population[3] :=
"111101" \# 61 \\
\>   next\_gen[3] := population[5] :=
"111110" \# 62 \\
\>   next\_gen[4] := population[5] :=
"111110" \# 62 \\
\>   next\_gen[5] := population[3] :=
"111101" \# 61 \\
\>   next\_gen[6] := population[4] :=
"111011" \# 59
}

Notice how there are two copies of \texttt{population[3]} and
\texttt{population[5]} in \texttt{next\_gen}. On the other hand, there
are no copies of \texttt{population[2]}.

\subsection*{Crossover}

Mating, more technically known as crossover, involves the sharing of
information between members of the population. Again there are many
different types of mating schemes, but this chapter describes one
called two-point crossover (Figure 17-1) that has proven to be very
robust for a wide range of GA applications. Once again, randomly pair
up the individuals, but this time instead of competing, individuals
will mate. First you must transform the linear strings into circular
rings. For each pair, randomly select two points in the ring and cut
the rings at the two selected points. Then swap the ring segments to
form two new rings.\\

\begin{center}
\includegraphics[width=3.322in,height=3.9453in]{ub-img/ub-img45.png}
\end{center}

{\sffamily\bfseries Figure 17-1:}
{\sffamily Two-point crossover.}

The code for two-point crossover is presented below. The
\texttt{?(lchrom+1)} expression picks a random number between one and
the length of the chromosome. Variables \texttt{a} and \texttt{b} are
initialized to two different values in this range; \texttt{a} is made
the smaller of the two indices. Two children in the new generation are
formed by splicing portions of \texttt{parent1} and \texttt{parent2}
within the range from \texttt{a} to \texttt{b}.

\iconcode{
\>   \ \ \ a := ?(lchrom+1) \\
\>   \ \ \ while ((b := ?(lchrom+1)) = a) \\
\>   \ \ \ if a {\textgreater} b then a :=: b \\
\>   \ \ \ ncross +:= 1 \\
\>   \ \ \ child1 := parent1[1:a] {\textbar}{\textbar} parent2[a:b]
{\textbar}{\textbar} parent1[b:0] \\
\>   \ \ \ child2 := parent2[1:a] {\textbar}{\textbar} parent1[a:b]
{\textbar}{\textbar} parent2[b:0]
}

\subsection*{Mutation}

The last GA operation is mutation. Mutation works at the independent
level of single binary digits. To implement mutation, take a look at
each bit of each individual of the population. With a fixed
probability, flip the value of the bit. This is the basic mechanism for
injecting completely new information into the population. Almost all of
that information will be useless, but as the GA evolves it will weed
out the useless information and keep the useful information.

\section{The GA Process}

Now that you have a handle on the basic operations, it is time to
describe the basic GA algorithm for applying these operations.

\ \ 1.\ \ Generate a random population of n individuals each with
\textit{l}{}-bits.

\ \ 2.\ \ Calculate the fitness of each individual.

\ \ 3.\ \ Perform tournament selection on the population to produce a
new generation.

\ \ 4.\ \ With probability p\textsubscript{c}, mate pairs of individuals
using two-point crossover.

\ \ 5.\ \ With probability p\textsubscript{m}, mutate the bits in the
population.

\ \ 6.\ \ Replace the old population with the new generation.

\ \ 7.\ \ Go to step 2, until the population meets some desired
condition.

Even with this mechanical algorithm, applying a GA to any specific
problem remains an art. For example, step 7 leaves the desired
stopping condition up to the implementer. Typically the stopping
condition might be something like: until the average fitness has not
risen significantly in the last five generations. This is one of
many implementation choices you have to make,
from variations on crossover to adjusting the
mutation and mating rates. Here are some time-tested rules of thumb:

\ \ 1.\ \ Encode the solutions to a problem with as few bits as possible
but not at the expense of making the encoding very complex.

\ \ 2.\ \ Let the size of the population be at least twenty but not so
large that your computer program is intolerably slow.

\ \ 3.\ \ Mating rates between 30 percent and 90 percent work for a
large range of problems.

\ \ 4.\ \ Mutation rates should be near 1/\textit{(the number of bits)},
so that each
individual undergoes about one mutation on average per generation.

\ \ 5.\ \ Once the average fitness of the population does not change
significantly after ten generations, the population has converged on
the solution. At this point, stop the GA and study the population of
solutions.

\section{\texttt{ga\_eng}: a Genetic Algorithm Engine}

The \texttt{ga\_eng} engine is a general purpose reusable GA engine that
can quickly and easily be adapted to solve a variety of problems. This
section presents its key elements. The full \index{source code}source
code for \texttt{ga\_eng} is on the book's web site.

From the preceding sections, you can tell that a GA maintains a large
amount of information as it transitions from one generation to the
next. What is the state information that the engine needs to track?
\ Below is a list of the most obvious things to record:

\begin{itemize}
\item n $-$ the size of the population
\item \textit{l} $-$ the length of the individual's bit
representation
\item p\textsubscript{c} $-$ the probability of crossover mating
\item p\textsubscript{m} $-$ the probability of mutation
\item population $-$ a list of a current population's
individuals
\end{itemize}
Given the above state information, what can a GA engine do?

\begin{itemize}
\item \texttt{init()} $-$ initialize the state information and
population
\item \texttt{evolve()} $-$ move a population from one generation to the
next
\item \texttt{tselect()} $-$ perform tournament of selection on the
population
\item \texttt{stats()} $-$ collect statistics about the current
population to monitor progress
\end{itemize}

\subsection*{The fitness function}

A key application-specific interface issue is the fitness function. The
user of the GA engine sets the fitness values of each member of the
population. If that value is not set, the default value will be the
average of the fitness of each of the parents. Each
parent's contribution to the fitness is weighted by
how many bits it contributed to the offspring. This has the nice
property of not requiring that each individual's
fitness be computed at every generation.

To use the GA engine, the programmer supplies an application-specific
fitness function \texttt{f(x)} that is applied to each individual of
the population every generation. Given a binary string \texttt{s},
\texttt{f(s)} would return a numeric fitness value.

\subsection*{Methods and attributes of class \texttt{ga\_eng}}

Class \texttt{ga\_eng} implements the engine. It provides
two public methods, \texttt{evolve()} and \texttt{set\_params()}, as
well as eight private methods:

\iconcode{
method evolve() \\
method set\_params(fitness\_func, popsize, lchrom, pcross, pmutation, log\_file) \\
method tselect() \\
method crossover2(parent1, parent2) \\
method generation() \\
method stats(Pop) \\
method report(Pop) \\
method random\_chrom() \\
method initpop() \\
method init()
}

The \texttt{set\_params()} method sets the fitness function, the
population size, the length of the chromosomes, the probability of
crossover, the probability of mutation, and the log file for generating
a trace of \ a run. The \index{constructor}constructor
\texttt{ga\_eng()} takes the same input parameters as
\texttt{set\_params()} but it also re-initializes the engine by
creating a new population from scratch. The \texttt{evolve()} method
moves the population from one generation to the next.

The \texttt{tselect()} method operates on the whole population by
performing tournament selection. The \texttt{crossover2()} method takes
two individuals and returns two new individuals in a list after doing
two-point crossover. The \texttt{stats()} method collects statistics on
the given population, and \texttt{report()} writes the results out to
the log file if it is not set to \texttt{\&null}. The \texttt{init()}
and \texttt{initpop()} methods initialize the GA.

You might be asking what is a population of \textit{individuals}? \ The
key pieces of information about an individual are its chromosomes,
which are represented as a string of zeros and ones. For implementation
reasons, we bundle the following information for each individual using
the following Unicon record:

\iconcode{
record individual(chrom, fitness, parent1, parent2, xsite)}

This stores the fitness values, the index of two parents, and the
crossover sites from when the parents were mated.

The class \texttt{ga\_eng} makes use of the following
\index{instance}instance variables:

\begin{itemize}
\item \texttt{oldpop}, \texttt{newpop} $-$ two populations; selection
goes from \texttt{oldpop} into \texttt{newpop}
\item \texttt{popsize}, \texttt{lchrom} $-$ the population size and the
length of the chromosomes
\item \texttt{gen}, \texttt{maxgen} $-$ current and max generation
number
\item \texttt{pcross}, \texttt{pmutation} $-$ the probability of
crossover and mutation
\item \texttt{sumfitness} $-$ the sum of the fitness of the entire
population
\item \texttt{nmutation} $-$ number of mutation in the current
generation
\item \texttt{ncross} $-$ number of crossovers (or matings) in the
current generation
\item \texttt{avg}, \texttt{max}, \texttt{min} $-$ average, maximum, and
minimum fitness in the population
\item \texttt{best} $-$ the location of the individual with the highest
fitness
\item \texttt{log\_file} $-$ a text file where statistics are written
during a GA run
\item \texttt{fitness\_func} $-$ the user-supplied fitness function. It
reads a string of zeros and ones and returns a number representing the
fitness.
\end{itemize}

\subsection*{A Closer Look at the \texttt{evolve()} method}

Space limitations preclude discussing every method of
\texttt{ga\_eng}, but \texttt{evolve()} is a method that is called
by user code, and defines the basic architecture of the
engine. The \texttt{evolve()} method initializes the population the
first time the method is invoked by each instance of \texttt{ga\_eng}.
After initialization and in subsequent calls, \texttt{evolve()}
does three things: it collects statistics, writes the results to a log
file, and then evolves the population for one generation. The method
\texttt{generation()} then becomes the focus of activity. The code for
\texttt{evolve()} is:

\iconcode{
method evolve() \\
\>   if /initialized := 1 then \{ \\
\>\>    gen := 0 \\
\>\>    init() \\
\>\>    statistics(oldpop) \\
\>\>    if {\textbackslash}log\_file then report(oldpop) \\
\>\>    \} \\
\>   gen +:= 1 \\
\>   generation() \\
\>   statistics(newpop) \\
\>   if {\textbackslash}log\_file then report(newpop) \\
\>   oldpop := newpop \\
end
}

Generation consists of three high-level operations. Tournament
selection is performed via a call to \texttt{tselect()}, which makes
copies of selected individuals from \texttt{oldpop} to
\texttt{newpop}.
After this, all operations take place on individuals in
\texttt{newpop}. The next two operations are performed in a
loop, on pairs of individuals. The \texttt{crossover2()} method does
the mating; it encapsulates the relatively low-level operation of
mutation. The last high-level operation that a GA does is call the user
supplied \texttt{fitness\_func} to assign a fitness value to each
individual in the new population. The GA keeps track of only two
generations for the \texttt{evolve()} method to continue as long as
needed. Once each individual has been evaluated, that generation is
complete. The \texttt{oldpop} variable is assigned the value of the
\texttt{newpop}, and the process is ready to start again. Listing 17-1
shows the code for the method \texttt{generation()}:

\bigskip

{\sffamily\bfseries Listing 17-1}

{\sffamily\bfseries A method for producing a new generation.}

\iconcode{
method generation() \\
\>   local j := 1, mate1, mate2, jcross, kids, x, fitness1, fitness2, selected
\\
\>   newpop \ \ \ := list(popsize) \\
\>   nmutation := ncross := 0 \\
\>   selected \ \ := tselect() \\
\>   repeat \ \{ \\
\>   \ \ \ mate1 := selected[j] \\
\>   \ \ \ mate2 := selected[j+1] \\
\>   \ \ \ kids := crossover2(oldpop[mate1].chrom, oldpop[mate2].chrom ) \\
\>   \ \ \ fitness1 := fitness\_func(oldpop[mate1].chrom) \\
\>   \ \ \ fitness2 := fitness\_func(oldpop[mate2].chrom) \\
\>   \ \ \ newpop[j]:= individual(kids[1], fitness1, mate1, mate2, kids[3]) \\
\>   \ \ \ newpop[j+1] := individual(kids[2], fitness2, mate1, mate2, kids[3])
\\
\>   \ \ \ if j {\textgreater} popsize then break \\
\>   \ \ \ j +:= 2 \\
\>   \ \ \ \} \\
end
}

\subsection*{Using \texttt{ga\_eng}}

GAs are extremely robust. It is easy to create a buggy GA that
works so well that the bugs go undetected. Masking of
bugs by robust algorithms is not unique to GA; it occurs in many
numerical algorithms. To prevent this, the following code tests
the GA on a simple problem where having one bit increases the fitness
values by one. This is a ready to compile and run GA application,
albeit a very simple one. As you can see, \texttt{ga\_eng()} and
\texttt{evolve()} are all the interface methods you need to build a
complete genetic algorithm. Listing 17-2 shows all the code needed
to use the GA engine for a simple test problem.

\bigskip

{\sffamily\bfseries Listing 17-2}

{\sffamily\bfseries Using \texttt{ga\_eng()} for a simple test problem}

\iconcode{
procedure decode(chrom) \\
\>   local i := 0 \\
\>   every !chrom == "1" do i +:= 1 \\
\>   return i \\
end \\
\ \\
\# a simple test of the engine, fitness is based on number of 1 bits in chrom \\
procedure main() \\
\>   log\_file \ := open("test\_ga.log",
"w") {\textbar} stop("cannot open log\_file.log") \\
\>   ga := ga\_eng(decode, 100, 20, 0.99, 1.0/real(20), log\_file) \\
\>   every 1 to 100 do ga.evolve() \\
\>   write(log\_file, "best location ={\textgreater} ", ga.best) \\
\>   write(log\_file, "best fitness \ ={\textgreater} ",
           ga.newpop[ga.best].fitness) \\
end
}

\subsection*{Log files}

The log file \texttt{test\_ga.log} contains a lot of statistics. Listing
17-3 is a fragment of the log file that is generated. The complete log
file has over twelve thousand lines.

\bigskip

{\sffamily\bfseries
Listing 17-3}

{\sffamily\bfseries
A trace of the GA for a simple test problem}

\iconcode{
Log\_File for Genetic Algorithm (GA) \\
\ \\
\ Population size \ \ \ \ \ \ \ \ \ = 100 \\
\ Chromosome length \ \ \ \ \ \ \ = 20 \\
\ Maximum \# of generations =  \\
\ Crossover probability \ \ \ = 0.99 \\
\ Mutation probability \ \ \ \ = 0.05 \\
\ \\
\ Initial population maximum fitness = \ 5.00e-1  \\
\ Initial population average fitness = \ 5.00e-1  \\
\ Initial population minimum fitness = \ 5.00e-1  \\
\ Initial population sum of \ fitness = \ 5.00e1 \  \\
{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}- \\
Population Report \\
\>   \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Generation 0 \\
\ \\
\>   \ \ \# \ \ \ parents \ \ \ \ \ \ xsite \ \ chromo
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ fitness \\
{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}- \\
\ \ \ \ \ \ \ 1) ( \ 0, \ \ 0) \ \ \ \ \ \ 0 \ \ \ 00010001110001000010
\ 5.00e-1  \\
\ \ \ \ \ \ \ 2) ( \ 0, \ \ 0) \ \ \ \ \ \ 0 \ \ \ 11011101011010001100
\ 5.00e-1  \\
\ \ \ \ \ \ \ 3) ( \ 0, \ \ 0) \ \ \ \ \ \ 0 \ \ \ 01101100000110100111
\ 5.00e-1  \\
..........  \\
\>   \ \ 100) ( \ 0, \ \ 0) \ \ \ \ \ \ 0 \ \ \ 11011100110011001010
\ 5.00e-1  \\
{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}- \\
Statistics:  \\
min = \ 5.00000000e-1 \ avg = \ 5.00000000e-1 \ max = \ 5.00000000e-1  \\
\ no. of mutations \ \ \ \ \ \ \ = 0 \\
\ no. of crossovers \ \ \ \ \ \ = 0 \\
\ location of best chromo = 1 \\
{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}- \\
dateline = Wednesday, January 27, 1999 \ 10:37 pm \\
................. \\
Population Report \\
\>   \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Generation
100 \\
\ \\
\>   \ \ \# \ \ \ parents \ \ \ \ \ \ xsite \ \ chromo
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ fitness  \\
{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}- \\
\ \ \ \ \ \ \ 1) ( \ 1, \ \ 2) \ \ 15:21 \ \ \ 00011011011000010100
\ 9.19e0 \  \\
\ \ \ \ \ \ \ 2) ( \ 1, \ \ 2) \ \ 15:21 \ \ \ 11011001011111100010
\ 1.08e1 \  \\
\ \ \ \ \ \ \ 3) ( 90, \ 72) \ \ 14:18 \ \ \ 10011111111111111111
\ 1.78e1 \ \  \\
..... \\
\>   \ \ 100) ( 57, \ 27) \ \ 13:15 \ \ \ 11111010110110111110 \ 1.78e1
\  \\
{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}- \\
Statistics:  \\
min = \ 9.19999999e0 \ \ avg = \ 1.77200000e1 \ \ max = \ 1.99500000e1 \\
\ no. of mutations \ \ \ \ \ \ \ = 111 \\
\ no. of crossovers \ \ \ \ \ \ = 51 \\
\ location of best chromo = 46 \\
{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}-{}- \\
dateline = Wednesday, January 27, 1999 \ 10:37 pm \\
best location ={\textgreater} 46 \\
best fitness \ ={\textgreater} 19.95
}

\section{Color Breeder: a GA Application}

Normally, the human eye is capable of distinguishing millions of
different colors. If you have a device capable of producing a large
number of different colors such as a PC with a color monitor or color
printer, then finding exactly the color that you have in mind can be a
tricky task. The color space is quite large. For example, imagine that
you want to create a Web page with a blue background. But this is no
ordinary blue; you want a blue that is like the blue you saw on the
Mediterranean skyline on your last trip to Greece.

One option is to look at a color wheel and click on the part that
represents the blue that you have in mind. Unfortunately,
the color you choose is typically only a very small part of the
color wheel. Also the color is surrounded by many different colors that
may be distracting or misleading in your evaluation. When you use the
same color for the entire background, you get a different sense of the
color.

A second option is to tinker with numeric color codes by hand. Through
trial and error, by examining a large number of colors you can select
the right one. This can be very time consuming and you may become
impatient and only experiment with a small number of colors, in which
case you settle for a quick approximation of the color you intended.

The following \textit{color breeder} program has properties from both of
the color wheel and the tinkering methods. It uses a GA to explore the
problem space. The program \texttt{cb} first displays sixteen randomly
generated colors. The fitness of an individual color is based entirely
on user preference. Using scrollbars, the user ranks the individuals,
and then hits a breed button to generate a new population of colors.

Scrollbars \texttt{sbar[\textit{i}]} are indexed with
\texttt{\textit{i}}, where \texttt{\textit{i}} runs from 1 to
16. The value of the scrollbar is inverted because scrollbar values
correspond to the y-axis by default, which grows from top to bottom;
when the tab is lowered the value of scrollbar goes up. The user of
\texttt{cb} sets higher tabs to indicate higher fitness.

\iconcode{
\>   every i := 1 to 16 do \{ \\
\>   \ \ \ f :=
VGetState(vidgets["sbar"{\textbar}{\textbar}i]) \\
\>   \ \ \ ga.newpop[i].fitness := 1 - f \\
\>   \}
}

Gradually the population evolves from a set of random colors to a set of
colors that look alike, with ever so slight variations on that
Mediterranean sky blue you were thinking of. You can then save a
snapshot of the screen in a GIF format if you wish, and can see the
numeric codes that represents the color by clicking on a color.

Figure 17-2 is a screenshot of \texttt{cb}. The higher the user
slides the scrollbar tab, the more fit the color. There are three
\index{color}color resolutions to select from: 12-, 24-, and 48-bit.
The bits are equally divided into three segments, each representing a
color intensity. There are two mutually exclusive advanced modes:
patterns and text. Patterns are square bit patterns that specify the
mix of foreground and background colors in three
resolutions: 6x6, 8x8, and 10x10. In text mode, the
foreground color is displayed as text atop the background.

\begin{center}
\includegraphics[width=4.0in,height=3.0in]{ub-img/ub-img46.png}
\end{center}

{\sffamily\bfseries Figure 17-2:}
{\sffamily cb - a genetic algorithm for picking colors.}

\subsection*{Breeding textures}

The user can turn on bi-level patterns of the form: \texttt{\textit{width,
\#data}} as described in Chapter 7. A bi-level pattern is tiled using
the foreground and background color to fill an area; a 1 in the
pattern denotes the foreground, and a 0 denotes the background. The
patterns come in three resolutions: 8x8, 10x10,
and 12x12. In low resolutions it is hard to find interesting patterns;
in high resolution, most patterns look like a random mixture and are
hard to evolve into something more structured. Figure 17-3 (left) shows
\texttt{cb} in pattern mode.

\bigskip

\noindent \includegraphics[width=3.125in,height=2.62in]{ub-img/ub-img47.png}
\includegraphics[width=3.125in,height=2.62in]{ub-img/ub-img48.png}

{\sffamily\bfseries Figure 17-3:}
{\sffamily \texttt{cb} in pattern mode (left) and text mode (right)}

\bigskip

The user can click on the snap shot button to create a GIF image file
that is a snapshot of the whole screen. If you left click on a color
region, the color and patterns specification will be displayed in a pop
window. For the sake of compactness, the colors and patterns are
represented with hexadecimal strings as opposed to binary strings.

\section{Picking Colors for Text Displays}

Perhaps the most fun use of \texttt{cb} is to pick colors for your Web
pages. Figure 17-3 (right) is a snapshot of \texttt{cb} in text mode.
In text mode with 24-bit color, the user can right click on a color
region to generate a sample HTML document such as the one shown in
Figure 17-4 that demonstrates how to incorporate the selected color
combination inside of a Web page. It is also possible to get the
hexadecimal representation of a pattern or a color by right clicking on
it.

\begin{center}
\includegraphics[width=3.7256in,height=3.0047in]{ub-img/ub-img49.png}
\end{center}

{\sffamily\bfseries Figure 17-4:}
{\sffamily A sample HTML page generated by \texttt{cb}}

\section*{Summary}

GAs are a branch of evolutionary computation that allow a
general-purpose optimization technique. The three main GA operations
are selection, mating, and mutation. Object-oriented techniques enable
a generic GA engine that can be adapted to a large variety of problems.
By making the engine very general purpose it is possible to create
applications with novel properties like using user preferences to set
the fitness of a color!