\clearpage
\section{Options}
\label{section:options}
\setcounter{footnote}{0}

The whole list of options can be found using

\user{R-scape -h}\\

Some important options are:
\subsection{Covariation statistic options}

\subsubsection{\prog{-E <x>}}  Target E-value is $x\geq 0$.

\subsubsection{\prog{--GT, --MI,  --MIr, --MIg, --CHI, --OMES, --RAF, --RAFS, }}
We favor the G-test covariation statistic, but a total of eight
covariation statistics are currently implemented in \rscape. For each
covariation statistic (GT, for instance), \rscape\ can also calculate
its average product correction (GTp) and its average sum corrections
(GTa). For each option above, appending ``p'' or ``a'' chooses one of the
corrections. For example, \prog{--GT} does the G-test statistic,
\prog{--GTp} does the APC-corrected G-test statistic, \prog{--GTa}
does the ASC-corrected G-test statistic.\\

The \rscape\ default is \prog{--GTp}.\\

Details of the definition and provenance of the different covariation
statistics can be found in the \rscape\ manuscript: Rivas, E. \& Eddy
S.~E., \textit{``A statistical test for conserved RNA structure shows
lack of evidence for structure in lncRNAs''}.

\noindent
In a nutshell, given two
alignment columns $i,j$,
%
\[
\begin{array}{lrcl}
  \mbox{G-test:\citep{Woolf1957}}                                       & \mathrm{GT}(i,j)   & = & 2 \, \sum_{a,b} \mathrm{Obs}^{ab}_{ij} \, \log \frac{ \mathrm{Obs}^{ab}_{ij} } { \mathrm{Exp}^{ab}_{ij} }, \\
  \mbox{Pearson's chi-square:}                                          & \mathrm{CHI}(i,j)  & = &      \sum_{a,b} \frac{ \left(\mathrm{Obs}^{ab}_{ij} - \mathrm{Exp}^{ab}_{ij}\right)^2 }{\mathrm{Exp}^{ab}_{ij}},\\
  \mbox{Mutual information:\citep{Shannon48,Gutell94b}}                 & \mathrm{MI}(i,j)   & = &      \sum_{a,b} P^{ab}_{ij} \, \log \frac{ P^{ab}_{ij} }{ p^{a}_{i} \, p^{b}_{j}},                       \\
  \mbox{MI normalized:\citep{Martin05}}                                 & \mathrm{MIr}(i,j)  & = & \frac{\mathrm{MI}(i,j)} {H(i,j)} = \frac{\mathrm{MI}(i,j)} { -\sum_{a,b} P^{ab}_{ij} \log  P^{ab}_{ij}},        \\
  \mbox{MI with gap penalty:\citep{LindgreenKrogh06}}                   & \mathrm{MIg}(i,j)  & = & \mathrm{MI}(i,j) - \frac{N^G_{ij}} {N},                                                                   \\
  \mbox{Obs-Minus-Exp-Squared:\citep{Fodor04}}                          & \mathrm{OMES}(i,j) & = &      \sum_{a,b} \frac{ \left(\mathrm{Obs}^{ab}_{ij} - \mathrm{Exp}^{ab}_{ij}\right)^2 }{N_{ij}},                \\
  \mbox{RNAalifold (RAF):\citep{Hofacker02}}                            & \mathrm{RAF}(i,j)  & = & \mathrm{B}_{i,j},                                      \\
  \mbox{RNAalifold Stacking (RAFS):\citep{LindgreenKrogh06}}            & \mathrm{RAFS}(i,j) & = & \frac{1}{4}\left(\mathrm{B}_{i-1,j+1}+2\,\mathrm{B}_{i,j}+\mathrm{B}_{i+1,j-1}\right).                                      \\
\end{array}
\]
%
\noindent
where $a,b$ are (non-gap) residues; $N$ is the total number of aligned
sequences; $\mathrm{Obs}^{ab}_{ij}$ is the observed count of $a:b$
pairs in columns $i,j$ (only counting when both a,b are residues);
$N_{ij}$ is the total number of residue pairs in columns $i,j$ (only
counting when both a,b are residues); $P^{ab}_{ij}$ is the observed
frequency of pair $a:b$ in columns $i,j$
($P^{ab}_{ij}=\frac{Obs^{ab}_{ij}}{N_{ij}}$); $\mathrm{Exp}^{ab}_{ij}=
N_{ij} p^a_ip^b_j$ is the expected frequency of pair $a:b$ assuming
$i,j$ are independent, where $p^a_i$ are the marginal frequencies of
$a$ residues in column $i$ (averaged to all other positions) ($p^a_i
= \frac{1}{L-1}\sum_{j\neq i} \sum_b P^{ab}_{ij}$); $N^G_{ij} = N -
N_{ij}$ is the number of pairs involving at least one gap symbol; the
definition of $\mathrm{B}_{i,j}$ used in the RAF and RAFS statistics
is involved, a concise definition can be found
elsewhere~\citep{LindgreenKrogh06}.

The background corrections~\citep{DunnGloor07} for a given
covariation statistic above $\mathrm{COV}(i,j)$ are,
%
\[
\begin{array}{lrcl}
  \mbox{\small Average product correction} & \mathrm{COVp}(i,j) & = &  \mathrm{COV}(i,j) - \frac { \mathrm{COV}(i) \mathrm{COV}(j) } { \mathrm{COV} }, \\
  \mbox{\small Average sum correction}     & \mathrm{COVa}(i,j) & = &  \mathrm{COV}(i,j) - \left( \mathrm{COV}(i) + \mathrm{COV}(j) - \mathrm{COV} \right). \\
\end{array}
\]

\subsubsection{\prog{--C2, --C16}}
For all the covariation statistics (except RAF and RAFS), one can do a
16-component (C16) or a two-component (C2) calculation, depending on
whether it uses the 16 possible pair combinations, or those are group
in two classes depending on whether they form a Watson-Crick pair (6
cases, including U:G and G:U), or whether they do not (10 cases).

\rscape's default is the 16 component covariation statistic, unless
the number of sequences in the alignment is $\leq$ 8 or the length of
the alignment is $\leq$ 50, in which case it uses the two-class
covariation statistic.

\subsection{Search options}

\subsubsection{\prog{-s}} The ``two-set test'' option.
This option requires that a structure is provided with the alignment.
If option \prog{-s} is used, \rscape\ performs two independent test,
one for the given structure, another for all other possible pairs.
The default is a ``one-set test'' in which all possible pairs in the
alignment are tested equivalently.

\subsubsection{\prog{--fold}} A CaCoFold structure is computed that includes all significant base pairs. All files related to this
CaCoFold structure include the suffix \prog{.fold.}

When option \prog{--fold} is used, a file with the original alignment  annotated with the R-scape structure in Stockholm format is produced.
This alignment has the suffix \prog{.fold.sto}.

\subsubsection{\prog{--naive}} Reports the laundry list of all covariation scores, without any statistical significance (E-value)
associated to them. No null alignments are created.

\subsubsection{\prog{--tstart <n>}} Analyze starting from position $n >= 1$ in the alignment.

\subsubsection{\prog{--tend <n>}} Analyze ending at position $n <= L$ in the alignment.

\subsubsection{\prog{--window <n>}} \rscape\ can be run in a window scanning version for long alignments.
The window size is $n>0$.

\subsubsection{\prog{--slide <n>}} In scanning mode, this options sets the number of positions to move from window to window, $n >0$.

\subsubsection{\prog{--vshuffle}} Vertical shuffle, a developers tool. Before performing any analysis, it shuffles all residues in each alignment column independently.

\subsubsection{\prog{--cshuffle}} Column shuffle, a developers tool. Before performing any analysis, it shuffles all columns in the alignment.

\subsubsection{\prog{--givennull <f>}} Use histrogram provided in file $<f>$ as null.

\subsection{Input alignment options}

\subsubsection{\prog{-I <x>}} Only sequences with less than $0<x\leq 1$
pairwise similarity are considered in the analysis.  Pairwise \%
identity is defined as the ratio of identical positions divided by the
minimum length of the two sequences. If this option is not used all
(weighted) sequences are used in the analysis.

\subsubsection{\prog{--gapthresh <x>}} Only columns with less than $0<x\leq 1$ fraction of gaps are considered in the analysis.

\subsubsection{\prog{--consensus}} If the alignment has a GC ``seq\_cons'' field, only consensus positions will be analyzed.

\subsubsection{\prog{--submsa <n>}} Analyzes a random subset of the input alignment.

\subsubsection{\prog{--treefile <f>}} A phylogenetic tree in Newick format can be given (by default a tree is created
from the alignment using the program FastTree~\citep{Price10}).  \rscape\ checks that the  number of taxa and the names
of the taxa matches for all alignments analyzed.

\subsection{Options for producing a CaCoFold structure}

\subsubsection{\prog{--fold}}
When using the option \prog{--fold}, R-scape engages the CaCoFold
algorithm to produce a predicted structure. The CaCoFold algorithm
incorporates all positive (significantly covarying) basepairs, and
prevents any negative pair (pairs that have power of covariation but
not covariation) from happening. The CaCoFold algorithm uses a
recursive cascade of constrained foldings. The first fold uses the RBG probabilistic grammar, the rest use the G6X probabilistic grammar.

\noindent
Regarding the predicted structure, CaCoFold can use one two algorithms:

\subsubsection{\prog{--cyk}}        Default option. Each folding reports the structure with the best probability using the CYK algorithm.

\subsubsection{\prog{--decoding}}   This options returns the structure obtained by posterior decoding.

\noindent
Posterior decoding usually performs better than CYK. Both algorithm has the same algorithmic complexity. CYK is faster.

\noindent
Several additional options can be used in combination with   \prog{--fold},

\subsubsection{\prog{--refseq}}   By default the CaCoFold algorithm folds a profile sequence built from the alignment. Using this option, the sequence to fold is a consensus reference sequence.

\subsubsection{\prog{--E\_neg <x>}}   Pairs with E-value larger than the E-value cutoff but smaller than <x> will not be called negatives regardless of their covariation power. Default for E\_neg is 1.0.

\subsubsection{\prog{--lastfold}} This option forces one last alternative fold (using grammar G6X) after all covarying basepairs have already been
integrated into the structure. By default this last fold is not
performed. In the absence of any covarying basepair, one fold is
performed using grammar RBG.

\subsubsection{\prog{--show\_hoverlap}} This option leaves the alternative helices unmodified. By default, alternative structures are trimmed down to show no overlap with helices from the previous layers.

\subsubsection{\prog{--covmin <n>}} Minimum distance between position to report significant covariations. Default is 1, which means that significant covariations between
contiguous positions are reported.

\subsubsection{\prog{--allow\_negatives}} This option (just for developers) allows all basepairs to form regardless of their power.

\subsection{Options for importing a structure}

\rscape\ does not require to input a structure (either a RNA structure
or a protein contact map). By default \rscape\ analyzes all possible
pairs in the alignment.
\vspace{1mm}

\noindent
There are two ways to provide a contact map (or structure):

\begin{itemize}

\item By providing the alignment in Stockholm format with a ``ss\_cons'' field including the consensus
structure for the alignment. (For RNA alignments only.)

\item By analyzing a 3D structure provided in a PDB file. (For either RNA or peptide alignments.)\\
\end{itemize}

\noindent
These two methods can be combined together. For a nucleotide
alignment, if both a consensus structure is present in the alignment,
and a PDB file is provided (using option \prog{--pdb}), the
consensus structure will be extended by the information provided by
the pdbfile.  To ignore the consensus structure use
option \prog{--onlypdb}.\\

\noindent
From the PDB file we obtain three types of structural pairs:

\begin{itemize}
\item \textbf{Contacts:} defined as those two residues at a close spatial distance (specified by the user with option \prog{--cntmaxD}).
\item \textbf{Basepair:} RNA basepairs. \\
RNA basepairs are calculated using the program \prog{rnaview}~\citep{YangWesthof03}.

These RNA basepairs can be further classified in two types:
\begin{itemize}
\item \textbf{Watson-Crick basepairs:} the canonical RNA basepairs. mostly A:U, G:C, or G:U pairs. (H-bond interactions between two W-C faces in cis).
\item \textbf{Other basepairs:} the non-canonical RNA basepairs (all other types of H-bond interactions, 12 different types).
\end{itemize}
\end{itemize}

Contacts and RNA basepairs are extracted as follows:
\begin{itemize}

\item The spatial distance between any two residues is calculated as the  minimal Euclidean distance between any two atoms (excluding
 H atoms). Any two pairs at a distance not larger than a maximum value
(\texttt{contmaxD}) are called a ``contact''.

\item RNA basepairs are obtained using the program \texttt{rnaview}~\citep{YangWesthof03}\\
(http://ndbserver.rutgers.edu/ndbmodule/services/download/rnaview.html).\\
The RNA basepair annotation takes precedent over the annotation as ``contact''.
\end{itemize}

\vspace{2mm}
\noindent
The options that control the input of a structure or contact map are:

\subsubsection{\prog{--pdb <s>}} Reads a pdbfile associated to the alignment, and extracts the contacts
from it.

A ``.cmap'' file is produced reporting the structure obtained from the PDB file.

Option \prog{--pdb} is incompatible with \prog{--fold}.

\subsubsection{\prog{--cntmaxD <x>}} Maximum distance (in Angstroms) allowed between two residues to define a ``contact'' is $\langle x\rangle$.

\subsubsection{\prog{--cntmind <n>}} Minimum distance (in residue positions) in the backbone between two residues required to define a ``contact'' is $\langle n\rangle$.
\vspace{5mm}

\subsubsection{\prog{--onlypdb}} Reads the structure from the pdbfile and ignores the alignment consensus structure (if provided).

\subsubsection{\prog{--draw\_nonWC}} Adds the non-canonical basepairs into the structure graphical output. For clarity, the default is to draw only the Watson-Crick basepairs.
This option affects only the drawing of the structure. All basepairs (canonical or not) are used as part of the structure to perform the two-set statistical test.
\\\\

\noindent
Example of reading a structure from a PDB file for the FMN riboswitch:

\user{bin/R-scape --cntmaxD 4 --cntmind 3 --pdb tutorial/3f2q.pdb -s --onlypdb tutorial/RF00050.sto}\\

\noindent
This command line extracts contacts from the pdb file that are at a
Euclidean distance $\leq 4 \AA$ in the PDB structure, and such that
they are at least 3 residues apart in the backbone.

\noindent
The output is

\begin{sreoutput}
# R-scape :: RNA Structural Covariation Above Phylogenetic Expectation
# R-scape 0.8.1 (Jul 2018)
# Copyright (C) 2016 Howard Hughes Medical Institute.
# Freely distributed under the GNU General Public License (GPLv3).
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Two-set statistical test (one test for annotated basepairs, another for all other pairs)
#
# Structure obtained from the pdbfile
# ij in alignment | ij in pdbsequence | basepair type
# 3 218 | 1 112 | WWc
# 4 216 | 2 110 | CONTACT
# 4 217 | 2 111 | WWc
# 4 218 | 2 112 | CONTACT
# 5 216 | 3 110 | WWc
# 5 217 | 3 111 | CONTACT
# 6 215 | 4 109 | WWc
# 6 216 | 4 110 | CONTACT
# .
# .
# .
# 192 202 | 87 96 | WWc
# 192 203 | 87 97 | CONTACT
# 193 198 | 88 92 | CONTACT
# 193 201 | 88 95 | WWc
# 193 202 | 88 96 | CONTACT
# 195 197 | 89 91 | CONTACT
# 195 198 | 89 92 | WHt
# 198 200 | 92 94 | CONTACT
# 198 201 | 92 95 | CONTACT
# 205 207 | 99 101 | CONTACT
# PDB:      versions/rscape/rscape_v0.8/tutorial/3f2q.pdb
# contacts  169 (49 bpairs 35 wc bpairs)
# maxD      4.00
# mind      3
# distance  MIN
# L         139
# alen      221
# pdblen    112
# ::[[[[[[[[,,,,,,<<<_______>>>,((((<<<<_________________AA>>>>,,<<<----------<____>>>>,,,<<<<<_______>>>>>aa))AAAA----))aaaa,,,,,,]]]]]]]]::
# MSA RF00050_FMN.3f2q nseq 144 (144) alen 139 (221) avgid 69.18 (68.15) nbpairs 49 (0)
#
# Method Target_E-val [cov_min,conv_max] [FP | TP True Found | Sen PPV F]
# GTp    0.05         [-9.78,216.11]     [1 | 14 49 15 | 28.57 93.33 43.75]
#
#       left_pos       right_pos        score           E-value
#------------------------------------------------------------------
*	     171	     183	216.11095	1.6421e-10
*	     170	     184	211.69081	2.76699e-10
*	     192	     202	168.72417	4.95548e-08
*	       8	     213	149.71776	4.89982e-07
*	     172	     182	138.66664	1.84675e-06
*	     169	     185	137.23189	2.21548e-06
**	      16	      30	133.44999	3.53772e-06
*	       5	     216	131.02575	4.70876e-06
*	      84	     186	125.60806	9.0169e-06
*	      17	      29	112.04610	4.62895e-05
*	       7	     214	111.12654	5.13519e-05
*	       6	     215	96.43781	0.00029929
*	      36	      87	96.32752	0.00029929
*	      94	     163	78.81578	0.0024303
 	       7	     213	107.68588	0.0147937
\end{sreoutput}

\noindent
All coordinates are relative to the input alignment. The annotation of
all types of RNA basepairs (WWc, WWt, WHc,...) is produced by the
program \prog{rnaview}~\citep{YangWesthof03}.

\subsection{Options for type of pairs tested}

When performing the two-class statistical test (option \prog{-s})
using a pdbfile to read the structure, there are different options as
to which types of basepairs are used to define the sample size for the
basepairs test.

The options are:

\subsubsection{\prog{--samplecontacts}}
The basepair statistical test includes all the contacts identified in a PDB
or/and as a RNA secondary structure included with a input alignment in
Stockholm format.  This is the default option for amino acid
alignments if a PDB file is provided.

\subsubsection{\prog{--samplebp}} For RNA alignments with only.
The basepair statistical test includes basepairs of all 12 possible types.
This is the default option for RNA/DNA alignments if a PDB file is
provided.

\subsubsection{\prog{--samplewc}} For RNA alignments only.
The basepair statistical test includes only the canonical
(Watson-Crick/Watson-Crick type) basepairs (A:U, G:C, G:U).  This is
the default option for RNA/DNA alignments if a consensus secondary
structure is provided.

\subsection{Output options}

\subsubsection{\prog{--roc}}

Produces a tabular output that provides statistics for each score value.

File \emprog{tutorial/updated\_Arisong.roc} looks like:

\user{more tutorial/updated\_Arisong.roc}
\begin{sreoutput}
# MSA nseq 95 alen 65 avgid 66.352419 nbpairs 20 (20)
# Method: GTp
#cov_score  FP  TP Found  True  Negatives  Sen   PPV     F       E-value
121.79543   0   2  2      20    2060       10.00 100.00  18.18   4.07104e-05
121.44018   0   2  2      20    2060       10.00 100.00  18.18   4.29443e-05
121.08494   0   2  2      20    2060       10.00 100.00  18.18   4.53006e-05
120.72970   0   2  2      20    2060       10.00 100.00  18.18   4.53006e-05
...
\end{sreoutput}

This file produces a tabular output for each alignment as a function
of the covariation score, for plotting ROC curves. The values in the
file are described by the comment line. Notice that the number of
Trues (column 5) and Negatives (column 6) are fixed for a given
secondary structure and do not change.

\subsubsection{\prog{--outmsa <f>}} The actual alignment analyzed can be saved in Stockholm format to file $<$f$>$.

\subsubsection{\prog{--outtree <f>}} The phylogenetic tree (created using the program FastTree) can be saved in Newick format to file $<$f$>$.

\subsubsection{\prog{--savenull}} Saves a histogram with the null distribution to file rnafile msaname.null.

\subsection{Plotting options}

\subsubsection{\prog{--nofigures}} None of the graphical outputs are produced using this option.

\subsubsection{\prog{--r2rall}} Forces R2R to draw all positions in the alignment. By default only
those that are more than 50\% occupied or are base paired are
depicted.

\subsection{Other options}

\subsubsection{\prog{--seed <n>}} Sets the seed of the random number generator to $<$n$>$. Use n = 0 for a random seed.


\documentclass{article}
\usepackage[utf8]{inputenc}
\setlength\parindent{2pt}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage{siunitx}
\usepackage{float}
\usepackage{derivative}
\usepackage{amsmath,amssymb}

\title{SURP Computing Project 2020: Lechun Xing \\ CTA200H \\ Supervisor: Jonathan Braden}
\author{Lechung Xing - 1004705170 }
\date{May 8th 2020}

\begin{document}

\maketitle
% Part 1
\section{Clone folder from Github}
git branch:\\ * dev(development branch)\\   master(main branch)
% Part 2
\section{Units}
\subsection{(a) find dimensions of field $\phi$ and potential energy density $V(\phi)$}
Given: $[c]=light speed=LT^{-1}$ and
$[\hbar]=ML^2{T^{-1}}=[S]$

\vspace{3mm}
Action $S=\int{d^d{x}dt({(\dot{\phi}})^2}/{2c^2}-{(\partial_x{\phi})^2}/2-V(\phi))$

\vspace{3mm}
Replace
$[\dot{\phi}]=[\phi]T^{-1}$ and
$[\partial_x{\phi}]=[\phi]/x=[\phi]x^{-1}$ where [x]=[L]

\vspace{3mm}
Dimension $[S]=L^d{T}({[\phi]^2{T^{-2}}}/L^2{T^{-2}}-[\phi]^2{L^{-2}}-[V(\phi)]$

\vspace{3mm}
Simplify $[S]={L^{d-2}T}({[\phi]^2}-[\phi]^2-L^2[V(\phi)])=[\hbar]=ML^2{T^{-1}}$

\vspace{3mm}
After cancelling the ${[\phi]^2}$ terms and collecting the $L$ terms:

\vspace{3mm}
We derived the dimension of
$[V(\phi)]=ML^{2-d}T^{-2}$

\vspace{5mm}
Recall PDE: $\frac{1}{c^2}{\frac{\partial^2\phi}{\partial{t^2}}}-{\frac{\partial^2\phi}{\partial{x^2}}}+{\frac{\partial{V}}{\partial{\phi}}}=0$

\vspace{3mm}
The first 2 terms differ only by a constant $A$, they share the same dimension as in the Action expression [S] listed above. Thus, we can rewrite PDE:

\vspace{3mm}
$A[\phi]L^{-2}+[V][\phi]^{-1}=0$ Replace the dimension of [V] and collect $[\phi]$ terms:

\vspace{3mm}
We derived the dimension of
$[\phi]=(-A)^{-1/2}L[V]^{1/2}=(L^{4-d})^{1/2}M^{1/2}T^{-1}$, in which $d$ is spatial dimensions.

\subsection{(b) introduce scalar $\Lambda, x_0, t_0$, rewrite action S}

Given: $\phi=\Lambda{\bar{\phi}}$ and $x=x_0{\bar{x}}$ and $t=t_o{\bar{t}}$

\vspace{3mm}
Derive: $\dot{\phi}=\Lambda{\dot{\bar\phi}}=\Lambda{\partial_t}{\bar\phi}$ and $\partial_x{\phi}=\frac{\partial\phi}{\partial{x}}=\frac{\partial\phi}{\partial}{{(x_0\bar{x})}}=\frac{1}{x_0}{\partial_\bar{x}}{(\Lambda{\bar\phi})}=\frac{\Lambda}{x_0}{\partial_\bar{x}}{\bar\phi}$

\vspace{3mm}
Similarly convert the differentials: $dx=x_0{d\bar{x}}$ and $dt=t_0{d\bar{t}}$

\vspace{3mm}
Action $S=\int{{x_0}^d{(d\bar{x}})^d{(t_0{d\bar{t}})}[\frac{\Lambda^2{\dot{\bar{\phi}}}^2}{2c^2}-\frac{1}{2}{(\frac{\Lambda}{x_0})^2(\partial_{\bar{x}}{\bar\phi}})^2-V(\Lambda\bar\phi)]}$

\subsection{(c) rewrite PDE in terms of  derivatives of $\bar{t}, \bar{x}$}
Substitute 3 scalars into PDE: $\frac{1}{c^2}{\frac{\partial^2}{\partial{t^2}}(\Lambda\bar\phi)}-\frac{\partial^2}{\partial{x^2}}(\Lambda\bar\phi)+\frac{\partial{V}}{\partial(\Lambda\bar\phi)}=0$

\vspace{3mm}
Pull out $\Lambda$: $\frac{\Lambda}{{c^2}{t_0}^2}{{\partial^2}_{\bar{t}}\bar\phi}-{\frac{\Lambda}{{x_0}^2}{\partial^2_{\bar{x}}}}{\bar\phi}+{\frac{\partial{V(\bar\phi)}}{\partial{\bar\phi}}}=0$

%Which code file? Which eqn?
\subsection{(d) rewrite dimensionless eqn. in the code for  $\hbar=c=1$}
For $\hbar=c=1$:
$\frac{\Lambda}{{t_0}^2}{{\partial^2}_{\bar{t}}\bar\phi}-{\frac{\Lambda}{{x_0}^2}{\partial^2_{\bar{x}}}}{\bar\phi}+{\frac{\partial{V}}{\partial{\bar\phi}}}=0$ all expressed in terms of scaling factors or dimensionless quantities.

%Part 3
\section{Plot field evolution $\phi(x, t)$}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Phi_evo_default.pdf}
    \caption{Field evolution $\phi(t, x)$ plotted from $<fields.dat>$ with the normalized lattice sites $x\in{[0, 1024*dx]}$ and the normalized time steps $t\in{[0, 200]}$.}
    \label{fig:Q3}
\end{figure}

$\phi(t, x)$ values drop across all 1024 lattice sites as t increases. The slanted bands extending over Fig.1 indicate the fluctuations of $\phi(t, x)$ values among the overall decreasing trend for larger t.

% Part 4
\section{Energy conservation}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{E_dev_alph=4}
    \caption{Violation of Energy conservation vs. output time steps $dt_{out}$. Blue data is calculated from $<fields.dat>$ according to $E=\sum_i(\frac{1}{2}{\dot\phi_i}^2+\frac{1}{2}{(\nabla{\phi})^2}_i+V(\phi_i))$, $i$ means summation over 1024 lattice sites. Red dashed line is the power fitting model $E = a*{t}^b$, a, b are parameters. $t=dt'$ array, determined by $\alpha=dx/dt$ in $<evolve-scalar.f90>$}
    \label{fig:Q4}
\end{figure}

The size of Energy deviation $|E(t) - E(0)|$ seems to increases linearly with t. When $\alpha = 0.95$ or any value smaller than 1., the code does not output data file.

% Part 5
\section{Energy violation: adjust $\alpha$}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{E_dev_alph_all}
    \includegraphics[scale=0.7]{E_dev_alph_all_log}
    \caption{Combined $|E(t) - E(0)|$ plot for 6 $\alpha$ values. The bottom plot is with log y axis.}
    \label{fig:Q5}
\end{figure}

Fig.3 shows $|E(t) - E(0)|$ decreases as $\alpha$ increases from 1. to 32..
b = 1 if rounded to the nearest integer except for the $\alpha=32$ plot. However, today I realized this part is asking for how Energy conservation error is scaled with dt' instead of the time evolution of $E(t) - E(0)$. I noticed from Fig.3.(log scale) that for each dt', the Error size satisfies $\frac{Error[i]}{Error[j]}\propto{\frac{dt'[i]}{dt'[j]}}^{b}$, where i,j are indices from dt' array. Averaging over the 6 b values, I conclude $b\approx9$. This is the answer for Part 6.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.35]{E_dev_alph=1}
    \includegraphics[scale=0.35]{E_dev_alph=2}
    \includegraphics[scale=0.35]{E_dev_alph=4}
    \includegraphics[scale=0.35]{E_dev_alph=8}
    \includegraphics[scale=0.35]{E_dev_alph=16}
    \includegraphics[scale=0.35]{E_dev_alph=32}
    \caption{Power fit for different $\alpha$ curves as in Fig.3.}
    \label{fig:Q1}
\end{figure}

$\alpha=1., a =  10.603, b =  0.928$ with the variance in a and b to be 0.064 and 0.001, respectively. $\alpha=2., a = 0.019, b =  1.011$ with variance 2.408e-05 and 0.0003. $\alpha=4., a = 3.807e-05, b =  1.011$ with variance, 4.926e-08 and 0.0003. $\alpha=8., a = 7.518e-08, b =  1.011$ with variance, 9.774e-11 and 0.0003. $\alpha=16., a = 1.491e-10, b =  1.010$ with variance 1.927e-13 and 0.0003. $\alpha=32., a = 4.536e-12, b =  0.440$ with variance 9.780e-13 and 0.045.

\vspace{3mm}
Above fitting are irrelevant to this part. Please see Part 5 instead.

\section{Field evolution: adjust $\lambda$ the shape of $V(\phi)$}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Phi_evo_L1point5}
    \includegraphics[scale=0.7]{Phi_evo_L1point3}
    \caption{$\lambda=1.5 and 1.3$}
    \label{fig:Q1}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Phi_evo_Lpoint9}
    \includegraphics[scale=0.7]{Phi_evo_Lpoint8}
    \caption{$\lambda=0.9 and 0.8$}
    \label{fig:Q1}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Phi_evo_L1}
    \caption{$\lambda=1.$}
    \label{fig:Q1}
\end{figure}

When $\lambda$ is far away from $1.2$, I no longer see the clear cut (wall) between True vacuum and the False vacuum. especially when $\lambda=1.5>1.2$, field evolves to a stage filled with fluctuations.
\section{Field evolution: adjust initial conditions}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Phi_evo_8617}
    \includegraphics[scale=0.7]{Phi_evo_8618}
    \caption{(86, 17), (86,18) initial conditions}
    \label{fig:Q1}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Phi_evo_8719}
    \includegraphics[scale=0.7]{Phi_evo_8818}
    \caption{(87, 19), (88,18) initial conditions}
    \label{fig:Q1}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.7]{Phi_evo_8710}
    \caption{(87,10) initial conditions}
    \label{fig:Q1}
\end{figure}

For random seeds close to the default (87, 18), the field evolution does not change much in terms of the $\phi$ value difference above and below the wall.
\end{document}


{\color{indiagreen}\subsection{Prosti pad}}
\begin{center}
	\includegraphics[width=7cm, height=15cm,keepaspectratio=true]{Prosti_pad.PNG}
\end{center}
\begin{align*}
	v &= gt\\
	h &= \frac{gt^2}{2}\\
	v^2 &= 2gh\\
\end{align*}


\subsection{Moment}
\begin{description}
\item[Type:] Moment
\item[Description:]
 A moment of emission computed from a spectral map.  Available moments
 are:
 \begin{itemize}
 \item Zeroth Moment : Integrated emission over some channel range
 \item First Moment :  Centroid velocity over some channel range
 \item Second Moment : Velocity dispersion over some channel range
 \end{itemize}

\item[Constituents:]: BDP\_Image

\item[ADMIT Task:] AT\_Moment
  Keywords:  \\
     - moment to compute - 0,1,2 \\
     - channel (velocity, frequency) range over which to compute \\
     - method to compute, possible methods include:\\
           simple clip  \\
           primary beam weighted clip \\
           Gaussian fit  \\
           Using emission features from FeatureList BDP \\
     - optionally correct velocity dispersion for mean motion \\

\item[CASA Task(s):]  TBD

\item[Input BDP(s):]
A spectral cube.  Optional: FeatureList if using that method

\item[Output BDP(s):]

Two-dimensional image of the requested moment of emission, and thumbnail image

\end{description}


\chapter{APPENDIX A.1}
\vglue6pt
% For Appendix A.1
% Format the equation environment
\renewcommand{\theequation}{A.1.\arabic{equation}}
% Reset the counter
\setcounter{equation}{0}

\begin{table*}[!ht]
	{\setlength{\tabcolsep}{14pt}
		\caption{Example table in appendix.}
		\begin{center}
			\vspace{-6mm}
			\begin{tabular}{cccc}
				\hline \\[-2.45ex] \hline \\[-2.1ex]
				Column A & Column B & Column C & Column D \\
				\hline \\[-1.8ex]
				Row A & Row A & Row A & Row A \\
				Row B & Row B & Row B & Row B \\
				Row C & Row C & Row C & Row C \\
				[-0ex] \hline
			\end{tabular}
			\vspace{-6mm}
		\end{center}
		\label{TableA.1}}
\end{table*}

\begin{equation}
y_{t} = \phi_{1} y_{t-1} + \epsilon_{t}
\label{EqA.1.1}
\end{equation}

Each parameter is described. As seen in equation \eqref{EqA.1.1}, or in \ref{EqA.1.1}.

\begin{equation}
y_{t} = \phi_{1} y_{t-1} + \epsilon_{t}
\label{EqA.1.2}
\end{equation}

\vspace*{12pt}

\section*{APPENDIX A.2}
\vglue6pt
% For Appendix A.2
% Format the equation environment
\renewcommand{\theequation}{A.2.\arabic{equation}}
% Reset the counter
\setcounter{equation}{0}

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed ac augue vel dui adipiscing placerat et nec metus. Donec bibendum sodales mollis. Cras in lacus justo, at vestibulum quam. Sed semper, est sit amet consectetur ornare, leo est lacinia velit, adipiscing elementum lectus felis at sem.

\begin{equation}
y_{t} = \phi_{1} y_{t-1} + \epsilon_{t}
\label{EqA.2.1}
\end{equation}

Each parameter is described. As seen in equation \eqref{EqA.2.1}, or in \ref{EqA.2.1}.

\newpage

\chapter{APPENDIX B.1}
\vglue12pt
% For Appendix B.1
% Format the equation environment
\renewcommand{\theequation}{B.1.\arabic{equation}}
% Reset the counter
\setcounter{equation}{0}

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed ac augue vel dui adipiscing placerat et nec metus. Donec bibendum sodales mollis. Cras in lacus justo, at vestibulum quam. Sed semper, est sit amet consectetur ornare, leo est lacinia velit, adipiscing elementum lectus felis at sem.

\begin{equation}
y_{t} = \phi_{1} y_{t-1} + \epsilon_{t}
\label{EqB.1.1}
\end{equation}
Each parameter is described. As seen in equation \eqref{EqB.1.1}, or in \ref{EqB.1.1}.

\begin{equation}
y_{t} = \phi_{1} y_{t-1} + \epsilon_{t}
\label{EqB.1.2}
\end{equation}

Each parameter is described. As seen in equation \eqref{EqB.1.2}, or in \ref{EqB.1.2}.

\begin{table*}[!ht]
	{\setlength{\tabcolsep}{14pt}
		\caption{Example table in appendix.}
		\begin{center}
			\vspace{-6mm}
			\begin{tabular}{cccc}
				\hline \\[-2.45ex] \hline \\[-2.1ex]
				Column A & Column B & Column C & Column D \\
				\hline \\[-1.8ex]
				Row A & Row A & Row A & Row A \\
				Row B & Row B & Row B & Row B \\
				Row C & Row C & Row C & Row C \\
				[-0ex] \hline
			\end{tabular}
			\vspace{-6mm}
		\end{center}
		\label{TableB.1}}
\end{table*}

\vspace*{18pt}

\section*{APPENDIX B.2}
\vglue6pt
% For Appendix B.2
% Format the equation environment
\renewcommand{\theequation}{B.2.\arabic{equation}}
% Reset the counter
\setcounter{equation}{0}

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed ac augue vel dui adipiscing placerat et nec metus. Donec bibendum sodales mollis. Cras in lacus justo, at vestibulum quam. Sed semper, est sit amet consectetur ornare, leo est lacinia velit, adipiscing elementum lectus felis at sem.

\begin{equation}
y_{t} = \phi_{1} y_{t-1} + \epsilon_{t}
\label{EqB.2.1}
\end{equation}

Each parameter is described. As seen in equation \eqref{EqB.2.1}, or in \ref{EqB.2.1}.

\begin{table*}[!ht]
	{\setlength{\tabcolsep}{14pt}
		\caption{Example table in appendix.}
		\begin{center}
			\vspace{-6mm}
			\begin{tabular}{cccc}
				\hline \\[-2.45ex] \hline \\[-2.1ex]
				Column A & Column B & Column C & Column D \\
				\hline \\[-1.8ex]
				Row A & Row A & Row A & Row A \\
				Row B & Row B & Row B & Row B \\
				Row C & Row C & Row C & Row C \\
				[-0ex] \hline
			\end{tabular}
			\vspace{-6mm}
		\end{center}
		\label{TableB.2}}
\end{table*}


\section{Programming Security}

\subsection{Challenge-Response}\label{chalres}
You insert a car key into a car engine. The engine would now like to know
that this key is valid.

Let $K$ be the key, and $E$ the engine. Then:
\begin{gather}
    \begin{align}
        &E \rightarrow K: N \\
        &K \rightarrow E: {\{E, N\}}_{K}
    \end{align}
\end{gather}

Here, N is a challenge sent by the engine to the key in step (1).
The key responds by encrypting the text. Iff the engine can decrypt
and get valid number, will it start.

\subsection{Needham-Schroeder Protocol}
A primitive, three-way~\nameref{keydistribution} process. Also known
to inspire Kerberos%
\footnote{originally from Greek/Roman mythology, a three-headed dog, or ``hellhound'',
    which guards the entrance of Hades.},
the security protocol in windows. The word \textit{primitive} is used to
emphasize that the protocol comes from an age where attacks were
mostly thought of as external-to-internal penetrations, i.e.\
one needs to assume that attacks come from external networks,
by people without means of authentication. Hence, the protocol
fails to defend against users of it's own system.

Having defined key-distribution in section~\ref{keydistribution}, the following should
be somewhat intuitive:
\setcounter{equation}{0}
\begin{gather}
    \begin{align}
        &A \rightarrow S: A,B,N_{A}  \\
        &S \rightarrow A: {{\{N_A, K_{AB}, B, \{K_{AB}, A\}}_{K_{BS}}\}}_{K_{AS}} \\
        &A \rightarrow B: \{K_{AB}, A\}_{K_{BS}} \\
        &B \rightarrow A: \{N_B\}_{K_{AB}} \\
        &A \rightarrow B: \{N_B-1\}_{K_{AB}}
    \end{align}
\end{gather}
The difference from plain~\nameref{keydistribution} is that we here introduce the
concept of \nameref{nonce}s.
In step 1, Alices attaches her nonce $N_{A}$ when talking to Sam.
The nonce will be used in Sam's reply. That way, Alice can know that
she is not recieving an old/corrupted message.

The entitiy $\kappa = $ $\{K_{AB}, A\}_{K_{BS}}$ represents the the shared key
between Alice and Bob, encrypted such that only Bob can read it (denoted by the
subscript $_{K_{BS}}$).
$K_{AB}$ is the key, and the $A$ is included such that Bob knows it is
inteded for his use to Alice.
In step 3, Alice sends $\kappa$ to Bob. Bob can decrypt with his key, given
to him earlier by Sam. Finally, to ensure that no data has been corrupted
by Eve or other errors, the last two steps verify the legimiticay of the keys.
Should Alice fail to return back $N_{B} - 1$, that is, a nonce from Bob$ - 1$,
then Bob can know that he is not talking to a valid user.

One easy to understand flaw in this system is that if someone is able
to obtain Alice's identifier, they could easily impersonate Alice.
The only required step is to contact Sam, get keys, and then communicate.
Alice cannot be aware of this since she never initiated the requests.
To revoke the damage done, Sam would have to keep logs of every issued key
and thereafter alert everyone who has recieved Alice's key that she
was compromised.


\chapter{Parameter-Based Policy Gradient}
\label{ch:parameter_based_policy_gradient}

The learning algorithms derived from the policy gradient theorems look for a set of optimal controller parameters by perturbing the control signal. Indeed, the policy gradient theorem holds only for stochastic policies, which explore in the action space.
However, a significant problem with this sampling strategy is that the high variance in their gradient estimates leads to slow convergence. On the other hand, parameter-based algorithms such as \gls{PGPE} look for an optimal solution to the control problem by directly perturbing the controller parameters. These algorithms thus employ a deterministic controller and explore in the parameters space. However, in its original version, \gls{PGPE} was conceived for episodic environments and extended to continuing environment by artificially truncating the experiments lifespan. In this section we propose a parameter-based version of the policy gradient theorem, which can be used to derive online learning algorithms both in the risk-neutral and the risk-sensitive frameworks. The following discussion starts from and generalizes the results presented in \cite{miyamae2010natural}, where the authors propose an online natural \gls{PGPE} algorithm for risk-neutral problems. Given the difficulties to extend the policy gradient theorem to the risk-sensitive discounted formulation, here we consider only the average reward formulation.

\section{Risk-Neutral Framework}
Let us consider a deterministic controller $F: \S \times
\Theta \to \A$ that, given a set of parameters $\theta \in \Theta \subseteq
\R^{D_\theta}$, maps a state $s \in \S$ to an action $a = F(s; \theta) =
F_\theta(s) \in \A$. The policy parameters are drawn from a probability distribution $p_\xi$, with hyper-parameters $\xi \in \Xi \subseteq \R^{D_\xi}$. Combining these two
hypotheses, the agent follows a stochastic policy $\pi_\xi$ defined by
\begin{equation}
	\forall B \in \calA ,\ \pi_\xi(s,B) = \pi(s, B; \xi) = \int_\Theta p_\xi(\theta)
	\ind{F_{\theta}(s)\in B} d\theta
\end{equation}
In the risk-neutral setting, the goal of the agent is to find a set of hyper-parameters $\xi_*$ that maximizes the average reward obtained over a single time-step
\begin{equation*}
	\xi_* = \argmax_\xi \rho(\xi)
\end{equation*}
Following the policy gradient approach, we iteratively update the parameters in the gradient ascent direction
\begin{equation*}
	\xi_{k+1} = \xi_k + \alpha_k \nabla_\xi \rho(\xi_k)
\end{equation*}
\begin{theorem}[Risk-Neutral Parameter-Based Policy Gradient]
	Let $p_\xi$ be differentiable with respect to $\xi$, then the gradient of the average reward is given by
	\begin{equation}
		\nabla_\xi \rho(\xi) = \E[\substack{S \sim d^\xi\\\theta \sim p_\xi}]{\nabla_\xi \log p_\xi(\theta) Q_{\pi_\xi}(S, \theta)}
	\end{equation}
	where we denoted $Q_\xi(S, \theta) = Q_\xi(S, F_\theta(S))$.
\end{theorem}
\begin{proof}
Thanks to the likelihood ratio technique, we have
\begin{equation*}
	\begin{split}
	\nabla_\xi \pi_\xi(s,a) &= \int_\Theta \nabla_\xi p_\xi(\theta) \ind{F_\theta(s)=a} d\theta
		\\
		&= \int_\Theta p_\xi(\theta) \nabla_\xi \log p_\xi(\theta) \ind{F_\theta(s)=a} d\theta
		\\
		&= \E[\theta \sim p_\xi]{\nabla_\xi \log p_\xi(\theta) \ind{F_\theta(s)=a}}
	\end{split}
\end{equation*}
hence, the policy $\pi_\xi$ is differentiable with respect to $\xi$ and the standard policy gradient theorem holds
\begin{equation*}\label{eq:policy_gradient}
	\begin{split}
		\nabla_\xi \rho(\xi) &= \E[\substack{S \sim d^\xi\\A \sim \pi_\xi}]{\nabla_\theta\log
				\pi_\theta(S,A) Q_{\theta}(S, A)}\\
						  &= \int_{\S} d^\xi(s) \int_{\A} \pi_{\xi}(s, a)
		\nabla_{\xi} \log \pi_{\xi}(s,a) Q_{\pi_\xi}(s,a) da ds\\
		&= \int_{\S} d^\xi(s) \int_{\A}
				\nabla_{\xi} \pi_{\xi}(s,a) Q_{\pi_\xi}(s,a) da ds
	\end{split}
\end{equation*}
Plugging the first equation in the inner integral and exchanging the integral over the action space with the expectation yields
\begin{equation*}
	\begin{split}
		 \int_{\A} \nabla_{\xi} \pi_{\xi}(s,a) Q_{\pi_\xi}(s,a) da
			&= \E[\theta \sim p_\xi]{\nabla_\xi \log p_\xi(\theta) \int_\A \ind{F_\theta(s)=a}
			Q_{\pi_\xi}(s,a) da}\\
			&= \E[\theta \sim p_\xi]{\nabla_\xi \log p_\xi(\theta) Q_{\pi_\xi}(s, F_\theta(s))}\\
			&= \E[\theta \sim p_\xi]{\nabla_\xi \log p_\xi(\theta) Q_{\pi_\xi}(s, \theta))}
	\end{split}
\end{equation*}
Finally, plugging this equation in the policy gradient theorem, we obtain the result
\begin{equation*}
	\nabla_\xi \rho(\xi) = \E[\substack{S \sim d^\xi\\\theta \sim p_\xi}]{\nabla_\xi \log p_\xi(\theta) Q_{\pi_\xi}(S, \theta)}
\end{equation*}
\end{proof}
This expression is very similar to the original policy gradient theorem, but
the expectation is taken over the controller parameters instead of the action space and we have the likelihood score the controller parameters distribution instead of that of the stochastic policy. Thus, we might interpret this result as if the agent directly selected the parameters $\theta$ according to a policy $p_\xi$, which then lead to an action through the deterministic mapping $F_\theta$. Therefore, it is as if the agent's policy was in the parameters space and not in the control space. As in the standard policy gradient methods, we can add a state-dependent baseline $B_\xi(S)$ to the gradient without increasing the bias
\begin{equation}
	\nabla_\xi \rho(\xi) = \E{\nabla_\xi \log p_\xi(\theta) \left(Q_{\pi_\xi}(S,
						\theta) - B_\xi(S)\right)}
\end{equation}
Indeed,
\begin{equation*}
	\begin{split}
		\E{\nabla_\xi \log p_\xi(\theta) B(S)}
			&= \int_\S d^\xi(s) \int_\Theta p_\xi(\theta) \nabla_\xi \log
		p_\xi(\theta) B_\xi(s) d\theta ds\\
		&= \int_\S d^\xi(s) B_\xi(s) ds \underbrace{\int_\Theta \nabla_\xi p_\xi(\theta)
	d\theta}_{\nabla_\xi 1 = 0} = 0
	\end{split}
\end{equation*}
This result can be used to design several actor-only or actor-critic algorithms that are the parameter-based equivalents of the traditional control-based policy-gradient algorithms discussed in the previous sections. Algorithm \ref{algo:ACPGPE} reports the pseudocode for a parameter-based actor-critic algorithm, which is one of the learning methods that will be used in the numerical applications.

\begin{algorithm}[t!]
	\caption{Actor-Critic PGPE}
	\label{algo:ACPGPE}
	\begin{algorithmic}[0]
			\Require{\\
				\begin{itemize}
					\item Deterministic controller $F: \S \times \Theta \to \A$
					\item Initial hyper-parameters $\xi_0$
					\item Initial critic parameters $\psi_0$
					\item Learning rate $\{\alpha_k\}$
					\item Momentum parameter $\lambda$
				\end{itemize}}
			\Ensure Approximation of the optimal hyper-parameters $\xi_*$
			\begin{algorithmic}[1]
			\State Initialize $k = 0$, average reward $\widehat{\rho}_0 = 0$ and eligibility trace $e_{-1} = 0$
			\Repeat
				\State Observe current state $s_k$
				\State Simulate controller parameters $\theta_k \sim p_{\xi_k}$
				\State Perform action $a_k = F_{\theta_k}(s_k)$ and receive reward $r_{k+1}$
				\State Update average reward estimate $\widehat{\rho}_{k+1} = \widehat{\rho}_{k} + \alpha_k (r_{k+1} - \widehat{\rho}_{k})$
				\State Compute TD error $\widehat{\delta}_k = r_{k+1} - \widehat{\rho}_{k+1} + \widehat{V}_{\psi_k}(s_{k+1}) - \widehat{V}_{\psi_k}(s_k)$
				\State Update critic parameters $	\psi_{k+1} = \psi_k + \alpha_k \widehat{\delta}_k \nabla_\psi \widehat{V}_{\psi_k}(s_k)$ .
				\State Update eligibility trace $e_k = \lambda e_{k-1} + \nabla_\xi\log p_\xi(\theta_k)$
				\State Update hyper-parameters $\xi_{k+1} = \xi_k + \alpha_k \widehat{\delta}_{\psi_k}^k e_k $.
				\State $k \leftarrow k + 1$
			\Until{converged}
		\end{algorithmic}
	\end{algorithmic}
\end{algorithm}

\subsection{Parameter-Based Natural Policy Gradient}
Given the interpretation of $p_\xi$ as a parameter-based stochastic policy, we can easily generalize the natural policy gradient methods by introducing the parameter-based Fischer information matrix
\begin{equation}
	F_\xi = \E[\theta\sim p_\xi]{\nabla_\xi\log p_\xi(\theta) \nabla_\theta\log p_\xi(\theta)^T}
\end{equation}
Natural parameter-based policy gradient methods update the hyper-parameters following the natural gradient direction
\begin{equation}
	\widetilde{\nabla}_\xi \rho(\xi) = F_\xi^{-1} \nabla_\xi \rho(\xi)
\end{equation}
The advantage of this approach is that, for some particular probability distributions, the inverse of the Fisher information matrix can be computed analytically. This is for instance the case if the controller parameters are normally distributed. This choice leads to an efficient online algorithm called \glsfirst{NPGPE} \cite{miyamae2010natural}. More in detail, suppose that
\begin{equation*}
	\theta \sim \calN(\mu, \Gamma \Gamma^T)
\end{equation*}
where $\mu\in\R^n$ is the controller parameters mean and $\Gamma \in \R^{n\times n}$ denotes the Cholesky factor of the distribution covariance matrix $\Sigma$. The hyper-parameters are thus given by
\begin{equation*}
	\xi = (\mu, \Gamma_1, \Gamma_2, ... , \Gamma_n) \in \R^\frac{n^2 + 3n}{2}
\end{equation*}
where we denoted $\Gamma_k = \Gamma_{k,k:n}^T$ the $k$-th column of the Cholesky factor. For this distribution, the parameter-based policy gradient is given by
\begin{equation}
	\nabla_\mu \log p_\xi(\theta) = \Sigma^{-1} (\theta - \mu)
\end{equation}
\begin{equation}
	\nabla_{\Gamma_k} \log p_\xi(\theta) = \begin{bmatrix} 0 & I_{\bar{k}} \end{bmatrix} \left(\Gamma^{-1} Y - \diag\left(\Gamma^{-1}\right)\right) e_k
\end{equation}
where $Y = \Gamma^{-T} (\theta - \mu) (\theta - \mu)^T \Gamma^{-1}$. These formulas might be used in a standard \gls{PGPE} algorithm. However, the inversion of $\Gamma$ is computationally expensive except for some very simple cases, for instance when $\Sigma$ is a diagonal matrix. In \cite{sun2009efficient}, the authors proved that the Fisher information matrix for this distribution is a block diagonal matrix $F_\xi = \diag\{B_0, B_1, ..., B_n\}$ where
\begin{equation*}
	\begin{cases}
		B_0 &= \Sigma^{-1}\\
		B_k &= \begin{bmatrix}
				0 & I_{\bar{k}}
		\end{bmatrix} \Gamma^{-1} (e_k e_k^T + I) \Gamma^{-T} \begin{bmatrix}
						0 \\ I_{\bar{k}}
				\end{bmatrix}
	\end{cases}
\end{equation*}
where $e_k$ is the $k$-th element of the canonical basis of $\R^n$ and $I_{\bar{k}}$ is $n - k + 1$-dimensional identity matrix. In \cite{akimoto2010bidirectional}, the authors found the following analytical expression for the inverse of each block
\begin{equation*}
	\begin{cases}
		B_0^{-1} &= \Sigma\\
		B_k^{-1} &= \begin{bmatrix}
				0 & I_{\bar{k}}
		\end{bmatrix} \Gamma^T \left(\begin{bmatrix}
			0 & 0 \\
			0 & I_{\bar{k}}\\
		\end{bmatrix}   -\frac{1}{2} e_k e_k^T\right) \Gamma \begin{bmatrix}
						0 \\ I_{\bar{k}}
				\end{bmatrix}
	\end{cases}
\end{equation*}
Multiplying the policy gradients by the inverse Fisher informatio matrix $F_\xi^{-1} = \diag\{B_0^{-1}, B_1^{-1}, ..., B_n^{-1}\}$, we obtain the following natural policy gradients
\begin{equation}
	\widetilde{\nabla}_\mu \log p_\xi(\theta) = \theta - \mu
\end{equation}
\begin{equation}
	\widetilde{\nabla}_\Gamma \log p_\xi(\theta) = \left(\triu(Y) - \frac{1}{2} \diag(Y) - \frac{1}{2} I \right) \Gamma
\end{equation}
The natural policy gradients can be computed in $O(n^3)$, which is a sensible improvement compared to the $O(n^6)$ complexity for the standard \gls{PGPE} updates. Let us remark that for an independent multi-variate Gaussian distribution, i.e. a diagonal covariance matrix $\Sigma$, the updates can be performed in $O(n)$. However, this approach neglects the dependences among parameters and could lead to suboptimal performances. On the other hand. updating the full Cholesky factor introduces many more parameters to be learned which may slow down the convergence of the method. The resulting \gls{NPGPE} algorithm is reported in Algorithm \ref{algo:NPGPE}. Given the parameter-based policy gradient algorithm, we can easily combine the actor-critic approach and the natural gradient technique in a parameter-based natural actor-critic algorithm, which is outlined in Algorithm \ref{algo:NACPGPE}.
\begin{algorithm}[t!]
	\caption{NPGPE}
	\label{algo:NPGPE}
	\begin{algorithmic}[0]
		\Require{\\
			\begin{itemize}
				\item Deterministic controller $F: \S \times \Theta \to \A$
				\item Initial hyper-parameters $\xi^0$
				\item Learning rate $\{\alpha_k\}$
				\item Momentum parameter $\lambda$
			\end{itemize}}
		\Ensure Approximation of the optimal hyper-parameters $\xi_*$
		\begin{algorithmic}[1]
		\State Initialize $k = 0$, average reward $\widehat{\rho}_0 = 0$ and eligibility trace $e_{-1} = 0$
		\Repeat
			\State Observe current state $s_k$
			\State Draw $\zeta_k \sim \calN(0, I_n)$
			\State Compute controller parameters $\theta_k = \mu_k + \Gamma^T \zeta_k$
			\State Perform action $a_k = F_{\theta_k}(s_k)$ and receive reward $r_{k+1}$
			\State Update average reward estimate $\widehat{\rho}_{k+1} = \widehat{\rho}_{k} + \alpha_k (r_{k+1} - \widehat{\rho}_{k})$
			\State Compute natural policy gradients
				\begin{equation*}
					\begin{split}
						\widetilde{\nabla}_\mu \log p_{\xi_k}(\theta_k) &= \theta_k - \mu_k\\
						\widetilde{\nabla}_\Gamma \log p_{\xi_k}(\theta_k) &= \left(\triu(\zeta_k \zeta_k^T) - \frac{1}{2} \diag(\zeta_k \zeta_k^T) - \frac{1}{2} I \right) \Gamma
					\end{split}
				\end{equation*}
			\State Update eligibility trace $e_{k} = \lambda e_{k-1} + \nabla_\xi \log p_{\xi_k}(\theta_k)$
			\State Update hyper-parameters $\xi_{k+1} = \xi_k + \alpha_k (r_{k+1} - \widehat{\rho}_{k}) e_k$
			\State $k \leftarrow k + 1$
		\Until{converged}
		\end{algorithmic}
	\end{algorithmic}
\end{algorithm}

\begin{algorithm}[t!]
	\caption{Natural Actor-Critic PGPE}
	\label{algo:NACPGPE}
	\begin{algorithmic}[0]
		\Require{\\
			\begin{itemize}
				\item Deterministic controller $F: \S \times \Theta \to \A$
				\item Initial hyper-parameters $\xi^0$
				\item Initial critic parameters $\psi_0$
				\item Actor learning rate $\{\alpha_k\}$
				\item Critics learning rate $\{\beta_k\}$
				\item Momentum parameter $\lambda$
			\end{itemize}}
		\Ensure Approximation of the optimal hyper-parameters $\xi_*$
		\begin{algorithmic}[1]
		\State Initialize $k = 0$, average reward $\widehat{\rho}_0 = 0$ and eligibility trace $e_{-1} = 0$
		\Repeat
			\State Observe current state $s_k$
			\State Draw $\zeta_k \sim \calN(0, I_n)$
			\State Compute controller parameters $\theta_k = \mu_k + \Gamma^T \zeta_k$
			\State Perform action $a_k = F_{\theta_k}(s_k)$ and receive reward $r_{k+1}$
			\State Update average reward estimate $\widehat{\rho}_{k+1} = \widehat{\rho}_{k} + \alpha_k (r_{k+1} - \widehat{\rho}_{k})$
			\State Compute TD error $\widehat{\delta}_k = r_{k+1} - \widehat{\rho}_{k+1} + \widehat{V}_{\psi_k}(s_{k+1}) - \widehat{V}_{\psi_k}(s_k)$
			\State Update critic parameters $	\psi_{k+1} = \psi_k + \alpha_k \widehat{\delta}_k \nabla_\psi \widehat{V}_{\psi_k}(s_k)$ .
			\State Compute natural policy gradients
				\begin{equation*}
					\begin{split}
						\widetilde{\nabla}_\mu \log p_{\xi_k}(\theta_k) &= \theta_k - \mu_k\\
						\widetilde{\nabla}_\Gamma \log p_{\xi_k}(\theta_k) &= \left(\triu(\zeta_k \zeta_k^T) - \frac{1}{2} \diag(\zeta_k \zeta_k^T) - \frac{1}{2} I \right) \Gamma
					\end{split}
				\end{equation*}
			\State Update eligibility trace $e_{k} = \lambda e_{k-1} + \widetilde{\nabla}_\xi \log p_{\xi_k}(\theta_k)$
			\State Update hyper-parameters $\xi_{k+1} = \xi_k + \alpha_k \widehat{\delta}_k e_k$.
			\State $k \leftarrow k + 1$
		\Until{converged}
		\end{algorithmic}
	\end{algorithmic}
\end{algorithm}
\clearpage

\section{Risk-Sensitive Framework}
In the risk-sensitive setting, the agent tries to find a policy that maximizes the Sharpe ratio. Following the same reasoning of the control-based approach, we simply need to estimate the gradient of the average square reward $\eta(\xi)$. The parameter-based policy gradient generalizes without any additional effort
\begin{theorem}[Risk-Sensitive Parameter-Based Policy Gradient]
	Let $p_\xi$ be differentiable with respect to $\xi$, then the gradient of the average square reward is given by
	\begin{equation}
		\nabla_\xi \eta(\xi) = \E[\substack{S \sim d^\xi\\\theta \sim p_\xi}]{\nabla_\xi \log p_\xi(\theta) W_{\pi_\xi}(S, \theta)}
	\end{equation}
	where we denoted $W_\xi(S, \theta) = W_\xi(S, F_\theta(S))$.
\end{theorem}
Again, we can add a state-dependent baseline $B_\xi(S)$ to the gradient without increasing the bias
\begin{equation}
	\nabla_\xi \eta(\xi) = \E{\nabla_\xi \log p_\xi(\theta) \left(W_{\pi_\xi}(S,
						\theta) - B_\xi(S)\right)}
\end{equation}
This leads in a straightforward way to a \gls{RSPGPE} algorithm.

\subsection{Parameter-Based Natural Policy Gradient}
The natural policy gradient idea can be trivially extended to the gradient of the average square reward. This leads to the following natural gradient estimate
\begin{equation}
	\begin{split}
	\widetilde{\nabla}_\xi \eta(\xi) &=  F_\xi^{-1} \nabla_\xi \eta(\xi)\\
									 &=  F_\xi^{-1} \E[\substack{S \sim d^\xi\\\theta \sim p_\xi}]{\nabla_\xi \log p_\xi(\theta) \left(W_{\pi_\xi}(S, \theta) - B_\xi(S)\right)}\\
									 &=\E[\substack{S \sim d^\xi\\\theta \sim p_\xi}]{\widetilde{\nabla}_\xi \log p_\xi(\theta) \left(W_{\pi_\xi}(S, \theta) - B_\xi(S)\right)}
	\end{split}
\end{equation}
which can be easily estimated via Monte Carlo. Combining this natural gradient with the one for the average reward, we obtain the natural policy gradient for the Sharpe ratio
\begin{equation}
	\widetilde{\nabla}_\xi \text{Sh}(\xi) = F_\xi^{-1} \nabla_\xi \text{Sh}(\xi) = \frac{\eta(\xi) \widetilde{\nabla}_\xi \rho(\xi) - \frac{1}{2} \rho(\xi) \widetilde{\nabla}_\xi \eta(\xi)}{\Lambda(\xi) \sqrt{\Lambda(\xi)}}
\end{equation}
Hence, it is trivial to derive a \gls{RSNPGPE} and its pseudocode reported in Algorithm \ref{algo:RSNPGPE}. Similarly to the risk-neutral case, it is simple to introduce a critic both for the average reward and for the average square reward.

\begin{algorithm}[t!]
	\caption{Risk-Sensitive NPGPE}
	\label{algo:RSNPGPE}
	\begin{algorithmic}[0]
		\Require{\\
			\begin{itemize}
				\item Deterministic controller $F: \S \times \Theta \to \A$
				\item Initial hyper-parameters $\xi^0$
				\item Learning rate $\{\alpha_k\}$
				\item Momentum parameter $\lambda$
			\end{itemize}}
		\Ensure Approximation of the optimal hyper-parameters $\xi_*$
		\begin{algorithmic}[1]
		\State Initialize $k = 0$, average reward $\widehat{\rho}_0 = 0$, average square reward $\widehat{\eta}_0 = 0$ and eligibility trace $e_{-1} = 0$
		\Repeat
			\State Observe current state $s_k$
			\State Draw $\zeta_k \sim \calN(0, I_n)$
			\State Compute controller parameters $\theta_k = \mu_k + \Gamma^T \zeta_k$
			\State Perform action $a_k = F_{\theta_k}(s_k)$ and receive reward $r_{k+1}$
			\State Update average reward estimate $\widehat{\rho}_{k+1} = \widehat{\rho}_{k} + \alpha_k (r_{k+1} - \widehat{\rho}_{k})$
			\State Update average square reward estimate $\widehat{\eta}_{k+1} = \widehat{\eta}_{k} + \alpha_k (r_{k+1}^2 - \widehat{\eta}_{k})$
			\State Compute natural policy gradients
				\begin{equation*}
					\begin{split}
						\widetilde{\nabla}_\mu \log p_{\xi_k}(\theta_k) &= \theta_k - \mu_k\\
						\widetilde{\nabla}_\Gamma \log p_{\xi_k}(\theta_k) &= \left(\triu(\zeta_k \zeta_k^T) - \frac{1}{2} \diag(\zeta_k \zeta_k^T) - \frac{1}{2} I \right) \Gamma
					\end{split}
				\end{equation*}
			\State Update eligibility trace $e_{k} = \lambda e_{k-1} + \nabla_\xi \log p_{\xi_k}(\theta_k)$
			\State Compute natural gradient $\widetilde{\nabla}_\xi \rho(\xi_k) = e_k \left(r_{k+1} - \widehat{\rho}_{k}\right)$
			\State Compute natural gradient $\widetilde{\nabla}_\xi \eta(\xi_k) = e_k \left(r_{k+1}^2 - \widehat{\eta}_{k}\right)$
			\State Compute Sharpe ratio natural gradient
			\begin{equation*}
				\widetilde{\nabla}_\xi \text{Sh}(\xi_k) = \frac{\widehat{\eta}_{k} \widetilde{\nabla}_\xi \rho(\xi_k) - \frac{1}{2} \widehat{\rho}_{k} \widetilde{\nabla}_\xi \eta(\xi_k)}{\widehat{\Lambda}_k \sqrt{\widehat{\Lambda}_k}}
			\end{equation*}
			\State Update hyper-parameters $\xi_{k+1} = \xi_k + \alpha_k \widetilde{\nabla}_\xi \text{Sh}(\xi_k)$
			\State $k \leftarrow k + 1$
		\Until{converged}
		\end{algorithmic}
	\end{algorithmic}
\end{algorithm}


\section{Introduction}
\begin{frame}{Basic Problem Setting}
  \onslide<+->{\blue{\textbf{Goal}}: Construct a 20~newsgroups document (binary) classifier}
  \vfill
  \onslide<+->{\textbf{Standard Assumption}: Given random variables, ${\X \in \domainX}$ and ${\y \in \domainY}$}
  \onslide<+->{%
    \begin{equation}\label{eq:BaseSample}
      \train \sim \pDist(\X, \y) \onslide<+->{= \pDist(\X) \pDist(\y \vert \X)}
    \end{equation}
    \vspace{-6pt}
  }

  \onslide<+->{By Bayes' rule:}
  \onslide<+->{%
    \begin{equation}
      \pDist(\X) = \pDist(\y = \pcls) \pDist(\X \vert \y = \pcls) + \pDist(\y = \ncls) \pDist(\X \vert \y = \ncls)
    \end{equation}
    \vspace{-6pt}
  }

  \onslide<+->{Training set partitions as: ${\train = \ptrain \sqcup \ntrain}$ where}
  \begin{itemize}[<+->]
    \item ${\ptrain \sim \pDist(\X \vert \y = \pcls)}$
    \item ${\ntrain \sim \pDist(\X \vert \y = \ncls)}$
  \end{itemize}
\end{frame}

\begin{frame}{Key Terminology}
  \small
  \onslide<+->{%
    \begin{block}{Covariate Shift}
      \blue{\textbf{Def.}} Training \& test set joint distributions

      \begin{center}
        ${\pDist_{\text{tr}}(\X,\y) = \blue{\pDist_{\text{tr}}(\X)}\green{\pDist(\y \vert \X)}}$ \textbf{\&} ${\pDist_{\text{\red{te}}}(\X,\y) = \blue{\pDist_{\text{\red{te}}}(\X)}\green{\pDist(\y \vert \X)}}$
      \end{center}

      resp.\ differ in marginals (${\blue{\pDist_{\text{tr}}(\X)}}$ vs.\ ${\blue{\pDist_{\red{\text{te}}}(\X)}}$) but have identical posterior~${\green{\pDist(\y \vert \X)}}$.
    \end{block}
  }
  \vfill
  \onslide<+->{%
    \begin{block}{Negative Covariate Shift}
      \blue{\textbf{Def.}} Covariate shift only \textit{biases} training samples drawn from the negative class~i.e.,~${\pDist(\X \vert \y = \ncls)}$. \only<+->{When under such bias, the negative training samples~${\ntrain \rightarrow \bntrain}$}
    \end{block}
  }
  % \vfill
  % \onslide<+->{%
  %   \textit{Example}: \onslide<+->{For a document-authorship classifier, it is usually impossible to collect a \textit{representative} sample of the negative class}
  %   \begin{itemize}[<+->]
  %     \item Entire document types may be missing from~$\bntrain$
  %     \item $\bntrain$'s document proportions may be \textit{biased}
  %   \end{itemize}
  % }
\end{frame}

\section{Risk Estimation}
\begin{frame}{Risk Estimation}
  \small
  \onslide<1->{}
  \onslide<2->{%
    \begin{block}{Risk Estimator}
      \blue{\textbf{Def}.} Given classifier~$\func{\dec}{\domainX}{\real}$ and loss function~$\func{\loss}{\real \times \domainY}{\real_{{\geq}0}}$ (e.g.,~logistic loss), a \textit{risk estimator} quantifies the expected loss as:
      \begin{equation}\label{eq:RiskEstimator}
        \risk_{\mathcal{D}}(\dec) = \mathbb{E}_{(\X,\y) \sim \pDist_{\mathcal{D}}(\X,\y)}\sbrack{\floss{\decX}{\y}} \text{.}
      \end{equation}
    \end{block}
  }
  \vfill
  \onslide<3->{\textbf{Restating our Original Goal}: Minimize the expected \textit{test} loss, i.e.,~${\mathcal{D} = \text{te}}$}
  \begin{itemize}
    \setlength{\itemsep}{7pt}
    \item \onslide<4->{\red{\textbf{Problem}}: Test joint distribution is unknown}
    \item \onslide<5->{\textbf{Solution}: Use an \textit{empirical estimate} as a surrogate}
  \end{itemize}
  \onslide<6->{%
    \begin{equation}\label{eq:}
      \emprisk(\dec) = \frac{1}{\abs{\ptrain \sqcup \only<6-8>{\ntrain}\only<9-9>{\green{\ntrain}}\only<10->{\red{\bntrain}}}} \sum_{(\X, \y) \sim \ptrain \sqcup \only<6-8>{\ntrain}\only<9-9>{\green{\ntrain}}\only<10->{\red{\bntrain}}} \floss{\dec(\X)}{\y}
    \end{equation}
    \vfill
    \begin{itemize}
      \item \onslide<7->{Refer to this as \textbf{\green{unbiased} \blue{positive-negative}} risk estimator}
      \item \onslide<8->{When $\bntrain$ replaces~$\ntrain$, then \textbf{\red{biased} \blue{positive-negative}} risk estimator}
    \end{itemize}
  }
\end{frame}

\begin{frame}{Negative Covariate Shift in Text Classification}
  \onslide<+->{}
  \onslide<+->{Three primary approaches in the literature\ldots}
  \vfill
  \onslide<+->{\textbf{Option~\#1}: Biased positive\-/negative learning~\cite{Li:2010}}
  \begin{itemize}[<+->]
    \item Already saw on previous slide
  \end{itemize}
  \vfill
  \onslide<+->{\textbf{Option~\#2}: Use an unlabeled set~${\utrain \sim \pDist(\X)}$}
  \begin{enumerate}[<+->][a]
    \setlength{\itemsep}{12pt}
    \item \textbf{\blue{Positive\-/Unlabeled} Estimator}: Train classifier using $\ptrain$ and~$\utrain$~\cite{Fei:2015}
      \begin{itemize}
        \item \textbf{Risk Estimator}: Non\-/Negative PU (nnPU)~\cite{Kiryo:2017}
      \end{itemize}
    \item \textbf{\blue{Positive\-/Unlabeled Biased Negative} Estimator}: Train using $\ptrain$, $\bntrain$ and~$\utrain$
      \begin{itemize}
        \item \textbf{Risk Estimator}: PUbN~\cite{Hsieh:2018}
      \end{itemize}
  \end{enumerate}
\end{frame}

\section{Dataset}
\begin{frame}{20 Newsgroups -- What is it?}
  \onslide<+->{Collection of internet message board posts}
  \begin{itemize}
    \item Examples vary in length from a few dozen tokens to over ${{>}1,000}$
    \item 20~labels partition into 7~categories \onslide<+->{-- 4~positive, 3~negative}
  \end{itemize}
  \vfill
  \onslide<+->{\textbf{Dataset Statistics}:} \onslide<+->{ ${\pDist(\y = \pcls) = 0.56}$}
  \begin{itemize}[<+->]
    \item \textasciitilde18.8k~examples divided including fixed \textasciitilde7.5k~test set
  \end{itemize}
  \vfill
  {
    \begin{center}
      \scriptsize
      \onslide<+->{\input{tables/beamer_20newsgroups}}
    \end{center}
  }
\end{frame}

\section{Architecture}
\begin{frame}{Classifier Architectures}
  \onslide<+->{Risk estimators can be applied to (almost) any learning architecture}
  \vfill
  \begin{itemize}[<+->]
    \item \blue{\textbf{ELMo Preprocessed}}: Encode entire document into a static representation with ELMo (a ``document embedding'').  Train a feedforward to classify the static vectors.
  \end{itemize}
\end{frame}

\section{Experiments}
\subsection{Overview}
\begin{frame}{Experimental Setup Overview}
  \onslide<+->{}
  \onslide<+->{\textbf{Two Negative Covariate Shift Types}:}
  \begin{itemize}[<+->]
    \setlength{\itemsep}{6pt}
    \item $\bntrain$~covers only \red{subset} of ${\pDist(\X \vert \y = \ncls)}$'s support
    \item $\bntrain$~biases the 20~newsgroup categories' marginal probabilities
  \end{itemize}
  \vfill
  \onslide<+->{\textbf{Results Overview}:}
  \begin{itemize}[<+->]
    \setlength{\itemsep}{6pt}
    \item \textit{Inductive} -- Test set classification accuracy
    % \item Discuss ELMo results only \onslide<+->{(significantly better than LSTM)}
  \end{itemize}
  \vfill
  \onslide<+->{\textbf{Hyperparameter Info}:}
  \begin{itemize}[<+->]
    \setlength{\itemsep}{6pt}
    \item Tuned via grid search on disjoint validation set 20\%~size of train set
    % \item ${\abs{\ptrain} = \abs{\ntrain} = \abs{\bntrain} = 500}$
    % \item ${\abs{\utrain} = 6,000}$
  \end{itemize}
\end{frame}

\subsection{Results}
\begin{frame}{Experiments -- Effect of Bias}
  \begin{minipage}[t]{0.69\linewidth}
    \begin{table}
      \centering
      \caption{ELMo preprocessed test accuracy results}
      \input{tables/beamer_elmo_results}
    \end{table}
    \vfill
    \onslide<9->{ $\red{\dagger}$ \textbf{Test Set Marginal}: ${\sbrack{48.2\%, 12.1\%, 39.7\%}}$}
  \end{minipage}
  \begin{minipage}{0.02\linewidth}
    \hspace{\fill}
  \end{minipage}
  \begin{minipage}[t]{0.26\linewidth}
    \vspace{40pt}
    \onslide<11->{%
      % \flushright
      \begin{block}{Major Takeaway}
        \small
        Type of bias determines best risk estimator
      \end{block}
    }
  \end{minipage}
\end{frame}

\begin{frame}[noframenumbering]
  \tableofcontents
\end{frame}

\begin{frame}{List of References}
  \bibliography{bib/ref.bib}
  \bibliographystyle{ieeetr}
\end{frame}

\appendix
\transitionFrame{Appendix}

%% ========================================================================= %%
%% ========================================================================= %%

\begin{frame}{Key Terminology}
  \small
  \onslide<+->{%
    \begin{block}{Covariate Shift}
      \blue{\textbf{Def.}} Training \& test set joint distributions

      \begin{center}
        ${\pDist_{\text{tr}}(\X,\y) = \blue{\pDist_{\text{tr}}(\X)}\green{\pDist(\y \vert \X)}}$ \textbf{\&} ${\pDist_{\text{\red{te}}}(\X,\y) = \blue{\pDist_{\text{\red{te}}}(\X)}\green{\pDist(\y \vert \X)}}$
      \end{center}

      resp.\ differ in marginals (${\blue{\pDist_{\text{tr}}(\X)}}$ vs.\ ${\blue{\pDist_{\red{\text{te}}}(\X)}}$) but have identical posterior~${\green{\pDist(\y \vert \X)}}$.
    \end{block}
  }
  \vfill
  \onslide<+->{%
    \begin{block}{Negative Covariate Shift}
      \blue{\textbf{Def.}} Covariate shift only \textit{biases} training samples drawn from the negative class~i.e.,~${\pDist(\X \vert \y = \ncls)}$. \only<+->{When under such bias, the negative training samples~${\ntrain \rightarrow \bntrain}$}
    \end{block}
  }
  \vfill
  \onslide<+->{%
    \textit{Example}: \onslide<+->{For a document-authorship classifier, it is usually impossible to collect a \textit{representative} sample of the negative class}
    \begin{itemize}[<+->]
      \item Entire document types may be missing from~$\bntrain$
      \item $\bntrain$'s document proportions may be \textit{biased}
    \end{itemize}
  }
\end{frame}

\begin{frame}{Classifier Architectures}
  \onslide<+->{We considered two deep architectures:}
  \vfill
  \begin{itemize}[<+->]
    \setlength{\itemsep}{24pt}
    \item \blue{\textbf{``Classic'' RNN Classifier}}: Tokenize sentence, encode with embedding matrix, classify with RNN + feedforward
      \begin{itemize}[<+->]
        \item Will not discuss since much worse results
      \end{itemize}

    \item \blue{\textbf{ELMo Preprocessed}}: Encode entire document into a static representation with ELMo (a ``document embedding'').  Train a feedforward to classify the static vectors.
  \end{itemize}
  \vfill
  \onslide<+->{\green{\textbf{How Can We Summarize the Architectural Difference?}} \blue{\textbf{Transfer learning}}}
\end{frame}

\begin{frame}{What is Transfer Learning?}
  \onslide<+->{\blue{\textbf{Def.}}: Taking knowledge learned from solving one problem and applying it to a different (related) one}
  \vfill
  \onslide<+->{Let's look at how each architecture uses transfer learning\ldots}
  \begin{itemize}[<+->]
    \setlength{\itemsep}{20pt}
    \item \textbf{``Classic'' RNN}: \red{Limited}. \onslide<+->{Only (GloVe) embedding matrix}
    \item \textbf{ELMo Preprocessed}: \green{Extensive}. \onslide<+->{Entire ELMo network parameters static.  Only train a couple feed forward layers.}
  \end{itemize}
  \vfill
  \onslide<+->{\textbf{\green{Question}}: Is transfer learning a \textit{free lunch}?}

  \vspace{6pt}
  \onslide<+->{\textbf{Answer}: No -- there's never a free lunch.  \onslide<+->{Transfer learning is just an \textit{inductive bias} and like all biases limits some flexibility -- which may be good or bad.}}
\end{frame}

\begin{frame}{Experiments -- Architectural Comparison}
  \begin{minipage}[t]{0.67\linewidth}
    \centering
    \vspace{-18pt}
    \onslide<+->{
      \begin{table}
        \caption{\small ELMo preprocessed test accuracy results}
        \onslide<+->{
          {\footnotesize \input{tables/elmo_results}}
        }
      \end{table}
    }
    \vspace{-20pt}
    \onslide<+->{
      \begin{table}
        \caption{\small ${\left(\text{Accuracy}_{ELMo} - \text{Accuracy}_{RNN}\right)}$\onslide<+->{: ${{>}0}$ means ELMO better}}
        \onslide<+->{
          {\footnotesize \input{tables/results_compare}}
        }
      \end{table}
    }
    \vspace{-5pt}  % Need to fix false warning on frame size
  \end{minipage}
  \begin{minipage}{0.07\linewidth}
    \hspace{\fill}
  \end{minipage}
  \begin{minipage}[t]{0.23\linewidth}
    \vspace{40pt}
    \onslide<+->{%
      % \flushright
      \begin{block}{Major Takeaway}
        If training data limited/biased, transfer learning's benefits compound
      \end{block}
    }
  \end{minipage}
\end{frame}