text:
\section{Introduction}

\IEEEPARstart{I}{n} this paper we focus on the application of deep
learning to structured output problems where the task is to map the input to an
output that possesses its own structure. The task is therefore not only to map
the input to the correct output (e.g. the classification task in object recognition), but also to
model the structure within the output sequence.

A classic example of a
structured output problem is machine translation: to automatically translate a
sentence from the \emph{source language} to the \emph{target language}. To
accomplish this task, not only does the system need to be concerned with
capturing the semantic content of the source language sentence, but also with
forming a coherent and grammatical sentence in the target language. In other words, given
an input source sentence, we cannot choose the elements of the output
(i.e. the individual words) independently: they have a complex joint distribution.

Structured output problems represent a large and important class of problems
that include classic tasks such as speech recognition and many natural language
processing problems (e.g. text summarization and paraphrase generation). As the
range of capabilities of deep learning systems increases, less established forms
of structured output problems, such as image caption generation and video
description generation (\cite{Kulkarni2013} and references therein,) are being
considered.

One important aspect of virtually all structured output tasks is that {\em the
structure of the output is imtimately related to the structure of the
input}. A central challenge to these tasks is therefore the problem of
\emph{alignment}. At its most fundamental, the problem of alignment is the
problem of how to relate sub-elements of the input to sub-elements of the
output. Consider again our example of machine translation. In order to
translate the source sentence into the target language we need to first
decompose the source sentence into its constituent semantic parts. Then we
need to map these semantic parts to their counterparts in the target
language. Finally, we need to use these semantic parts to compose the
sentence following the grammatical regularities of the target
language. Each word or phrase of the target sentence can be aligned to a
word or phrase in the source language.

In the case of image caption generation, it is often appropriate for the output sentence to accurately describe
the spatial relationships between elements of the scene represented in the image. For this, we need to \emph{align} the output words to spatial
regions of the source image.

In this paper we focus on a general approach to the alignment problem known
as the soft attention mechanism. Broadly, attention mechanisms are components of
prediction systems that allow the system to sequentially focus on different
subsets of the input. The selection of the subset is typically conditioned
on the state of the system which is itself a function of the previously
attended subsets.

Attention mechanisms are employed for two purposes. The first is to reduce
the computational burden of processing high dimensional inputs by selecting
to only process subsets of the input. The second is to allow the system to
focus on distinct aspects of the input and thus improve its ability to
extract the most relevant information for each piece of the output, thus
yielding improvements in the quality of the generated outputs.

As the name suggests, soft attention mechanisms avoid a hard selection of
which subsets of the input to attend and instead uses a soft weighting of the
different subsets. Since all subset are processed,
these mechanisms offer no computation advantage. Instead, the advantage brought
by the soft-weighting is that it is readily amenable to efficient learning via
gradient backpropagation.

In this paper, we present a review of the recent work in applying the soft
attention to structured output tasks and spectulate about the future course of this line of research. The soft-attention mechanism is part of a growing litterature on more flexible deep learning architectures that embed a certain amount of \emph{distributed decision making}.

\section{Background: \\ Recurrent and Convolutional Neural Networks}

\subsection{Recurrent Neural Network}
\label{sec:rnn}

A recurrent neural network (RNN) is a neural network specialized at handling a
variable-length input sequence $x=(\vx_1, \dots, \vx_T)$ and optionally a
corresponding variable-length output sequence $y=(\vy_1, \dots, \vy_T)$, using
an internal hidden state $\vh$. The RNN sequentially reads each symbol $\vx_t$
of the input sequence and updates its internal hidden state $\vh_t$ according to
\begin{align}
    \label{eq:encoding}
    \vh_t = \phi_{\theta}\left( \vh_{t-1}, \vx_t \right),
\end{align}
where $\phi_{\theta}$ is a nonlinear activation function parametrized by a set of
parameters $\theta$. When the target sequence is given, the RNN can be trained to
sequentially make a prediction $\hat{\vy}_t$ of the actual output $\vy_t$ at each time step $t$:
\begin{align}
    \label{eq:seq_pred}
    \hat{\vy}_t = g_{\theta}\left( \vh_{t}, \vx_t \right),
\end{align}
where $g_{\theta}$ may be an arbitrary, parametric function that is learned
jointly as a part of the whole network.

The recurrent activation function $\phi$ in Eq.~\eqref{eq:encoding} may be as
simple as an affine transformation followed by an element-wise logistic function
such that
\begin{align*}
    \vh_t = \tanh \left( \mU \vh_{t-1} + \mW \vx_t \right),
\end{align*}
where $\mU$ and $\mW$ are the learned weight matrices.\footnote{
    We omit biases to make the equations less cluttered.
}

It has recently become more common to use more sophisticated recurrent
activation functions, such as a long short-term memory
(LSTM,~\cite{Hochreiter+Schmidhuber-1997}) or a gated recurrent unit
(GRU,~\cite{Cho-et-al-EMNLP2014,Chung-et-al-NIPSDL2014-small}), to reduce the
issue of vanishing gradient~\cite{Bengio-et-al-TNN1994,Hochreiter+al-2000}.
Both LSTM and GRU avoid the vanishing gradient by introducing gating units that
adaptively control the flow of information across time steps.

The activation of a GRU, for instance, is defined by
\begin{align*}
    \vh_t = \vu_t \odot \tilde{\vh}_t + (1 - \vu_t) \odot \vh_{t-1},
\end{align*}
where $\odot$ is an element-wise multiplication, and the update gates $\vu_t$
are
\begin{align*}
    \vg_t = \sigma\left( \mU_u \vh_{t-1} + \mW_u \vx_t \right).
\end{align*}
The candidate hidden state $\tilde{\vh}_t$ is computed by
\begin{align*}
    \tilde{\vh}_t = \tanh\left( \mU \vh_{t-1} + \mW \left(\vr_t \odot \vx_t\right) \right),
\end{align*}
where the reset gates $\vr_t$ are computed by
\begin{align*}
    \vr_t = \sigma\left( \mU_r \vh_{t-1} + \mW_r \vx_t \right).
\end{align*}

All the use cases of the RNN in the remaining of this paper use either the GRU
or LSTM.

\subsection{RNN-LM: Recurrent Neural Network Language Modeling}
\label{sec:rnnlm}

In the task of language modeling, we let a model learn the probability
distribution over natural language sentences. In other words, given a model, we
can compute the probability of a sentence $s=(w_1, w_2, \ldots, w_T)$ consisting
of multiple words, i.e., $p(w_1, w_2, \ldots, w_T)$, where the sentence is $T$
words long.

This task of language modeling is equivalent to the task of predicting the next
word. This is clear by rewriting the sentence probability into
\begin{align}
    \label{eq:next_word_pred}
    p(w_1, w_2, \ldots, w_T) = \prod_{t=1}^T p(w_t \mid w_{<t}),
\end{align}
where $w_{<t}=(w_1, \ldots, w_{t-1})$.  Each conditional probability on the
right-hand side corresponds to the predictive probability of the next word $w_t$
given all the preceding words ($w_1, \ldots, w_{t-1}$).

A recurrent neural network (RNN) can, thus, be readily used for language
modeling by letting it predict the next symbol at each time step $t$
(RNN-LM, \cite{Mikolov-ICASSP-2011}). In other words, the RNN predicts the probability
over the next word by
\begin{align}
    \label{eq:rnn_lm}
    p(w_{t+1} = w|w_{\leq t}) = g_{\theta}^w \left( \vh_{t}, \vw_t
    \right),
\end{align}
where $g_{\theta}^w$ returns the probability of the word $w$ out of all possible
words. The internal hidden state $\vh_{t}$ summarizes all the preceding symbols
$w_{\leq t} = (w_1,\ldots,w_{t})$.

We can generate an exact sentence sample from an RNN-LM by iteratively sampling
from the next word distribution $p(w_{t+1}|w_{\leq t})$ in
Eq.~\eqref{eq:rnn_lm}. Instead of stochastic sampling, it is possible to
approximately find a sentence sample that maximizes the probability $p(s)$
using, for instance, beam search~\cite{Graves2012,Boulanger2013}.

The RNN-LM described here can be extended to learn a {\it conditional} language
model. In conditional language modeling, the task is to model the distribution
over sentences given an additional input, or context. The context may be
anything from an image and a video clip to a sentence in another language. Examples
of textual outputs associated with these inputs by the {\it conditional RNN-LM} include
respectively an image caption, a video description and a translation. In these cases,
the transition function of the RNN will take as an additional input the context
$c$ such that
\begin{align}
    \label{eq:encoding_cond}
    \vh_t = \phi_{\theta}\left( \vh_{t-1}, \vx_t , c\right).
\end{align}
Note the $c$ at the end of the r.h.s. of the equation.

This conditional language model based on RNNs will be at the center of later
sections.

\subsection{Deep Convolutional Network}
\label{sec:cnn}

A convolutional neural network (CNN) is a special type of a more general
feedforward neural network, or multilayer perceptron, that has been specifically
designed to work well with two-dimensional images \cite{LeCun+98}. The CNN often
consists of multiple convolutional layers followed by a few fully-connected
layers.

At each convolutional layer, the input image of width $n_i$, height $n_j$ and $c$
color channels ($\vx \in \RR^{n_i \times n_y \times c}$) is first convolved with a
set of local filters $\vf \in \RR^{n'_i \times n'_y \times c \times d}$. For each
location/pixel $(i,j)$ of $\vx$, we get
\begin{align}
    \label{eq:conv}
    \vz_{i,j} = \sum_{i'=1}^{n'_i} \sum_{j'=1}^{n'_j} f\left(\vf_{i',j'}^{\top}
    \vx_{i+i',j+j'}\right),
\end{align}
where $\vf_{i',j'} \in \RR^{c \times d}$, $\vx_{i+i',j+j'} \in \RR^{c}$ and
$\vz_{i,j} \in \RR^{d}$. $f$ is an element-wise nonlinear activation function.

The convolution in Eq.~\eqref{eq:conv} is followed by local max-pooling:
\begin{align}
    \label{eq:maxpool}
    \vh_{i,j} = \max_{
        \begin{array}{c}
            i'\in\left\{ ri, \ldots, (r+1)i - 1\right\}, \\
            j'\in\left\{ rj, \ldots, (r+1)j - 1\right\}
        \end{array}
    } \vz_{i',j'},
\end{align}
for all $i\in\left\{ 1, \ldots, n_i/r \right\}$ and $j\in\left\{ 1, \ldots, n_j/r
\right\}$. $r$ is the size of the neighborhood.

The pooling operation has two desirable properties. First, it reduces the
dimensionality of a high-dimensional output of the convolutional layer.
Furthermore, this spatial max-pooling summarizes the activation of the
neighbouring feature activations, leading to the (local) translation invariance.

After a small number of convolutional layers, the final feature map from the
last convolutional layer is flattened to form a vector representation $\vh$ of
the input image. This vector $\vh$ is further fed through a small number of
fully-connected nonlinear layers until the output.

Recently, the CNNs have been found to be excellent at the task of large-scale
object recognition. For instance, the annual ImageNet Large Scale Visual
Recognition Challenge (ILSVRC) has a classification track where more than a
million annotated images with 1,000 classes are provided as a training set.  In
this challenge, the CNN-based entries have been dominant since
2012~\cite{Krizhevsky-2012,ZeilerFergus14,Simonyan2015,Szegedy-et-al-arxiv2014}.

\subsection{Transfer Learning with Deep Convolutional Network}
\label{sec:imagenet}

Once a deep CNN is trained on a large training set such that the one provided as
a part of the ILVRC challenge, we can use any intermediate representation, such
as the feature map from any convolutional layer or the vector representation
from any subsequent fully-connected layers, of the whole network for tasks other
than the original classification.

It has been observed that the use of these intermediate representation from the
deep CNN as an image descriptor significantly boosts subsequent tasks such as
object localization, object detection, fine-grained recognition, attribute
detection and image retrieval (see, e.g., \cite{Sermanet14,razavian2014cnn}.)
Furthermore, more non-trivial tasks, such as image caption generation
\cite{Karpathy+Li-CVPR2015,Fang-et-al-CVPR2015,Mao-et-al-ICLR2015,Vinyals-et-al-CVPR2015,Kiros-et-al-arxiv2014},
have been found to benefit from using the image descriptors from a pre-trained
deep CNN. In later sections, we will discuss in more detail how image
representations from a pre-trained deep CNN can be used in these non-trivial
tasks such as image caption generation~\cite{Xu-et-al-ICML2015} and video
description generation~\cite{Li-et-al-ARXIV2015}.

\section{Attention-based Multimedia Description}

Multimedia description generation is a general task in which a model generates a
natural language description of a multimedia input such as speech, image and
video as well as text in another language, if we take a more general view. This
requires a model to capture the underlying, complex mapping between the
spatio-temporal structures of the input and the complicated linguistic
structures in the output. In this section, we describe a neural network based
approach to this problem, based on the encoder--decoder framework with the
recently proposed attention mechanism.

\subsection{Encoder--Decoder Network}
\label{sec:encdec}

An encoder--decoder framework is a general framework based on neural networks that
aims at handling the mapping between highly structured input and output. It was
proposed recently in
\cite{Kalchbrenner+Blunsom-EMNLP2013,Cho-et-al-EMNLP2014,Sutskever-et-al-NIPS2014}
in the context of machine translation, where the input and output are natural
language sentences written in two different languages.

As the name suggests, a neural network based on this encoder--decoder framework
consists of an encoder and a decoder. The encoder $f_{\text{enc}}$ first reads
the input data $x$ into a continuous-space representation $c$:
\begin{align}
    \label{eq:enc_gen}
    c = f_{\text{enc}}(x),
\end{align}

The choice of $f_{\text{enc}}$ largely depends on the type of input. When $x$ is
a two-dimensional image, a convolutional neural network (CNN) from
Sec.~\ref{sec:imagenet} may be used. A recurrent neural network (RNN) in
Sec.~\ref{sec:rnn} is a natural choice when $x$ is a sentence.

The decoder then generates the output $y$ conditioned on the continuous-space
representation, or context $c$ of the input. This is equivalent to computing the
conditional probability distribution of $y$ given $x$:
\begin{align}
    \label{eq:dec_gen}
    p(Y|x) = f_{\text{dec}}(c).
\end{align}
Again, the choice of $f_{\text{dec}}$ is made based on the type of the output.
For instance, if $y$ is an image or a pixel-wise image segmentation, a conditional
restricted Boltzmann machine (CRBM) can be used \cite{taylor-eccv-10-small}.
When $y$ is a natural language description of the input $x$, it is natural to
use an RNN which is able to model natural languages, as described in
Sec.~\ref{sec:rnnlm}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.65\columnwidth]{figures/rnn_encdec.pdf}
    \caption{Graphical illustration of the simplest form encoder-decoder model for machine
    translation from \cite{Cho-et-al-EMNLP2014}. $x=(x_1, \ldots, x_T)$,
$y=(y_1, \ldots, y_{T'})$ and $c$ are respectively the input sentence, the
output sentence and the continuous-space representation of the input sentence.}
    \label{fig:encdec_mt}
\end{figure}

This encoder--decoder framework has been successfully used in
\cite{Sutskever-et-al-NIPS2014,Cho-et-al-EMNLP2014} for machine translation. In
both work, an RNN was used as an encoder to summarize a source sentence (where
the summary is the last hidden state $\vh_T$ in Eq.~\eqref{eq:encoding}) from
which a conditional RNN-LM from Sec.~\ref{sec:rnn} decoded out the corresponding
translation. See Fig.~\ref{fig:encdec_mt} for the graphical illustration.

In \cite{Mao-et-al-ICLR2015,Vinyals-et-al-CVPR2015}, the authors used a
pre-trained CNN as an encoder and a conditional RNN as a decoder to let model
generate a natural language caption of images. Similarly, a simpler
feedforward log-bilinear language model~\cite{Mnih+Hinton-2007} was used as a
decoder in \cite{Kiros-et-al-arxiv2014}. The authors of
\cite{Venugopalan-et-al-arxiv2014} applied the encoder--decoder framework to
video description generation, where they used a pre-trained CNN to extract a
feature vector from each frame of an input video and averaged those vectors.

In all these recent applications of the encoder--decoder framework, the
continuous-space representation $c$ of the input $x$ returned by an encoder, in
Eq.~\eqref{eq:enc_gen} has been a fixed-dimensional vector, regardless of the
size of the input.\footnote{
    Note that in the case of machine translation and video description
    generation, the size of the input varies.
} Furthermore, the context vector was not structured by design, but rather an
arbitrary vector, which means that there is no guarantee that the context vector
preserves the spatial, temporal or spatio-temporal structures of the input.
Henceforth, we refer to an encoder--decoder based model with a fixed-dimensional
context vector as a {\it simple encoder--decoder model}.

\subsection{Incorporating an Attention Mechanism}
\label{sec:attention}

\subsubsection{Motivation}

A naive implementation of the encoder--decoder framework, as in the simple
encoder--decoder model, requires the encoder to compress the input into a single
vector of predefined dimensionality, regardless of the size of or the amount of
information in the input. For instance, the recurrent neural network (RNN) based
encoder used in \cite{Cho-et-al-EMNLP2014,Sutskever-et-al-NIPS2014} for machine
translation needs to be able to summarize a variable-length source sentence into
a single fixed-dimensional vector. Even when the size of the input is fixed, as
in the case of a fixed-resolution image, the amount of information contained in
each image may vary significantly (consider a varying number of objects in each
image).

In \cite{Cho2014a}, it was observed that the performance of the neural machine
translation system based on a simple encoder--decoder model rapidly degraded as
the length of the source sentence grew. The authors of \cite{Cho2014a}
hypothesized that it was due to the limited capacity of the simple
encoder--decoder's fixed-dimensional context vector.

Furthermore, the interpretability of the simple encoder--decoder is extremely
low. As all the information required for the decoder to generate the output is
compressed in a context vector without any presupposed structure, such structure
is not available to techniques designed to inspect the representations captured by the
model~\cite{ZeilerFergus14,Springenberg2015,Denil2014c}.

\subsubsection{Attention Mechanism for Encoder--Decoder Models}
\label{sec:attention:details}

We the introduction of an attention mechanism in between the encoder and
decoder, we address these two issues, i.e., (1) limited capacity of a
fixed-dimensional context vector and (2) lack of interpretability.

The first step into introducing the attention mechanism to the encoder--decoder
framework is to let the encoder return a structured representation of the input.
We achieve this by allowing the continuous-space representation to be a set of
fixed-size vectors, to which we refer as a {\it context set}, i.e.,
\[
    c=\left\{\vc_1, \vc_2, \ldots, \vc_M \right\}
\]
See Eq.~\eqref{eq:enc_gen}. Each vector in the context set is localized to a
certain spatial, temporal or spatio-temporal component of the input. For instance, in the
case of an image input, each context vector $\vc_i$ will summarize a certain
spatial location of the image (see Sec.~\ref{sec:capgen}), and with machine
translation, each context vector will summarize a phrase centered around a
specific word in a source sentence (see Sec.~\ref{sec:nmt}.) In all cases, the
number of vectors $M$ in the context set $c$ may vary across input examples.

The choice of the encoder and of the kind of context set it will return is governed
by the application and the type of the input considered.
In this paper, we assume that the decoder is a conditional RNN-LM from
Sec.~\ref{sec:rnnlm}, i.e., the goal is to describe the input in a natural
language sentence.

The attention mechanism controls the input actually seen by the decoder and requires
another neural network, to which refer as the attention model. The main job of
the attention model is to score each context vector $\vc_i$ with respect to the
current hidden state $\vz_{t-1}$ of the decoder:\footnote{
    We use $\vz_t$ to denote the hidden state of the decoder to distinguish it
    from the encoder's hidden state for which we used $\vh_t$ in
    Eq.~\eqref{eq:encoding}.
}
\begin{align}
    \label{eq:att_score_gen}
    e_i^t = f_{\text{ATT}}(\vz_{t-1}, \vc_i, \{ \alpha_j^{t-1}\}_{j=1}^M ),

\end{align}
where $\alpha_j^{t-1}$ represents the attention weights computed
at the previous time step, from the scores $e_i^{t-1}$, through
a softmax that makes them sum to 1:
\begin{align}
    \label{eq:att2}
    \alpha_i^t = \frac{\exp(e_i^t)}{\sum_{j=1}^M \exp(e_j^t)},
\end{align}
This type of scoring can be
viewed as assigning a probability of being {\it attended} by the decoder to
each context, hence the name of the attention model.

Once the attention weights are computed, we use them to compute the new context
{\it vector} $\vc^t$:
\begin{align}
    \label{eq:att_gen}
    \vc^t = \varphi\left(\left\{ \vc_i \right\}_{i=1}^M,
        \left\{ \alpha_i^t \right\}_{i=1}^M\right),
\end{align}
where $\varphi$ returns a vector summarizing the whole context set $c$ according
to the attention weights.

A usual choice for $\varphi$ is a simple weighted sum of the context vectors
such that
\begin{align}
    \label{eq:att_wsum}
    \vc^t = \varphi\left(\left\{ \vc_i \right\}_{i=1}^M,
    \left\{ \alpha_i^t \right\}_{i=1}^M\right) = \sum_{i=1}^M \alpha_i \vc_i.
\end{align}
On the other hand, we can also force the attention model to make a hard decision
on which context vector to consider by sampling one of the context vectors
following a categorical (or multinoulli) distribution:
\begin{align}
    \label{eq:att_hard}
    \vc^t = \vc_{r^t}, \text{ where }
    r^t \sim \text{Cat}(M, \left\{ \alpha_i^t \right\}_{i=1}^M).
\end{align}

With the newly computed context vector $\vc_t$, we can update the hidden state
of the decoder, which is a conditional RNN-LM here, by
\begin{align}
    \label{eq:encoding_cond_att}
    \vh_t = \phi_{\theta}\left( \vh_{t-1}, \vx_t , \vc_t\right).
\end{align}

This way of computing a context vector at each time step $t$ of the decoder
frees the encoder from compressing any variable-length input into a single
fixed-dimensional vector. By spatially or temporally dividing the
input\footnote{
    Note that it is possible, or even desirable to use overlapping regions.
}, the encoder can represent the input into a set of vectors of which each
needs to encode a fixed amount of information focused around a particular
region of the input. In other words, the introduction of the attention
mechanism bypasses the issue of limited capacity of a fixed-dimensional context
vectors.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.85\columnwidth]{figures/EEA.pdf}
    \caption{Visualization of the attention weights $\alpha_j^t$ of the
    attention-based neural machine translation model \cite{Bahdanau-et-al-ICLR2015-small}. Each row corresponds to
the output symbol, and each column the input symbol. Brighter the higher
$\alpha_j^t$.}
    \label{fig:alpha_viz}
\end{figure}

Furthermore, this attention mechanism allows us to directly inspect the
internal working of the whole encoder--decoder model. The magnitude of the
attention weight $\alpha_j^t$, which is positive by construction in
Eq.~\eqref{eq:att2}, highly correlates with how predictive the spatial,
temporal or spatio-temporal region of the input, to which the $j$-th context
vector corresponds, is for the prediction associated with the $t$-th output
variable $y_t$. This can be easily done by visualizing the attention matrix
$\left[ \alpha_j^t \right]_{t,j} \in \RR^{T' \times M}$, as in
Fig.~\ref{fig:alpha_viz}.

This attention-based approach with the weighted sum of the context vectors (see
Eq.~\eqref{eq:att_wsum}) was originally proposed in
\cite{Bahdanau-et-al-ICLR2015-small} in the context of machine translation,
however, with a simplified (content-based) scoring function:
\begin{align}
    \label{eq:att1}
    e_i^t = f_{\text{ATT}}(\vz_{t-1}, \vc_i).
\end{align}
See the missing $\{ \alpha_j^{t-1}\}_{j=1}^M$ from Eq.~\eqref{eq:att_score_gen}.
In \cite{Xu-et-al-ICML2015}, it was further extended with the hard attention
using Eq.~\eqref{eq:att_hard}.  In~\cite{Chorowski2015} this attention mechanism
was extended to be by taking intou account the past values of the attention
weights as the general scoring function from Eq.~\eqref{eq:att_score_gen},
following an approach based purely on those weights introduced
by~\cite{Graves-arxiv2013}. We will discuss more in detail these three
applications/approaches in the later sections.

\subsection{Learning}

As usual with many machine learning models, the attention-based encoder--decoder
model is also trained to maximize the log-likelihood of a given training set
with respect to the parameters, where the log-likelihood is defined as
\begin{align}
    \label{eq:loglikelihood}
    \LL\left(D=\left\{ (x^n, y^n) \right\}_{n=1}^N, \Theta\right) =
    \frac{1}{N} \sum_{n=1}^N \text{log} p(y^n \mid x^n, \Theta),
\end{align}
where $\Theta$ is a set of all the trainable parameters of the model.

\subsubsection{Maximum Likelihood Learning}
\label{sec:mle}

When the weighted sum is used to compute the context vector, as in
Eq.~\eqref{eq:att_wsum}, the whole attention-based encoder--decoder model
becomes one large differentiable function. This allows us to compute the
gradient of the log-likelihood in Eq.~\eqref{eq:loglikelihood} using
backpropagation~\cite{Rumelhart86b}. With the computed gradient, we can use, for
instance, the stochastic gradient descent (SGD) algorithm to iteratively update the
parameters $\Theta$ to maximize the log-likelihood.

\subsubsection{Variational Learning for Hard Attention Model}
\label{sec:vari}

When the attention model makes a hard decision each time as in
Eq.~\eqref{eq:att_hard}, the derivatives through the stochastic decision
are zero, because those decisions are discrete. Hence, the information
about how to improve the way to take those focus-of-attention decisions is
not available from back-propagation, while it is needed to train the
attention mechanism. The question of training neural networks
with stochastic discrete-valued hidden units has a long history,
starting with Boltzmann machines~\cite{Hinton-bo86}, with recent
work studying how to deal with such units in a system trained using back-propagated
gradients \cite{bengio2013estimating,Tang+Salakhutdinov-NIPS2013,ba+mnih,Raiko2015}.
Here we briefly describe the variational learning approach from
\cite{ba+mnih,Xu-et-al-ICML2015}.

With stochastic variables $\vr$ involved in the computation from
inputs to outputs, the log-likelihood in
Eq.~\eqref{eq:loglikelihood} is re-written into
\begin{align*}
    \LL\left(D=\left\{ (x^n, y^n) \right\}_{n=1}^N, \Theta\right) =
    \frac{1}{N} \sum_{n=1}^N l(y^n, x^n, \Theta),
\end{align*}
where
\begin{align*}
    l(y, x, \Theta) = \text{log} \sum_{\vr} p(y, \vr | x, \Theta)
\end{align*}
and $\vr=(r_1, r_2, \ldots, r_T')$.  We derive a lowerbound of $l$ as
\begin{align}
    l(y, x) &= \text{log} \sum_{\vr} p(y| \vr, x) p(\vr|x)
    \nonumber \\
    &\geq
    \sum_{\vr} p(\vr|x) \text{log} p(y| \vr, x).
    \label{eq:lowerbound}
\end{align}
Note that we omitted $\Theta$ to make the equation less cluttered.

The gradient of $l$ with respect to $\Theta$ is then
\begin{align}
    \nabla l(y, x) = \sum_{\vr} p(\vr|x) &\left[ \nabla \text{log} p(y | \vr, x) \right.
    \nonumber
    \\
                    &+ \left. \text{log} p (y| \vr,x) \nabla \text{log} p(\vr|x) \right]
    \label{eq:grad_lb}
\end{align}
which is often approximated by Monte Carlo sampling:
\begin{align}
    \label{eq:approx_grad_lb}
    \nabla l(y, x) \approx \frac{1}{M} \sum_{m=1}^M &\nabla \text{log} p(y | \vr^m, x)
    \nonumber
    \\
    &+\text{log} p (y| \vr^m,x) \nabla \text{log} p(\vr^m|x).
\end{align}
As the variance of this estimator is high, a number of variance reduction
techniques, such as baselines and variance normalization, are often used in
practice~\cite{Mnih+Gregor-2014arxiv,ba+mnih}.

Once the gradient is estimated, any usual gradient-based iterative optimization
algorithm can be used to approximately maximize the log-likelihood.

\section{Applications}

In this section, we introduce some of the recent work in which the
attention-based encoder--decoder model was applied to various multimedia
description generation tasks.

\subsection{Neural Machine Translation}
\label{sec:nmt}

Machine translation is a task in which a sentence in one language (source) is
translated into a corresponding sentence in another language (target).
{\em Neural} machine translation aims at solving it with a single neural network based
model, jointly trained end-to-end. The encoder--decoder framework described in
Sec.~\ref{sec:encdec} was proposed for neural machine translation recently in
\cite{Kalchbrenner+Blunsom-EMNLP2013,Cho-et-al-EMNLP2014,Sutskever-et-al-NIPS2014}.
Based on these works, in \cite{Bahdanau-et-al-ICLR2015-small}, the
attention-based model  was proposed to make neural machine translation systems
more robust to long sentences. Here, we briefly describe the model from
\cite{Bahdanau-et-al-ICLR2015-small}.

\subsubsection{Model Description}

The attention-based neural machine translation in
\cite{Bahdanau-et-al-ICLR2015-small} uses a bidirectional recurrent neural
network (BiRNN) as an encoder. The forward network reads the input sentence
\mbox{$x=(x_1, \ldots, x_T)$} from the first word to the last, resulting in
a sequence of state vectors
\[
    \left\{ \overrightarrow{\vh}_1, \overrightarrow{\vh}_2, \ldots, \overrightarrow{\vh}_T \right\}.
\]
The backward network, on the other hand, reads the input sentence in the reverse
order, resulting in
\[
    \left\{ \overleftarrow{\vh}_T, \overleftarrow{\vh}_{T-1}, \ldots, \overleftarrow{\vh}_1 \right\}.
\]
These vectors are concatenated per step to form a context set (see
Sec.~\ref{sec:attention:details}) such that $\vc_t = \left[ \overrightarrow{\vh}_t;
\overleftarrow{\vh}_t \right]$.

\begin{figure}[ht]
    \centering
    \begin{minipage}{0.45\columnwidth}
        \centering
        \includegraphics[width=\columnwidth]{figures/rnnsearch.pdf}
    \end{minipage}
    \hfill
    \begin{minipage}{0.47\columnwidth}
        \caption{Illustration of a single step of decoding in
            attention-based neural machine translation
        \cite{Bahdanau-et-al-ICLR2015-small}.}
        \label{fig:nmt}
    \end{minipage}
\end{figure}

The use of the BiRNN is crucial if the content-based attention mechanism is
used. The content-based attention mechanism in
Eqs.~\eqref{eq:att1}~and~\eqref{eq:att2} relies solely on a so-called {\it
content-based scoring}, and without the context information from the whole
sentence, words that appear multiple times in a source sentence cannot be
distinguished by the attention model.

The decoder is a conditional RNN-LM that models the target language given the
context set from above. See Fig.~\ref{fig:nmt} for the graphical illustration of
the attention-based neural machine translation model.

\begin{table}[ht]
    \centering
    \caption{The translation performances and the relative improvements over the
    simple encoder-decoder model on an English-to-French translation task,
measured by BLEU \cite{Bahdanau-et-al-ICLR2015-small,Jean-et-al-ACL2015}.
$\star$: an ensemble of multiple attention-based models. $\circ$: the
state-of-the-art phrase-based statistical machine translation system~\cite{Durrani2014}.}
    \label{tbl:att_nmt}
    \begin{tabular}{c|c|c}
    Model & BLEU & Rel. Improvement \\
    \hline
    \hline
    Simple Enc--Dec & 17.82 &  -- \\
    Attention-based Enc--Dec & 28.45 & +59.7\% \\
    Attention-based Enc--Dec (LV) & 34.11 & +90.7\% \\
    Attention-based Enc--Dec (LV)$^\star$ & {\bf 37.19} & {\bf +106.0\%} \\
    \hline
    State-of-the-art SMT$^\circ$ & 37.03 & --
    \end{tabular}
\end{table}

\subsubsection{Experimental Result}

Given a fixed model size, the attention-based model proposed in
\cite{Bahdanau-et-al-ICLR2015-small} was able to achieve a relative
improvement of more than 50\% in the case of the English-to-French translation
task, as shown in Table~\ref{tbl:att_nmt}. When the very same model was extended with
a very large target vocabulary~\cite{Jean-et-al-ACL2015}, the relative
improvement over the baseline without the attention mechanism was 90\%.
Additionally, the very same model was recently tested on a number of European
language pairs at the WMT'15 Translation
Task.\footnote{\url{http://www.statmt.org/wmt15/}}.  See Table~\ref{tbl:wmt15}
for the results.

\begin{table}
    \centering
    \caption{The performance of the attention-based neural machine translation
            models with the very large target vocabulary in the WMT'15 Translation
            Track \cite{Jean-et-al-ACL2015}. We show the results on two
            representative language pairs. For the complete result, see
        http://matrix.statmt.org/.}
        \label{tbl:wmt15}
    \begin{tabular}{c | c c c }
        Language Pair & Model & BLEU & Note \\
        \hline
        \hline
        \multirow{2}{*}{En-$>$De} & NMT & 24.8 & \\
                                & Best Non-NMT & 24.0 & Syntactic SMT (Edinburgh) \\
        \hline
        \multirow{2}{*}{En-$>$Cz} & NMT & 18.3 & \\
                                & Best Non-NMT & 18.2 & Phrase SMT (JHU) \\
    \end{tabular}
\end{table}

The authors of \cite{Gulcehre-Orhan-et-al-2015} recently proposed a method for
incorporating a monolingual language model into the attention-based neural
machine translation system. With this method, the attention-based model was
shown to outperform the existing statistical machine translation systems on
Chinese-to-English (restricted domains) and Turkish-to-English translation
tasks as well as other European languages they tested.

\subsection{Image Caption Generation}
\label{sec:capgen}

Image caption generation is a task in which a model looks at an input image and
generates a corresponding natural language description. The encoder--decoder
framework fits well with this task. The encoder will extract the
continuous-space representation, or the context, of an input image, for
instance, with a deep convolutional network (see Sec.~\ref{sec:cnn},) and from
this representation the conditional RNN-LM based decoder generates a natural
language description of the image. Very recently (Dec 2014), a number of
research groups
independently proposed to use the simple encoder--decoder model to solve the
image caption generation
\cite{Fang-et-al-CVPR2015,Karpathy+Li-CVPR2015,Mao-et-al-ICLR2015,Vinyals-et-al-CVPR2015}.
Instead, here we describe a more recently proposed approach based on the
attention-based encoder--decoder framework in \cite{Xu-et-al-ICML2015}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.8\columnwidth]{figures/nmt_att_caption.pdf}
    \caption{Graphical illustration of the attention-based encoder--decoder
    model for image caption generation.}
    \label{fig:capgen}
\end{figure}

\subsubsection{Model Description}

The usual encoder--decoder based image caption generation models use the
activation of the last fully-connected hidden layer as the continuous-space
representation, or the context vector, of the input image (see
Sec.~\ref{sec:imagenet}.) The authors of \cite{Xu-et-al-ICML2015} however
proposed to use the activation from the last {\it convolutional} layer of the
pre-trained convolutional network, as in the bottom half of
Fig.~\ref{fig:capgen}.

Unlike the fully-connected layer, in this case, the context set consists of
multiple vectors that correspond to different spatial regions of the input image
on which the attention mechanism can be applied. Furthermore, due to convolution
and pooling, the spatial locations in pixel space represented by each context vector overlaps
substantially with those represented by the neighbouring context vectors, which
helps the attention mechanism distinguish similar objects in an image using its
context information with respect to the whole image, or the neighbouring pixels.

Similarly to the attention-based neural machine translation in
Sec.~\ref{sec:nmt}, the decoder is implemented as a conditional RNN-LM. In
\cite{Xu-et-al-ICML2015}, the content-based attention mechanism (see
Eq.~\eqref{eq:att1}) with
either the weighted sum (see Eq.~\eqref{eq:att_wsum})
or hard decision (see Eq.~\eqref{eq:att_hard} was
tested by training a model with the maximum likelihood estimator from
Sec.~\ref{sec:mle} and the variational learning from Sec.~\ref{sec:vari},
respectively. The authors of \cite{Xu-et-al-ICML2015} reported the similar
performances with these two approaches on a number of benchmark datasets.

\begin{table}[ht]
    \centering
    \caption{The performances of the image caption generation models in the
        Microsoft COCO Image Captioning Challenge. ($\star$)
        \cite{Vinyals-et-al-CVPR2015}, ($\bullet$) \cite{Fang-et-al-CVPR2015},
        ($\circ$) \cite{Devlin2015}, ($\diamond$) \cite{Donahue-et-al-arxiv2014} and
        ($\ast$) \cite{Xu-et-al-ICML2015}. The rows are sorted according to M1.}
    \label{tbl:capgen}
    \begin{tabular}{c|c c|c c}
        & \multicolumn{2}{c|}{Human} & \multicolumn{2}{c}{Automatic} \\
        Model & M1 & M2 & BLEU & CIDEr \\
        \hline
        \hline
        Human & 0.638 & 0.675 & 0.471 & 0.91\\
        Google$^\star$ & 0.273 & 0.317 & 0.587 & 0.946 \\
        MSR$^\bullet$ & 0.268 & 0.322 & 0.567 & 0.925 \\
        Attention-based$^\ast$ & 0.262 & 0.272 & 0.523 & 0.878 \\
        Captivator$^\circ$ & 0.250 & 0.301 & 0.601 & 0.937 \\
        Berkeley LRCN$^\diamond$ & 0.246 & 0.268 & 0.534 & 0.891 \\
    \end{tabular}
\end{table}

\subsubsection{Experimental Result}

In \cite{Xu-et-al-ICML2015}, the attention-based image caption generator was
evaluated on three datasets; Flickr 8K~\cite{Hodosh-et-al-JAIR2013}, Flickr
30K~\cite{Young-et-al-TACL2014} and MS CoCo~\cite{Lin-et-al-ECCV2014}. In
addition to the self-evaluation, an ensemble of multiple attention-based models
was submitted to Microsoft COCO Image Captioning
Challenge\footnote{\url{https://www.codalab.org/competitions/3221}} and
evaluated with multiple automatic evaluation metrics\footnote{
    BLEU~\cite{Papineni2002}, METEOR~\cite{Denkowski2014},
    ROUGE-L~\cite{Lin2004} and CIDEr~\cite{Vedantam2014}.
} as well as by human evaluators.

In this Challenge, the attention-based approach ranked third based on the
percentage of captions that are evaluated as better or equal to human caption
(M1) and the percentage of captions that pass the Turing Test (M2).
Interestingly, the same model was ranked eighth according to the most recently
proposed metric of CIDEr and ninth according to the most widely used metric of
BLEU.\footnote{\url{http://mscoco.org/dataset/\#leaderboard-cap}}
It means that this model has better relative performance in terms of human
evaluation than in terms of the automatic metrics, which only look at
matching subsequences of words, not directly at the meaning of the generated sentence.
The performance of the top-ranked systems, including the attention-based model
from \cite{Xu-et-al-ICML2015}, are listed in Table~\ref{tbl:capgen}.

The attention-based model was further found to be highly interpretable,
especially, compared to the simple encoder--decoder models. See
Fig.~\ref{fig:capgen_alignment} for some examples.

\begin{figure*}[!ht]
    \centering
    \includegraphics[width=0.87\textwidth]{figures/capgen_good.pdf}
    \caption{Examples of the attention-based model attending to the correct
    object (\textit{white} indicates the attended regions, \textit{underlines}
indicated the corresponding word) \cite{Xu-et-al-ICML2015}}
    \label{fig:capgen_alignment}

\end{figure*}

\subsection{Video Description Generation}
\label{sec:capgen_vid}

Soon after the neural machine translation based on the simple encoder--decoder
framework was proposed in \cite{Sutskever-et-al-NIPS2014,Cho-et-al-EMNLP2014},
it was further applied to video description generation, which amounts to
translating a (short) video clip to its natural language description
\cite{Venugopalan-et-al-arxiv2014}. The authors of
\cite{Venugopalan-et-al-arxiv2014} used a pre-trained convolutional network
(see Sec.~\ref{sec:imagenet}) to extract a feature vector from each frame of the
video clip and average all the frame-specific vectors to obtain a single
fixed-dimensional context vector of the whole video. A conditional RNN-LM from
Sec.~\ref{sec:rnnlm} was used to generate a description based on this context
vector.

Since any video clip clearly has both temporal and spatial structures, it is
possible to exploit them by using the attention mechanism described throughout
this paper. In \cite{Li-et-al-ARXIV2015}, the authors proposed an approach based
on the attention mechanism to exploit the global and local temporal structures
of the video clips. Here we briefly describe their approach.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\columnwidth,clip=True,trim=0 25 0 20]{figures/NewStructure.png}
    \caption{The 3-D convolutional network for motion from
    \cite{Li-et-al-ARXIV2015}.}
    \label{fig:3dconv}
\end{figure}

\subsubsection{Model Description}

In \cite{Li-et-al-ARXIV2015}, two different types of encoders are tested. The
first one is a simple frame-wise application of the pre-trained convolutional
network. However, they did not pool those per-frame context vectors as was done
in \cite{Venugopalan-et-al-arxiv2014}, but simply form a context set consisting
of all the per-frame feature vectors. The attention mechanism will work to
select one of those per-frame vectors for each output symbol being decoded. In
this way, the authors claimed that the overall model captures the {\it global}
temporal structure (the structure across many frames, potentially across the
whole video clip.)

The other type of encoder in \cite{Li-et-al-ARXIV2015} is a so-called 3-D
convolutional network, shown in Fig.~\ref{fig:3dconv}. Unlike the usual convolutional network which often works
only spatially over a two-dimensional image, the 3-D convolutional network
applies its (local) filters across the spatial dimensions as well as the
temporal dimensions. Furthermore, those filters work not on pixels but on local
motion statistics, enabling the model to concentrate on motion rather than
appearance. Similarly to the strategy from Sec.~\ref{sec:imagenet}, the model
was trained on larger video datasets to recognize an action from each video
clip, and the activation vectors from the last convolutional layer were used as
context. The authors of \cite{Li-et-al-ARXIV2015} suggest that this encoder
extracts more local temporal structures complementing the global structures
extracted from the frame-wise application of a 2-D convolutional network.

The same type of decoder, a conditional RNN-LM, used in \cite{Xu-et-al-ICML2015}
was used with the content-based attention mechanism in Eq.~\eqref{eq:att1}.

\subsubsection{Experimental Result}

In \cite{Li-et-al-ARXIV2015}, this approach to video description generation has
been tested on two datasets; (1) Youtube2Text~\cite{Chen2011} and (2) Montreal
DVS~\cite{Torabi2015}. They showed that it is beneficial to have both types of
encoders together in their attention-based encoder--decoder model, and that the
attention-based model outperforms the simple encoder--decoder model. See
Table~\ref{tbl:capgen_vid} for the summary of the evaluation.

\begin{table}[ht]
    \centering
    \caption{The performance of the video description generation models on
    Youtube2Text and Montreal DVS. ($\star$)~Higher the better. ($\circ$)~Lower
the better.}
    \label{tbl:capgen_vid}
    \begin{tabular}{c|c c |c c}
        & \multicolumn{2}{c}{Youtube2Text} & \multicolumn{2}{c}{Montreal DVS} \\
        Model & METEOR$^\star$ & Perplexity$^\circ$ & METEOR & Perplexity \\
    \hline
    \hline
Enc-Dec & 0.2868 & 33.09       & 0.044  & 88.28       \\
+ 3-D CNN & 0.2832 & 33.42       & 0.051  & 84.41       \\
+ Per-frame CNN & 0.2900 & 27.89       & .040  & 66.63       \\
+ Both & 0.2960 & 27.55       & 0.057  & 65.44       \\
    \end{tabular}
\end{table}

Similarly to all the other previous applications of the attention-based model,
the attention mechanism applied to the task of video description also
provides a straightforward way to inspect the inner
workings of the model. See
Fig.~\ref{fig:capgen_vid_alignment} for some examples.

\begin{figure}[ht]
\centering
\includegraphics[width=0.9\columnwidth]{./figures/results_main.png}
\vspace{-2mm}
\caption{Two sample videos and their corresponding generated and ground-truth
    descriptions from Youtube2Text. The bar plot under each frame corresponds to
    the attention weight $\alpha_j^t$ (see
    Eq.~\eqref{eq:att2}) for the frame when the corresponding
    word (color-coded) was generated. Reprinted from
\cite{Li-et-al-ARXIV2015}.}
    \label{fig:capgen_vid_alignment}
\end{figure}

\subsection{End-to-End Neural Speech Recognition}
\label{sec:speech}

Speech recognition is a task in which a given speech waveform is translated
into a corresponding natural language transcription. Deep neural networks have
become a standard for the acoustic part of speech recognition systems
~\cite{deepSpeechReviewSPM2012}.
Once the input speech (often in the form of spectral
filter response) is processed with the deep neural network based acoustic
model, another model, almost always a hidden Markov model (HMM), is used to map
correctly the much longer sequence of speech into a shorter sequence of
phonemes/characters/words. Only recently, in
\cite{Graves-et-al-2006,Graves2012,Graves+Jaitly-ICML2014,Hannun2014}, fully neural
network based speech recognition models were proposed.

Here, we describe the recently proposed attention-based fully neural speech
recognizer from \cite{Chorowski2015}. For more detailed comparison between the
attention-based fully speech recognizer and other neural speech recognizers,
e.g., from \cite{Graves+Jaitly-ICML2014}, we refer the reader to
\cite{Chorowski2015}.

\subsubsection{Model Description--Hybrid Attention Mechanism}

The basic architecture of the attention-based model for speech recognition in
\cite{Chorowski2015} is similar to the other attention-based models described
earlier, especially the attention-based neural machine translation model in
Sec.~\ref{sec:nmt}. The encoder is a stacked bidirectional recurrent neural
network (BiRNN) \cite{Pascanu-et-al-ICLR2014} which reads the input sequence of
speech frames, where each frame is a 123-dimensional vector consisting of 40
Mel-scale filter-bank response, the energy and first- and second-order temporal
differences. The context set of the concatenated hidden states from the
top-level BiRNN is used by the decoder based on the conditional RNN-LM to
generate the corresponding transcription, which in the case of
\cite{Chorowski2015}, consists in a sequence of phonemes.

The authors of \cite{Chorowski2015} however noticed the peculiarity of speech
recognition compared to, for instance, machine translation. First, the lengths
of the input and output differ significantly; thousands of input speech frames
against a dozen of words. Second, the alignment between the symbols in the input
and output sequences is monotonic, where this is often not true in the case of
translation.

These issues, especially the first one, make it difficult for the content-based
attention mechanism described in Eqs.~\eqref{eq:att1}~and~\eqref{eq:att2} to
work well. The authors of \cite{Chorowski2015} investigated these issues more
carefully and proposed that the attention mechanism with location awareness are
particulary appropriate (see Eq.~\eqref{eq:att_score_gen}. The location
awareness in this case means that the attention mechanism directly takes into
account the previous attention weights to compute the next ones.

The proposed location-aware attention mechanism scores each context vector by
\begin{align*}
    e_i^t = f_{\text{ATT}}(\vz_{t-1}, \vc_i, f_{\text{LOC}}^i(\left\{
        \alpha_j^{t-1}
    \right\}_{j=1}^T),
\end{align*}
where $f_{\text{LOC}}^j$ is a function that extracts information from the
previous attention weights $\left\{ \alpha_j^{t-1} \right\}$ for the $i$-th context
vector. In other words, the location-aware attention mechanism takes into
account both the content $\vc_i$ and the previous attention weights $\left\{
\alpha_j^{t-1} \right\}_{j=1}^T$.

In \cite{Chorowski2015}, $f_{\text{LOC}}^j$ was implemented as
\begin{align}
    \label{eq:locatt2}
    f_{\text{LOC}}^j(\left\{ \alpha_j^t \right\}) =
    \sum_{k=j-\frac{K}{2}}^{j+\frac{K}{2}} \vv_k \alpha_{k}^{t-1},
\end{align}
where $K$ is the size of the window, and $\vv_k \in \RR^d$ is a
learned vector.

Furthermore, the authors of \cite{Chorowski2015} proposed additional
modifications to the attention mechanism, such as sharpening, windowing and
smoothing, which modify Eq.~\eqref{eq:att2}. For more details of each of these,
we refer the reader to \cite{Chorowski2015}.

\subsubsection{Experimental Result}

In \cite{Chorowski2015}, this attention-based speech recognizer was evaluated on
the widely-used TIMIT corpus~\cite{garofolo1993darpa}, closely following the
procedure from \cite{Graves-et-al-ICASSP2013}. As can be seen from
Table~\ref{tbl:speech}, the attention-based speech recognizer with the
location-aware attention mechanism can recognize a sequence of phonemes given a
speech segment can perform better than the conventional fully neural speech
recognition. Also, the location-aware attention mechanism helps the model
achieve better generalization error.

\begin{table}[ht]
    \caption{Phoneme error rates (PER). The bold-faced PER corresponds to the
    best error rate achieved with a fully neural network based model. From
\cite{Chorowski2015}.}
    \label{tbl:speech}
    \centering
    \begin{tabular}{l|c|c}

        \multicolumn{1}{c|}{\bf Model}  &\multicolumn{1}{c|}{\bf Dev} &\multicolumn{1}{c}{\bf Test} \\
        \hline
        \hline

        Attention-based Model & 15.9\% & 18.7\% \\
        Attention-based Model + Location-Awareness & 15.8\% & {\bf 17.6\%} \\
        \hline

        RNN Transducer~\cite{Graves-et-al-ICASSP2013} & N/A & 17.7\% \\
        \hline\hline
        Time/Frequency Convolutional Net+HMM
        \cite{Toth2014} & 13.9\% & 16.7\%
    \end{tabular}
\end{table}

Similarly to the previous applications, it is again possible to inspect the
model's behaviour by visualizing the attention weights. An example is shown in
Fig.~\ref{fig:speech}, where we can clearly see how the model attends to a
roughly correct window of speech each time it generates a phoneme.

\begin{figure*}[ht]
  \centering
  \includegraphics[width=0.9\textwidth]{figures/michael_smooth.pdf}

  \caption{
      Attention weights by the attention-based model with
      location-aware attention mechanism.  The vertical
      bars indicate ground-truth phone location. For more details, see
      \cite{Chorowski2015}.
  }
  \label{fig:speech}
\end{figure*}

\subsection{Beyond Multimedia Content Description}

We briefly present three recent works which applied the described
attention-based mechanism to tasks other than multimedia content description.

\subsubsection{Parsing--Grammar as a Foreign Language}

Parsing a sentence into a parse tree can be considered as a variant of machine
translation, where the target is not a sentence but its parse tree. In
\cite{Vinyals2014}, the authors evaluate the simple encoder--decoder model and
the attention-based model on generating the linearized parse tree associated with a natural
language sentence. Their experiments revealed that the attention-based parser
can match the existing state-of-the-art parsers which are often highly
domain-specific.

\subsubsection{Discrete Optimization--Pointer Network}

In \cite{Vinyals2015}, the attention mechanism was used to (approximately) solve
discrete optimization problems. Unlike the usual use of the described attention
mechanism where the decoder generates a sequence of output symbols, in their
application to discrete optimization, the decoder predicts which one of the
source symbols/nodes should be chosen at each time step.  The authors achieve
this by considering $\alpha_i^t$ as the probability of choosing the $i$-th input
symbol as the selected one, at each time step $t$.

For instance, in the case of travelling salesperson problem (TSP), the model
needs to generate a sequence of cities/nodes that cover the whole set of input
cities so that the sequence will be the shortest possible route in the input map
(a graph of the cities) to cover every single city/node. First, the encoder
reads the graph of a TSP instance and returns a set of context vectors, each of
which corresponds to a city in the input graph.  The decoder then returns a
sequence of probabilities over the input cities, or equivalently the context
vectors, which are computed by the attention mechanism. The model is trained to
generate a sequence to cover all the cities by correctly {\it attending} to each
city using the attention mechanism.

As was shown already in \cite{Vinyals2015}, this approach can be applied to any
discrete optimization problem whose solution is expressed as a subset of the
input symbols, such as sorting.

\subsubsection{Question Answering--Weakly Supervised Memory Network}
\label{sec:wsmn}

The authors of \cite{Sukhbaatar2015} applied the attention-based model to a
question-answering (QA) task. Each instance of this QA task consists of a set of
facts and a question, where each fact and the question are both natural language
sentences. Each fact is encoded into a continuous-space representation, forming a
context set of fact vectors. The attention mechanism is applied to the context
set given the continuous-space representation of the question so that the model
can focus on the relevant facts needed to answer the question.

\section{Related Work: Attention-based Neural Networks}

The most related, relevant model is a neural network with {\it location-based}
attention mechanism, as opposed to the content-based attention mechanism
described in this paper. The content-based attention mechanism computes the
relevance of each spatial, temporal or spatio-temporally localized region of the
input, while the location-based one directly returns to which region the model
needs to attend, often in the form of the coordinate such as the $(x,y)$-coordinate
of an input image or the offset from the current coordinate.

In \cite{Graves-arxiv2013}, the location-based attention mechanism was
successfully used to model and generate handwritten text. In
\cite{ba+mnih,Mnih2014}, a neural network is designed to use the location-based
attention mechanism to recognize objects in an image. Furthermore, a generative
model of images was proposed in \cite{Gregor2015}, which iteratively reads and
writes portions of the whole image using the location-based attention mechanism.
Earlier works on utilizing the attention mechanism, both content-based and
location-based, for object recognition/tracking can be found in
\cite{Larochelle2010,Denil2012,Zheng2014}.

The attention-based mechanim described in this paper, or its variant, may be
applied to something other than multimedia input. For instance, in
\cite{Graves2014}, a neural Turing machine was proposed, which implements a
memory controller using both the content-based and location-based attention
mechanisms. Similarly, the authors of \cite{Weston2014} used the content-based
attention mechanism with hard decision (see, e.g., Eq.~\eqref{eq:att_hard}) to
find relevant memory contents, which was futher extended to the weakly
supervised memory network in \cite{Sukhbaatar2015} in Sec.~\ref{sec:wsmn}.

\section{Looking Ahead...}

In this paper, we described the recently proposed attention-based
encoder--decoder architecture for describing multimedia content. We started by
providing background materials on recurrent neural networks (RNN) and
convolutional networks (CNN) which form the building blocks of the
encoder--decoder architecture. We emphasized the specific variants of those
networks that are often used in the encoder--decoder model; a conditional
language model based on RNNs (a conditional RNN-LM) and a pre-trained CNN for
transfer learning. Then, we introduced the simple encoder--decoder model
followed by the attention mechanism, which together form the central topic of
this paper, the attention-based encoder--decoder model.

We presented four recent applications of the attention-based encoder--decoder
models; machine translation (Sec.~\ref{sec:nmt}), image caption generation
(Sec.~\ref{sec:capgen}), video description generation (Sec.~\ref{sec:capgen_vid}) and
speech recognition (Sec.~\ref{sec:speech}). We gave a concise
description of the attention-based model for each of these applications together
with the model's performance on benchmark datasets. Furthermore, each
description was accompanied with a figure visualizing the behaviour of the
attention mechanism.

In the examples discussed above, the attention mechanism was primarily considered as a means to
building a model that can describe the input multimedia content in natural
language, meaning the ultimate goal of the attention mechanism was to aid the
encoder--decoder model for multimedia content description. However, this should
not be taken as the only possible application of the attention mechanism.
Indeed, as recent work such as the pointer networks \cite{Vinyals2015} suggests,
future applications of attention mechanisms could run the range of AI-related tasks.

Beside superior performance it delivers, an attention mechanism can be used to extract
the underlying mapping between two entirely different modalities {\it without}
explicit supervision of the mapping.  From Figs.~\ref{fig:alpha_viz},
\ref{fig:capgen_alignment}, \ref{fig:capgen_vid_alignment} and \ref{fig:speech},
it is clear that the attention-based models were able to infer -- \emph{in an unsuperivsed way} -- alignments
between different modalities (multimedia and its text description) that
agree well with our intuition. This suggests that this type of attention-based
model can be used solely to extract these underlying, often complex, mappings
from a pair of modalities, where there is not much prior/domain knowledge. As an
example, attention-based models can be used in neuroscience to temporally
and spatially map between the neuronal activities and a sequence of
stimuli~\cite{Wehbe2014}.

\section*{Acknowledgment}

The authors would like to thank the following for research funding and
computing support: NSERC, FRQNT, Calcul Qu\'{e}bec, Compute Canada, the Canada Research
Chairs, CIFAR and Samsung.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


text:
ISSUES/LOCATIONS
Select ISSUES Aesthetics General Human rights Impacts Economics Emissions Environment Grid Health Noise Property values Tourism Wildlife Law Contracts Filings Ordinances Regulations Safety Siting Technology LOCATIONS Africa Egypt South Africa Americas Canada Alberta British Columbia Manitoba New Brunswick Nova Scotia Ontario Prince Edward Island Saskatchewan Chile Mexico U.S. Alaska Arizona California Colorado Connecticut Delaware Florida Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Maine Maryland Massachusetts Michigan Minnesota Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Puerto Rico Rhode Island South Carolina South Dakota Tennessee Texas Vermont Virginia Washington West Virginia Wisconsin Wyoming Asia China India Iran Japan Taiwan Turkey Europe Belgium Denmark Finland France Germany Greece Ireland Italy Latvia Lithuania Netherlands Norway Poland Portugal Romania Slovenia Spain Sweden Switzerland U.K. England Northern Ireland Scotland Wales Oceania Australia New Caledonia New Zealand Photos Videos
View PDF, DOC, PPT, and XLS files on line
Add NWW documents to your site (click here)
posted: August 27, 2018 • Wildlife, Bats
Migratory bats are attracted by red light but not by warm-white light: Implications for the protection of nocturnal migrants
Author: Voight, Christian; et al. | Wildlife
The replacement of conventional lighting with energy‐saving light emitting diodes (LED) is a worldwide trend, yet its consequences for animals and ecosystems are poorly understood. Strictly nocturnal animals such as bats are particularly sensitive to artificial light at night (ALAN). Past studies have shown that bats, in general, respond to ALAN according to the emitted light color and that migratory bats, in particular, exhibit phototaxis in response to green light. As red and white light is frequently used in outdoor lighting, we asked how migratory bats respond to these wavelength spectra. At a major migration corridor, we recorded the presence of migrating bats based on ultrasonic recorders during 10‐min light‐on/light‐off intervals to red or warm‐white LED, interspersed with dark controls. When the red LED was switched on, we observed an increase in flight activity for Pipistrellus pygmaeus and a trend for a higher activity for Pipistrellus nathusii. As the higher flight activity of bats was not associated with increased feeding, we rule out the possibility that bats foraged at the red LED light. Instead, bats may have flown toward the red LED light source. When exposed to warm‐white LED, general flight activity at the light source did not increase, yet we observed an increased foraging activity directly at the light source compared to the dark control. Our findings highlight a response of migratory bats toward LED light that was dependent on light color. The most parsimonious explanation for the response to red LED is phototaxis and for the response to warm‐white LED foraging. Our findings call for caution in the application of red aviation lighting, particularly at wind turbines, as this light color might attract bats, leading eventually to an increased collision risk of migratory bats at wind turbines.
Christian C. Voigt, Katharina Rehnig, Oliver Lindecke, Gunārs Pētersons
Leibniz Institute for Zoo and Wildlife Research, Berlin, Germany (CCV, KR, OL)
Institute of Biology, Freie Universität Berlin, Germany (CCV, OL)
Faculty of Life Science, University of Vienna, Austria (KR)
Faculty of Veterinary Medicine, Latvia University of Life Sciences and Technologies, Jelgava (GP)
Ecology and Evolution. Published online August 25, 2018. doi: 10.1002/ece3.4400
Download original document: "Migratory bats are attracted by red light but not by warm-white light: Implications for the protection of nocturnal migrants"
This material is the work of the author(s) indicated. Any opinions expressed in it are not necessarily those of National Wind Watch.
The copyright of this material resides with the author(s). As part of its noncommercial effort to present the environmental, social, scientific, and economic issues of large-scale wind power development to a global audience seeking such information, National Wind Watch endeavors to observe "fair use" as provided for in section 107 of U.S. Copyright Law and similar "fair dealing" provisions of the copyright laws of other nations. Queries e-mail.
[ Short URLs: https://wind-watch.org/doc/?p=5208 | http://wndfo.net/D5208 ]
Tags: Wind power, Wind energy, Bats

redpajama_set_name: RedPajamaCommonCrawl


Hi! My name is Hope. I am a stay-at-home mom of 3 rowdy boys. A wife to one awesome husband (my high school sweetheart). A passionate newborn photographer that is a little nerdy. And most importantly, a follower of God.


Our commercial office furniture includes items designed for individual office spaces and products intended for use in larger sitting areas. Commercial furniture includes boardroom tables, conference room equipment and sitting-area furniture for use in employee break-out areas. Selections for individual office spaces include office desks, office chairs, office partitions and desk accessories. Our range includes hard-to-find items, such as computer monitor mounts and monitor risers. Office Plus Furniture offers a variety of reception desks and counters that will make a notable first impression with all visitors to your business location. We offer discounted pricing on projects and bulk orders. Ask our staff today!
Office Plus Furniture offers a complete line of affordable high-quality commercial office furniture to suit every business need. The professional staff at this Victoria-based company will design an office fitout and help you to customize product selections according to your design plan. Their proprietary e-book, 5 Things You Must Know Before Hiring an Office Fitout Company, is available as a free download.
In addition to commercial office furniture, Office Plus Furniture offers a full range of school furniture. Products to choose from in this category include not only teachers' chairs, but student desks and large workstations. The company's specialty offerings include shelves, partitions and storage components. Office Plus Furniture makes every attempt to minimize disruptions to your school's activities and provides school furniture installation and aftercare with every purchase.
Great customer service at Office Plus Furniture includes free consultations, free measurements and free quotes. The company will match competitors' prices if you provide them with a written quote from another supplier. All commercial furniture on their website is available in different sizes and colors and designs can be customized to meet your vision of the perfect office space. Office Plus Furniture offers quick order turnaround and can often deliver your order within three to five business days. All products are guaranteed for five to ten years and two showrooms offer products that are not found on the website.


text:
Living Electronics for Bio-interfacing
Speaker: Prof. Xiaocheng Jiang, Tufts University
Date: Nov 15, 2019; Time: 2:30pm Location: UTEB 175
Abstract: Electronic and biological systems represent two limiting thermodynamic models in terms of functioning and information processing. By converging the dynamic and self-adaptable features of bio-machinery and the rationally defined/programmed functionalities of electronic components, there is potential to evolve new capabilities to effectively interrogate and direct biologically significant processes, as well as novel bio-inspired systems/device concepts for a range of engineering applications. The intrinsic mismatches in physiochemical properties and signaling modality at biotic/abiotic interfaces, however, have made the seamless integration challenging. In this talk, I will present our recent effort in forging their structural and functional synergy through the design and development of: (1) bio-hybrid electronics, where living transducers, such as functional biomolecules, organelles, or cells, are integrated with electronic transducers using spatially-defined, biocompatible hydrogel as the interfacing material; and (2) biosynthetic electronics, where biogenic electron pathways are utilized to naturally bridge the gap between internal biological and external electrical circuits. Blurring the distinction between livings and non-livings, these efforts have the potential to facilitate the cross-system communication and broadly impact how complex structures/functions may be designed/engineered.
Biographical Sketch: Xiaocheng Jiang is an Assistant Professor in the Department of Biomedical Engineering at Tufts University. He received his Ph.D. in physical chemistry from Harvard University with Professor Charles Lieber, with a focus on the design and application of nanoscale materials and nanoelectronic devices. Prior to joining Tufts, he was an American Cancer Society postdoctoral fellow at Massachusetts General Hospital, where he worked with Prof. Mehmet Toner on functional microfluidics for early cancer diagnostics. His current research concentrates broadly at the interface of materials and biomedical science, with specific interests in bio-inspired/bio-integrable electronics. He is a recipient of NSF CAREER award (2017) and AFOSR young investigator award (2018).


•	Streamlined ability to deploy .NET applications to Stackato as a result of integration with the Iron Foundry platform. Developers can now use Stackato's automatic configuration tool to link Stackato to Iron Foundry, the implementation of VMware's Cloud Foundry that supports .NET.
•	"Containerization technology" that creates more secure applications and improves application performance by allowing applications to operate with less resources, in a production environment featuring multiple containers for each virtual machine.
•	An upgraded web-based management interface that provides increased visibility into the deployment of PaaS applications in Stackato.
Stackato now supports Java, Ruby, Python, Perl, PHP, Node.JS, Clojure, Scala, Erlang, and .NET and is emerging as a key player in the rapidly exploding PaaS space as evidenced by its aggressive roll-out cycle and clear commitments to multiple languages as well as performance.


text:
package org.rdv;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.rdv.data.Channel;
import org.rdv.datapanel.DataPanel;
import org.rdv.rbnb.RBNBController;
import org.rdv.ui.DataPanelContainer;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;

/**
 * A class to manage all the data panels.
 *
 * @author  Jason P. Hanley
 * @author  Wei Deng
 */
public class DataPanelManager {

	/**
	 * The logger for this class.
	 */
	static Log log = LogFactory.getLog(DataPanelManager.class.getName());

  /** the single instance of this class */
  private static DataPanelManager instance;

	/**
	 * A reference to the data panel container for the data panels
	 * to add their ui component too.
	 */
	private DataPanelContainer dataPanelContainer;

	/**
	 * A list of all the data panels.
	 */
	private List<DataPanel> dataPanels;

	/**
	 * A list of registered extensions
	 */
	private List<Extension> extensions;

	/**
	 * The name of the extensions configuration file
	 */
	private static String extensionsConfigFileName = "config/extensions.xml";

  /**
   * Gets the single instance of this class.
   *
   * @return  the single instance of this class
   */
  public static DataPanelManager getInstance() {
    if (instance == null) {
      instance = new DataPanelManager();
    }

    return instance;
  }

	/**
	 * The constructor for the data panel manager. This initializes the
	 * data panel container and the list of registered data panels.
	 */
	private DataPanelManager() {
		dataPanelContainer = new DataPanelContainer();

		dataPanels = new ArrayList<DataPanel>();
		extensions = new ArrayList<Extension>();

		loadExtenionManifest();
	}

	/**
	 * Load the the  extension manifest file describing all the
	 * extensions and their properties.
	 */
	private void loadExtenionManifest() {
		try {
			DocumentBuilder documentBuilder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
			Document xmlDocument = documentBuilder.parse(DataViewer.getResourceAsStream(extensionsConfigFileName));
			NodeList extensionNodes = xmlDocument.getElementsByTagName("extension");
			for (int i=0; i<extensionNodes.getLength(); i++) {
				Node extensionNode = extensionNodes.item(i);
				String extensionID = null;
				String extensionName = null;
				List<String> mimeTypes = new ArrayList<String>();
				NodeList parameters = extensionNode.getChildNodes();
				for (int j=0; j<parameters.getLength(); j++) {
					Node parameterNode = parameters.item(j);
					String parameterName = parameterNode.getNodeName();
					if (parameterName.equals("id")) {
						extensionID = parameterNode.getFirstChild().getNodeValue();
					} else if (parameterName.equals("name")) {
						extensionName = parameterNode.getFirstChild().getNodeValue();
					} else if (parameterName.equals("mimeTypes")) {
						NodeList mimeTypeNodes = parameterNode.getChildNodes();
						for (int k=0; k<mimeTypeNodes.getLength(); k++) {
							Node mimeTypeNode = mimeTypeNodes.item(k);
							if (mimeTypeNode.getNodeName().equals("mimeType")) {
								mimeTypes.add(mimeTypeNode.getFirstChild().getNodeValue());
							}
						}
					}
				}

				Extension extension = new Extension(extensionID, extensionName, mimeTypes);
				extensions.add(extension);
				log.info("Registered extension " + extensionName + " (" + extensionID + ").");
			}
		} catch (Exception e) {
			log.error("Failed to fully load the extension manifest.");
			e.printStackTrace();
		}
	}

	/**
	 * Return the registered extension specified by the given class.
	 *
	 * @param extensionClass  the class of the desired extension
	 * @return                the extension, or null if the class wasn't found
	 */
	public Extension getExtension(Class<?> extensionClass) {
		for (int i=0; i<extensions.size(); i++) {
			Extension extension = extensions.get(i);
			if (extension.getID().equals(extensionClass.getName())) {
				return extension;
			}
		}

		return null;
	}

  /**
   * Return the registered extension specified by the given class name.
   *
   * @param extensionID              the class name of the desired extension
   * @return                         the extension, or null if the class wasn't found
   * @throws ClassNotFoundException
   */
  public Extension getExtension(String extensionID) throws ClassNotFoundException {
    Class<?> extensionClass = Class.forName(extensionID);
    return getExtension(extensionClass);
  }

	/**
	 * Return a list of registered extensions.
	 *
	 * @return  an array list of extensions
	 * @see     Extension
	 */
	public List<Extension> getExtensions() {
		return extensions;
	}

  /**
   * Return a list of extension that support the channel.
   *
   * @param channelName  the channel to support
   * @return             a list of supported extensions
   */
  public List<Extension> getExtensions(String channelName) {
    Channel channel = RBNBController.getInstance().getChannel(channelName);

    String mime = null;
    if (channel != null) {
      mime = channel.getMetadata("mime");
    }

    List<Extension> usefulExtensions = new ArrayList<Extension>();
    for (int i=0; i<extensions.size(); i++) {
      Extension extension = extensions.get(i);
      List<String> mimeTypes = extension.getMimeTypes();
      for (int j=0; j<mimeTypes.size(); j++) {
        String mimeType = mimeTypes.get(j);
        if (mimeType.equals(mime)) {
          usefulExtensions.add(extension);
        }
      }
    }

    return usefulExtensions;
  }

  /**
   * Returns a list of extensions that support the channels. An extension must
   * support each channel to be included on this list.
   *
   * @param channels  the channels to support
   * @return          a list of supported extensions
   */
  public List<Extension> getExtensions(List<String> channels) {
    if (channels == null || channels.isEmpty()) {
      return Collections.emptyList();
    }

    List<Extension> extensions = getExtensions(channels.get(0));

    for (int i=1; i<channels.size(); i++) {
      String channel = channels.get(i);
      List<Extension> channelExtensions = getExtensions(channel);
      for (int j=extensions.size()-1; j>=0; j--) {
        Extension extension = extensions.get(j);
        if (!channelExtensions.contains(extension)) {
          extensions.remove(extension);
        }
      }
    }

    return extensions;
  }

  /**
   * Return the default extension for the channel or null if there is none.
   *
   * @param channelName  the channel to use
   * @return             the default extension or null if there is none
   */
  public Extension getDefaultExtension(String channelName) {
    Channel channel = RBNBController.getInstance().getChannel(channelName);
    String mime = null;
    if (channel != null) {
      mime = channel.getMetadata("mime");
    }

    return findExtension(mime);
  }

	/**
	 * Returns the data panel container where data panels can
	 * add their ui components too.
	 *
	 * @return  the data panel container
	 */
	public DataPanelContainer getDataPanelContainer() {
		return dataPanelContainer;
	}

	/**
	 * Calls the closePanel method on the specified data panel.
	 *
	 * After this method has been called this data panel instance has
	 * reached the end of its life cycle.
	 *
	 * @param dataPanel  the data panel to be closed
	 * @see              DataPanel#closePanel()
	 */
	public void closeDataPanel(DataPanel dataPanel) {
		dataPanel.closePanel();
		dataPanels.remove(dataPanel);
	}

	/**
	 * Calls the closePanel method on each registered data panel.
	 *
	 * @see    DataPanel#closePanel()
	 */
	public void closeAllDataPanels() {
		DataPanel dataPanel;
		for (int i=dataPanels.size()-1; i>=0; i--) {
			dataPanel = dataPanels.get(i);
			closeDataPanel(dataPanel);
		}
	}

	/**
	 * View the channel specified by the given channel name. This will
	 * cause the data panel manager to look for the default viewer for
	 * the mime type of the viewer.
	 *
	 * If the channel has no mime type, it is assumed to be numeric data
	 * unless the channel ends with .jpg, in which case it is assumed to
	 * be image or video data.
	 *
	 * @param channelName  the name of the channel to view
	 */
	public void viewChannel(String channelName) {
    viewChannel(channelName, getDefaultExtension(channelName));
	}

  /**
   * View the channel given by the channel name with the specified extension.
   *
   * @param channelName  the name of the channel to view
   * @param extension    the extension to view the channel with
   */
  public void viewChannel(String channelName, Extension extension) {
    if (extension != null) {
      log.info("Creating data panel for channel " + channelName + " with extension " + extension.getName() + ".");
      try {
        DataPanel dataPanel = createDataPanel(extension);
        if (!dataPanel.setChannel(channelName)) {
          log.error("Failed to add channel " + channelName + ".");
          closeDataPanel(dataPanel);
        }
      } catch (Exception e) {
        log.error("Failed to create data panel and add channel.");
        e.printStackTrace();
        return;
      }
    } else {
      log.warn("Failed to find data panel extension for channel " + channelName + ".");
    }
  }

  /**
   * View the list of channels with the specified extension. If the channel
   * supports multiple channels, all the channels will be viewed in the same
   * instance. Otherwise a new instance will be created for each channel.
   *
   * If adding a channel to the same instance of an extension fails, a new
   * instance of the extension will be created and the channel will be added
   * this this.
   *
   * @param channels   the list of channels to view
   * @param extension  the extension to view these channels with
   */
  public void viewChannels(List<String> channels, Extension extension) {
    DataPanel dataPanel = null;
    try {
      dataPanel = createDataPanel(extension);
    } catch (Exception e) {
      e.printStackTrace();
      return;
    }
    boolean supportsMultipleChannels = dataPanel.supportsMultipleChannels();

    for (String channel : channels) {
      if (supportsMultipleChannels) {
        boolean subscribed = dataPanel.addChannel(channel);

        if (!subscribed) {
          try {
            dataPanel = createDataPanel(extension);
          } catch (Exception e) {
            e.printStackTrace();
            return;
          }

          dataPanel.addChannel(channel);
        }
      } else {
        if (dataPanel == null) {
          try {
            dataPanel = createDataPanel(extension);
          } catch (Exception e) {
            e.printStackTrace();
            return;
          }
        }

        dataPanel.setChannel(channel);
        dataPanel = null;
      }
    }
  }

	/**
	 * Return an extension that is capable of viewing the
	 * provided mime type.
	 *
	 * @param mime  the mime type that the extension must support
	 * @return      an extension to view this mime type, or null if none is found
	 */
	private Extension findExtension(String mime) {
		for (int i=0; i<extensions.size(); i++) {
			Extension extension = extensions.get(i);
			List<String> mimeTypes = extension.getMimeTypes();
			for (int j=0; j<mimeTypes.size(); j++) {
				String mimeType = mimeTypes.get(j);
				if (mimeType.equals(mime)) {
					return extension;
				}
			}
		}

		return null;
	}

  /**
   * Creates the data panel referenced by the supplied extension ID. This first
   * finds the extension onbject then calls createDataPanel with this.
   *
   * @param extensionID  the ID of the extension to create
   * @throws Exception
   * @return             the newly created data panel
   */
  public DataPanel createDataPanel(String extensionID) throws Exception {
    return createDataPanel(getExtension(extensionID));
  }

	/**
	 * Creates the data panel referenced by the supplied extension
	 * ID. This ID is by convention, the name of the class implementing
	 * this extension.
	 *
	 * @param extension   the extension to create
	 * @throws Exception
	 * @return            the newly created data panel
	 */
	public DataPanel createDataPanel(Extension extension) throws Exception {
		Class<?> dataPanelClass;
		try {
			dataPanelClass = Class.forName(extension.getID());
		} catch (ClassNotFoundException e) {
			throw new Exception("Unable to find extension " + extension.getName());
		}

		DataPanel dataPanel = (DataPanel)dataPanelClass.newInstance();

		dataPanel.openPanel(this);
		dataPanels.add(dataPanel);

		return dataPanel;
	}

  /**
   * See if any data panel are subscribed to at least one of these channels.
   *
   * @param channels  the names of the channels to check
   * @return             true if any data panel is subscribed to the channels,
   *                     false otherwise
   */
  public boolean isAnyChannelSubscribed(List<String> channels) {
    for (DataPanel dataPanel : dataPanels) {
      for (String channel : channels) {
        if (dataPanel.isChannelSubscribed(channel)) {
          return true;
        }
      }
    }

    return false;
  }

  /**
   * Unsubscribe all data panels from these channels. Data panels will be
   * closed if they are subscribed to no additional channels.
   *
   * @param channel   the channel to unsubscribe
   */
  public void unsubscribeChannels(List<String> channels) {
    for (DataPanel dataPanel : dataPanels) {
      for (String channel : channels) {
        dataPanel.removeChannel(channel);
      }
    }

    closeEmptyDataPanels();
  }

  /**
   * Close data panels that aren't subscribed to any channels.
   */
  private void closeEmptyDataPanels() {
    for (int i=dataPanels.size()-1; i>=0; i--) {
      DataPanel dp = dataPanels.get(i);
      if (dp.subscribedChannelCount() == 0) {
        closeDataPanel(dp);
      }
    }
  }

  /**
   * Return a list of data panels.
   *
   * @return  a list of data panels
   */
  public List<DataPanel> getDataPanels() {
    return dataPanels;
  }
}

redpajama_set_name: RedPajamaGithub


Providing heating and air conditioning repair service and HVAC system installation to Hugo, MN 55038 and surrounding areas.
No matter the HVAC problem, Comfort Plus Heating & Cooling can help!
If you live in Hugo, MN or near the 55038 zip code and you need repair service on your furnace or air conditioner, CALL 651-426-4233.


One of only three men (along with John Young, Eugene Cernan) to circle the Moon on two separate flights. Lovell was aboard Apollo VIII, the first mission to orbit the Moon in December 1968. Due to land on Apollo XIII, the landing was aborted when an oxygen tank exploded on the way to the Moon. The subsequent effort to return the crew safely to Earth was one of the high points of America's Space Program.


We have a couple of Bike Rack Hitch Receivers for Open Range Trailers or others that have same width - 79". Minor scratches on one, other is still wrapped, but have never been installed. Receiver Tube is 1-1/4".


text:
Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul • Academy Festival Istanbul
ACADEMYACADEMY
Click here to download the program booklet for 2021 Season.
↳BACKBACK
Elena Rozanova | Piano
Elena Rozanova was born in Odessa, Soviet Union. Both her parents were musicians. She started her education at the Gnessine School and then was admitted at the Tchaikovsky Conservatory in Moscow. Her teachers were Tatiana Zelikman, Alexei Nassedkine and Evgueni Moguilevski. She won many major prizes in international competitions, including the Marguerite Long - Jacques Thibaud in Paris, the Eduard Flipse in Rotterdam, and the Takahiro Sonoda of Fukoaka in Japan. She is also a laureate of the Cziffra Foundation.
She performed as a soloist with many orchestras, such as the Orchestre National de France, the Royal Philarmonia of Flanders, the orchestra of the Beethovenhalle in Bonn, the Symphonic Orchestra of Novosibirsk and the Camerata of Saint Petersburg, la Camerata Ireland and worked with conductors such as Jean-Claude Casadesus, Saulus Sondeckis and Arnold Katz. She also has been invited to many prestigious festivals, such as La Roque d'Anthéron, La Folle Journée de Nantes, Île de France, Schleswig-Holstein, Carintische Sommer, Portogruaro, Ravinia.
She has a trio with violinist Svetlin Roussev and cellist François Salque. Her other partners for chamber music include Josef Silverstein, Raphael Oleg, Vladimir Mendelssohn, Dora Schwarzberg, Barry Douglas, le Quatuor Ysaÿe among others.
Ms. Rozanova performs regularly in prestigious concert halls such as the Théâtre des Champs- Elysées, le Théâtre de la Ville, the Théâtre du Châtelet in Paris, the Centro Cultural de Belém in Lisbon, the Cité de la Musique in Moscow, the Konzerthaus in Vienna. Her recording with violinist Graf Mourja was published in France by Harmonia Mundi in their collection of "Les Nouveaux Interprètes" (Ravel, Schnittke, Szymanowsky) and it has been awarded with "Choc de la Musique" by French magazine Le Monde de la Musique. Her CD which was dedicated to works of Shostakovich, Ravel and Prokofiev, published also by Harmonia Mundi, was awarded with 5 Stars by the German Magazine Fono Forum and was the object of a radio programme on channel Bayern 4 Klassik. The CD she recorded with violinist Svetlin Roussev, dedicated to works of Vladiguerov, was awarded with Diapason d'Or by Diapason Magazine, 10 of Classica/Répertoire and with "Coup de Cœur" from Piano Magazine. Her CD for Satirino Records has been acclaimed by the French and international press and chosen among the CDs of the month by cultural TV Channel ARTE.
Between 1999 and 2005 Elena Rozanova was a professor of chamber music with piano at the Musik Hochschule in Frankfurt and she is currently a piano professor at the Conservatoire á Rayonnement Régional de Paris. Her repertoire includes the works from baroque to contemporary period and she also made the debut of many new creations.
Ms. Rozanova is the artistic director of the festivals "Les Classiques au Val d'Isère", "Chatel Classics" and the director of the Conservatoire d'Agglomeration Dracenoise (Draguignan, South of France) where she also manages classical concert series.
InstagramInstagramDESIGN:LocksBridgeLocksBridgePRIVACYPRIVACY©2021


In the dying days of the last Labour government in 2010 the Child Poverty Act was passed, setting a target of reducing child poverty by 10% before 2020. But when the Conservative government was elected, doing something about it didn't quite fit in with George Osborne's plea for austerity: eradicating poverty is expensive so since 2010 the Conservative government has shied away from their predecessor's commitment. Today the numbers of children in poverty has barely declined at all and currently stands at 2.3 million—a rather shocking one in six of all children.
George Osborne's solution is creative as ever—redefine the term. Today poverty means living in a household with an income below 60% of the median. The proposed Welfare Reform and Work Bill, will replace this with new measures such as levels of unemployment, education, family breakdown and drug dependence and remove the link with income. For Osborne the beauty of the change is that in future no child with a working parent would be defined as "poor", thereby removing 63% of those currently on the registrar.
It would also allow the government to freeze benefit payments without increasing the number of people defined as poor.
Curiously, there is strong opposition to this change from the red leather benches on the other side of the House of Commons and the good people in ermine are fighting a rearguard action: last month the Lords voted by 290 to 198 to keep the original targets, requiring MPs to reconsider.
Whilst the House of Lords is not normally seen as an especially militant or radical body, currently Labour and the Lib Dems outnumber Conservatives – and in consequence the Lords have been flexing their muscles (such as they are) since the election.
It may seem curious that it is the Lords who are fighting for the rights of the poorest, but just maybe that's exactly what the most privileged in our society should be doing.
How wonderful it is to be rich.


Pope Francis has officially deemed the murder of more than 1 million Armenians in 1915-17 by the Ottoman Turks as "genocide" Sunday - a definitive step in ensuring recognition of the tragedy on the international scale.
"In the past century, our human family has lived through three massive and unprecedented tragedies," Francis stated, at the beginning of a Mass in the Armenian Catholic rite held at the Vatican.
"The first, which is widely considered 'the first genocide of the 20th century', struck your own Armenian people," he said. He named the other two tragedies as the Holocaust and Stalinism.
Francis's words could cause tensions between the Vatican and Turkey, but the Pope stressed that the declaration is a step toward justice.
"Concealing or denying evil is like allowing a wound to keep bleeding without bandaging it," he said, according to the Associated Press. In response, Turkey's embassy to the Vatican has cancelled a scheduled press conference for Sunday.
Only 22 countries recognize the systemic murder of Armenians as "genocide," and Israel and the US are not among them. Armenia marks the genocide every year on April 24.
Between 1-1.5 million Armenians were killed by the Turks in various massacres, ultimately leading to the genocide of over one-third of the Armenian population.
The carnage began after Turkish authorities ordered the executions of much of the Armenian elite in Istanbul on April 24, 1915.
Men, women, and children were later murdered by various means, including through forced marches, starvation, and poison.
The Ottoman government set up some 25 concentration camps as well throughout the period, and mass graves of up to 60,000 people were found in some locations.


On behalf of Neville & Cunat, LLP posted in wrongful death on Tuesday, December 11, 2018.
On behalf of Neville & Cunat, LLP posted in wrongful death on Friday, April 20, 2018.
Can I file criminal charges against someone for a wrongful death?
On behalf of Neville & Cunat, LLP posted in wrongful death on Friday, January 5, 2018.


text:
Wed	Jun 26
Problem Plastic: Investigating Microplastic Pollution in Nevada's Waterways
Category: Featured Projects
Microplastics are plastic pieces ranging in size from 5mm to microscopic particles, in other words, the size of a pencil's eraser or smaller. They've been found everywhere from the air we breathe to the deep ocean. A team of DRI researchers have begun to investigate these increasingly troubling pollutants in two bodies of freshwater in Nevada: Lake Tahoe and the Las Vegas Wash.
Read more: Problem Plastic: Investigating Microplastic Pollution in Nevada's Waterways
Tue	Jun 25
Meet Graduate Researcher Dante Staten
Category: Featured Researchers
Meet Dante Staten, a Ph.D. student in environmental science with an emphasis in environmental chemistry. Staten is working with Dr. Andrey Khlystov in the Organic Analytical Laboratory to study the human-caused air pollutant emissions and their effects on public health.
Read more: Meet Graduate Researcher Dante Staten
Thu	May 23
Meet Kristin VanderMolen, Ph.D.
Kristin is an assistant research professor and social scientist. Right now, she is studying how warnings about extreme heat and heat waves are received and acted upon by vulnerable populations in southern California and northwestern Mexico. Learn more about her work in this profile.
Read more: Meet Kristin VanderMolen, Ph.D.
Mon	Apr 29
Preserving Nevada's Lost City using drones
Nevada's "Lost City," located northeast of Las Vegas on a terrace above the Muddy River, has been lost twice before – first abandoned by the native people who built it, then later flooded beneath the waters of Lake Mead – but a team of archaeologists from the Desert Research Institute's Las Vegas campus hopes to ensure that it isn't lost a third time.
Read more: Preserving Nevada's Lost City using drones
Meet Josh Sackett, Ph.D.
Joshua Sackett, Ph.D., is a postdoctoral researcher with the Division of Hydrologic Sciences at DRI's campus in Las Vegas. Josh specializes in the study of microbes that inhabit Earth's deep subsurface environments. Learn more about his research in this Behind the Science profile.
Read more: Meet Josh Sackett, Ph.D.
Mon	Mar 11
Meet John Watson, Ph.D.
John Watson, Ph.D., is a research professor of air quality science with the Division of Atmospheric Sciences at the Desert Research Institute in Reno. John specializes in air quality measurements, source apportionment (tracing pollutants to their sources), and adverse effects of air pollutants...
Read more: Meet John Watson, Ph.D.
Wed	Feb 6
Meet Jim Hudson, Ph.D.
Jim Hudson, Ph.D., is a research professor of physics with the Division of Atmospheric Sciences at the Desert Research Institute in Reno. Jim specializes in cloud physics, and has worked throughout his career to gather and analyze field measurements of cloud condensation nuclei (CCN) from around the world...
Read more: Meet Jim Hudson, Ph.D.
Tue	Jan 29
Putting the Knowledge Fund to Work
Created in 2013, the Knowledge Fund spurs research, innovation, and commercialization in Nevada. Thanks to this support, researchers across the Silver State have been busy cultivating the intellectual property that will continue the diversification of our economy for years to come. DRI researchers have been awarded over $11 million by the Knowledge Fund. Learn more about DRI's Knowledge Fund supported projects in this blog.
Read more: Putting the Knowledge Fund to Work
Thu	Jan 17
DRI ice core research makes Discover magazine's list of top breakthroughs in 2018
Category: Awards and Honors
For the second time, research out of the Ultra-trace Ice Core Chemistry Laboratory at the Desert Research Institute (DRI) in Reno, Nevada, has been named one of the year's biggest scientific discoveries by Discover magazine.
Read more: DRI ice core research makes Discover magazine's list of top breakthroughs in 2018
Q&A with Charlene Martin, DRI's 2018 Technical Employee of the Year
Each year, faculty, students, and staff have the opportunity to nominate those technologists they believe go above and beyond to make DRI a great place to work for the Technical Employee of the Year award. From those nominations, a council of technical employees selects the recipient of the award. This year, the recipient is Charlene Martin, a financial accounting specialist who has served DRI for 24 years.
Read more: Q&A with Charlene Martin, DRI's 2018 Technical Employee of the Year
12 Days of DRI: 2018 in Review
Meet Erick Bandala, Ph.D.
Meet Graduate Researcher Megan Rennie
DRI at AGU 2018
Q & A with AGU Presenter Christine Albano
Featured Researchers


text:
Our Solar System (Scie...
Cart Subtotal $3.49
Astronomy and Space Science
View cart "Our Solar System (Science for Toddlers)" has been added to your cart.
Maximum Power Point Tracking System for solar PV array
Mike Holt's Illustrated Guide to Understanding NEC Requirements for Solar Photovoltaic Systems Based on the 2014 NEC
MPPT Controllers of Power Converters in Solar PV System
Aeronautics and Space, Astronomy, Astronomy and Space, Books, Children's Books, Education and Reference, Encyclopedias, Encyclopedias and Subject Guides, Science, Nature and How It Works
National Geographic Kids First Big Book of Space (National Geographic Little Kids First Big Books)
Off-Grid Living: How To Build Wind Turbine, Solar Panels And Micro Hydroelectric Generator To Power Up Your House: (Wind Power, Hydropower, Solar Energy, Power Generation)
Organic, Inorganic and Hybrid Solar Cells: Principles and Practice
Astronomy, Astronomy and Space, Books, Children's Books, Science, Nature and How It Works
Our Solar System (Science for Toddlers)
Passive Solar Energy House Projects: A How-To Guide
Photovoltaic Design and Installation For Dummies
Photovoltaic Power System: Modeling, Design, and Control
Photovoltaic Solar Energy: From Fundamentals to Applications
Photovoltaic Systems
Photovoltaic Systems Engineering, Fourth Edition
Photovoltaic Systems Engineering, Third Edition
Photovoltaics System Design and Practice
Top Innovative Solar Products (0)
Solar Panel Efficiency and Solar Panel Cost (0)
DYE-SENSITIZED SOLAR PANEL (0)
Top 5 Solar Products (0)
Solar Panel Installation (0)
POLYCRYSTALLINE SILICON SOLAR PANEL (0)
Solar Energy in India – Future Challenges (0)