text:
\section{Introduction}
Languages such as English, which have plenty of annotated resources at their disposal have better Natural Language Processing (NLP) capabilities than other languages that are not so fortunate in terms of annotated resources. For example, high quality POS taggers \cite{Toutanova:2003:FPT:1073445.1073478}, parsers \cite{socher-EtAl:2013:ACL2013}, sentiment analyzers \cite{DBLP:series/synthesis/2012Liu} are already available for English but this is not the case for many other languages such as Hindi, Marathi, Bodo, Farsi, Urdu, \textit{etc}. This situation was acceptable in the past when only a few languages dominated the digital content available online and elsewhere. However, the ever increasing number of languages on the web today has made it important to accurately process natural language data in such lesser-fortunate languages also. An obvious solution to this problem is to improve the annotated inventory of these languages but the involved cost, time and effort act as a natural deterrent to this.

Another option is to exploit the unlabeled data available in a language. In this context, vectorial text representations have proven useful for multiple NLP
tasks~\citep{Turian+Ratinov+Bengio-2010,CollobertR2011}. It's been shown that meaningful representations, capturing
syntactic and semantic similarity, can be learned from unlabeled
data. Along with a (usually smaller) set of labeled data, these
representations allow to exploit unlabeled data and improve
the generalization performance on some given task, even allowing to
generalize out of the vocabulary observed in the labeled data only (thereby, partly alleviating the problem of data sparsity).

While the majority of previous work on vectorial text representations has concentrated on the
monolingual case, recent work has started looking at learning word
and document representations that are aligned across languages ~\citep{KlementievA2012,ZhouW2013,MikolovT2013}. Such aligned representations can potentially allow the use of resources from a resource fortunate language to develop NLP capabilities in a resource deprived language ~\citep{a1,a2,a3,a4,a5}.
For example, if a common representation model is learned for representing English and German documents, then a classifier trained on annotated English documents can be used to classify German documents (provided we use the learned common representation model for representing documents in both languages).

Such reuse of  resources across languages has been tried in the past by projecting parameters learned from the annotated data of one language to another language ~\cite{a1,a2,a3,a4,a5}
These projections are enabled by a bilingual resource such as a Machine Translation (MT) system. Recent attempts at learning common bilingual representations ~\cite{KlementievA2012,ZhouW2013,MikolovT2013} aim to eliminate the need of such a MT system. Such bilingual
representations have been applied to a variety of problems, including
cross-language document classification~\cite{KlementievA2012} and phrase-based
machine translation~\cite{ZhouW2013}. A common property of these approaches
is that a {\it word-level alignment} of translated sentences is leveraged,
\textit{e.g.},\ to derive a regularization term relating word embeddings
across languages~\cite{KlementievA2012,ZhouW2013}. Such methods not only eliminate the need for an MT system but also outperform MT based projection approaches.

In this paper, we experiment with a method to learn
bilingual word representations that {\it does without word-to-word
alignment} of bilingual corpora during training. Unlike previous approaches ~\cite{KlementievA2012}, we only require
aligned sentences and do not rely on word-level alignments
(e.g.,\ extracted using GIZA++, as is usual), which simplifies the learning procedure. To do so, we propose a
bilingual autoencoder model, that learns hidden
encoder representations of paired bag-of-words sentences which are not only informative of the original bag-of-words
but also predictive of each other. Word representations can then easily be extracted from the encoder
and used in the context of a supervised NLP task. Specifically, we demonstrate the quality of these representations for
the task of cross-language document
classification, where a labeled data set can be available in one language, but not in another one.
As we'll see, our approach is able to reach state-of-the-art performance, achieving up to 10-14 percentage point
improvements over the best previously reported results.

\section{Autoencoder for Bags-of-Words}
\label{sec:autoencoder}

Let ${\bf x}$ be the bag-of-words representation of a
sentence. Specifically, each $x_i$ is a word index from a fixed
vocabulary of $V$ words. As this is a bag-of-words, the order of the
words within ${\bf x}$ does not correspond to the word order in the
original sentence. We wish to learn a $D$-dimensional vectorial
representation of our words from a training set of sentence
bag-of-words $\{{\bf x}^{(t)}\}_{t=1}^T$.

We propose to achieve this by using an autoencoder model that encodes
an input bag-of-words ${\bf x}$ with a sum of the representations
(embeddings) of the words present in ${\bf x}$, followed by a nonlinearity. Specifically, let matrix ${\bf W}$ be the $D\times V$
matrix whose columns are the
vector representations for each word. The encoder's computation will involve summing over the columns of ${\bf W}$ for each
word in the bag-of-word. We will note this encoder function ${\bm \phi}({\bf x})$.
Then, using a decoder, the autoencoder will be trained to optimize
a loss function that measures how predictive of the original bag-of-words the
 encoder representation ${\bm \phi}({\bf x})$ is.

There are different variations we can consider, in the design of the encoder/decoder and the choice of loss function. One must be careful however, as certain choices can be inappropriate for training on word observations, which
are intrinsically sparse and high-dimensional. In this paper, we explore and compare two different approaches, described in the next two sub-sections.

\subsection{Binary bag-of-words reconstruction training with merged mini-batches}
\label{sec:reconstruction_sampling}
In the first approach, we start from the conventional autoencoder architecture, which minimizes a cross-entropy loss that compares a binary vector observation with a decoder reconstruction. We thus convert the bag-of-words ${\bf x}$ into a fixed-size but sparse binary vector ${\bf v}({\bf x})$, which is such that $v({\bf x})_{x_i}$ is 1 if word $x_i$ is present in ${\bf x}$ or otherwise 0.

From this representation, we obtain an encoder representation by multiplying ${\bf v}({\bf x})$ with the word representation matrix ${\bf W}$
\begin{equation}
{\bm \phi}({\bf x}) = {\bf h}({\bf c} + {\bf W} {\bf v}({\bf x}))
\end{equation}
where ${\bf h}(\cdot)$ is an element-wise non-linearity such as the
sigmoid or hyperbolic tangent, and ${\bf c}$ is a
$D$-dimensional bias vector. Encoding thus involves summing the word representation of the words present at least once in the bag-of-word.

To produce a reconstruction, we parametrize the decoder using the following non-linear form:
\begin{equation}
  \widehat{{\bf v}}({\bf x}) = \mathrm{sigm}({\bf V}{\bm \phi}({\bf x}) + {\bf b})
\end{equation}
where ${\bf V}$ = ${\bf W}^T$ and ${\bf b}$ is the bias vector of the reconstruction layer and $\mathrm{sigm}(a) = 1 /
(1+\exp(-a))$ is the sigmoid non-linearity.

Then, the reconstruction
is compared to the original binary bag-of-words as follows:
\begin{equation}
\ell({\bf v}({\bf x})) = - \sum_{i=1}^{V} v({\bf x})_i \log( \widehat{v}({\bf x})_i) + (1 - v({\bf x})_i) \log (1 - \widehat{v}({\bf x})_i)~.
\end{equation}
Training then proceeds by optimizing the sum of reconstruction cross-entropies across the training set, \textit{e.g.},\
using stochastic or mini-batch gradient descent.

Note that, since the binary bag-of-words are very high-dimensional (the dimensionality corresponds to the
size of the vocabulary, which is typically large), the above training procedure which aims at reconstructing the complete binary bag-of-word, will be slow. Since we will later be training on millions of sentences, training on each individual sentence bag-of-words will be expensive.

Thus, we propose a simple trick, which exploits the bag-of-words structure of the input. Assuming we are performing mini-batch training (where a mini-batch contains a list of bag-of-words), we simply propose to merge the bags-of-words of the mini-batch into a single bag-of-word, and revert back to stochastic gradient descent. The resulting effect is that each update is as efficient as in stochastic gradient descent, but the number of updates per training epoch is divided by the mini-batch size. As we'll see in the experimental section, we've found this trick to still produces good word representations, while sufficiently reducing training time.

We note that, additionally, we could have used the stochastic approach proposed by \citet{DauphinY2011} for reconstructing binary bag-of-words representations of documents, to further improve the efficiency of training. They use importance sampling to avoid reconstructing the whole $V$-dimensional input vector.

\subsection{Tree-based decoder training}
\label{sec:tree-based_training}

The previous autoencoder architecture worked with a binary vectorial
representation of the input bag-of-word. In the
second autoencoder architecture we investigated, we considered
an architecture that instead works with the bag (unordered list) representation
more directly.

Firstly, the encoder representation will now involve a sum of the representation of all words, reflecting the relative frequency of each word:
\begin{equation}
{\bm \phi}({\bf x}) = {\bf h}\left({\bf c} + \sum_{i=1}^{|{\bf x}|} {\bf W}_{\cdot,x_i}\right)~.
\end{equation}
Notice that this implies that the scaling of the pre-activation (the input
to the nonlinearity) can vary between bags-of-words, depending on how many
words it contains. Thus, we'll optionally consider using the average of the representations, as opposed to the sum (this choice is cross-validated in our experiments).

Moreover, decoder training will assume that, from
the decoder's output, we can obtain a probability distribution over any word
$\widehat{x}$ observed at the reconstruction output layer
$p(\widehat{x}|{\bm \phi}({\bf x}))$. Then, we can treat the input bag-of-words as
a $|{\bf x}|$-trials multinomial sample from that distribution and use as
the reconstruction loss its negative log-likelihood:
\begin{equation}
  \ell({\bf x}) = \sum_{i=1}^{V} -\log p(\widehat{x}=x_i|{\bm \phi}({\bf x}))~.
\end{equation}
We now must ensure that the decoder can compute $p(\widehat{x}=x_i|{\bm \phi}({\bf x}))$
efficiently from ${\bm \phi}({\bf x})$. Specifically, we'd like to avoid a
procedure scaling linearly with the vocabulary size $V$, since $V$ will
be very large in practice. This precludes any procedure that would
compute the numerator of $p(\widehat{x}=w|{\bm \phi}({\bf x}))$ for each possible word $w$
separately and normalize so it sums to one.

We instead opt for an approach borrowed from the work on neural
network language models~\cite{Morin+al-2005,Mnih+Hinton-2009}. Specifically, we use a probabilistic
tree decomposition of $p(\widehat{x}=x_i|{\bm \phi}({\bf x}))$.  Let's assume each
word has been placed at the leaf of a binary tree.  We can then treat
the sampling of a word as a stochastic path from the root of the tree
to one of the leaves.

We denote as ${\bf l}(x)$ the sequence of
internal nodes in the path from the root to a given word $x$, with
$l(x)_1$ always corresponding to the root. We will denote as
${\bm \pi}(x)$ the vector of associated left/right branching choices on
that path, where $\pi(x)_k=0$ means the path branches left at internal
node $l(x)_k$ and branches right if $\pi(x)_k=1$ otherwise.  Then, the
probability $p(\widehat{x}=x|{\bm \phi}({\bf x}))$ of reconstructing a certain word $x$ observed in the
bag-of-words is computed as
\begin{equation}
  p(\widehat{x}|{\bm \phi}({\bf x})) = \prod_{k=1}^{|{\bm \pi}(\hat{x})|} p(\pi(\widehat{x})_k|{\bm \phi}({\bf x}))
\end{equation}
where $p(\pi(\widehat{x})_k|{\bm \phi}({\bf x}))$ is output by the decoder. By using a full
binary tree of words, the number of different decoder outputs required
to compute $p(\widehat{x}|{\bm \phi}({\bf x}))$ will be logarithmic in the vocabulary size
$V$. Since there are $|{\bf x}|$ words in the bag-of-words, at most
$O(|{\bf x}| \log V)$ outputs are required from the decoder. This is
of course a worst case scenario, since words will share internal nodes
between their paths, for which the decoder output can be computed just
once. As for organizing words into a tree, as in \citet{LarochelleH2012} we used a
random assignment of words to the leaves of the full binary tree,
which we have found to work well in practice.

Finally, we need to choose a parametrized form for the decoder.
We choose the following non-linear form:
\begin{equation}
  p(\pi(\widehat{x})_k=1|{\bm \phi}({\bf x})) = \mathrm{sigm}(b_{l(\hat{x}_i)_k} + {\bf V}_{l(\hat{x}_i)_k,\cdot} {\bm \phi}({\bf x}))
\end{equation}
where ${\bf b}$ is a ($V$-1)-dimensional bias
vector and ${\bf V}$ is a $(V-1)\times D$ matrix. Each left/right
branching probability is thus modeled with a logistic regression model
applied on the encoder representation of the input
bag-of-words ${\bm \phi}({\bf x})$.

\section{Bilingual autoencoders}
\label{sec:bilingual_autoencoder}

Let's now assume that for each sentence bag-of-words ${\bf x}$ in some
source language ${\cal X}$, we have an associated bag-of-words ${\bf y}$ for the same
sentence translated in some target language ${\cal Y}$ by a human expert.

Assuming we have a training set of such $({\bf x},{\bf y})$ pairs, we'd like
to use it to learn representations in both languages that are aligned,
such that pairs of translated words have similar representations.

To achieve this, we propose to augment the regular autoencoder
proposed in Section~\ref{sec:autoencoder} so that, from the sentence
representation in a given language, a reconstruction can be attempted
of the original sentence in the other language. Specifically, we now define language specific word representation
matrices ${\bf W}^{x}$ and ${\bf W}^{y}$, corresponding to the languages
of the words in ${\bf x}$ and ${\bf y}$ respectively. Let $V^{\cal X}$ and
$V^{\cal Y}$ also be the number of words in the vocabulary of both languages,
which can be different. The word representations however are of
the same size $D$ in both languages. For the binary reconstruction
autoencoder, the bag-of-words representations
extracted by the encoder becomes
\begin{eqnarray*}
  {\bm \phi}({\bf x}) = {\bf h}\left({\bf c} + {\bf W}^{\cal X} {\bf v}({\bf x})\right)~,~~ {\bm \phi}({\bf y}) = {\bf h}\left({\bf c} + {\bf W}^{\cal Y} {\bf v}({\bf y})\right)
\end{eqnarray*}
and are similarly extended for the tree-based autoencoder.
Notice that we share the
bias ${\bf c}$ before the nonlinearity across encoders,
to encourage the encoders in both languages to produce representations
on the same scale.

From the sentence in either languages, we want to be able to perform a
reconstruction of the original sentence in any of the languages. In
particular, given a representation in any language, we'd like a
decoder that can perform a reconstruction in language ${\cal X}$ and
another decoder that can reconstruct in language ${\cal Y}$.  Again,
we use decoders of the form proposed in either Section~\ref{sec:reconstruction_sampling}~or~\ref{sec:tree-based_training} (see Figures \ref{fig:errorbased_bilingual_autoencoder} and \ref{fig:treebased_bilingual_autoencoder}),
but let the decoders of each language have their own parameters
$({\bf b}^{\cal X},{\bf V}^{\cal X})$ and $({\bf b}^{\cal Y},{\bf V}^{\cal Y})$.

This encoder/decoder decomposition structure allows us to learn a mapping within
each language and across the languages. Specifically, for a given pair
$({\bf x},{\bf y})$, we can train the model to (1) construct ${\bf y}$ from ${\bf x}$ (loss $\ell({\bf x},{\bf y})$),
(2) construct ${\bf x}$ from ${\bf y}$ (loss $\ell({\bf y},{\bf x})$), (3) reconstruct ${\bf x}$ from itself (loss $\ell({\bf x})$) and
(4) reconstruct ${\bf y}$ from itself (loss $\ell({\bf y})$). We follow this approach
in our experiments and optimize the sum of the corresponding 4 losses during training.

\subsection{Cross-lingual correlation regularization}

\label{sec:correlation}
The bilingual encoder proposed above can be further enriched by ensuring that the embeddings learned for a given pair $({\bf x},{\bf y})$ are highly correlated. We achieve this by adding a correlation term to the objective function. Specifically, we could optimize
\begin{equation}
  \ell({\bf x},{\bf y}) +\ell({\bf y},{\bf x})  - \lambda \cdot cor({\bm \phi}({\bf x}), {\bm \phi}({\bf y}))
\end{equation}
where $cor({\bm \phi}({\bf x}), {\bm \phi}({\bf y}))$ is the correlation between the encoder representations learned for ${\bf x}$ and ${\bf y}$ and $\lambda$ is a scaling factor which ensures that the three terms in the loss function have the same range. Note that this approach could be used for either the binary bag-of-words or the tree-based reconstruction autoencoders.

\begin{figure}[t]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=0.4\textwidth]{autoen.png}}
\caption{Illustration of a binary reconstruction error based bilingual autoencoder that learns to reconstruct
the binary bag-of-words of the English sentence ``{\it the dog barked}''
from its French translation ``{\it le chien a jappé}''.}
\label{fig:errorbased_bilingual_autoencoder}
\end{center}
\vskip -0.2in
\end{figure}

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.4\textwidth]{autoencoder_framed.pdf}
\end{center}
\caption{Illustration of a tree-based bilingual autoencoder that learns to construct
the bag-of-words of the English sentence ``{\it the dog barked}''
from its French translation ``{\it le chien a jappé}''. The
horizontal blue line across the input-to-hidden connections
highlights the fact that these connections share the same parameters
(similarly for the hidden-to-output connections).}
\label{fig:treebased_bilingual_autoencoder}
\end{figure}

\subsection{Document representations}
\label{sec:docrep}
Once we learn the language specific word representation
matrices ${\bf W}^{x}$ and ${\bf W}^{y}$ as described above, we can use them to construct document representations, by using their columns as vector representations for words in both languages. Now, given a document ${\bf d}$ written in language ${\cal Z}\in \{{\cal X},{\cal Y}\}$ and containing $m$ words, $z_1, z_2, \dots, z_m$, we represent it as the tf-idf weighted sum of its words' representations:

\begin{equation}
  \psi({\bf d}) = \sum_{i=1}^{m} \textit{tf-idf}(z_i)\cdot{\bf W}_{.,z_i}^{\cal Z}
\end{equation}

We use the document representations thus obtained to train our document classifiers, in the cross-lingual document classification task described in Section~\ref{exp}.

\section{Related Work}
\label{sec:related_work}

Recent work that has considered the problem of learning bilingual representations of words usually has relied on word-level
alignments. \citet{KlementievA2012} propose to train simultaneously two neural
network languages models, along with a regularization term that
encourages pairs of frequently aligned words to have similar word
embeddings.  \citet{ZhouW2013} use a similar approach, with a different form
for the regularizer and neural network language models as
in~\cite{CollobertR2011}. In our work, we specifically investigate whether
a method that does not rely on word-level alignments can
learn comparably useful multilingual embeddings in the
context of document classification.

Looking more generally at neural networks that learn multilingual
representations of words or phrases, we mention the work of \citet{GaoJ2013}
which showed that a useful linear mapping between {\it separately
  trained} monolingual skip-gram language models could be learned.
They too however rely on the specification of pairs of words
in the two languages to align. \citet{MikolovT2013} also propose a method
for training a neural network to learn useful representations
of phrases (i.e.\ short segments of words), in the context of
a phrase-based translation model. In this case, phrase-level
alignments (usually extracted from word-level alignments)
are required.

\section{Experiments}
\label{exp}

The techniques proposed in this paper enable us to learn bilingual embeddings which capture cross-language similarity between words. We propose to evaluate the quality of these embeddings by using them for the task of cross-language document classification. We follow the same setup as used by \citet{KlementievA2012} and compare with their method. The set up is as follows. A labeled data set of documents in some language ${\cal X}$ is available to train a classifier, however we are interested in classifying documents in a different language ${\cal Y}$ at test time. To achieve this, we leverage some bilingual corpora, which importantly is not labeled with any document-level categories. This bilingual corpora is used instead to learn document representations in both languages ${\cal X}$ and ${\cal Y}$ that are encouraged to be invariant to translations from one language to another. The hope is thus that we can successfully apply the classifier trained on document representations for language ${\cal X}$ directly to the document representations for language ${\cal Y}$. We use English (EN) and German (DE) as the language pair for all our experiments.

\subsection{Data}
For learning the bilingual embeddings, we used the English German section of the Europarl corpus \cite{europ} which contains roughly 2 million parallel sentences. As mentioned earlier, unlike \citet{KlementievA2012}, we do not use any word alignments between these parallel sentences. We use the same pre-processing as used by \citet{KlementievA2012}. Specifically, we tokenize the sentences using NLTK \cite{nltk}, remove punctuations and lowercase all words. We do not remove stopwords (similar to ~\citet{KlementievA2012}).

Note that \citet{KlementievA2012} use the word-aligned Europarl corpus to first learn an interaction matrix between the words in the two languages. This interaction matrix is then used in a multitask learning setup to induce bilingual embeddings from English and German sections of the Reuters RCV1/RCV2 corpora. Note that these documents are not parallel. Each document is assigned one or more categories from a pre-defined hierarchy of topics. Following \citet{KlementievA2012}, we consider only those documents which were assigned exactly one of the 4 top level categories in the topic hierarchy. These topics are CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social) and MCAT (Markets). The number of such documents sampled by \citet{KlementievA2012} for English and German is 34,000 and 42,753 respectively. In contrast to \citet{KlementievA2012}, we do not require a two stage approach (of learning an interaction matrix and then inducing bilingual embeddings). We directly learn the embeddings from the Europarl corpus which is parallel. Further, in addition to the Europarl corpus, we also considered feeding the same RCV1/RCV2 documents (34000 EN and 42,753 DE) to the autoencoders. These non-parallel documents are used only to reinforce the monolingual embeddings (by reconstructing ${\bf x}$ from ${\bf x}$ or ${\bf y}$ from ${\bf y}$). So, in effect, we use the same amount of data as that used by \citet{KlementievA2012} but our model/training procedure is completely different.

Next for the cross language classification experiments, we again follow the same setup as used by \citet{KlementievA2012}. Specifically, we use 10,000 single-topic documents for training and 5000 single-topic documents for testing in each language. These documents are also pre-processed using a similar procedure as that used for the Europarl corpus.

\begin{table*}[!t]
\begin{center}
\caption{Example English words along with 10 closest words both in English (en) and German (de), using the Euclidean distance between the embeddings learned by BAE-cr/corr}
\begin{tabular}{cc|cc|cc}

   \multicolumn{2}{c|}{january} & \multicolumn{2}{|c|}{president} & \multicolumn{2}{|c}{said}\\
  \hline
  en & de & en & de & en & de \\
  \hline
  january&januar&president&präsident&said&gesagt\\
march&märz&i&präsidentin&told&sagte\\
october&oktober&mr&präsidenten&say&sehr\\
july&juli&presidents&herr&believe&heute\\
december&dezember&thank&ich&saying&sagen\\
1999&jahres&president-in-office&ratspräsident&wish&heutigen\\
june&juni&report&danken&shall&letzte\\
month&1999&voted&danke&again&hier\\
year&jahr&colleagues&bericht&agree&sagten\\
september&jahresende&ladies&kollegen&very&will\\
\hline
  \hline

   \multicolumn{2}{c|}{oil} & \multicolumn{2}{|c|}{microsoft} & \multicolumn{2}{|c}{market}\\
  \hline
  en & de & en & de & en & de \\
  \hline
  oil&öl&microsoft&microsoft&market&markt\\
supply&boden&cds&cds&markets&marktes\\
supplies&befindet&insider&warner&single&märkte\\
gas&gerät&ibm&tageszeitungen&commercial&binnenmarkt\\
fuel&erdöl&acquisitions&ibm&competition&märkten\\
mineral&infolge&shareholding&handelskammer&competitive&handel\\
petroleum&abhängig&warner&exchange&business&öffnung\\
crude&folge&online&veranstalter&goods&binnenmarktes\\
materials&ganze&shareholder&geschäftsführer&sector&bereich\\
causing&nahe&otc&aktiengesellschaften&model&gleichzeitig\\
  \hline

  \end{tabular}

  \label{Tabd}
  \end{center}
\end{table*}

\subsection{Cross language classification}
Our overall procedure for cross language classification can be summarized as follows:

\begin{itemize}
\item Train bilingual word representations ${\bf W}^{x}$ and ${\bf W}^{y}$ on sentence pairs extracted from Europarl-v7 for languages ${\cal X}$ and ${\cal Y}$. Optionally, we also use the monolingual documents from RCV1/RCV2 to reinforce the monolingual embeddings (this choice is cross-validated, as described in Section~\ref{sec:emb-models}).
\item Train document classifier on the Reuters training set for language ${\cal X}$, where documents are represented using the word representations ${\bf W}^{x}$.
\item Use the classifier trained in the previous step on the Reuters test set for language ${\cal Y}$, using the word representations ${\bf W}^{y}$ to represent the documents.
\end{itemize}

As in \citet{KlementievA2012} we used an averaged perceptron to train a multi-class classifier for 10 epochs, for all the experiments (\citet{KlementievA2012} report that the results were not sensitive to the number of epochs). The English and German vocabularies contained 43,614 and 50,110 words, respectively. Each document is represented with the tf-idf weighted linear combination of its word's embeddings, as described in Section~\ref{sec:docrep}, where only the words belonging to the above vocabulary are considered.

\subsection{Different models for learning embeddings}
\label{sec:emb-models}
From the different autoencoder architectures and the optional correlation-based regularization term proposed earlier, we trained 3 different models for learning bilingual embeddings. Each of these models is described below.

\begin{itemize}
\item BAE-tr: uses tree-based decoder training (see Section \ref{sec:tree-based_training}).
\item BAE-cr: uses reconstruction error based decoder training (see Section \ref{sec:reconstruction_sampling}).
\item BAE-cr/corr: uses reconstruction error based decoder training (see Section \ref{sec:reconstruction_sampling}), but unlike BAE-cr it uses the correlation based regularization term (see Section \ref{sec:correlation}).
\end{itemize}
As we'll see, BAE-cr is our worse performing model, thus this experiment will allow us to observe whether the correlation regularization can play an important role in improving the quality of the representations.

All of the above models were trained for up to 20 epochs using the same data as described earlier. All results are for word embeddings of size $D=40$, as in \citet{KlementievA2012}. Further, to speed up the training for BAE-cr and BAE-cr/corr we merged mini-batches of 5 adjacent sentence pairs into a single training instance, as described in Section~\ref{sec:reconstruction_sampling}.

Other hyperparameters were selected using a training/validation set split of 80\% and 20\% and using the performance on the validation set of an averaged perceptron trained on the smaller training set portion (notice that this corresponds to a monolingual classification experiment, since the general assumption is that no labeled data is available in the test set language).

\section{Results and Discussions}
Before discussing the results of cross language classification, we would first like to give a qualitative feel for the embeddings learned by our method. For this, we perform a small experiment where we select a few English words and list the top 10 English and German words which are most similar to these words (in terms of the Euclidean distance between their embeddings as learned by BAE-cr/corr). Table \ref{Tabd} shows the result of this experiment. For example, Table \ref{Tabd} shows that in all the cases the German word which is closest to a given English word is actually the translation of that English word. Also, notice that the model is able to capture semantic similarity between words by embedding semantically similar words (such as, \textit{(january, march)}, \textit{(gesagt, sagte)}, \textit{(market, commercial)}, \textit{etc.}) close to each other. The results of this experiment suggest that these bilingual embeddings should be useful for any cross language classification task as indeed shown by the results presented in the next section. The supplementary material also includes a 2D visualization of the word embeddings in both languages, generated using the t-SNE dimensionality reduction algorithm~\citep{VanDerMaaten08}.

\subsection{Comparison of the performance of different models}

We now present the cross language classification results obtained by using the embeddings produced by each of the 3 models described above. We also compare our models with the following approaches, for which the results are reported in \citet{KlementievA2012}:

\begin{itemize}

\item Klementiev et al. : This model uses word embeddings learned by a multitask neural network language model with a regularization term that
encourages pairs of frequently aligned words to have similar word
embeddings. From these embeddings, document representations are computed as described in Section~\ref{sec:docrep}.

\item MT: Here, test documents are translated to the language of the training documents using a Machine Translation (MT) system. MOSES\footnote{\url{http://www.statmt.org/moses/}}, a standard phrase-based MT system, using default parameters and a 5-gram language model was trained on the Europarl v7 corpus (same as the one used for inducing our bilingual embeddings).

\item Majority Class: Every test document is simply assigned the Majority class prevalent in the training data.

\end{itemize}

Table \ref{tab:title} summarizes the results obtained using 1K training data with different models. We report results in both directions, \textit{i.e.}, EN-DE and DE-EN. Between BAE-tr and BAE-cr, we observe that BAE-tr provides better performance and is comparable to the embeddings learned by the neural network language model of \citet{KlementievA2012} which, unlike BAE-tr, relies on word-level alignments. We also observe that the use of the correlation regularization is very beneficial. Indeed, it is able to improve the performance of BAE-cr and make it the best performing method, with more than 10\% in accuracy over other methods for the EN to DE task.

\begin {table}[t]
\caption {Classification Accuracy for training on English and German with 1000 labeled examples} \label{tab:title}
\begin{center}
\begin{tabular}{|l|l|l|}
\hline
{\bf } & EN $\rightarrow$ DE & DE $\rightarrow$ EN \\
\hline
BAE-tr & 80.2 & 68.2 \\
\hline
BAE-cr  & 78.2 & 63.6 \\
\hline
BAE-cr/corr  & {\bf 91.8} & {\bf 72.8} \\
\hline
Klementiev et al.  & 77.6 & 71.1 \\
\hline
MT  & 68.1 & 67.4 \\
\hline
Majority Class  & 46.8 & 46.8 \\
\hline
\end{tabular}
\end{center}
\end{table}

\subsection{Effect of varying training size}
Next, we evaluate the effect of varying the amount of supervised training data for training the classifier, with either BAE-tr, BAE-cr/corr or \citet{KlementievA2012} embeddigns. We experiment with training sizes of 100, 200, 500, 1000, 5000 and 10000.  These results for EN-DE and DE-EN are summarized in Figure \ref{fig:x} and Figure \ref{fig:y} respectively. We observe that BAE-cr/corr clearly outperforms the other models at almost all data sizes. More importantly, it performs remarkably well at very low data sizes ($t$=100) which suggests that it indeed learns very meaningful embeddings which can generalize well even at low data sizes.

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.48\textwidth]{classificationCurvesEN-DE.pdf}
\end{center}
\caption{Crosslingual classification accuracy results with English documents for the train set and German documents for the test set}
\label{fig:x}
\end{figure}

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.48\textwidth]{classificationCurvesDE-EN.pdf}
\end{center}
\caption{Crosslingual classification accuracy results with German documents for the train set and English documents for the test set}
\label{fig:y}
\end{figure}

\subsection{Effect of coarser alignments}
The excellent performance of BAE-cr/corr suggests that merging mini-batches
into single bags-of-words does not significantly impact the quality of the word embeddings. In other words, not only we do not need to rely on word-level alignments, but exact sentence-level alignment is also not essential to reach good performances. It is thus natural to ask the effect of using even coarser level alignments. We check this by varying the size of the merged mini-batches from 5, 25 to 50, for both BAE-cr/corr and BAE-tr. The cross language classification results obtained by using these coarser alignments are summarized in Table~\ref{tab:title2}.

Surprisingly, the performance of BAE-tr does not significantly decrease, by using merged mini-batches of size 5 (in fact, the performance even improves for the EN to DE task). However, with larger mini-batches, the performance can deteriorate, as is observed on the DE to EN task, for the BAE-cr/corr embeddings.

\begin {table}[t]
\caption {Classification Accuracy for training on English and German with coarser alignments for 1000 labeled examples} \label{tab:title2}
\begin{center}
\begin{tabular}{|l|l|l|l|}
\hline
{\bf } & Sent. per doc &  EN $\rightarrow$ DE & DE $\rightarrow$ EN \\
\hline
BAE-tr & 5 & 84.0 & 67.7 \\
\hline
BAE-tr & 25 & 83.0 & 63.4 \\
\hline
BAE-tr & 50 & 75.9 & 68.6 \\
\hline

BAE-cr/corr  & 5 & 91.75 & 72.78 \\
\hline
BAE-cr/corr  & 25 & 88.0 & 64.5 \\
\hline
BAE-cr/corr  &50 & 90.2 & 49.2 \\
\hline
\end{tabular}
\end{center}
\end{table}

\section{Conclusion and Future Work}

We presented evidence that meaningful bilingual word
representations could be learned without relying on word-level
alignments and can even be successful on fairly coarse sentence-level alignments. In particular, we showed that even though our model does not use word level alignments, it is able to outperform a state of the art word representation learning method that exploits word-level alignments. In addition, it also outperforms a strong Machine Translation based baseline. We observed that using a correlation based regularization term leads to better bilingual embeddings which are highly correlated and hence perform better for cross language classification tasks.

As future work we would like to investigate extensions of our bag-of-
words bilingual autoencoder to bags-of-n-grams, where the model would also have to learn representations for short phrases. Such a model  should be particularly useful in the context of a machine translation system. We would also like to explore the possibility of converting our bilingual model to a multilingual model which can learn common representations for multiple languages given different amounts of parallel data between these languages.

\bibliographystyle{unsrtnat}

\section{Introduction}
Languages such as English, which have plenty of annotated resources at their disposal have better Natural Language Processing (NLP) capabilities than other languages that are not so fortunate in terms of annotated resources. For example, high quality POS taggers \cite{Toutanova:2003:FPT:1073445.1073478}, parsers \cite{socher-EtAl:2013:ACL2013}, sentiment analyzers \cite{DBLP:series/synthesis/2012Liu} are already available for English but this is not the case for many other languages such as Hindi, Marathi, Bodo, Farsi, Urdu, \textit{etc}. This situation was acceptable in the past when only a few languages dominated the digital content available online and elsewhere. However, the ever increasing number of languages on the web today has made it important to accurately process natural language data in such lesser-fortunate languages also. An obvious solution to this problem is to improve the annotated inventory of these languages but the involved cost, time and effort act as a natural deterrent to this.

Another option is to exploit the unlabeled data available in a language. In this context, vectorial text representations have proven useful for multiple NLP
tasks~\citep{Turian+Ratinov+Bengio-2010,CollobertR2011}. It's been shown that meaningful representations, capturing
syntactic and semantic similarity, can be learned from unlabeled
data. Along with a (usually smaller) set of labeled data, these
representations allow to exploit unlabeled data and improve
the generalization performance on some given task, even allowing to
generalize out of the vocabulary observed in the labeled data only (thereby, partly alleviating the problem of data sparsity).

While the majority of previous work on vectorial text representations has concentrated on the
monolingual case, recent work has started looking at learning word
and document representations that are aligned across languages ~\citep{KlementievA2012,ZhouW2013,MikolovT2013}. Such aligned representations can potentially allow the use of resources from a resource fortunate language to develop NLP capabilities in a resource deprived language ~\citep{a1,a2,a3,a4,a5}.
For example, if a common representation model is learned for representing English and German documents, then a classifier trained on annotated English documents can be used to classify German documents (provided we use the learned common representation model for representing documents in both languages).

Such reuse of  resources across languages has been tried in the past by projecting parameters learned from the annotated data of one language to another language ~\cite{a1,a2,a3,a4,a5}
These projections are enabled by a bilingual resource such as a Machine Translation (MT) system. Recent attempts at learning common bilingual representations ~\cite{KlementievA2012,ZhouW2013,MikolovT2013} aim to eliminate the need of such a MT system. Such bilingual
representations have been applied to a variety of problems, including
cross-language document classification~\cite{KlementievA2012} and phrase-based
machine translation~\cite{ZhouW2013}. A common property of these approaches
is that a {\it word-level alignment} of translated sentences is leveraged,
\textit{e.g.},\ to derive a regularization term relating word embeddings
across languages~\cite{KlementievA2012,ZhouW2013}. Such methods not only eliminate the need for an MT system but also outperform MT based projection approaches.

In this paper, we experiment with a method to learn
bilingual word representations that {\it does without word-to-word
alignment} of bilingual corpora during training. Unlike previous approaches ~\cite{KlementievA2012}, we only require
aligned sentences and do not rely on word-level alignments
(e.g.,\ extracted using GIZA++, as is usual), which simplifies the learning procedure. To do so, we propose a
bilingual autoencoder model, that learns hidden
encoder representations of paired bag-of-words sentences which are not only informative of the original bag-of-words
but also predictive of each other. Word representations can then easily be extracted from the encoder
and used in the context of a supervised NLP task. Specifically, we demonstrate the quality of these representations for
the task of cross-language document
classification, where a labeled data set can be available in one language, but not in another one.
As we'll see, our approach is able to reach state-of-the-art performance, achieving up to 10-14 percentage point
improvements over the best previously reported results.

\section{Autoencoder for Bags-of-Words}
\label{sec:autoencoder}

Let ${\bf x}$ be the bag-of-words representation of a
sentence. Specifically, each $x_i$ is a word index from a fixed
vocabulary of $V$ words. As this is a bag-of-words, the order of the
words within ${\bf x}$ does not correspond to the word order in the
original sentence. We wish to learn a $D$-dimensional vectorial
representation of our words from a training set of sentence
bag-of-words $\{{\bf x}^{(t)}\}_{t=1}^T$.

We propose to achieve this by using an autoencoder model that encodes
an input bag-of-words ${\bf x}$ with a sum of the representations
(embeddings) of the words present in ${\bf x}$, followed by a nonlinearity. Specifically, let matrix ${\bf W}$ be the $D\times V$
matrix whose columns are the
vector representations for each word. The encoder's computation will involve summing over the columns of ${\bf W}$ for each
word in the bag-of-word. We will note this encoder function ${\bm \phi}({\bf x})$.
Then, using a decoder, the autoencoder will be trained to optimize
a loss function that measures how predictive of the original bag-of-words the
 encoder representation ${\bm \phi}({\bf x})$ is.

There are different variations we can consider, in the design of the encoder/decoder and the choice of loss function. One must be careful however, as certain choices can be inappropriate for training on word observations, which
are intrinsically sparse and high-dimensional. In this paper, we explore and compare two different approaches, described in the next two sub-sections.

\subsection{Binary bag-of-words reconstruction training with merged mini-batches}
\label{sec:reconstruction_sampling}
In the first approach, we start from the conventional autoencoder architecture, which minimizes a cross-entropy loss that compares a binary vector observation with a decoder reconstruction. We thus convert the bag-of-words ${\bf x}$ into a fixed-size but sparse binary vector ${\bf v}({\bf x})$, which is such that $v({\bf x})_{x_i}$ is 1 if word $x_i$ is present in ${\bf x}$ or otherwise 0.

From this representation, we obtain an encoder representation by multiplying ${\bf v}({\bf x})$ with the word representation matrix ${\bf W}$
\begin{equation}
{\bm \phi}({\bf x}) = {\bf h}({\bf c} + {\bf W} {\bf v}({\bf x}))
\end{equation}
where ${\bf h}(\cdot)$ is an element-wise non-linearity such as the
sigmoid or hyperbolic tangent, and ${\bf c}$ is a
$D$-dimensional bias vector. Encoding thus involves summing the word representation of the words present at least once in the bag-of-word.

To produce a reconstruction, we parametrize the decoder using the following non-linear form:
\begin{equation}
  \widehat{{\bf v}}({\bf x}) = \mathrm{sigm}({\bf V}{\bm \phi}({\bf x}) + {\bf b})
\end{equation}
where ${\bf V}$ = ${\bf W}^T$ and ${\bf b}$ is the bias vector of the reconstruction layer and $\mathrm{sigm}(a) = 1 /
(1+\exp(-a))$ is the sigmoid non-linearity.

Then, the reconstruction
is compared to the original binary bag-of-words as follows:
\begin{equation}
\ell({\bf v}({\bf x})) = - \sum_{i=1}^{V} v({\bf x})_i \log( \widehat{v}({\bf x})_i) + (1 - v({\bf x})_i) \log (1 - \widehat{v}({\bf x})_i)~.
\end{equation}
Training then proceeds by optimizing the sum of reconstruction cross-entropies across the training set, \textit{e.g.},\
using stochastic or mini-batch gradient descent.

Note that, since the binary bag-of-words are very high-dimensional (the dimensionality corresponds to the
size of the vocabulary, which is typically large), the above training procedure which aims at reconstructing the complete binary bag-of-word, will be slow. Since we will later be training on millions of sentences, training on each individual sentence bag-of-words will be expensive.

Thus, we propose a simple trick, which exploits the bag-of-words structure of the input. Assuming we are performing mini-batch training (where a mini-batch contains a list of bag-of-words), we simply propose to merge the bags-of-words of the mini-batch into a single bag-of-word, and revert back to stochastic gradient descent. The resulting effect is that each update is as efficient as in stochastic gradient descent, but the number of updates per training epoch is divided by the mini-batch size. As we'll see in the experimental section, we've found this trick to still produces good word representations, while sufficiently reducing training time.

We note that, additionally, we could have used the stochastic approach proposed by \citet{DauphinY2011} for reconstructing binary bag-of-words representations of documents, to further improve the efficiency of training. They use importance sampling to avoid reconstructing the whole $V$-dimensional input vector.

\subsection{Tree-based decoder training}
\label{sec:tree-based_training}

The previous autoencoder architecture worked with a binary vectorial
representation of the input bag-of-word. In the
second autoencoder architecture we investigated, we considered
an architecture that instead works with the bag (unordered list) representation
more directly.

Firstly, the encoder representation will now involve a sum of the representation of all words, reflecting the relative frequency of each word:
\begin{equation}
{\bm \phi}({\bf x}) = {\bf h}\left({\bf c} + \sum_{i=1}^{|{\bf x}|} {\bf W}_{\cdot,x_i}\right)~.
\end{equation}
Notice that this implies that the scaling of the pre-activation (the input
to the nonlinearity) can vary between bags-of-words, depending on how many
words it contains. Thus, we'll optionally consider using the average of the representations, as opposed to the sum (this choice is cross-validated in our experiments).

Moreover, decoder training will assume that, from
the decoder's output, we can obtain a probability distribution over any word
$\widehat{x}$ observed at the reconstruction output layer
$p(\widehat{x}|{\bm \phi}({\bf x}))$. Then, we can treat the input bag-of-words as
a $|{\bf x}|$-trials multinomial sample from that distribution and use as
the reconstruction loss its negative log-likelihood:
\begin{equation}
  \ell({\bf x}) = \sum_{i=1}^{V} -\log p(\widehat{x}=x_i|{\bm \phi}({\bf x}))~.
\end{equation}
We now must ensure that the decoder can compute $p(\widehat{x}=x_i|{\bm \phi}({\bf x}))$
efficiently from ${\bm \phi}({\bf x})$. Specifically, we'd like to avoid a
procedure scaling linearly with the vocabulary size $V$, since $V$ will
be very large in practice. This precludes any procedure that would
compute the numerator of $p(\widehat{x}=w|{\bm \phi}({\bf x}))$ for each possible word $w$
separately and normalize so it sums to one.

We instead opt for an approach borrowed from the work on neural
network language models~\cite{Morin+al-2005,Mnih+Hinton-2009}. Specifically, we use a probabilistic
tree decomposition of $p(\widehat{x}=x_i|{\bm \phi}({\bf x}))$.  Let's assume each
word has been placed at the leaf of a binary tree.  We can then treat
the sampling of a word as a stochastic path from the root of the tree
to one of the leaves.

We denote as ${\bf l}(x)$ the sequence of
internal nodes in the path from the root to a given word $x$, with
$l(x)_1$ always corresponding to the root. We will denote as
${\bm \pi}(x)$ the vector of associated left/right branching choices on
that path, where $\pi(x)_k=0$ means the path branches left at internal
node $l(x)_k$ and branches right if $\pi(x)_k=1$ otherwise.  Then, the
probability $p(\widehat{x}=x|{\bm \phi}({\bf x}))$ of reconstructing a certain word $x$ observed in the
bag-of-words is computed as
\begin{equation}
  p(\widehat{x}|{\bm \phi}({\bf x})) = \prod_{k=1}^{|{\bm \pi}(\hat{x})|} p(\pi(\widehat{x})_k|{\bm \phi}({\bf x}))
\end{equation}
where $p(\pi(\widehat{x})_k|{\bm \phi}({\bf x}))$ is output by the decoder. By using a full
binary tree of words, the number of different decoder outputs required
to compute $p(\widehat{x}|{\bm \phi}({\bf x}))$ will be logarithmic in the vocabulary size
$V$. Since there are $|{\bf x}|$ words in the bag-of-words, at most
$O(|{\bf x}| \log V)$ outputs are required from the decoder. This is
of course a worst case scenario, since words will share internal nodes
between their paths, for which the decoder output can be computed just
once. As for organizing words into a tree, as in \citet{LarochelleH2012} we used a
random assignment of words to the leaves of the full binary tree,
which we have found to work well in practice.

Finally, we need to choose a parametrized form for the decoder.
We choose the following non-linear form:
\begin{equation}
  p(\pi(\widehat{x})_k=1|{\bm \phi}({\bf x})) = \mathrm{sigm}(b_{l(\hat{x}_i)_k} + {\bf V}_{l(\hat{x}_i)_k,\cdot} {\bm \phi}({\bf x}))
\end{equation}
where ${\bf b}$ is a ($V$-1)-dimensional bias
vector and ${\bf V}$ is a $(V-1)\times D$ matrix. Each left/right
branching probability is thus modeled with a logistic regression model
applied on the encoder representation of the input
bag-of-words ${\bm \phi}({\bf x})$.

\section{Bilingual autoencoders}
\label{sec:bilingual_autoencoder}

Let's now assume that for each sentence bag-of-words ${\bf x}$ in some
source language ${\cal X}$, we have an associated bag-of-words ${\bf y}$ for the same
sentence translated in some target language ${\cal Y}$ by a human expert.

Assuming we have a training set of such $({\bf x},{\bf y})$ pairs, we'd like
to use it to learn representations in both languages that are aligned,
such that pairs of translated words have similar representations.

To achieve this, we propose to augment the regular autoencoder
proposed in Section~\ref{sec:autoencoder} so that, from the sentence
representation in a given language, a reconstruction can be attempted
of the original sentence in the other language. Specifically, we now define language specific word representation
matrices ${\bf W}^{x}$ and ${\bf W}^{y}$, corresponding to the languages
of the words in ${\bf x}$ and ${\bf y}$ respectively. Let $V^{\cal X}$ and
$V^{\cal Y}$ also be the number of words in the vocabulary of both languages,
which can be different. The word representations however are of
the same size $D$ in both languages. For the binary reconstruction
autoencoder, the bag-of-words representations
extracted by the encoder becomes
\begin{eqnarray*}
  {\bm \phi}({\bf x}) = {\bf h}\left({\bf c} + {\bf W}^{\cal X} {\bf v}({\bf x})\right)~,~~ {\bm \phi}({\bf y}) = {\bf h}\left({\bf c} + {\bf W}^{\cal Y} {\bf v}({\bf y})\right)
\end{eqnarray*}
and are similarly extended for the tree-based autoencoder.
Notice that we share the
bias ${\bf c}$ before the nonlinearity across encoders,
to encourage the encoders in both languages to produce representations
on the same scale.

From the sentence in either languages, we want to be able to perform a
reconstruction of the original sentence in any of the languages. In
particular, given a representation in any language, we'd like a
decoder that can perform a reconstruction in language ${\cal X}$ and
another decoder that can reconstruct in language ${\cal Y}$.  Again,
we use decoders of the form proposed in either Section~\ref{sec:reconstruction_sampling}~or~\ref{sec:tree-based_training} (see Figures \ref{fig:errorbased_bilingual_autoencoder} and \ref{fig:treebased_bilingual_autoencoder}),
but let the decoders of each language have their own parameters
$({\bf b}^{\cal X},{\bf V}^{\cal X})$ and $({\bf b}^{\cal Y},{\bf V}^{\cal Y})$.

This encoder/decoder decomposition structure allows us to learn a mapping within
each language and across the languages. Specifically, for a given pair
$({\bf x},{\bf y})$, we can train the model to (1) construct ${\bf y}$ from ${\bf x}$ (loss $\ell({\bf x},{\bf y})$),
(2) construct ${\bf x}$ from ${\bf y}$ (loss $\ell({\bf y},{\bf x})$), (3) reconstruct ${\bf x}$ from itself (loss $\ell({\bf x})$) and
(4) reconstruct ${\bf y}$ from itself (loss $\ell({\bf y})$). We follow this approach
in our experiments and optimize the sum of the corresponding 4 losses during training.

\subsection{Cross-lingual correlation regularization}

\label{sec:correlation}
The bilingual encoder proposed above can be further enriched by ensuring that the embeddings learned for a given pair $({\bf x},{\bf y})$ are highly correlated. We achieve this by adding a correlation term to the objective function. Specifically, we could optimize
\begin{equation}
  \ell({\bf x},{\bf y}) +\ell({\bf y},{\bf x})  - \lambda \cdot cor({\bm \phi}({\bf x}), {\bm \phi}({\bf y}))
\end{equation}
where $cor({\bm \phi}({\bf x}), {\bm \phi}({\bf y}))$ is the correlation between the encoder representations learned for ${\bf x}$ and ${\bf y}$ and $\lambda$ is a scaling factor which ensures that the three terms in the loss function have the same range. Note that this approach could be used for either the binary bag-of-words or the tree-based reconstruction autoencoders.

\begin{figure}[t]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=0.4\textwidth]{autoen.png}}
\caption{Illustration of a binary reconstruction error based bilingual autoencoder that learns to reconstruct
the binary bag-of-words of the English sentence ``{\it the dog barked}''
from its French translation ``{\it le chien a jappé}''.}
\label{fig:errorbased_bilingual_autoencoder}
\end{center}
\vskip -0.2in
\end{figure}

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.4\textwidth]{autoencoder_framed.pdf}
\end{center}
\caption{Illustration of a tree-based bilingual autoencoder that learns to construct
the bag-of-words of the English sentence ``{\it the dog barked}''
from its French translation ``{\it le chien a jappé}''. The
horizontal blue line across the input-to-hidden connections
highlights the fact that these connections share the same parameters
(similarly for the hidden-to-output connections).}
\label{fig:treebased_bilingual_autoencoder}
\end{figure}

\subsection{Document representations}
\label{sec:docrep}
Once we learn the language specific word representation
matrices ${\bf W}^{x}$ and ${\bf W}^{y}$ as described above, we can use them to construct document representations, by using their columns as vector representations for words in both languages. Now, given a document ${\bf d}$ written in language ${\cal Z}\in \{{\cal X},{\cal Y}\}$ and containing $m$ words, $z_1, z_2, \dots, z_m$, we represent it as the tf-idf weighted sum of its words' representations:

\begin{equation}
  \psi({\bf d}) = \sum_{i=1}^{m} \textit{tf-idf}(z_i)\cdot{\bf W}_{.,z_i}^{\cal Z}
\end{equation}

We use the document representations thus obtained to train our document classifiers, in the cross-lingual document classification task described in Section~\ref{exp}.

\section{Related Work}
\label{sec:related_work}

Recent work that has considered the problem of learning bilingual representations of words usually has relied on word-level
alignments. \citet{KlementievA2012} propose to train simultaneously two neural
network languages models, along with a regularization term that
encourages pairs of frequently aligned words to have similar word
embeddings.  \citet{ZhouW2013} use a similar approach, with a different form
for the regularizer and neural network language models as
in~\cite{CollobertR2011}. In our work, we specifically investigate whether
a method that does not rely on word-level alignments can
learn comparably useful multilingual embeddings in the
context of document classification.

Looking more generally at neural networks that learn multilingual
representations of words or phrases, we mention the work of \citet{GaoJ2013}
which showed that a useful linear mapping between {\it separately
  trained} monolingual skip-gram language models could be learned.
They too however rely on the specification of pairs of words
in the two languages to align. \citet{MikolovT2013} also propose a method
for training a neural network to learn useful representations
of phrases (i.e.\ short segments of words), in the context of
a phrase-based translation model. In this case, phrase-level
alignments (usually extracted from word-level alignments)
are required.

\section{Experiments}
\label{exp}

The techniques proposed in this paper enable us to learn bilingual embeddings which capture cross-language similarity between words. We propose to evaluate the quality of these embeddings by using them for the task of cross-language document classification. We follow the same setup as used by \citet{KlementievA2012} and compare with their method. The set up is as follows. A labeled data set of documents in some language ${\cal X}$ is available to train a classifier, however we are interested in classifying documents in a different language ${\cal Y}$ at test time. To achieve this, we leverage some bilingual corpora, which importantly is not labeled with any document-level categories. This bilingual corpora is used instead to learn document representations in both languages ${\cal X}$ and ${\cal Y}$ that are encouraged to be invariant to translations from one language to another. The hope is thus that we can successfully apply the classifier trained on document representations for language ${\cal X}$ directly to the document representations for language ${\cal Y}$. We use English (EN) and German (DE) as the language pair for all our experiments.

\subsection{Data}
For learning the bilingual embeddings, we used the English German section of the Europarl corpus \cite{europ} which contains roughly 2 million parallel sentences. As mentioned earlier, unlike \citet{KlementievA2012}, we do not use any word alignments between these parallel sentences. We use the same pre-processing as used by \citet{KlementievA2012}. Specifically, we tokenize the sentences using NLTK \cite{nltk}, remove punctuations and lowercase all words. We do not remove stopwords (similar to ~\citet{KlementievA2012}).

Note that \citet{KlementievA2012} use the word-aligned Europarl corpus to first learn an interaction matrix between the words in the two languages. This interaction matrix is then used in a multitask learning setup to induce bilingual embeddings from English and German sections of the Reuters RCV1/RCV2 corpora. Note that these documents are not parallel. Each document is assigned one or more categories from a pre-defined hierarchy of topics. Following \citet{KlementievA2012}, we consider only those documents which were assigned exactly one of the 4 top level categories in the topic hierarchy. These topics are CCAT (Corporate/Industrial), ECAT (Economics), GCAT (Government/Social) and MCAT (Markets). The number of such documents sampled by \citet{KlementievA2012} for English and German is 34,000 and 42,753 respectively. In contrast to \citet{KlementievA2012}, we do not require a two stage approach (of learning an interaction matrix and then inducing bilingual embeddings). We directly learn the embeddings from the Europarl corpus which is parallel. Further, in addition to the Europarl corpus, we also considered feeding the same RCV1/RCV2 documents (34000 EN and 42,753 DE) to the autoencoders. These non-parallel documents are used only to reinforce the monolingual embeddings (by reconstructing ${\bf x}$ from ${\bf x}$ or ${\bf y}$ from ${\bf y}$). So, in effect, we use the same amount of data as that used by \citet{KlementievA2012} but our model/training procedure is completely different.

Next for the cross language classification experiments, we again follow the same setup as used by \citet{KlementievA2012}. Specifically, we use 10,000 single-topic documents for training and 5000 single-topic documents for testing in each language. These documents are also pre-processed using a similar procedure as that used for the Europarl corpus.

\begin{table*}[!t]
\begin{center}
\caption{Example English words along with 10 closest words both in English (en) and German (de), using the Euclidean distance between the embeddings learned by BAE-cr/corr}
\begin{tabular}{cc|cc|cc}

   \multicolumn{2}{c|}{january} & \multicolumn{2}{|c|}{president} & \multicolumn{2}{|c}{said}\\
  \hline
  en & de & en & de & en & de \\
  \hline
  january&januar&president&präsident&said&gesagt\\
march&märz&i&präsidentin&told&sagte\\
october&oktober&mr&präsidenten&say&sehr\\
july&juli&presidents&herr&believe&heute\\
december&dezember&thank&ich&saying&sagen\\
1999&jahres&president-in-office&ratspräsident&wish&heutigen\\
june&juni&report&danken&shall&letzte\\
month&1999&voted&danke&again&hier\\
year&jahr&colleagues&bericht&agree&sagten\\
september&jahresende&ladies&kollegen&very&will\\
\hline
  \hline

   \multicolumn{2}{c|}{oil} & \multicolumn{2}{|c|}{microsoft} & \multicolumn{2}{|c}{market}\\
  \hline
  en & de & en & de & en & de \\
  \hline
  oil&öl&microsoft&microsoft&market&markt\\
supply&boden&cds&cds&markets&marktes\\
supplies&befindet&insider&warner&single&märkte\\
gas&gerät&ibm&tageszeitungen&commercial&binnenmarkt\\
fuel&erdöl&acquisitions&ibm&competition&märkten\\
mineral&infolge&shareholding&handelskammer&competitive&handel\\
petroleum&abhängig&warner&exchange&business&öffnung\\
crude&folge&online&veranstalter&goods&binnenmarktes\\
materials&ganze&shareholder&geschäftsführer&sector&bereich\\
causing&nahe&otc&aktiengesellschaften&model&gleichzeitig\\
  \hline

  \end{tabular}

  \label{Tabd}
  \end{center}
\end{table*}

\subsection{Cross language classification}
Our overall procedure for cross language classification can be summarized as follows:

\begin{itemize}
\item Train bilingual word representations ${\bf W}^{x}$ and ${\bf W}^{y}$ on sentence pairs extracted from Europarl-v7 for languages ${\cal X}$ and ${\cal Y}$. Optionally, we also use the monolingual documents from RCV1/RCV2 to reinforce the monolingual embeddings (this choice is cross-validated, as described in Section~\ref{sec:emb-models}).
\item Train document classifier on the Reuters training set for language ${\cal X}$, where documents are represented using the word representations ${\bf W}^{x}$.
\item Use the classifier trained in the previous step on the Reuters test set for language ${\cal Y}$, using the word representations ${\bf W}^{y}$ to represent the documents.
\end{itemize}

As in \citet{KlementievA2012} we used an averaged perceptron to train a multi-class classifier for 10 epochs, for all the experiments (\citet{KlementievA2012} report that the results were not sensitive to the number of epochs). The English and German vocabularies contained 43,614 and 50,110 words, respectively. Each document is represented with the tf-idf weighted linear combination of its word's embeddings, as described in Section~\ref{sec:docrep}, where only the words belonging to the above vocabulary are considered.

\subsection{Different models for learning embeddings}
\label{sec:emb-models}
From the different autoencoder architectures and the optional correlation-based regularization term proposed earlier, we trained 3 different models for learning bilingual embeddings. Each of these models is described below.

\begin{itemize}
\item BAE-tr: uses tree-based decoder training (see Section \ref{sec:tree-based_training}).
\item BAE-cr: uses reconstruction error based decoder training (see Section \ref{sec:reconstruction_sampling}).
\item BAE-cr/corr: uses reconstruction error based decoder training (see Section \ref{sec:reconstruction_sampling}), but unlike BAE-cr it uses the correlation based regularization term (see Section \ref{sec:correlation}).
\end{itemize}
As we'll see, BAE-cr is our worse performing model, thus this experiment will allow us to observe whether the correlation regularization can play an important role in improving the quality of the representations.

All of the above models were trained for up to 20 epochs using the same data as described earlier. All results are for word embeddings of size $D=40$, as in \citet{KlementievA2012}. Further, to speed up the training for BAE-cr and BAE-cr/corr we merged mini-batches of 5 adjacent sentence pairs into a single training instance, as described in Section~\ref{sec:reconstruction_sampling}.

Other hyperparameters were selected using a training/validation set split of 80\% and 20\% and using the performance on the validation set of an averaged perceptron trained on the smaller training set portion (notice that this corresponds to a monolingual classification experiment, since the general assumption is that no labeled data is available in the test set language).

\section{Results and Discussions}
Before discussing the results of cross language classification, we would first like to give a qualitative feel for the embeddings learned by our method. For this, we perform a small experiment where we select a few English words and list the top 10 English and German words which are most similar to these words (in terms of the Euclidean distance between their embeddings as learned by BAE-cr/corr). Table \ref{Tabd} shows the result of this experiment. For example, Table \ref{Tabd} shows that in all the cases the German word which is closest to a given English word is actually the translation of that English word. Also, notice that the model is able to capture semantic similarity between words by embedding semantically similar words (such as, \textit{(january, march)}, \textit{(gesagt, sagte)}, \textit{(market, commercial)}, \textit{etc.}) close to each other. The results of this experiment suggest that these bilingual embeddings should be useful for any cross language classification task as indeed shown by the results presented in the next section. The supplementary material also includes a 2D visualization of the word embeddings in both languages, generated using the t-SNE dimensionality reduction algorithm~\citep{VanDerMaaten08}.

\subsection{Comparison of the performance of different models}

We now present the cross language classification results obtained by using the embeddings produced by each of the 3 models described above. We also compare our models with the following approaches, for which the results are reported in \citet{KlementievA2012}:

\begin{itemize}

\item Klementiev et al. : This model uses word embeddings learned by a multitask neural network language model with a regularization term that
encourages pairs of frequently aligned words to have similar word
embeddings. From these embeddings, document representations are computed as described in Section~\ref{sec:docrep}.

\item MT: Here, test documents are translated to the language of the training documents using a Machine Translation (MT) system. MOSES\footnote{\url{http://www.statmt.org/moses/}}, a standard phrase-based MT system, using default parameters and a 5-gram language model was trained on the Europarl v7 corpus (same as the one used for inducing our bilingual embeddings).

\item Majority Class: Every test document is simply assigned the Majority class prevalent in the training data.

\end{itemize}

Table \ref{tab:title} summarizes the results obtained using 1K training data with different models. We report results in both directions, \textit{i.e.}, EN-DE and DE-EN. Between BAE-tr and BAE-cr, we observe that BAE-tr provides better performance and is comparable to the embeddings learned by the neural network language model of \citet{KlementievA2012} which, unlike BAE-tr, relies on word-level alignments. We also observe that the use of the correlation regularization is very beneficial. Indeed, it is able to improve the performance of BAE-cr and make it the best performing method, with more than 10\% in accuracy over other methods for the EN to DE task.

\begin {table}[t]
\caption {Classification Accuracy for training on English and German with 1000 labeled examples} \label{tab:title}
\begin{center}
\begin{tabular}{|l|l|l|}
\hline
{\bf } & EN $\rightarrow$ DE & DE $\rightarrow$ EN \\
\hline
BAE-tr & 80.2 & 68.2 \\
\hline
BAE-cr  & 78.2 & 63.6 \\
\hline
BAE-cr/corr  & {\bf 91.8} & {\bf 72.8} \\
\hline
Klementiev et al.  & 77.6 & 71.1 \\
\hline
MT  & 68.1 & 67.4 \\
\hline
Majority Class  & 46.8 & 46.8 \\
\hline
\end{tabular}
\end{center}
\end{table}

\subsection{Effect of varying training size}
Next, we evaluate the effect of varying the amount of supervised training data for training the classifier, with either BAE-tr, BAE-cr/corr or \citet{KlementievA2012} embeddigns. We experiment with training sizes of 100, 200, 500, 1000, 5000 and 10000.  These results for EN-DE and DE-EN are summarized in Figure \ref{fig:x} and Figure \ref{fig:y} respectively. We observe that BAE-cr/corr clearly outperforms the other models at almost all data sizes. More importantly, it performs remarkably well at very low data sizes ($t$=100) which suggests that it indeed learns very meaningful embeddings which can generalize well even at low data sizes.

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.48\textwidth]{classificationCurvesEN-DE.pdf}
\end{center}
\caption{Crosslingual classification accuracy results with English documents for the train set and German documents for the test set}
\label{fig:x}
\end{figure}

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.48\textwidth]{classificationCurvesDE-EN.pdf}
\end{center}
\caption{Crosslingual classification accuracy results with German documents for the train set and English documents for the test set}
\label{fig:y}
\end{figure}

\subsection{Effect of coarser alignments}
The excellent performance of BAE-cr/corr suggests that merging mini-batches
into single bags-of-words does not significantly impact the quality of the word embeddings. In other words, not only we do not need to rely on word-level alignments, but exact sentence-level alignment is also not essential to reach good performances. It is thus natural to ask the effect of using even coarser level alignments. We check this by varying the size of the merged mini-batches from 5, 25 to 50, for both BAE-cr/corr and BAE-tr. The cross language classification results obtained by using these coarser alignments are summarized in Table~\ref{tab:title2}.

Surprisingly, the performance of BAE-tr does not significantly decrease, by using merged mini-batches of size 5 (in fact, the performance even improves for the EN to DE task). However, with larger mini-batches, the performance can deteriorate, as is observed on the DE to EN task, for the BAE-cr/corr embeddings.

\begin {table}[t]
\caption {Classification Accuracy for training on English and German with coarser alignments for 1000 labeled examples} \label{tab:title2}
\begin{center}
\begin{tabular}{|l|l|l|l|}
\hline
{\bf } & Sent. per doc &  EN $\rightarrow$ DE & DE $\rightarrow$ EN \\
\hline
BAE-tr & 5 & 84.0 & 67.7 \\
\hline
BAE-tr & 25 & 83.0 & 63.4 \\
\hline
BAE-tr & 50 & 75.9 & 68.6 \\
\hline

BAE-cr/corr  & 5 & 91.75 & 72.78 \\
\hline
BAE-cr/corr  & 25 & 88.0 & 64.5 \\
\hline
BAE-cr/corr  &50 & 90.2 & 49.2 \\
\hline
\end{tabular}
\end{center}
\end{table}

\section{Conclusion and Future Work}

We presented evidence that meaningful bilingual word
representations could be learned without relying on word-level
alignments and can even be successful on fairly coarse sentence-level alignments. In particular, we showed that even though our model does not use word level alignments, it is able to outperform a state of the art word representation learning method that exploits word-level alignments. In addition, it also outperforms a strong Machine Translation based baseline. We observed that using a correlation based regularization term leads to better bilingual embeddings which are highly correlated and hence perform better for cross language classification tasks.

As future work we would like to investigate extensions of our bag-of-
words bilingual autoencoder to bags-of-n-grams, where the model would also have to learn representations for short phrases. Such a model  should be particularly useful in the context of a machine translation system. We would also like to explore the possibility of converting our bilingual model to a multilingual model which can learn common representations for multiple languages given different amounts of parallel data between these languages.

\bibliographystyle{unsrtnat}

redpajama_set_name: RedPajamaArXiv


text:
Bobby Allison Net Worth
in Richest Athletes › Race Car Drivers
Bobby Allison Net Worth:
💰 Compare Bobby Allison's Net Worth
Net Worth: $5 Million
Date of Birth: Dec 3, 1937 (85 years old)
Place of Birth: Miami
Profession: Race car driver
Bobby Allison net worth: Bobby Allison is an American former professional stock car racing driver and owner who has a net worth of $5 million. Bobby Allison was born in Miami, Florida in December 1937. He has been named one of NASCAR's 50 greatest drivers. Allison won the Daytona 500 in 1978, 1982 and 1988. He also was the 1983 Winston Cup champion. His sons Clifford and Davey Allison both became racing drivers but both died within a year of each other. Bobby Allison was the 1980 IROC Champion, the Southern 500 winner in 1971, 1972, 1975, and 1983, the World 600 Winner in 1981 and 1984, and the Winston 500 winner in 1979, 1981, and 1986. He was named the Winston Cup Series Most Popular Driver seven times. Bobby Allison was inducted into the NASCAR Hall of fame in 2011, the International Motorsports Hall of Fame in 1983, and the Motorsports Hall of Fame of America in 1992.
Darrell Waltrip Net Worth
Bubba Wallace Net Worth
Bruton Smith Net Worth
Dale Earnhardt Net Worth
Bill Elliott Net Worth
Chris Buescher Net Worth
Ray Evernham Net Worth
Larry McReynolds Net Worth
David Gilliland Net Worth

redpajama_set_name: RedPajamaCommonCrawl


Home.ca helps you find all homes listed for sale in the Kawartha Lakes MLS® system. The Norland neighbourhood has 10 homes on the market for sale or rent. Of the 538 total properties listed in Kawartha Lakes, Norland makes up just 1.9%. The average asking price of a property in Norland is $1,024,440, with an estimated mortgage of $3,470 per month. That is 1.5 times of the average asking price of $663,232 in Kawartha Lakes. Properties listed in Norland are an average of 870 square feet, with 1.9 beds and 2.5 baths.
The Norland neighbourhood has 0% apartments relative to all the other listings in this neighbourhood. Nearby neighbourhoods you should also look at include Coboconk, Burnt River, Kinmount, Rural Fenelon, Fenelon Falls.


text:
Snow blows.
By the end of last night, Jonathan and I were one nut short of the crazy house. It's one thing when you choose to have a day at home, do nothing...but completely different when you have no choice in the decision!
So this morning when the snow was six inches deep and I decided to wait until daylight to leave for work (instead of at 5 like normal), I almost lost my mind when the snow started falling again.
From the front porch this morning --
Maggie showing Viper that there is no water to be had in their metal water dish. Just snow.
Viper, after two minutes, deciding she'd had enough. Trust me, Viper, I feel you.
The absolutely beautiful Father Christmas that Kathie made for us...we snuck him outside for thirty seconds for an appropriate snow laden image...everything that's inside his bag is just for us...down to the small plastic camera tucked near his heart. :)
And the silver nativity scene, flanked by the lights of the newly decorated Christmas tree.
We're heading out in a few to test the roads. Wish us luck.
blogged by shelley at 8:18 AM
find more | the farm
it was quite the hassle for tacoma schools today. Oh well, there's a reason for everything right...I just need to figure out what it was! :)
A glimpse into the world of the Jonathan and Shelley Ski: with three whirlwinds known as the Wee Ski 1.0, 2.0, and 3.0. In 2007, we met and married. Over the next five years, we've had the joy of adding three sons to the mix: four-year-old Thomas, two-year-old John Paul and the newest Wee Ski, Alexander Xavier, who was born in early December 2012.
Here, you'll find a little of everything we love. Our family, our friends, our farm, our faith, our life through Shelley's photography and our passion to cure cancer through the Relay For Life.
Because, well, we're here, at least virtually. And sometimes it's easier to find us on the web than drive to the farm. Unless you have a tractor. We have one. It's pretty sweet. We roll on the farm in it.
the skis
Jonathan | Facebook
Shelley | Facebook
Shelley's Photography: Finian Road Image & Design, LLC
Finian Road: Recent Work
Finian Road's Online Shoppe
Our Farm: Sprouffske Trees
Through My Eyes: A 500 Day Photo Project
Our Relay Fundraising Progress
blogs of sweet peeps
Finian Road: The Blog
2016 Shoot to Fight Cancer - win a photo session from Finian Road!
Kili • little piece of aloha
Spring Break and Opening Day
Kathy • All About Charlie
Turning 47....
Sunday • WYSIWYG
Jual Batu Split Murah
Kate • Random Musings
Cream La Tansa | Cream Pemutih Wajah Alami
Mike & Kristine • The Fire Within
Peace Out, sniffle sniffle, happy happy joy joy
The Mallahans • As the family goes...
The best way to spend 15 minutes
Spencer • Bellamere Cottage
Where Bloggers Create Studio Tour!
a rosey outlook on life
Little House On The Hill
The Callons
How to put on daddy's shorts
Nancy • Mountain of dreams
Race Resume - 2013
The Sosinskis • From Sea to Shining Sea
A Soggy Blog- A Picture For Every Raindrop!
Tara • Miller's Paradise
Now we can sit with Dad
The Kang Family
Our Neglected Blog is Back!
Angie • The Kids Are Alright
Brenda • It's a Wonderful Life
The Slideshow from our Wedding
Nearly a decade of Christmas letters
An Open Letter to My In-Laws
Introducing Mr. & Mrs. Ski
Sleeping to Dream
Overheard • Between J & S
A Litany to Sanctify Work
Sweet Fireworks Images • 2007
Scenes from the Family Christmas Photo • 2006
(2005-2012) October 2015 (1) April 2014 (1) December 2013 (1) November 2013 (1) September 2013 (1) August 2013 (2) July 2013 (1) May 2013 (1) April 2013 (1) March 2013 (1) February 2013 (2) January 2013 (4) December 2012 (2) November 2012 (3) October 2012 (3) September 2012 (8) August 2012 (3) July 2012 (5) June 2012 (6) May 2012 (7) April 2012 (5) March 2012 (10) February 2012 (6) January 2012 (5) December 2011 (7) November 2011 (5) October 2011 (9) September 2011 (7) August 2011 (8) July 2011 (16) June 2011 (12) May 2011 (16) April 2011 (9) March 2011 (10) February 2011 (15) January 2011 (19) December 2010 (14) November 2010 (12) October 2010 (15) September 2010 (16) August 2010 (18) July 2010 (24) June 2010 (27) May 2010 (23) April 2010 (26) March 2010 (22) February 2010 (25) January 2010 (33) December 2009 (29) November 2009 (33) October 2009 (19) September 2009 (18) August 2009 (18) July 2009 (25) June 2009 (21) May 2009 (20) April 2009 (24) March 2009 (25) February 2009 (20) January 2009 (20) December 2008 (18) November 2008 (15) October 2008 (13) September 2008 (11) August 2008 (12) July 2008 (11) June 2008 (14) May 2008 (11) April 2008 (15) March 2008 (15) February 2008 (14) January 2008 (10) December 2007 (20) November 2007 (13) October 2007 (22) September 2007 (21) August 2007 (22) July 2007 (23) June 2007 (20) May 2007 (16) April 2007 (16) March 2007 (13) February 2007 (17) January 2007 (16) December 2006 (22) November 2006 (16) October 2006 (22) September 2006 (24) August 2006 (31) July 2006 (19) June 2006 (19) May 2006 (33) April 2006 (24) March 2006 (17) February 2006 (21) January 2006 (20) December 2005 (34) November 2005 (27) October 2005 (22) September 2005 (28) August 2005 (19) July 2005 (52) June 2005 (27) May 2005 (25) April 2005 (13) March 2005 (8) February 2005 (3) January 2005 (2)
things that make me happy (527)
the fam (279)
random musings (251)
baby ski (237)
the farm (207)
the prince (125)
relay for life (123)
e-props (87)
shelley mauss photography (50)
quick takes (49)
john paul (47)
good eats (38)
prayer requests (31)
small successes (26)
12 of 12 (23)
relay countdown (20)
circle journal (10)
december daily (10)
monthly letter to thomas (9)
advent daily (7)
the estrogen militia (7)
creative goodness (6)
finian road (6)
project life (5)
things that suck (3)
week in the life (3)
in the skitchen (2)
live from the blackberry (1)


GIGS EVERY NIEIIT TRON 20111-290! 0,. , WWW.
£5 or £3 concession See page 118 for more details . . .
0 Previews and reviews of the best events in music, theatre, , , 5.: ,g _- gist-254;3:423; '~ ~33 If: 1;; ; comedy, dance and the visual arts - " '7"??? POSIZCOde' ' if 0 Fun, definitive and hip guide to the club scene i _ i I. w:j ;' l f ' a ' 0 Book reviews and literary events ' ; Daytime talephone '. ' '_ "' f '70,": 0 Special supplements and festival guides s 4 r v » - ' «1 law-2;. - -.:;' T_ 0 Competitions and reader offers ',' ' . l r ' '4 - '" i Free postage and packing encloseacheque/poswiorderfeta-sf" \ l ' "


text:
Implementation in and by Switzerland
Multilateral institutions
Climate policy and financing
Finance and tax policy
Trade and investments
Free trade policy
InfoDoc
Core Concerns
as analysis
Having a voice is a prerequisite for development
© https://monitor.civicus.org
The CIVICUS Monitor measures the degree of freedom of civil society.
Political article
Eva Schmassmann
In functioning societies, civil society acts as a critical counterpart to the government. There is a worldwide trend toward restricting or even closing down spaces for participation and critical engagement by NGOs.
We read and hear it almost every day: autocrats such as Putin, Orbán or Erdoğan violently put down protests, order the arrest of critical bloggers and journalists and threaten them with death. Violence against those who stand up for their rights is daily fare in many countries. This is making it ever more dangerous to fight for one's own rights and for those of the underprivileged. A total of 312 human rights activists were killed in 27 countries last year. For that same period, 207 environmental activists lost their lives. An all-time high.
Other civil society organizations too are coming under intense pressure. This affects first and foremost organizations that are politically engaged, critical of the government, and demanding more accountability and transparency. In Kenya, for example, the government closed down some 1,500 NGOs between 2014 and 2015. Yet the underpinnings of sustainable development must include a voice for all stakeholders, transparency, and accountability of politicians to society, as political openness and development go hand in hand. Accordingly, countries where civil society has the benefit of an open, encouraging and enabling environment are ranked higher in the UN Human Development Index (HDI) than authoritarian countries, whose civil society must operate in a restricted, limited or even totally closed environment.
When NGOs do the work of governments
Such restrictions therefore also impact organizations that provide services and foster development, for example in education or health. They often fill gaps that developing country governments do not, whether on account of the remoteness of some regions, the lack of resources or because of bad governance. For beneficiaries, the work of these organizations is often crucial to survival, as it helps meet basic needs. If sustainable results are to be achieved, however, these services must go hand-in-hand with the political empowerment of beneficiaries, who must be in a position to claim their rights vis-a-vis the government and themselves take action to improve their lot and promote development in their region. To promote real, sustainable development, cooperation must be accompanied by the political work of these organizations.
As even authoritarian regimes recognize the service aspect of development work – whether at the local, national or international level – they do allow (at least ostensibly) free rein to non-political NGOs. After all, the funding and provision of basic services by others suits them just fine. The challenge in international cooperation therefore lies in encouraging and supporting civil society players without abetting the government and its principle of "divide and rule". Beneficiaries need to be politically empowered so that they can promote the necessary change in their society.
Good solutions include everyone
The role of civil society organizations in development work has long been recognized. In the 2030 Agenda for Sustainable Development, they are expressly mentioned as playing critical roles in implementation. It is important, in a functioning society, for development prospects to be debated collectively in an inclusive process. If instead debate takes place in a closed process within elite circles, the solutions favoured will be those that benefit the very few. The more people taken on board, the more inclusive the solutions will be.
This can be very easily recognized in resource-rich countries. The more people there are deciding over matters of distribution and use of natural resources, the more people will benefit from them.
The example of oil-rich Chad illustrates this correlation: since the start of oil mining in 2003 some US$13 billion have flowed into State coffers. Yet the country has fallen even lower down the UN Development Index and now occupies the ante-penultimate place. The earnings from oil exports are benefiting only a small elite around the country's President, Idriss Déby Itno, who has been in power for the past 27 years. According to CIVICUS, the global network for civilian participation, Chad's civil society is being stifled.
Leave no one behind, the guiding principle of the 2030 Agenda is central to the implementation of sustainable development for all: no one should be left behind, everyone must be a participant. This motto assumes that in particular the weakest will be included. According to the principle "nothing about us without us", they must be able to stand up for themselves and have a voice in the discussion of solutions. By definition, a critical civil society that champions sustainable and inclusive development is the counterpart to State authority and therefore needs space, recognition, access to funds and trust, so as to be able to play this role properly. There is no arguing away the dilemma whereby NGOs in fact operate in a space outside of State structures, at the same time being dependent on the State to protect that space.
It is precisely this space that is being eroded by the current trend toward shrinking space. This is occurring in various ways:
The government imposes excessive requirements relating to the registration and official recognition of an organization or the reporting on its work.
Access to funding is made difficult or even impossible. This may be through measures taken under the pretext of fighting terrorism or by applying the "foreign agent" stigma.
Violence, the threat of violence or toleration of violence by third parties.
The upshot is a climate of fear and insecurity, which often gives rise to self-censorship in which civil society players no longer venture to demand or decry things that are self-evident.
Joint learning process with SDC
Private and public development players in Switzerland have been closing ranks for many years now to push back against the trend towards shrinking space in developing countries. The first joint event was organized in 2016. On 14 September 2018 another event was held in Berne. On the basis of jointly conducted case studies in Myanmar, Tanzania and Cambodia, concrete options were discussed as to how governmental and private players could enhance the scope of action of their local partners.
The Alliance Sud position
In its development work in partner countries, Switzerland must push back against the trend towards restricting the space available to civil society. Protecting civil society is a core function of development cooperation. For it is only where open debate, inclusive and democratic decision-making processes are possible that rights can be won and enforced, elites can be held accountable and grievances overcome, thereby realizing sustainable development for all. In Switzerland too, much that we now take for granted first had to be fought for – be it women's suffrage or the rights of people with disabilities.
Political field
Aside content
President Civil Society Platform on the Agenda 2030
Politics - similar articles and issues
After the elections...
Laos: Unresolved fate of civil rights activist
Driver of development, neo-colonialism – or both?
Only diligent businesses accepted
Turn out and vote for change!
The obsolescence of the free trade model
Genuine development leaves no one behind
InfoDoc - Sources of related topics
Generationengespräch – Aktivismus in der Schweiz
Konzernverantwortung : Warum wir verbindliche Regeln für alle brauchen / Johannes Schorling ... [et al.]
Exploitation par le travail : recommandations du Conseil de l'Europe à la Suisse
Demokratie : Wie das Erfolgsmodell unter Druck geraten ist / Erhard Stackl ... [et al.]
Hope dies - action begins : Stimmen einer neuen Bewegung / Ingo Bittner [et all.]
Überblick: Gemeinsam für Gerechtigkeit : Prinzipien und Positionen zu Gerechtigkeit in der Arbeit von Brot für die Welt / Melanie Bleil [et all.] ; Brot für die Welt
Wendezeiten in Lateinamerika : Wird aus dem Umbruch auch ein politischer Aufbruch? / Günther Maihold / Stiftung Wissenschaft und Politik
I am interested in - Select -Alliance Sud InfoDocAlliance Sud Politics
I am (for our information) - Select -a studenta teacher/professorotherworking for an NGOworking for the media
Events of the civil society in Switzerland.
January 2020 – Neuchâtel
Peuple kurde: un otage de l'Histoire ?
January 2020 – Zürich
Filmabend: La Buena Vida -Das gute Leben
January 2020 – BFH-HAFL, Zollikofen
The role of plant health in sustainable food production
For media requests the responsible staff members are at your disposal.
Alliance Sud News No. 92 – Spring 2018
Former editions, subscription, advertisements.
Alliance Sud
PC 30-36070-0


text:
Architects & Designers >
Dressing rooms with a difference at Milan's Teatro degli Arcimboldi
Giulia Pellegrino
Milan, Metropolitan City of Milan, Italy
Here's your backstage pass to Teatro degli Arcimboldi. Wallpaper* talks to Milanese architect Giulia Pellegrino, who curated 'Vietato L'Ingresso', a project inviting 17 local practices to redesign dressing rooms in the legendary theatre
Milanese architect Giulia Pellegrino did not waste a moment during the pandemic. At a time when theatres were closed and social life was just a distant memory, she started what she calls a 'sofa project', which became a real, virtuous initiative presented during Milan Design Week 2021, and open for viewing on 18 September 2021. The heroes here are the dressing rooms of Milan's Teatro degli Arcimboldi (built between 1997 and 2002 by Italian architect Vittorio Gregotti, in collaboration with Mario Botta and Elisabetta Fabbri). They were in need of a restyling, and Pellegrino took care of it.
Presented by Show Bees (a Milan-based agency for the live arts) and Teatro degli Arcimboldi Milano (TAM), 'Vietato L'Ingresso' (meaning 'entry forbidden', a nod to the exclusive nature of the theatre's backstage areas) involved 17 local practices: Atelierzero, BDGS with Daniela Volpi, Calvi Ceschia Viganò, DAP Studio, Enrico Scaramellini, Francesco Rota, Galante Menichini, Gio Latis Studio, Lupettatelier, Monica Baio with Nicola Quadri, Studio Wok, NM Nick Maltese Studio, Paolo Volpato with Caterina Steiner, Park Associati, RGA studio, Studio Cagnato, and Senselab with Nicolò Gobini.
The rooms feature diverse styles, from the minimalist to the bold and colourful, with furniture designs by the likes of Living Divani, Molteni&C, USM, as well as Laufen and Zucchetti Kos.
Wallpaper* talked with Giulia Pellegrino to discover more about the project.
Wallpaper*: Keeping a theatre up and running without funds is a difficult thing. How did you start this project?
Giulia Pellegrino: This is what I call a sofa project: the months of total closure of the theatre due to the pandemic have forced us into an almost comfortable laziness. I was about to fall asleep lulled by this reassuring world where the 'nothing happens' was not followed by the sense of guilt when, suddenly, a light bulb started blinking. I am linked by a deep friendship with the founding members of Show Bees, Marzia Ginocchio and Gianmario Longoni, and together we worked on the project: the Arcimboldi theatre's management contract with the municipality of Milan was signed 15 days before the closure of the world, and from that moment I had closely followed the evolution of a situation much more complicated than that related to my own work environment.
W*: In your own words, what is the aim of 'Vietato L'Ingresso'?
GP: The dressing room is a place normally off-limits to the public, secret and private places where the artists become characters ready to go on stage. The project kept them in mind, those artists who stir our deepest emotions. The generosity of designers, sponsors and workers has allowed us to give strength to a strong gesture towards the sector with a permanent setting and not a temporary one. The intellectual and economic effort was enormous, but the fact that it is not being dismantled helped it to be successful. The places where magic was born have redone their make-up thanks to them.
W*: How did you select the architects involved?
GP: Once the idea was clear in my mind, I drew up a list of professional firms that were suitable for the project. The criteria were: to have a studio in Milan or the metropolitan area, as the Arcimboldi is owned by the municipality and therefore by the Milanese citizens; to work in interior design, even if on different scales and different design themes; and to have a good dose of recklessness, a huge one. The [project included] complete redesign and execution. In essence, they risked having to donate not only their work but also the realisation of it, the supply of objects and of furniture. Fortunately it did not go that way: we have been sponsored 100 per cent thanks to everyone's efforts.
W*: Did you give 'carte blanche' to the studios or did they receive a specific brief from your side?
GP: The project brief was very condensed: a mirror, a make-up table, some space or a furniture for relaxation and the inevitable pot for flowers. Nothing more. From there on, free pencils!
W*: Did the design interpretation have to be connected to Vittorio Gregotti's architecture?
GP: Since these are interior spaces, we did not set the condition that Gregotti's project had connections with theirs, but some of them did so spontaneously: holding a part of the original dressing room or graphically recalling the distinctive signs of the theatre and the surrounding Bicocca neighborhood.
W*: How did you work around the limitations of the pandemic?
GP: I wrote down the concept of 'Vietato L'Ingresso' in April 2021, the delivery of the projects was set in May, and the opening of the construction site was fixed on 6 June. My personal satisfaction was that of being an enlightened client, who does not ask to change the colour of the curtains nor is bothered by the choice of lamps. They had maximum freedom and my mantra was: the dressing rooms are all the same from the outside, just distinguished by different handles, but once you cross the threshold you will have 17 stories to be discovered.
W*: There is also an open contest, so that people can vote for the best dressing room; how important is it to involve the audience in such a kind of project?
GP: The technology of virtual visits, which has undergone an acceleration in the last year and a half, allows for an unlimited dissemination of any initiatives. The theatre is everyone's heritage, therefore everyone must be able to express their opinion. It is really about the audience, whether it's a concert, a ballet or an opera. The audience always imagines what's behind the scenes. Imagination always matters.
A changing room at Teatro Degli Arcimboldi, Milan, redesigned by Senselab as part of 'Vietato L'Ingresso'
Francesco Rota Architettura
Park Associati
Atelierzero
Calvi Ceschia Viganò Architetti Associati
Paolo Volpato and Caterina Steiner


Located on South Coast Hwy 101 in the Moonlight Plaza overlooking beautiful Moonlight Beach, we serve the communities of Encinitas, Cardiff-by-the-Sea, Olivenhain, La Costa, Carlsbad South, and Leucadia. Our Encinitas location, opened in 1980, was the first of our 3 restaurants. We provide great food and good service in a fun and casual setting. We welcome you to come in and celebrate all your special occasions with us.
Featuring fresh gourmet Italian food from locally-owned Leucadia Pizzeria. Make your next catered event easy by ordering the food that everyone loves… Italian! From an office party, to an at-home celebration, we can help with your next event.
The portions are so big and the food is so wonderful. The late dinner was a great choice. I like the people here friendly and attentive. The place is clean and the open patio is fantastic, you have a nice breeze and this is the perfect time to be outside. The lasagna was delicious, taking my leftovers home for later.
Perfect local family pizza restaurant for a large crowd. We love to sit in the enclosed outdoor patio that has a view of Moonlight Beach. They bring tons of delicious garlic bread to the table - before you've ordered. My family loves the BBQ chicken pizza. I like the caprese salad. Very attentive service.
This is the best pizza place for delivery, always my go-to for pizza delivery! Their cheese raviolis, cheese bread, garlic bread, and hot wings are delicious, too! They also always never forget to include carrots and celery with ranch when you order wings. It's always great!
This is my all time favorite restaurant. Grew up eating here and always make an effort to dine in when I am in town. Best garlic bread you will ever have. Had an amazing server today, can't remember his name at the moment. He took care of my mom and I and made our lunch really enjoyable. Other staff was really friendly and helpful as well!