text:
\section{Introduction}
The world we navigate through is a multimodal and multilingual kaleidoscope. While tremendous success has been realized in multimodal research with the advent of vision-and-language (V+L) pre-training~\cite{UNITER,OSCAR,vilbert,tan2019lxmert, huang2020pixelbert}, the majority of current literature is biased towards English. Although English-trained V+L models can be finetuned on each target language (given that there is sufficient language-specific data in downstream task), maintaining language-specific models for every language in the world (6,900+) is impossible given insurmountable development and maintenance cost \cite{hu2020xtreme}.
Naturally, a ``Tower of Babel'' strategy starts to gain interest in the community, aiming at building one giant model that can handle all languages, notable examples including massively multilingual neural machine translation~\cite{aharoni2019massively}, cross-lingual language model~\cite{lample2019cross}, and multilingual multimodal representation learning~\cite{gella2017image,huang2020m3p}.

\begin{figure}[t!]
\centering
\includegraphics[width=0.9\linewidth]{latex/image/intro_v2.png}
  \caption{ A topology comparison between existing work (M3P) and our proposed \textsc{UC$^2$}.

  M3P combines two types of pre-training tasks and the cross-modal Transformer works only on images and English captions. Our UC$^2$ builds a cross-lingual cross-modal Transformer over images and all the other languages.

  }
\label{fig:comp}
\end{figure}

\begin{figure*}[h!]
\centering
\includegraphics[width=16cm]{latex/image/uc2_model_v4.png}
\caption{An overview of UC$^2$ model. Figure (a) shows the construction of multilingual multimodal pre-training corpus via machine translation.
(b) depicts the overall UC$^2$ framework, which is pre-trained with a massive corpus of multilingual caption-image pairs.
Figure (c) and (d) illustrate details of four pre-training tasks.}

\label{fig:model}
\end{figure*}

Early works on cross-lingual multimodal tasks mainly focus on machine translation~\cite{hewitt2018learning,zhou2018visual,calixto-liu-2017-incorporating,yao-wan-2020-multimodal,DBLP:journals/corr/abs-1807-11605} and image-text retrieval~\cite{gella2017image,MULE,SMALR,Par.EMnb, S-LIWE}. The goal is to construct a common embedding space for vision and cross-lingual inputs, and draw visual concepts from images and similar semantics from languages close together in the feature space.
However, due to the scarcity of large-scale training corpora, these models are validated only on small task-specific datasets, thus scaling and generalizing these models to more languages is non-trivial.

Recent release of large-scale multimodal datasets ~\cite{conceptual-caption} and multilingual corpora (\eg Wikipedia in 100 languages) has served as a key impetus to accelerate fast advances in V+L pre-training ~\cite{UNITER,vilbert,tan2019lxmert,zhou2019vlp} and multilingual language modeling~\cite{conneau2019cross,XLMR,hu2020xtreme}, which makes pre-training large-scale multilingual V+L models possible. A pioneering work is $\text{M}^3\text{P}$~\cite{huang2020m3p}, which formulates the training process as alternating V+L pre-training between cross-modal monolingual corpus and mono-modal cross-lingual corpus.
It relies on English as the focal point to build a bridge between image and different languages, which inevitably introduces linguistic discrepancy into downstream tasks that rely on direct alignment between image and Non-English language (\eg, Image-to-German retrieval), as shown in Figure~\ref{fig:comp} (a).

In this paper, we propose a new pre-training framework, UC$^2$ (\textbf{U}niversal \textbf{C}ross-lingual \textbf{C}ross-modal pre-training), which pivots primarily on images and complementarily on English for multilingual multimodal representation learning (Figure~\ref{fig:comp} (b)). The major challenge is that pivoting on images requires paired image and aligned multilingual data (\eg, image-English, image-German), while existing V+L datasets only contain image-English pairs.
To fill this blank, we propose to augment English-only datasets with other languages via machine translation (MT), and leverage the augmented datasets for pre-training.
To the best of our knowledge, this is the first known effort in creating large-scale training datasets with multilingual image captions.

In addition to extending two widely-adopted pre-training tasks (Masked Language Modeling and Image-Text Matching) to a multilingual setting, we further propose two novel pre-training objectives, namely Masked Region-to-Token Language Modeling (MRTM) and Visual Translation Language  Modeling (VTLM). MRTM encourages fine-grained alignment between words and image regions, by sharing the embedding space of word tokens and region labels (\ie, object class predictions from an object detector). VTLM is designed to jointly learn cross-lingual cross-modal mapping from parallel textual corpora and paired images. Extensive experiments demonstrate that our proposed UC$^2$ framework achieves new state of the art over multiple mainstream benchmarks such as Multi30k~\cite{multi30k, multi30k_extension_1, multi30k_extension_2} and COCO~\cite{MSCOCO, MSCOCO_JA, MSCOCO_ZH} across multilingual image-text retrieval and visual question answering (VQA) tasks.

Our contributions are summarized as follows. $(i)$ We construct a multilingual V+L corpus, and propose the first MT-augmented cross-lingual cross-modal pre-training framework UC$^2$, which pivots on both images and English language for joint representation learning.
$(ii)$ We propose new pre-training tasks, Masked Region-to-Token Language Modeling and Visual Translation Language Modeling, two effective learning objectives for multilingual multimodal tasks.
$(iv)$ We achieve new state of the art on multiple multilingual image-text retrieval and VQA benchmarks, outperforming existing methods.
\section{Related Work}
\paragraph{Vision-Language Pre-training.}
There is a growing interest in building generic pre-trained BERT-like~\cite{BERT} models for V+L tasks. Early work such as VilBERT \cite{vilbert} and LXMERT \cite{tan2019lxmert} propose a two-stream architecture that encodes visual and textual input through two separate Transformers, and then fuse the two modalities by a cross-modal Transformer. Later work such as VL-BERT \cite{su2020vlbert}, Unicoder-VL\cite{unicodervl} and UNITER \cite{UNITER} introduce a single-stream architecture that uses one Transformer to encode concatenated input from both modalities simultaneously. Later, Unified VLP \cite{zhou2019vlp} applies to both understanding and generation tasks. Further improvements are proposed on using different input features \cite{OSCAR, huang2020pixelbert} and multi-task learning \cite{lu202012in1}.

\vspace{-3mm}
\paragraph{Multimodal Multilingual Learning.}
Existing studies arching over multilingual and multimodal aspects mainly focus on two tasks: cross-modal retrieval and multimodal machine translation (MT).
\cite{rajendran-etal-2016-bridge, calixto-liu-2017-sentence} introduces a multimodal multilingual approach by aligning images and captions in  different languages to English captions. Unlike previous work using languages as a pivoting point, \cite{gella2017image} learns a shared embedding space that forces representations of different languages towards the pivot image representation. Later work focuses on scaling to more languages via character-based word-embedding \cite{S-LIWE} or shared language-acoustic embedding \cite{MULE}. SMALR \cite{SMALR} proposes a scalable multilingual model to learn visually aligned word embeddings, for better balance between multilingual capacity and task performance.

Multimodal MT exploits visual information to improve language translations. Earlier work introduces vision to an LSTM-based neural MT model via attention to visual context \cite{calixto-etal-2017-doubly, helcl-libovicky-2017-cuni}, or fusion \cite{calixto-liu-2017-incorporating}, or multi-task learning\ \cite{elliott-kadar-2017-imagination, zhou2018visual}. Lately, Transformer-based~\cite{NIPS2017_3f5ee243} models are proposed~\cite{DBLP:journals/corr/abs-1807-11605, yao-wan-2020-multimodal}. There is also an growing interest in unsupervised multimodal MT \cite{huang-etal-2020-unsupervised,Sigurdsson_2020_CVPR}, where translation between monolingual corpus is augmented via pivoting on image.

While successful in individual tasks, these models are usually trained on small amount of data, which limits its extension to other tasks or languages. To learn task-agnostic universal representations across vision and multilingual text,  $\text{M}^3\text{P}$\cite{hu2020xtreme} introduces the first pre-training framework that alternatively optimizes the model on multimodal monolingual corpus and mono-modal multilingual corpus. While $\text{M}^3\text{P}$ achieves better performance compared to task-specific methods, the alignment between vision and Non-English languages is hard to capture, as the model is learned via using English as the anchor point. To strengthen the alignment between vision and all languages, we propose to pre-train a unified architecture where sentences in different languages are grounded on shared visual context.

\section{Cross-Lingual Cross-Modal Pre-training}

In this section, we start with introducing our machine translation augmented dataset that enables large-scale cross-lingual pre-training. We then go over the proposed UC$^2$ model and our designed pre-training objectives for universal representation learning across vision and languages.

\subsection{Machine Translation Augmented Dataset}
Our multilingual image-text paired data is collected via augmenting the captions
from the Conceptual Captions dataset~\cite{conceptual-caption} with a set of machine translated\footnote{We use Microsoft Azure Translation API Service and will release the translated captions.} captions in other languages $\boldsymbol{L} = \{l_1, l_2, \dots, l_n\}$.
Specifically, we translate the original English captions into five different languages (German, French, Czech, Japanese, and Chinese), which covers languages required for all the downstream tasks studied in this work. Note that with recent advances on machine translation for low-resource languages, we can further expand the dataset to more languages, which we leave for future work. With this data augmentation, we obtained 3.3 million images, each paired with captions in six languages, as the process shown in Figure~\ref{fig:model} (a).
This one-to-many mapping greatly facilitates the learning of alignment between visual content and semantics from each language through image as a shared anchor.
By introducing translated data into model pre-training, our method yields significant improvement over the baseline with MT tools applied only on downstream tasks.
Next, we elaborate how to leverage these data for cross-lingual cross-modal pre-training.

\subsection{Model Overview}
UC$^2$ extends monolingual language encoder of V+L frameworks, such as UNITER~\cite{UNITER}, to cross-lingual encoder~\cite{XLMR}, as shown in Figure~\ref{fig:model} (b).
 The visual feature is extracted from an image encoder and the language feature is obtained from a general cross-lingual language encoder.
The multimodal features are then combined into a sequence and fed to a multi-layer Transformer to produce contextualized cross-modal and cross-lingual representations.

\vspace{5pt}
\head{Image Encoder.}
Given an input image, we first obtain a sequence of image region features $\boldsymbol{v} = \{v_1, v_2, \cdots, v_m\}$ with Faster R-CNN \cite{faster-rcnn}. For each region, we also extract location features via a 7-dimensional vector: $\boldsymbol{p} = [x_1, y_1, x_2, y_2, w, h, w*h ]$, which denotes the normalized top left coordinates, bottom right coordinates, width, height, and the area of the detected region box. The region feature and location feature are fed through separate fully-connected (FC) layers to be projected into the same dimension as the text embedding space, followed by a layer-normalization (LN) layer. The final representation of the region feature is then obtained via summing up the projected region feature and location feature.

\vspace{5pt}
\head{Cross-lingual Language Encoder.}
We follow XLM-R \cite{XLMR} to tokenize an input sentence $T^{l_i}$ in language $l_i$ to BPE tokens $\boldsymbol{t^{l_i}} = \{t_1^{l_i}, t_2^{l_i}, \cdots, t_n^{l_i}\}$ using Sentence Piece model \cite{SentencePiece}. We then project each token to its embedding based on the XLM-R vocabulary and word embeddings.
The final representation of each token is obtained via summing up its word embedding, segment embedding, and position embedding as in XLM-R, followed by another Layer Normalization.

\subsection{Pre-training Tasks}
For model training, we employ four pre-training objectives to train on large multilingual image-text paired data: Masked Language Modeling (MLM), Image-Text Matching (ITM), Masked Region-to-Token Modeling (MRTM), and Visual Translation Language Modeling (VTLM), as shown in Figure~\ref{fig:model} (c) and (d). We continuously optimize our model with the four objectives on multilingual image-text pairs to capture the cross-modal alignment between vision and different languages. As the translated captions are associated to the same image, cross-lingual alignment is also enforced using visual context as the anchor.
\subsubsection{General Tasks}
Following previous V+L pre-training work \cite{UNITER, unicodervl, vilbert, su2020vlbert}, we consider Masked Language Modeling and Image-Text Matching as two of our pre-training tasks.

\vspace{5pt}
\head{Masked Language Modeling (MLM).}
 Given a set of image regions $\boldsymbol{v} = \{v_1, v_2, \cdots, v_m\}$ and its associated caption words $\boldsymbol{w^{l_i}} = \{w^{l_ii}_1, \cdots, w^{l_i}_{T}\}$ in language $l_i \in \boldsymbol{L}$, and mask indices as $m \in \mathbb{N}^M$, we randomly mask a word $w^{l_i}_m$ with the probability of $15\%$ and replace the masked word with a special token $\text{[mask]}$. The objective is to predict the masked word $w^{l_i}_m$ based on the surrounding words $w_{\textbackslash m}$ and all image regions $\boldsymbol{v},$ by minimizing the negative log-likelihood:
\begin{equation*}
    \mathcal{L}_{MLM}(\theta) = - \mathbb{E}_{(w^{l_i}, v)\sim D} \log{P_{\theta}(w^{l_i}_m|w^{l_i}_{\backslash m},v)},
\end{equation*}
where $\theta$ is the learnable parameters. Each pair $(\boldsymbol{w^{l_i}}, \boldsymbol{v})$ is sampled from the whole training set $D$. The caption for each language is sampled with even probability $p = 1/|\boldsymbol{L}|$.

\vspace{5pt}
\head{Image-Text Matching (ITM).}
ITM has been widely used in vision-and-language pre-training~\cite{UNITER, unicodervl,vilbert,su2020vlbert} to learn instance-level alignment between image and sentence.
The output of the special token $\text{[cls]}$ is fed through a FC layer and a sigmoid function to predict a score $s_\theta(w^{l_i}, v)$ between 0 and 1, which predicts whether the input image $\boldsymbol{v}$ and the text input $\boldsymbol{w^{l_i}}$ are semantically matched. During training, we sample positive and negative pairs from the dataset $D$ with equal probability at each step. The negative image-text pair is created by replacing the image or text in a matched pair with a randomly-selected distractor from the same mini-batch. The objective is optimized with binary cross-entropy loss:
\begin{equation*}
    \begin{split}
        \mathcal{L}_{ITM}(\theta) = & - \mathbb{E}_{(w^{l_i}, v)\sim D} [y \log{s_{\theta}(w^{l_i},v)} \\
        & + (1-y)log(1-s_{\theta}(w^{l_i},v))]
    \end{split}
\end{equation*}
where $y \in \{0, 1\}$ indicates whether the input image-text pair is a positive or negative sample. The deployment of MLM and ITM serves as our base model. Next, we introduce two novel objectives to further enhance cross-lingual cross-modal representation learning.

\subsubsection{Masked Region-to-Token Modeling}
Now that we have a learning objective for language (MLM), how about the vision counterpart? In existing VLP models, Masked Region Modeling (MRM) serves this purpose by predicting the top-1 or soft object label associated with the masked image region. The de facto approach to harvesting object labels is using the predictions from an off-the-shelf object detector (\eg, Faster R-CNN~\cite{faster-rcnn}). However, there are two limitations with this approach. First, the association between object labels from image and word tokens from text is not well utilized. While salient objects detected in the image are usually mentioned in the paired description, MRM misses this connection as it directly predicts masked image region to an index between 0 and 1600.
Second, the visual embedding extracted from an object detector can differ significantly from pre-trained word embedding due to different embedding space. Existing methods merely rely on \emph{weak} supervision from pre-training objectives to close the gap between these two disparate embedding spaces.
We argue that a well-aligned embedding space is indispensable for our problem, given its complex multilingual multimodal nature. Therefore, we propose to explicitly learn the correspondence between region and word tokens and tackle the aforementioned issues with two strategies.

\vspace{5pt}

\head{Masked Region-to-Token Modeling (MRTM).} This new objective aims to classify each masked region to its ``pseudo'' object label (e.g., ``dog'', ``cat'', provided by a pre-trained object detector), which is the (sub-word) token\footnote{When multiple tokens are associated to one label word, we randomly select one to decode during pre-training} in our word vocabulary that associates with the original object label.
Compared to the MRM objective from previous work~\cite{vilbert, unicodervl, UNITER}, MRTM leverages additional semantic association between object labels and captions to capture semantic alignment between vision and language. More formally, given an image region $\boldsymbol{v_i} \in \boldsymbol{v}$, we set its probability for being masked out as 15\% (as in ~\cite{BERT}). For each masked region, the region feature vector is either replaced by a zero-initialized vector $v_{m}$ ($90\%$ probability) or remains the same ($10\%$). Then we predict the associated ``pseudo'' object label $c_{v_m}^{l_i}$ on the masked region based on the observation of surrounding image regions $v_{\backslash m}$ and the paired caption $w^{l_i}$ in language $l_i$, by minimizing the negative log-likelihood:
\begin{equation*}
    \mathcal{L}_{MRTM}(\theta) = - \mathbb{E}_{(w^{l_i}, v)\sim D} \log{P_{\theta}(c_{v_m}^{l_i}|w^{l_i},v_{\backslash m})}
\end{equation*}

\head{Early Adaptation (EA).} To address the second limitation and facilitate learning of a joint embedding space between vision and language, we warm up the \textit{image encoder} to make sure the output visual embedding shares the same embedding space as word embeddings.
Specifically, each image region is projected to an image region feature $v_{i} \in \mathbb{R}^{p}$ through the \textit{image encoder}, with the same dimension as the word embedding vector. We then extract the word embedding vectors from XLM-R that correspond to the $k$ object categories $\boldsymbol{c} = \{c_1, c_2, \dots, c_k\}$ defined by the object detector. We compute the cosine similarity between the projected image feature $v_{i}$ with the $k$ word embedding vectors followed by a softmax function, resulting in a normalized distribution $h_{\theta_{I}}(v_i) \in \mathbb{R}^{k}$ that indicates the prediction on what semantics are mapped in the region. We then maximize the similarity between this predicted distribution and the ``GT'' object probability distribution from the object detector output $g(v_i) \in \mathbb{R}^{K}$, by minimizing their  KL divergence:
\begin{equation*}
    \mathcal{L}_{EA}(\theta_{I}) = D_{KL}(g(v_i)||h_{\theta_{I}}(v_i)),
\end{equation*}
where $\theta_I$ is the learnable parameters of the \textit{image Encoder}.

Note that a recent work named OSCAR~\cite{OSCAR} has made a similar effort to close the visual-textual embedding gap by inserting object tags into the input sequence. Compared to OSCAR \cite{OSCAR}, our method has two advantages. First, it does not rely on object tags for downstream tasks, which might not be applicable for image domains that cannot be well covered by the object categories from the pre-trained detector. Second, by forcing image representation to be similar to language representation with EA, our pre-training model can better leverage the initialized weights from language-only pre-trained model to adapt to image modality.

\subsubsection{Visual Translation Language Modeling}
All the objectives mentioned so far operate on image and \emph{monolingual} input, without considering cross-lingual objectives. The correspondence between languages is vital for cross-lingual generalization, clearly observed from existing work on language understanding~\cite{XLMR}. Our proposed methods so far unexceptionally learn cross-lingual correspondence \emph{indirectly} through the image focal point, which might not be sufficient.  We hence propose visual translation language modeling (VTLM), which directly and jointly learns the alignment between visual context and text in different languages.

In VTLM, given an image $\boldsymbol{v}$ and a pair of captions $(w^{l_i}, w^{l_j})$ in two different languages, the goal is to predict masked caption tokens from both languages. One of the two languages is always English, as English captions in our pre-training data are directly from~\cite{conceptual-caption}, while captions in other languages are translated by MT, therefore less reliable. Under this bilingual framework, model input size only grows linearly with more languages.

Besides, as our model is initialized with the weights of a powerful pre-trained multilingual model, it has already learned a good alignment between different linguistic words to some extent. Applying random masking strategy in VTLM is sub-optimal, as
the model can make a correct prediction by simply translating words from one language to another, without taking into account the visual information from image.
To encourage the model to fully consider visual context, we introduce a strategy called \emph{co-masking}, where we simultaneously mask out tokens with similar semantic meanings from paired captions to prevent easy translations.

There are a few steps in \emph{co-masking}. First, we apply Fast Align \cite{fast_align} to learn the word alignment between two different languages $(l_i, l_j)$ from the noisy parallel corpus that was created using machine translation. Then, during the pre-training stage, we follow the same strategy as in MLM to randomly mask a token $w^{l_i}_{m}$ from the caption of one language. For the paired caption in the other language $l_j$, we mask the aligned word tokens $w^{l_j}_{k}$ that are predicted from Fast Align. \cite{fast_align}
The final objective is again to predict masked tokens from both languages by minimizing the negative log-likelihood:
\begin{equation*}
    \mathcal{L}_{VTLM}(\theta) = - \mathbb{E}_{(w^{l_i}, w^{l_j}, v)\sim D} \log{P_{\theta}(w^{l_i}_m, w^{l_j}_k|w^{l_i}_{\backslash m}, w^{l_j}_{\backslash k}, v)}
\end{equation*}

\begin{table*}[!]
\centering
\small
\begin{tabular}{lcccccccc}
\hline
& \multicolumn{4}{c}{Flickr30K} & \multicolumn{3}{c}{MSCOCO} & \\
Method  & EN  & DE   & FR                       & CS                         & EN                            & ZH                          & JA                            & Meta-Ave                          \\ \hline
\multicolumn{9}{l}{\textit{SOTA without pre-training}}\\
\hline
EmbN\cite{EMnb} & 72.0 & 60.3 & 54.8 & 46.3 & 76.8 & 73.2 & 73.5 & 65.3\\
PAR.EmbN \cite{Par.EMnb} & 69.0 & 62.6 & 60.6 & 54.1 & 78.3 & 76.0 & 74.8 & 67.9\\
S-LIWE \cite{S-LIWE} & 76.3 & 72.1 & 63.4 & 59.4 & 80.9 & 73.6 & 70.0 & 70.8\\
MULE \cite{MULE} & 70.3 & 64.1 & 62.3 & 57.7 & {\color[HTML]{0000FF}79.0} & {\color[HTML]{0000FF}75.9} & {\color[HTML]{0000FF}75.6} & 69.3 \\
SMALR \cite{SMALR} & 74.5 & 69.8 & 65.9 & 64.8 & {\color[HTML]{0000FF}81.5} & {\color[HTML]{0000FF}77.5} & {\color[HTML]{0000FF}76.7} & 73.0 \\
\hline
\multicolumn{9}{l}{\textit{English-only Fine-tune}}\\
\hline
$\text{M}^3\text{P}$\cite{huang2020m3p}                & 87.4                                   & 58.5                                 & 46.0                               & 36.8                                 & 88.6                                 & 53.8                               & 56.0                                 & 60.7                                \\ \hline
UC$^2$   & 87.2 & 74.9 & 74 & 67.9                                 & 88.1 & 82 & 71.7 & 78.0 \\ \hline
\multicolumn{9}{l}{\textit{Translate-Test}} \\
\hline
$\text{UNITER}_{\text{CC}}$\cite{UNITER} & 87.7  & 81.2  & 81.9  & 80.2 & 88.4 & 87.3 & 82.2 & 84.1  \\ \hline
\multicolumn{9}{l}{\textit{Single-Language Fine-tune}}\\
\hline
$\text{M}^3\text{P}$\cite{huang2020m3p}               & 87.4                                  & 82.1                                 & 67.3                               & 65.0                                 & 88.6                                 &75.8                              &  80.1                              & 78.0                                 \\ \hline
UC$^2$   &  87.2 &  83.8 &  77.6 &  74.2                                 & 88.1 &  84.9 & 87.3 & 83.3 \\
\hline
\multicolumn{9}{l}{\textit{All-Language Fine-tune}} \\
\hline
$\text{M}^3\text{P}$\cite{huang2020m3p}               & 87.7                                  & 82.7                                 & 73.9                               & 72.2                                 & \textbf{88.7}                                 & 86.2                              &  \textbf{87.9}                              & 82.8                                 \\ \hline
UC$^2$   &  \textbf{88.2} &  \textbf{84.5} &  \textbf{83.9} &  \textbf{81.2}                                 & 88.1 &  \textbf{89.8} & 87.5 & \textbf{86.2} \\
\hline
\end{tabular}
\caption{Evaluation results on image-text retrieval over Flickr30K and MSCOCO datasets across different languages. We highlight the MSCOCO results for MULE and SMALR in {\color[HTML]{0000FF}blue} as they are using different dev/test splits of MSCOCO compared to other models.}
\label{tab:itm}
\vspace{-3mm}
\end{table*}

\section{Experiments}
In this section,
we provide detailed experiments to evaluate our proposed UC$^2$ model over multilingual image-text retrieval and multilingual VQA tasks.

\paragraph{Multilingual Image-Text Retrieval}
In the retrieval task, the model retrieves an image from a set of candidates given a caption in a certain language, or vice versa. We consider two datasets: Multi30K \cite{multi30k, multi30k_extension_1, multi30k_extension_2} and MSCOCO \cite{MSCOCO, MSCOCO_JA, MSCOCO_ZH}. Multi30K is built upon Flickr30K \cite{Flickr30K}, where English captions are manually translated to German, French, and Czech. It contains 31K images (each paired with 5 captions in English and German, 1 caption in French and Czech). Following Flickr30K\cite{Flickr30K}, we split the data into 29K/1K/1K images for train/val/test.

MSCOCO\cite{MSCOCO} consists of 123K images, with 5 English captions per image. STAIR \cite{MSCOCO_JA} extends MSCOCO dataset by collecting 820K Japanese captions for 165K COCO images. Similarly, Li et al. \cite{MSCOCO_ZH} collect Chinese captions for 20K COCO images with roughly 1 caption per image. We use the train/dev/test splits for English and Japanese defined in \cite{KarpathyF14}, and present results on the 1K test set. For MSCOCO Chinese, we follow the original split as in \cite{MSCOCO_ZH}. We compute Recall@K (recall of top K candidates)  for both image-to-text retrieval and text-to-image retrieval with $K=1, 5, 10$. The average of all these 6 evaluation scores, Average Recall (AR)\cite{huang2020m3p}, is used as the final evaluation metric.

\paragraph{Multilingual Visual Question Answering (VQA)}
In Multilingual VQA, given an image and a question in a certain language, the model predicts an answer based on the visual context in the image. We evaluate our model on two datasets: VQA v2.0~\cite{vqa_v2} and Japanese Visual Genome (VG) VQA~\cite{vqa_ja}. VQA v2.0 is a widely used benchmark for English VQA task. We follow the official partition to split the dataset and report results on Test-Dev set through the official evaluation server. Following \cite{UNITER}, our training is augmented by running on both the training and validation split of VQA v2.0 as well as the VQA from Visual Genome\cite{vg}.
Visual Genome VQA Japanese \cite{vqa_ja} expands the VG English VQA dataset \cite{vg} by collecting 793K Japanese question answering pairs on 99K images from VG. We use the train/test split in the original VG VQA to split the data into 61K/30K training/test images.

We formulate VQA as a multi-label classification problem, where the model predicts answer from the candidate pool.\footnote{We only consider top-3129 frequent answers for VQA v2.0 and top-3000 frequent answers for VQA VG Japanese.}
VQA score~\cite{vqa_v2} is used to compare model predictions against 10 human-annotated answers in VQA v2.0.
On Visual Genome VQA Japanese, which only has one ground-truth answer to each question, we use accuracy and BLEU score as the evaluation metrics.\footnote{BLEU score is used to compute a soft mapping score between the predicted answer and the ground-truth answer, assuming answers with many overlapping words should share similar semantic meaning.}

\paragraph{Implementation Details}
UC$^2$ consists of 12 layers of transformer blocks, where each block has 768 hidden units and 12 self-attention heads. Except for the image encoder, the model is initialized with XLM-R \cite{XLMR}. We run continuous pre-training with MLM, ITM, MRTM and VTLM objectives. We use Adam optimizer \cite{ADAM} with a linear warm-up for the first $5\%$ of training, and set the learning rate to $4e-4$. We use Horovod and NCCL for multi-node communications and apply gradient accumulation (every 3 steps) to reduce multi-GPU communication overheads. The batch-size for pre-training is set as 1024 and the dropout rate is 0.1. Pre-training experiments are conducted on 8 Nvidia V100 GPUs for 30 epochs, which takes 4 days to converge.

\subsection{Experimental Results}
We first compare UC$^2$ to various SOTA with or without pre-training on the two downstream tasks. Then, We conduct ablation experiments to study the effectiveness of MRTM and VTLM, as well as the impact of image pivoting. Finally, we visualize the alignments between visual context and cross-lingual text context learned by our pre-trained UC$^2$ model.

\subsubsection{Evaluation on Multilingual Retrieval}
We compare UC$^2$ with state-of-the-art methods on image retrieval and text retrieval in two different settings:
\begin{itemize}[noitemsep]
    \item \textbf{English-only Fine-tune:} We finetune the pre-trained model on just the English training data.
    \item \textbf{Single-Language Fine-tune:} We finetune the pre-trained model on training data for each target language.
    \item \textbf{All-Language Fine-tune:} We finetune the pre-trained model on merged training data of all languages.
\end{itemize}
Besides reporting AR on each language, we also compute the Meta-Ave (average of AR across all languages over two datasets) to reflect the overall performance in this task. Given that we have access to pre-trained machine translation models, we also introduce a strong translate-test baseline $\text{UNITER}_{\text{CC}}$ based on~\cite{UNITER}, which is pre-trained on Conceptual Conception English data and finetuned on English training data in the downstream tasks. By translating the test data from other languages to English, $\text{UNITER}_{\text{CC}}$ can be directly applied for text/image retrieval. Results are summarized in Table~\ref{tab:itm}.

Our model on the all-language setting achieves a significant improvement over all task-specific methods without pre-training, showing the effectiveness of cross-lingual cross-modal  pre-training in learning universal representation across vision and different languages. Our model also demonstrates a superior transferability. When finetuned on English dataset only, we observe an absolute gain of $17.3\%$ on  Meta-Ave
across different languages over M3P via better transition of the learned knowledge from English to other languages.
Compared to the best non-pretrained models trained on data in each language, our cross-lingual model under the English-only fine-tune setting is still $5\%$ better. We suspect the improvement comes from the in-domain pre-training objective: we use image as the grounding media in ITM to learn cross-modal mapping from one language to another. With strong transfer capability, our model could potentially generalize the learned knowledge from a high-resource language to downstream tasks in low-resource languages.

When we finetune UC${^2}$ model on all-language data, our model still demonstrates a consistent advantage over $\text{M}^3\text{P}$ on the majority of languages, with $3.4\%$ improvement on Meta-Ave. Our best model is also better than the strong translate-test baseline $\text{UNITER}_{\text{CC}}$ on all languages except English in MSCOCO. The slightly worse performance on COCO English is potentially due to lack of pre-training in English data, given that our pre-training time is evenly splitted to multiple languages. However, this does not overshadow the fact that we achieve overall better performance across all languages.  Thanks to the cross-lingual pre-training and finetuning, our model can leverage the complementary information captured in different languages to improve the performance on each language.

\begin{table}[!]
\small
\centering
\begin{tabular}{l|c|cc}
\hline
& VQA v2.0                              & \multicolumn{2}{c}{VG VQA JA} \\
method & Test-Dev Acc & Acc & BLEU \\
\hline
MCAN \cite{MCAN}                        & 70.63                                & -                                    & -                                    \\
PCATT  \cite{vqa_ja}                      & -                                     & 19.2                                 & -                                    \\ \hline
Vil-BERT \cite{vilbert}                    & 70.55                                 & -                                    & -                                    \\
VL-BERT \cite{su2020vlbert}                      & 71.16                                 & -                                    & -                                    \\ \hline
UNITER$_{\text{CC}}$ \cite{UNITER}                       &  71.22 & 22.7                             & 11.8                             \\ \hline
UC$^2$                       &  \textbf{71.48}          & \textbf{34.2} & \textbf{26.8} \\ \hline
\end{tabular}
\caption{\label{tab:vqa} Evaluation results on multilingual VQA task over VQA v2.0 and VG VQA Japanese datasets. We highlight the results for PCATT in {\color[HTML]{0000FF}blue} as they are using different dev/test splits.}

\end{table}
\subsubsection{Evaluation on Multilingual VQA}
For multilingual VQA, our pre-trained model is finetuned and evaluated on the target language for each dataset. Unlike image-text retrieval where the same output layer is shared across different languages, multilingual VQA has different classes of answers for each language, which makes joint training across different languages impossible. We compare our model with state-of-the-art methods without pre-training as well as V+L pre-training methods that use the same pre-training corpus. When evaluating the translate-test baseline $\text{UNITER}_{\text{CC}}$ on VG VQA Japanese dataset, we first finetune it on VQA v2.0~\cite{vqa_v2} with english answer candidates translated from VQA VG Japanese to ensure the same reference is used during evaluation as in UC$^2$. We then use machine translation model to translate the test dataset of VG VQA Japanese to English, and evaluate the finetuned translate-test model using classification accuracy and BLEU. Results are summarized in Table~\ref{tab:vqa}.

On VQA v2.0, our model achieves significant improvement over SOTA task-specific method,
and also outperforms existing monolingual models pre-trained on Conceptual Conception \cite{vilbert, su2020vlbert, UNITER} by an obvious margin.
On VG VQA Japanese, we finetune our model with a different data split from the original baseline method PCATT proposed in VG VQA Japanese, where we have much less training data than their split (ours: 61K images vs. PCATT: 91K images). Even under this disadvantage in an unfair comparison, our pre-trained model still achieves more than $10\%$ improvement on both accuracy and BLEU over baselines.
Although achieving better performance compared to the task-specific method, the translate test baseline ($\text{UNITER}_{\text{CC}}$) performs much worse than UC$^2$ on the translated VQA VG Japanese dataset. Despite strong performance on the VQA English dataset, the noisiness from the machine translated language would lead to unavoidable degradation especially for tasks like VQA that requires fine-grained level understanding and interpretation on multi-modal context. Hence, building unified cross-linugual cross-modal pre-training model like UC$^2$ is a better solution to directly work on tasks in target languages than a translate-test method.

\begin{table*}[!]
\centering
\small
\begin{tabular}{l|rrrrrrrr|rrr}
\hline
& \multicolumn{4}{c}{Flickr30K} & \multicolumn{3}{c}{MSCOCO} &  & VQA v2.0 & \multicolumn{2}{c}{VG VQA JA}\\
Pretraiing Objective  & EN  & DE   & FR   & CS    & EN & ZH   & JA    & Meta-Ave  & Test-Dev Acc & Acc & BLEU\\
\hline
UC${^2}$ (full model)   & \textbf{88.2} & \textbf{84.5} &  \textbf{83.9} & \textbf{81.2}    &  \textbf{88.1} & \textbf{89.8} & \textbf{87.5} & \textbf{86.2}  & \textbf{71.48} & \textbf{34.2} & \textbf{26.8}\\
\hline
\quad\quad -VTLM & 87.5  & 83.6  & 82.4 & 79.6  & 87.7   & 89.2  & 87.2 & 85.3  & 71.45 & 34.1 & 26.7
\\
\quad\quad -MRTM & 87.6  & 83.7  & 82.0 & 80.0  & 87.9   & 89.4  & 87.4 & 85.4  & 70.93 & 33.5 & 26.4
\\
\quad\quad-VTLM-MRTM    & 86.8    & 82.9   & 81.3   & 79.3   & 87.5   & 88.9  & 86.7 & 84.8 & 69.94 & 33.4 & 26.4 \\ \hline
\end{tabular}
\caption{\label{tab:itm_abs} Ablation study on pre-training objectives.}

\end{table*}
\begin{table*}[!]
\centering
\small
\begin{tabular}{l|rrrrrrrc}
\hline
& \multicolumn{4}{c}{Flickr30K} & \multicolumn{3}{c}{MSCOCO} & \\
Topology  & EN  & DE   & FR   & CS    & EN & ZH   & JA    & Meta-Ave  \\
\hline
UC${^2}$ (Image pivoting)   & \textbf{87.5} & \textbf{83.6} &  \textbf{82.4} & \textbf{79.6}    &  87.7 & \textbf{89.2} & 87.2 & \textbf{85.3}  \\
\hline
UC${^2}$ (English pivoting)    & 86.2    & 81.9   & 80.7   & 77.4   & \textbf{88.1}   & 88.5  & \textbf{87.3} & 84.2 \\ \hline
\end{tabular}
\caption{\label{tab:img_piv_abs} Comparison between the pre-training topology of pivoting on image against pivoting on English.}
\end{table*}

\subsubsection{Ablation Study}~\label{sec:ablate}
\vspace{-3mm}
\paragraph{Effect of Training Objectives} To validate the effectiveness of the proposed pre-training objective MRTM and VTLM, we conduct ablation study to verify their contributions to the model performance. We gradually remove the two proposed training objectives and evaluate these ablated models on our two downstream tasks.
When finetuning the pre-trained model on the image-text retrieval task, we follow the best experimental setting to train the model on all language data. On VQA task, the model is directly finetuned on the target language data.

From Table~\ref{tab:itm_abs}, we observe that MRTM has led to significant performance boost on multilingual VQA tasks over the two languages while gaining some incremental improvement on image-text retrieval tasks. VQA requires more fine-grained understanding about connections between language and visual context, therefore benefits more from the cross-modal local alignment captured by MRTM. When introduce VTLM to the pre-training of UC2, we observe similar improvement on image-text retrieval task, but the improvement on VQA VG Japanese is relatively incremental. We suspect the limited help is mainly due to the language gap between English and Japanese captions. Hence it is hard to capture the good alignment between English and Japanese via VTLM.
\paragraph{Effect of Pivoting on Image}  To validate the effectiveness of image pivoting, we conduct a controlled experiment where the model variant only pivots on English.
In this setting, we train UC2 with all the pre-training objectives on English Conceptual Caption data, except for VTLM which involves image as one of the pivoting points. To capture the alignment between English and other languages, we train \textsc{UC$^2$}  on pairs of captions in two different languages with one language fixed as English. The training objective is translated language modeling adopted from XLM~\cite{lample2019cross}.
From Table~\ref{tab:img_piv_abs}, we can see that \textsc{UC$^2$}  pre-trained by pivoting on image achieves overall better performance in multilingual image-text retrieval task. The advantage is particularly sound when the target language has limited training data. This indicates that the cross-lingual cross-modal representation learned by pivoting on images imbues stronger cross-modal mapping transfer across different languages.

\begin{figure}[h!]
\centering
\includegraphics[width=6.5cm]{latex/image/Attention_Visualization.png}
\vspace{-3mm}
\caption{Visualization of Text-to-Image Attention on aligned words across English, German and Czech (Flickr30K).}
\label{fig:fig2}
\end{figure}

\paragraph{Visualization} To visualize the  cross-lingual cross-modal alignment learned by UC$^2$, we provide examples of text-to-image attention from salient words in multilingual captions to the shared image context. As shown in figure \ref{fig:fig2}, words from different languages that share the same semantic meaning can attend to similar corresponding regions in the image. This shows that while our model can effectively capture cross-modal alignments between regions and words, it also connects different languages by grounding them to similar image regions.

\section{Conclusion}
We present the first MT-augmented pre-training model UC$^2$ that pivots primarily on images and complementary on English to learn cross-lingual cross-modal representation from large scale of multilingual image-to-text pairs. We propose two new pre-training tasks that facilitate our model to capture better alignment between vision and different languages. Our model achieves the new state-of-art performance on two  mainstream multilingual V+L tasks and demonstrate strong cross-lingual transfer capability.
For future work, we will continue exploring this topic and expanding the framework to include more families of languages. As more benchmarks \cite{Wang_2019_ICCV, HOW2,bagher-zadeh-etal-2020-cmu} on multilingual video-text pairs become available, we are interested in enhancing the grounding between vision and language by leveraging the temporal information from videos.

{\small
\bibliographystyle{ieee_fullname}

redpajama_set_name: RedPajamaArXiv


I don't tend to do reviews, but several people have asked me about this dishwasher I bought a few months ago, so I thought I'd do a bit about it here.
I've never been quite sure if I like dishwashers or not. When I was a teenager, I seem to recall my parents making me load ours, as a household chore. The biggest complaint I have with this, is that it's completely unnecessary. Instead of stacking your plates on the sideboard, why not just put them in the dishwasher yourself? Then nobody has to load the bloody thing. As I've never actually owned one as an adult though, maybe there was something magical about them I wasn't seeing through my grumpy-teenager-tinted spectacles?
While looking in Argos, Comet, Currys, etc. we noticed there was a cheap model in every store, always priced at £129.99, and practically identical inside and out. The main difference is that the "Bush" one that Argos/Homebase sell has a viewing window at the front. That's mediocre-ly cool, right? I mean it's the most exciting thing in dishwasher-world, right? Viewing-window-one, it was.
Me and girlfriend plumbed it in. It's a piece of piss to plumb in a dishwasher. Especially if there's already a hole through the cupboard for the washing machine, and you don't mind being able to see a bright blue water-in pipe forever. And I don't.
So first up, what of that viewing window? It's a waste of time really. There's no light in there, and you can't see much because it's full of stuff. The only time it came in useful was a couple of days ago when it made a funny jarring noise, and I could see that something was stopping the arm at the bottom from spinning (from the way the water splatters against the inside of the glass).
1. The number of times it beeps.
When your microwave finishes, how many times does it beep? 3? 4? The dishwasher beeps 8 times. And they are both an irritating pitch, and incredibly loud. So loud, you can hear it several rooms away on the other side of the flat. Why is it so important to be notified the second the dishwasher finishes, anyway? I'm not desperately sat here longing for the moment when my favourite mug will be clean again. And if it knows it has finished, they should train it to open the door and turn itself off.
The saving grace is if you use the delay timer when you start it (handily in 1 hour increments from 1-24), then it doesn't beep when it finishes.
2. The manual is a bit misleading.
That rack on the top left of my picture (or the far right of the manual picture) is not big enough for cups. It just isn't. I think the ones in their picture must be espresso mugs. And if that's the case, then how small are those 4 example glasses on the left. Are they shot glasses?
And that's the shortest cup I've got.
It won't hold half as much cutlery as you might think it would. Pack it in like the manual shows, and it'll all come out half dirty.
If you don't think that's so bad now, you should try it when you're trying to squeeze in one more plate, around loads of other things.
You will need to accept that oven trays don't fit. I end up washing all the oven trays by hand. And all the saucepans. And frying pans. And the wok. And some of the larger plates because they don't fit in there. FFS.
3. It needs a lot of help.
Despite the fact that this dishwasher was sold as using a reasonably small amount of water, it says in the manual you should "prepare your plates" before putting them in, by rinsing them all under water. Hmnph.
It needs dishwasher liquid (or powder if you prefer), rinse aid, and salt. The salt is in one of the most awkward locations possible. It's the big round white circle thing in the top right of this picture.
Really you can't put any salt in, without taking all the dishes and the rack out. Then they give you two scoops – one that measures 20g, and one that measures 100ml. The manual says you can add "up to 500g of salt". In that case, couldn't they have provided a bigger measuring device?
I've taken to measuring the 500g into the 100ml thing, and pouring it in.
Assume there will be spillages.
Some ends up going everywhere, but it's quick at least. What is more confusing is when it asks for more salt. Given there is some in there already, do you add the full 500g again, or just throw in a random amount.
You can buy 3-in-1 tablets with salt (and rinse aid) in them, but the "rapid" wash takes 70 minutes, and the 3-in-1 cycle takes 2 hours. I can't be doing with that much noise for that long. So your choice is either twice as long, every time, OR once a month you have to piss about pouring salt all over the place. Couldn't they have made that hole accessible through the top, or in the bottom. Just any way round where you pour it down, rather than sideways?
Oh and anything with a slight indentation (cups, etc.) holds a little pool of water, so needs to be dried manually.
There's a rotating arm at the bottom that spins round and throws water at your plates.
Despite having a built-in softener on our water supply, this clogs up with limescale regularly.
Bottom hole blocked, next up half-blocked.
5. It leaks, but not enough to bother fixing.
It's been doing this for a while now. I think it's some of the rinse aid leaking back out, but it's such a small amount, that I can't be bothered to look any further into what is causing it.
None of the water comes out, and I can't trace where it comes from. It doesn't leak fast enough that I pour it in, and it comes straight back out. Just a minor annoyance really.
6. The instructions are rubbish.
If you've been reading this far, you might think this is a bit like #2. But unlike the misleading nature of #2, this is actually quite amusing.
I'd be amazed if that could accurately display "high temperature".
No, probably not, but in a kitchen where you definitely can't have a full-size (or slimline) dishwasher, if it is your ONLY option, and you are a couple who don't eat the same food, so would have lots more to wash up. Maybe. Just.
Several people have asked me now about where to download the manual for this. It's rubbish, but as I can't find it online, I thought I'd photograph mine, and upload it (sorry I don't have a scanner, so apologies if it's a tiny bit blurry).
thanks for the tip on it not beeping at the end when you do the Delay. Really loud those beeps and so many lol. Trying to find someone that does spares as my door white fascia is cracked and bit came off, its in a cupboard fits well and still works so rather try and get a new plastic for door first.
No problem on the beeps. There might be a way to disable them completely by holding combinations of buttons or something, but as you'll know – the manual is rubbish. Even when choosing program/pressing start, the beeps are insanely loud, considering they know I'm stood right in front of it.
When you say fascia – which bit do you mean? Around the door, or underneath it? Because that's where I think mine is leaking rinse aid from.
I also noticed the other day that some of the plastic has started to come off the rack, and the underneath has started to rust. I'm umming and arring about calling the warranty people.
For something which is only about 9 months old, it's not exactly robust.
There is a number printed inside the manual of 0845 600 4632 for in-warranty repairs (or according to saynoto0870.com you can call 0800 0929129, press 5 and ask for repairs, and then the call won't cost you anything). If it's out of warranty, they'll presumably want a lot to repair it, but they might be able to sell you the part. Don't know how much it would cost to pay someone else to do it, or whether you might be able to do it yourself. Good luck.
Its the white plastic over the front door that goes around the window. Looks like from the screws you can remove it easy.
Argos gave me the their bush repairs number but they say they only have spares enough for warranty items, mines over 12 months old. So probably have to put up with it til it breaks down.
I don't know where it might be online. A quick google search just finds me other people asking for it.
To be honest, the manual is rubbish. There's very little of use in it anyway, but I'll photograph mine and send it to you.
hi read all comments and found them more better than them online thanx..
There is a link to download a .pdf of the full manual, at the end of the post. Just underneath "update 30/09/12". The liquid/powder/rinse-aid goes in the door compartments. The enormous screw on the right hand side when you open the door, is for topping up the salt.
If using powder, you pour it in the slight incline in the bottom right of the door (where the liquid goes, too – if you were using that). It says "detergent >20g" on it.
If using a tablet, it goes in the little tiny basket you get with the dishwasher.
I personally use liquid and separate rinse aid, topping up with salt every now and then, because although the 3-in-1 tablets are easier, the 3-in-1 wash cycle takes 2 hours. The rapid wash is only 70 minutes, and the machine is quite noisy.
I'll send you an email with a link to download the manual. It's a rubbish manual though.
Hi everyone, I just bought it second hand, and it's worked properly until I put salt in it. Now not cleaning the dishes but make more dirty. Anyone any ide how can I fix it? Many thanks.
I'm not sure. The salt area in the version I had is a completely contained area – just one opening to put salt in, but no way to empty it. However, if you fill it with water, it does overflow back into the dishwasher, so that might be a way to empty it, perhaps?
I'm assuming you've tried running it empty to see if that clears it?


text:
Sexual Assault Center Ransacked — Then Mysteriously Repaid
Published August 9, 2013 at 10:24 AM HST
MELISSA BLOCK, HOST:
Last week under cover of night, burglars broke into an office building in San Bernardino, California.
CANDY STALLINGS: It was about 10:30 on July 31st, and I received a call from my alarm company.
BLOCK: That's Candy Stallings. She's executive director of San Bernardino County Sexual Assault Services. The office of her nonprofit organization was a wreck - desks smashed; papers and files all over the floor; and six desktop computers and one laptop were gone. The very next morning, Stallings received another call, this time from the police.
STALLINGS: Candy, can you come out? We've had some suspicious activity happening. And I want you to see this.
BLOCK: Stallings drove back to the office, preparing for the worst, thinking...
STALLINGS: This is crazy, we're never going to be able - you know, they came back in, took the rest of everything else we might have had. But I just felt like so down. I was just like I'm done, this is just too much.
BLOCK: But when she got there, she learned that all seven computers had been returned in a shopping cart by the door.
STALLINGS: And then one of the officers opened up the computer laptop, and inside was a handwritten note of apology.
BLOCK: And here's what the letter said:
STALLINGS: We had no idea what we were taking. Here, your stuff back. We hope that you guys can continue to make a difference in people's lives. God bless.
BLOCK: Candy Stallings was shocked by the gesture. She says it was an emotional roller coaster. Just a few hours earlier, she thought the nonprofit had lost everything.
STALLINGS: But somebody that really needed that stuff and robbed our building brought it back.
BLOCK: Stallings was unable to keep the original note of apology, law enforcement collected it as evidence. But she says she plans to frame a copy of the letter and hang it in her office.
This is NPR News. Transcript provided by NPR, Copyright NPR.
NPR National News
Linah Mohammad
The horrific video of Memphis police fatally beating Tyre Nichols has dominated the media this weekend. But another video is circulating, showing Nichols happily skateboarding in his teen years.
Alex Hager
The seven states that share the Colorado River have until Tuesday to agree to voluntary water cutbacks, or have federal cuts imposed on them.

redpajama_set_name: RedPajamaCommonCrawl


The Valuation Division is headed by the Valuer General and is generally responsible for the valuation of properties and the revision of Land Rental and Property rates to reflect current market values. The Valuation Division also monitors outstanding land rental forfeiture, updating of provincial land rent for property rates, updating of provincial and Honiara valuation map. It plays a vital role in supporting government national projects especially in terms of valuation on properties. The objectives of the Valuation Division is to strengthen the capacity of the Valuation Unit with the view to effectively carry out its roles and responsibilities; revision and improvement in rental and property rates reflect current market values; and to support SIG national projects in the area of land/property valuation.
Contact us if you would like to get a copy of the above persons registered phone numbers.


Eric Decker is currently a Professor and Head of the Department of Food Science at the University of Massachusetts, Amherst. Dr. Decker is actively conducting research to characterize mechanisms of lipid oxidation, antioxidant protection of foods and the health implications of bioactive lipids. Dr. Decker has over 400 publications and he is listed as one of the Most Highly Cited Scientists in Agriculture. Dr. Decker has served on numerous committees for institutions such as FDA, Institute of Medicine, Institute of Food Technologist, USDA and the American Heart Association. He has received numerous recognitions for his research including awards from the American Oil Chemist Society, the Agriculture and Food Chemistry Division of ACS, the International Life Science Institute, and the Institute of Food Technologist.


We really enjoyed to stay there. Hotel is in great location. The staff in front desk tried to help us to check in early which was much appreciated. He was very kind to do that.
Booked this room for my sister and husband as an anniversary gift. They had a good time and completely enjoyed the view of the NEW YEAR fireworks. The room service food was above average. Think overall, it was a good choice!
The staff were extremely helpful and friendly. It takes 15 minutes to walk to the Australian Open venue. The King room with city view that we had was compact and the toilet even more compact. There is one restaurant in the hotel called Hop Sing. One can use the gym at Pullman on the Park free of charge. We booked the room in October 2018 for a three night stay and paid over AUD 200 more than if we had booked ithe room two days before going to attend the first two days of the Australian Open.
We've stayed here several times. Its location is second to none - quiet as a city place can be, and overlooking parklands, yet within walking distance of everywhere including MCG, tennis, trams and trains. The rooms are comfortable, clean and beds are great. The bathrooms are almost like a game of Tetris but they have everything you would need. We're more than likely to stay here again, the next time we visit Melbourne.
I had a really great experience at the hotel. The staff were really friendly and helpful, giving us instructions and recommendations on where to go. They were also able to help us book our airport transfer with the starbus. We were able to check in early even without requesting for it and that was great. I checked in on my birthday and they prepared a plate of chocolates and strawberries and brought it up to my room. That was really sweet and thoughtful. Thank you! The hotel room was quite small but not uncomfortably so, it was very cosy and had everything. We asked for bedroom slippers and they also brought it up to our room. Breakfast was great. We loved that there was a small Asian option everyday with the congee. My mom really appreciated that. The location of the hotel was great. It was in the CBD, there is a tram station just round the corner of the hotel and the metro within walking distance. We were able to walk to Chinatown, and also some of the shopping areas on Bourke St. and Little Bourke St. Overall, we had a very good stay!
Stayed during Australian Open. Perfect location as you can walk to the Open from the hotel. Very convenient transport to all areas of Melbourne. Clean rooms, comfy bed. Lemon water near entry/exit welcome service and was always topped up with glasses available. Nice touch and appreciated.
The room design is modern. I like the wall picture above the bed. Clean and tidy. Friendly receptionist. Only thing that the gap between the table and bed is too narrow. Cannot walk through if pushing out the chair. Always being hook by the chair leg.
Excellent stay. The staff were very friendly and professional. I had a room at the rear of the hotel on a high floor. It was very quiet. The room was very clean and just what I need on a business trip. Small complaint is that the remote control for the TV didn't work.
My mum and I stayed there for a night while we were shopping for a wedding dress and we loved it. The staff were very friendly and so helpful. The food at breakfast was one of the best buffets I have ever had.
The Mercure Gardens is a quite good hotel, centrally located. The rooms are nice, except for the bathrooms, which are quite small. Internet is not included in the room rate, and the extra price is extremely high !
Pillow selection is great. Decor in room was very nice. Loved the flinders st bedhead. Great location.
We only needed an overnight stay, and the Mercure was reasonably priced. Disappointed that our king bed was two separate singles so we had a very hard line up the centre of the bed. Did seem like the slightest noise outside the room was amplified and very loud. No view to speak of in our room on the 8th floor. But, we only needed somewhere nice to sleep after a night out in the city and the surprise free 2pm checkout was very appreciated.
My stay at the hotel was good. Only problem was the gym was located in another hotel, and those gym facilities were very poor. It shouldn't advertise a gym if it's not present. The photo of the building is deceiving, there's a view of another building giving the impression that it's the Mercure. It's not, you can barely see the hotel.
The carpark (directly) under the hotel is terribly small. Fortunately, there's another nearby carpark, which the hotel has association / link up with. The staffs are flexible and service oriented. When my rooms heater were not working, they upgraded us to a bigger room with working heater.
สถานีรถไฟ Parliament - ระยะ 0.34 กม.


As a resident of Pasig for the last six years I became accustomed to being just a stone's throw away from everything. With no less than five malls around the corner, my hunger pangs, coffee fixation, bag obsession and shoe addiction are easily satisfied, if not cured. These malls kept me sane (who ever invented retail therapy is a genius), fit (walking burns calories) and, of course, fashionable and happy (again, retail therapy is the key).
Because of the proliferation of malls in the country, not just in the capital city, kahit saan ka lumingon may SM. In fact, in the Ortigas area alone there are two SMs (SM Megamall and SM Hypermart).
Noong 80's, wala pang SM. Noon, ang tawag sa tindahan ni Henry Sy ay "ShoeMart". Lima (5) pa lang ang branches ng ShoeMart noon and they are strategically located in Manila, Makati, Cubao, Cebu and Ilo-ilo (I can still remember this because the names of the 5 branches were written at the bottom of their plastic bags and I used to recite, correction, sing about it. No kidding.). Since the branch in Araneta Center Cubao was the nearest to Rizal, that's where we go.
Sa loob nang isang taon, siguro mga apat o limang beses lang kaming pumupunta sa ShoeMart para mag-shopping. Naka-schedule na yun and, usually, in preparation for a special occasion. Here's our usual schedule March, before Holy Week (to buy Easter Sunday dress); June, before the start of classes (to buy school supplies); November, before All Saints/Souls Day (to buy candles and sementeryo outfit, hehehe, pag pangit damit mumultuhin) December, before Christmas (to buy Christmas clothes and Christmas gifts).
Pag December, super-special ang trip namin sa Cubao. Kasi, in addition to our usual ShoeMart visit, pumupunta din kami sa COD, to watch the Christmas window display, at sa Araneta Coliseum, to look at the giant Christmas tree and to watch Disney or Holiday on Ice.
Dahil sa ShoeMart kami namimili, akala ko sosyal ako. Akala ko mayaman kami. Hindi pala. Hindi pala kami "well to do family".
Noong 80's, ang politically correct term para sa mga mayayaman ay "well to do family". Let's use that in a sentence. 'Michelle is a beautiful girl, she hails from Antipolo and comes from a well to do family'. Oh 'di ba, politically correct at pang textbook yan.
Anyway, na-discover kong hindi kami 'well to do' kasi yung family ni Sheila, my grade school best friend, sa Rustans, Ansons at LandMark nag-sa-shopping. "Saan yon?", I asked. She told me, "sa Makati, near ShoeMart Makati". "Ahhh", sabi ko. I pretended that I knew what she was talking about kahit, sa totoo lang, hindi pa ako nakakatapak sa ShoeMart Makati.
When I got home that day, I asked Mama why we don't shop in Makati. She told me it was far from our place and that it was a bit expensive to shop there. That was the end of it. I didn't ask her to bring me there. Hindi ko na rin siya kinulit about the different shops in Makati, after all ShoeMart Cubao is good enough for me. Kahit halos kalahati ng mga girls sa class kapareho ko ng damit at sapatos pag Christmas party okay lang, mas maganda naman ang ipit ko.


text:
The History of MOBA Games
by admin / Friday, 03 March 2017 / Published in Home page, Uncategorized
A lot of players didn't get the chance to start out with earlier incarnations of MOBA games. MOBAs like League of Legends and DOTA 2 are part of a long legacy of quick, action-oriented strategy games which put the emphasis on a single champion rather than an army of controllable units—but anyone who plays League already knows that. What some younger players might not know is that the modern MOBA genre originated with Defense of the Ancients, a popular map mod for Blizzard Entertainment's highly-successful RTS, Warcraft 3. DOTA eclipsed the popularity of WC3's regular multiplayer map modes quickly, reaching cult status in Europe and the United States.
Arguably, if you reach back even further, the starting point really began in 1998 with a map mod for StarCraft, another Blizzard strategy game. The mod known as Aeon of Strife allowed players to control a single hero unit to take down a wave of NPCs attacking from three different lanes. The lanes were connected to the home bases of two different teams, and the ultimate objective of any given match was to destroy the enemy team's headquarters. Aeon of Strife provided the MOBA genre with the foundation behind its universally recognizable game mechanics—mechanics which emphasize both simplicity and a hidden complexity.
The meta strategy of MOBA games tends to reveal itself after hundreds of hours of play. Mechanics that were overlooked by new players begin to be revealed gradually as experience is accumulated. When you start out playing League of Legends, you'd be forgiven for not understanding how to weigh the best strategic choices and how to utilize your toolkit to the highest level of effectiveness. As you play more and more, the hidden aspects of the game's strategy overall begin to reveal a far more multifaceted experience.
We've come a long way from the days of Defense of the Ancients and Aeon of Strife. League has more champion choices than any previous game in the genre, and this results in a lot of trouble for the developers as they are constantly working hard to ensure the game is as balanced as possible. Keep in mind, attaining perfect balance in a game like League of Legends is a major effort, and if your particular champion feels a little bit anemic after a patch change, try and operate within the new parameters of what is possible with your favorite heroes—better yet, learn to play ones you might not have given the time of day before.
If you're interested in skipping all of the inconvenience of having to look up guides, learn strategies, and play with players who are dispassionate or toxic, consider enlisting the help of our professional boosters. We can bring you all the way up to diamond, if you desire it. We make sure you get where you need to go faster than any other provider of boosting services in the industry. Otherwise, continue playing the game on your own—but keep us in mind when you're looking for a boost!
Why you Need a Coach in League of Legends
Best Computer for League of Legends
Easter Eggs in League of Legends


Choices: Stories You Play – an interesting project that absorbed a huge number of interactive novels in various directions, but always with the presence of a romantic line. New in school, the investigation of a series of murders, the medieval war and many other stories are ready to embrace everyone who is tired of ordinary, paper volumes. A real library of fascinating adventures is already waiting for the book-keepers. If you want to play this game, you can download and play the game from the link below. Good luck.


In this video brought to you by Project Smart, I talk about the importance of risk management, discuss tools and common risk management mistakes.
What do you do when a customer complains about something on the project? Are you abrasive? Defensive? Apologetic? No matter what happens, no matter whose fault it is, we should always be apologetic. Not that the customer is always right, but if they are upset at all then we need to be concerned. Twice I did not take my own advice and I lost my clients over it. Lesson learned.
You're a marketing manager in the creative management industry. What skills do you need to do your job well? To be effective for your organization, to lead your teams, to successfully deliver for your clients or your own organization? Current marketing managers should be thinking in terms of the following list of broad-based skills that can be applied to different scenarios, in different industries, and for companies varying in size from small startups and mom and pop shops to large Fortune 500 organizations.
You know that new car smell or how a new romantic relationship feels? Ok, that may be a little over the top in trying to equate that to the feeling of a fresh new project, but it gets you close. Everything is new. Everything is perfect. Everything is fresh. The team has not experienced any conflicts. Hopefully the customer isn't complaining about anything yet. Senior management isn't wondering why you haven't delivered that big billable deliverable yet so they can have the revenue in 'x' quarter.
Ready to check it out for yourself?
Have I peaked your interest? You need to look into this one in more detail – trust me. Check out the OneDesk homepage and then signup for a demo and see for yourself.
Profitability on our projects is always a goal – especially as our senior management looks at our portfolio of projects. And there is probably no greater factor into that profitability than our own human resources that are charging to our projects. So, in terms of technical projects…one thing that many organizations consider is who to utilize when developing software on a project. Do we use in-house resources (if we even have in-house resources…depending on the organization type, of course)? Do we outsource it locally or nationwide? Or do we utilize inexpensive offshore development resources?
It is that last option – the offshore team – that I want to focus on for this article. Not about whether we should use them or not…that's a call each organization has to make. My focus is more on what the project manager needs to be considering and focusing on once you've already made that choice to go offshore. I've been there, and for me it comes down to five key things to look at and consider or incorporate into the project….
Now is our chance. Yes...our turn to complain. Think hard. When you're managing a project, what things really grind your gears? What bothers you and makes you want to blow your top? What behaviors or actions, questions or issues show up and really drive you crazy? Think hard now...this is your chance to gripe about them.
As a client, Do the programs and tools they use have the flexibility to be customized to your needs?


text:
Q: Why is property unaccessible from the setter defined on prototype I have this code:
function Test2() {
  this.data = 1;
  this.unaccesible = 7;
}

Object.defineProperty(Test2.prototype, "data", {
  get: function() {
    return this.data;
  },
  set: function(nv) {
    // here this.unaccesible is undefined - why?
    console.log(this.unaccesible);
  }
});

var o = new Test2();
o.data = 2;

Why the property is not available inside the setter?
console.log(this.unaccesible);

A: Because you define a property on the prototype, not an instance of Test2. this.data is only accessible because you are defining a property data. It's not the same data as the one set in the constructor of Test2.

A: When the instance of Test2 is created and line 2 is executed, which is  this.data = 1, this is defined and it has prototype which  has "data" property but the instance itself doesn't have "data" yet. That's why prototype's data-property setter method is called and it's happening before this.unaccesible = 7. that is why this.unaccesible is undefined at this point

A: You need to create a separate Object.defineProperty for each variable.
Check out the following fiddle for code example with comments. Hope this answers your query.
JSfiddle example for the problem
OR Check out the same code below
            function Test2() {
              this.data = 1;
              this.unaccesible = 5;
            }

            Object.defineProperty(Test2.prototype, "unaccesible", {
              get: function() {
                return this.data;
              },
              set: function(nv) {
                // here this.unaccesible is undefined - why?
                // You cannot access "unaccesible" directly as the parameter for the
                // setter function is nv.
                var test = nv;
                alert("Unaccessible:" + test);
              }
            });
            // Use object.defineProperty for each variable as follows to set data value
            Object.defineProperty(Test2.prototype, "data", {
              get: function() {
                return this.data;
              },
              set: function(nv) {
                // here this.unaccesible is undefined - why?
                // You cannot access "unaccesible" directly as the parameter for the
                // setter function is nv.
                var test = nv;
                alert("Data:" + test);
              }
            });

            var o = new Test2();
            //This sets the value of unaccesible to 2 after the intial value 5 is stored.
            o.unaccesible = 6;
            //This sets the value of data to 2 after the intial value 1 is stored.
            // This is executed after unaccesible
            o.data = 2;

redpajama_set_name: RedPajamaStackExchange


Registration is free. Participants must have at least 3-years experience in the fashion industry. Please include the following details: Name, Contact number, No. of years in fashion, and what you hope to get out of the seminar.
The British Council and LASALLE College of the Arts have collaborated to launch Asia Fashion Future.
The mission of Asia Fashion Future is to provide fashion professionals from Asia and the UK with the skills and knowledge necessary to increase the sustainability and growth of Asian fashion businesses.
This one day seminar will bring experts from the UK and Asia together to examine challenges and opportunities facing Asia's growing fashion sector.
Participants will benefit from a unique learning experience, gaining the best of UK expertise within an international context.
Explore how the popularity of web and social media has reshaped consumption patterns and transformed the fashion industry.
For more information or questions, email culturalskillsunit@britishcouncil.org.
Participants are responsible for covering the cost of their accommodation, travel and Singapore visa (if applicable).
Upon request the British Council can provide a letter of invitation for a visa.
Art, architecture, literature, film or design, we bring the best of UK culture to you.