text:
\section{Introduction}
\label{sec:intro}

Meta-embedding~\cite{Yin:ACL:2016,Bollegala:IJCAI:2018} is the task of producing embeddings from a given set of pretrained text embeddings such that the complementary strengths of the input (source) embeddings are preserved or further enhanced to obtain the best performance on a downstream NLP task.
Numerous methods have been proposed for creating a meta-embedding from a given set of static word embeddings such as GloVe \cite{Pennington:EMNLP:2014}, Skip-gram with Negative Sampling (SGNS) \cite{NIPS2013_9aa42b31} or Continuous Bag-of-Words (CBOW)  \cite{DBLP:journals/corr/abs-1301-3781} as the source embeddings.
In contrast to static word embeddings, contextualised word embeddings obtained from Neural Language Models (NLMs) such as BERT \cite{BERT}, RoBERTa \cite{RoBERTa} etc. represent the same word with different vectors in different contexts, thereby providing a richer context-dependent semantic representations.
Moreover, contextualised word embeddings can be pooled and fine-tuned to obtain accurate sentence-level representations.
Compared to static word embeddings, contextualised word/sentence embeddings have reported impressive performances in a wide range of NLP tasks.

\paragraph{Challenges:}
Compared to meta-embedding of static word embeddings, when computing meta-embeddings from contextualised word embeddings~\cite{poerner-etal-2020-sentence} one must overcome several challenges.
\begin{itemize}
 \item First, unlike static word embeddings, which represent the same word using the same embedding vector in all contexts in which the target word occurs, contextualised word embeddings depend on the context of the word.
Therefore, the same word will be represented by different embeddings in different contexts by a contextualised word embedding model.
This is particularly challenging when creating meta word embeddings because the meta embedding of a word will depend not only on the source embeddings, but also on the context in which the target word occurs.

\item Second, static word embeddings represent words using relatively smaller dimensional vector spaces compared to contextualised word embeddings.
For example, good performance can be obtained with as small as 300-600 dimensional GloVe static word embeddings, whereas BERT contextualised word embeddings require 768, 1024 or higher dimensionalities.
Therefore, methods such as concatenation that have been effective for creating meta-embeddings~\cite{coates-bollegala-2018-frustratingly} for static word embeddings become problematic when applied to contextualised word embeddings.
This issue is further aggravated by the fact that contextualised word embedding models require specialised hardware such as GPUs, which have limited memory buffers.

\item Third, although supervised approaches have been proposed for meta-embedding of static word embeddings that require labelled data for the target tasks~\cite{xie2019dynamic,he-etal-2020-learning,wu2020task}, adapting such supervised approaches for contextualised word embeddings remains a challenging task.
Because contextualised word embeddings such as BERT typically contain over 100 million parameters,\footnote{For example, the \texttt{BERT-small} model trained by \newcite{BERT} with 12 layers contain 110 million parameters}
it requires significantly more labelled data to fine-tune such models for a given target task.
Therefore, unsupervised methods are preferable over supervised ones when creating meta-embeddings from contextualised embedding models.
\end{itemize}

\paragraph{Proposal:}
To address the above-mentioned challenges, we propose an unsupervised method\footnote{Codes: \url{https://github.com/keigotak/unsup-att-based-meta-sent-emb}} to combine multiple contextualised word embedding models (referred to as \textbf{source embeddings}) to create a meta-embedding for a given target word in a sentence that it occurs.
Furthermore, we create sentence-level embeddings by applying parameter-free operations such as \emph{pooling} to create sentence-level meta-embeddings from the source embeddings.

In our proposed method, we first project token-level embeddings obtained for a word from a contextualised word embedding to a common meta-embedding space using linear transformations.
We then compute the linearly-weighted average of the projected source embeddings to compute their token-level meta-embedding.
The weighting coefficients can be seen as describing the significance we must impart on a source embedding in the ensemble.
Note that contextualised token-embeddings produced by different NLMs can in general have different dimensionalities.
Therefore, the source-specific linear projections learnt in the first step ensures all source embeddings are projected to a common meta-embedding dimensionality such that we can compute their linearly-weighted average.
Finally, we apply pooling operations such as max and mean pooling over the token-level meta-embeddings for the words in a sentence to create a sentence-level meta-embedding for the corresponding sentence.

\paragraph{Contributions:}
We evaluate our proposed meta embedding learning method using different types of source embeddings on Semantic Textual Similarity (STS) benchmarks \cite{cer-etal-2017-semeval}.
We compare our results against multiple baselines such as vector concatenation, averaging, singular value decomposition and the current state-of-the-art unsupervised sentence-level meta-embedding method proposed by \newcite{poerner-etal-2020-sentence}.
Experimental results show that our proposed unsupervised sentence-level meta-embedding learning method obtains the highest correlation with human similarity ratings in all STS benchmark datasets, outperforming all methods compared.

\section{Related Work}
\subsection{Sentence-Level Meta-embeddings}
Sentence-level meta-embedding learning methods aim to create a single vector representation for a given sentence by combining token-level representations produced by multiple source embeddings.
\newcite{kiela-etal-2018-dynamic} proposed dynamic meta-embeddings, which learns task-specific combinations of source embeddings using the attention mechanism~\cite{Bahdanau:2014}.
\newcite{poerner-etal-2020-sentence} proposed the use of Generalised Canonical Correlation Analysis (GCCA) to learn sentence-level meta-embeddings from contextualised word embeddings.
GCCA is an extension of Canonocal Correlation Analysis (CCA) to more than two random variables.
Specifically, a word in a sentence can be represented using multiple vectors by using multiple contextualised word embedding models.
GCCA learns linear transformations that can be applied to such vectors to maximise the correlation in the projected space.

Unlike our proposed method, prior work on sentence-level meta-embedding methods consider sentence embeddings as the input to the meta-embedding algorithms.
In contrast, we first create token-level meta-embeddings by combining multiple contextualised embeddings and then create sentence-level meta-embeddings by applying pooling operations (e.g. max and min pooling) over the words in a sentence.
GCCA-based meta sentence embedding method proposed by \newcite{poerner-etal-2020-sentence} is the current state-of-the-art unsupervised sentence-level meta-embedding learning method and is a direct comparison to ours which is also unsupervised method.
As we later see in \autoref{sec:exp}, our proposed method outperforms GCCA in multiple benchmark datasets.

\subsection{Word-Level Meta-embeddings}
Word meta-embedding has been studied more actively compared to sentence-level meta-embedding.
\newcite{Yin:ACL:2016} proposed 1\texttt{TO}N,
 by projecting source embeddings to a common space via source-specific linear transformations.
This method minimises squared $\ell_{2}$ distance between the meta and each source embedding assuming a common vocabulary.
1\texttt{TO}N+ overcomes this limitation by learning pairwise linear transformations between two given sources for predicting the embeddings for out of vocabulary (OOV) words.
Both of these methods can be seen as \emph{globally-linear} transformations because \emph{all} the words in a particular source are projected to the meta-embedding space using the \emph{same} transformation matrix.

In contrast, \emph{locally-linear} meta-embedding (LLE)~\cite{Bollegala:IJCAI:2018} computes the nearest neighbours for a particular word in each source and then represent a word as the linearly-weighted combination of its neighbours.
Next, meta-embeddings are learnt by preserving the neighbourhood constraints.
This method does not require all words to be represented by all sources, thereby obviating the need to predict missing source embeddings for OOVs.

\newcite{Bao:COLING:2018} modelled meta-embedding learning as an \emph{autoencoding} problem where information embedded in different sources are integrated at different levels to propose three types of meta-embeddings: Decoupled Autoencoded Meta-Embedding (DAEME) (independently encode each source and  concatenate), Concatenated Autoencoded Meta-Embedding (CAEME) (independently decode meta-embeddings to reconstruct each source), and Averaged Autoencoded Meta-Embedding (AAEME) (similar to DAEME but instead of concatenation use average).
In comparison to methods that learn globally or locally linear transformations~\cite{Bollegala:IJCAI:2018,Yin:ACL:2016}, autoencoders learn nonlinear transformations.
\newcite{Neill:2018} further extend this approach using squared cosine proximity loss as the reconstruction loss.

Vector concatenation has been used as a baseline for producing
meta-embeddings~\cite{Bao:COLING:2018,Yin:ACL:2016,Bollegala:IJCAI:2018}.
\newcite{AAAI:2016:Goikoetxea} concatenated independently learnt word embeddings from a corpus and the WordNet.
Moreover, applying Principal Component Analysis on the concatenation further improved their performance on similarity tasks.
Interestingly, \newcite{Coates:NAACL:2018} showed that accurate meta-embeddings can be produced by averaging source embeddings that exist in \emph{different} vector spaces.
Specifically, they showed that when word embeddings in each source are approximately orthogonal (they empirically validate for pre-trained embeddings), averaging can approximate concatenation.
Although averaging does not increase the dimensionality of the meta-embedding space as with concatenation, it does not consistently outperform concatenation, especially when the orthogonality condition does not hold.
Recent work~\cite{He:2020,jawanpuria-etal-2020-learning} show that learning orthogonal transformations prior to averaging can further improve accuracy.

\section{Method}
\label{sec:unsup}

Let us denote a sentence $s = w_1, w_2, \ldots, w_{|s|}$ consisting of tokens $w_1, w_2, \ldots, w_{|s|}$ in that order.
In the case where we have applied a subtokenisation method such as Byte Pair Encoding \newcite{Sennrich:2016} to tokenise sentences into subtokens, we will assume $w_j$ to be subtokens instead of tokens.
In the following discussion, we use the term \emph{word} to cover both tokens as well as subtokens unless otherwise explicitly stated.
Moreover, let us assume that we are given $n$ independently pretrained contextualised word embeddings of which the $i$-th source embedding is denoted by $\vec{f}_i (w, s) \in \R^{d_{i}}$, and returns a $d_i$ dimensional word embedding of $w$ in a given context $s$ (e.g. a sentence or a contextual window).
In particular, static word embeddings can be seen as a special case of this formulation where the word embedding does not depend on $s$ (i.e. $\forall s$ $\vec{f}_i(w,s) = \vec{f}_i(w)$).
Our goal in this paper is to create a sentence-level meta-embedding, $\vec{m}(s)$, representing the sentence $s$ using all word-level source embeddings, $\{\vec{f}_i(w,s)\}_{i=1}^{n}$, for the words $w$ in $s$.
Following the convention used in prior work on meta-embedding learning, we refer to individual $\vec{f}_i$ as source embeddings.

Note that in general the individual source embeddings will have different dimensionalities $d_i$.
To transform each source embedding to the same meta-embedding space with dimensionality $d_m$, we first transform each source embedding by a source-specific projection matrix $\mat{A}_i \in \R^{d_m \times d_i}$ as given by \eqref{eq:wo-attention}.
\begin{align}
    \label{eq:wo-attention}
    \vec{\phi}_i(w,s) = \mat{A}_i \vec{f}_i(w,s)
\end{align}
Here, the projected source embeddings, $\vec{\phi}(w,s) \in \R^{d_m}$, will be in the same common $d_m$ dimensional meta-embedding space for all source embeddings.
The total number of learnable parameters in this model is $\sum_{i=1}^n d_m d_i$, which corresponds to the sum of elements in all projection matrices $\{\mat{A}_i\}_{i=1}^{n}$.

Different source embeddings will have complementary strengths and weaknesses.
Therefore, we propose to assign weights to the source embeddings when creating a meta-embedding of those source embeddings as given in \eqref{eq:w-attention}.
\begin{align}
    \label{eq:w-attention}
    \vec{\phi}_i(w_j,s_k) = \alpha_{i} \mat{A}_i \vec{f}_i(w_j,s_k)
\end{align}
Here, $\alpha_i$ is the weight associated with the $i$-th source embedding such that $\sum_i \alpha_i = 1$.
Moreover, to reduce the degrees of freedom of the model, we require cross-correlations in projection matrices to be small.
This requirement can be implemented as a regularisation on the projection matrices such that $\norm{\mat{A}_i\T\mat{A}_i - \mat{I}}_F^2$ is minimised, where $\norm{\cdot}_F$ denotes the Frobenius norm.

We can consider the individual projection matrices $\mat{A}_i$ to be concerned with projecting source embeddings with different dimensionalities to a common meta-embedding space, whereas $\alpha_i$ determines the weight imparted on the projected source embedding in the meta-embedding space.
However, in the unsupervised meta-embedding learning setting that we consider in this paper, we do not assume the availability of any labelled data for downstream tasks nor our goal is to learn a meta-embedding that is specific to a particular task.
On the other hand, task specialisation of meta-embedding can be done as a subsequent task akin to fine-tuning of foundation models.
Therefore, we learn both $\alpha_i$ and $\mat{A}_i$ via an unsupervised learning method as described later.

\iffalse
Note that different words and sources should have different importance when determining the meta-embedding of that word in a sentence.
For example, a given word $w$ might be represented more accurately by some source embedding $f_i$ than the others because $f_i$ was pretrained on a corpus that contained various uses of $w$, thus enabling $f_i$ to learn more accurate embedding for $w$.
It is known that rare words in a corpus are represented by less accurate word embeddings.
Therefore, we might want to assign higher weight in the meta-embedding for the sources that are more confident about a particular word's embedding.
On the other hand, when creating a sentence embedding (which is our ultimate goal here) some words such as content words are more important than function words.
However, the formulation given in \eqref{eq:wo-attention} considers all words equally and cannot capture word- or source-specific importance.
To address this problem, we propose the variant given by \eqref{eq:w-attention}, where $\alpha_{ijk}$ is the attention score assigned to the $i$-th source embedding $f_i(w_j,s_k)$ of a word $w_j$ appearing in the sentence $s_k$.
\begin{align}
    \label{eq:w-attention}
    \vec{\phi}_i(w_j,s_k) = \alpha_{ijk} \mat{A}_i \vec{f}_i(w_j,s_k)
\end{align}

Following dot-product attention, we compute $\alpha_{ijk}$ as the inner-product between an attention vector $\vec{\theta}_i \in \R^d_i$ that depends only on the $i$-th source and the contextualised embedding $f_i(w_j,s_k)$ as defined in \eqref{eq:dot-prod}.
\begin{align}
    \label{eq:dot-prod}
    \alpha_{ijk} = \vec{f}_i(w_j,s_k)\T\vec{\theta}_i
\end{align}
Note that the factorisation assumption made in \eqref{eq:dot-prod} significantly reduces the number of cleanable parameters in the model because the attention vector $\vec{\theta}_i$ is shared among all words in the $i$-th source.
Specifically, we have $\sum_{i=1}^n d_i$ number of parameters to learn for the attention vectors in addition to the $\mat{A}_i$ matrices.
Note that the model defined by \eqref{eq:wo-attention} can be seen as a special case of the model defined by \eqref{eq:w-attention} that incorporates attention.
Therefore, in the remainder of this section we use the attention enabled model defined by \eqref{eq:w-attention} and later in \autoref{sec:exp} empirically compare the difference of the two models.
\fi

Given $n$ distinct source embeddings, $\{\vec{f}_i(w,s)\}_{i=1}^{n}$, of a word $w$ in a sentence $s$, we compute the word's meta-embedding, $\vec{m}_{\rm word}(w,s)$, as the average of the projected source embeddings as given by \eqref{eq:meta-word-embedding}.
\begin{align}
    \label{eq:meta-word-embedding}
    \vec{m}_{\rm word}(w,s) = \frac{1}{n} \sum_{i=1}^{n} \vec{\phi}_i(w,s)
\end{align}
Recall that the projection matrices ensure that source embeddings with different dimensionalities are projected to the same $d_m$ dimensional meta-embedding space such that they can be averaged to obtain a reliable estimate of the final word-level meta-embedding.

Armed with the word-level meta-embeddings, $\vec{m}_{\rm word}(w,s)$, for a word $w$ in a sentence $s$, we compute two types of  sentence-level meta-embeddings, $\vec{m}_{\rm mean}(s)$ (defined by \eqref{eq:max-pool}) and $ \vec{m}_{\rm max}(s)$ (defined by \eqref{eq:mean-pool}) by applying respectively mean or max pooling  over the word embeddings $\vec{m}_{\rm word}(w,s)$ corresponding to the words in the sentence $s$.
\begin{align}
     &\vec{m}_{\rm mean}(s) = \frac{1}{|s|} \sum_{w \in s} \vec{m}_{\rm word}(w,s) \label{eq:mean-pool} \\
    &\vec{m}_{\rm max}(s) = \max_{w \in s} \vec{m}_{\rm word}(w,s) \label{eq:max-pool}
\end{align}
The pooling operations are conducted elementwise over the set of word-level meta-embeddings.
We compare the performances of meta sentence embeddings created using mean and max pooling in \autoref{sec:exp}.

What remains in our model is to devise a method to learn the projection matrices, $\{\mat{A}_i\}_{i=1}^{n}$, and the weights $\{\alpha_i\}_{i=1}^{n}$.
Recall that in this paper we are considering \emph{unsupervised} sentence embeddings.
Therefore, we do not assume the availability of labelled data such as human-annotated similarity ratings used in supervised sentence embedding learning.
For this reason, next we derive an unsupervised training objective.

Let us consider the projected source embeddings of two words $w \in s$, $w' \in s'$ in two different sentences $s$ and $s'$ to be given respectively by $\vec{\phi}_i(w,s)$ and $\vec{\phi}_j(w',s')$ using two different sources $f_i$ and $f_j$.
We assume the following four criteria to be satisfied by these projected source embeddings.
\begin{description}
\item [Criterion 1:] \emph{The projected source embeddings $\vec{\phi}_i(w,s)$ and $\vec{\phi}_j(w,s)$ of the same word in the same sentence must be similar.}

This requirement stems from the definition of meta-embedding process because the purpose of the projection matrix in the first place was to ensure that different source embeddings of the same word are closer to each other in the meta-embedding space.
This requirement can be formulated as the minimisation of the squared $\ell_2$ distance between the projected source embeddings given by \eqref{eq:crit-1}.
\begin{align}
    \label{eq:crit-1}
    \norm{\vec{\phi}_i(w,s) - \vec{\phi}_j(w,s)}_2^2
\end{align}

\item [Criterion 2:] \emph{The projected source embeddings $\vec{\phi}_i(w,s)$ and $\vec{\phi}_j(w',s)$ of two different words $w$ and $w'$ that co-occur in the same sentence $s$ must be dissimilar.}

This requirement is important to make the meta-embeddings of two distinct words to be sufficiently dissimilar such that we can discriminate them in the meta-embedding space.
We formalise this requirement as the minimisation of the squared $\ell_2$ distance given by \eqref{eq:crit-2}
\begin{align}
    \label{eq:crit-2}
    \norm{\vec{\phi}_i(w,s) - \vec{\phi}_j(w',s)}_2^2
\end{align}

\item [Criterion 3:] The projected source embeddings $\vec{\phi}_i(w,s)$ and $\vec{\phi}_j(w,s')$ of the same word $w$ in two different sentences $s$ and $s'$ must be dissimilar.

This is particularly true for contextualised source embeddings, which consider both the word as well as the context in which the word occurs when representing the word.
Moreover, we are considering embeddings produced by two distinct source embeddings here.
This requirement is useful for creating different meta-embeddings for different sentences eventhough they might share some words in common.
We formalise this requirement as the minimisation of the squared $\ell_2$ distance given by \eqref{eq:crit-3}.
\begin{align}
    \label{eq:crit-3}
    \norm{\vec{\phi}_i(w,s) - \vec{\phi}_j(w,s')}_2^2
\end{align}

\item [Criterion 4:] \emph{The projected source embeddings $\vec{\phi}_i(w,s), \vec{\phi}_j(w',s')$ of two different words $w, w'$ each occurring in different sentences $s, s'$ must be dissimilar.}

This requirement can be seen as the most extreme dissimilar case involving distinct sources, distinct sentences and distinct words.
It can also be seen as imposing a regularisation on the parameter learning objective that prevents degenerated solutions such as projecting all words and sentences to the same point in the meta-embedding space.
We formalise this requirement as the minimisation of the squared $\ell_2$ loss given by \eqref{eq:crit-4}.

\begin{align}
    \label{eq:crit-4}
    \norm{\vec{\phi}_i(w,s) - \vec{\phi}_j(w',s')}_2^2
\end{align}
\end{description}

Finally, we combine the above-mentioned four constraints into a single training objective including the regulariser for the projection matrices as in \eqref{eq:unsup-loss}.
\begin{align}
    \label{eq:unsup-loss}
    &L(\{\mat{A}_i\}_{i=1}^{n}, \{\alpha_i\}_{i=1}^{n}) \nonumber \\
    =& \norm{\vec{\phi}_i(w,s) - \vec{\phi}_j(w,s)}_2^2 \nonumber \\
    & - \lambda \norm{\vec{\phi}_i(w,s) - \vec{\phi}_j(w',s)}_2^2 \nonumber \\
    & - \mu \norm{\vec{\phi}_i(w,s) - \vec{\phi}_j(w,s')}_2^2 \nonumber \\
    & - \nu \norm{\vec{\phi}_i(w,s) - \vec{\phi}_j(w',s')}_2^2 \nonumber \\
    &+ \xi \sum_{i=1}^{n}\norm{\mat{A}_i\T\mat{A}_i - \mat{I}}_F^2
\end{align}
Here, the regularisation coefficients $\lambda$, $\mu$, $\nu$ and $\xi$ are all optimised during training via backpropagation.

We randomly initialise all projection matrices and attention vectors and use Stochastic Gradient Descent (SGD) to optimise.
Further details on learning settings are provided in \autoref{sec:hyperparameters}.

Note that we do not assume any constraints for the projections of a single source embedding such as for two words in the same sentence, or the same word in different sentence for two reasons.
First, many source embedding learning methods already use these constraints during pretraining.
For example, masked language models such as BERT, masks a word in a sentence and requires that it can be predicted correctly using the embeddings of the other words in the sentence.
This requirement forces the words in the same sentence to have similar embeddings.
Second, during meta-embedding learning, we would like to preserve as much information as possible in the original sources, without imposing constraints for the individual sources, beyond what they have been already trained with.

Note that we require no human labelled data such as labels for sentences or similarity ratings for creating sentence-level meta-embedding from the above-described proposed method.
Therefore, we denote our unsupervised proposed method by \textbf{UNSUP} in the remainder of the paper.

\section{Experiments}
\label{sec:exp}

\subsection{Source Embeddings}

We use two source embeddings in our experiments --
BERT\footnote{1024-dimensional token embeddings. Labeled as \texttt{bert-large-nil-stsb-mean-tokens} in sBERT.} and
RoBERTa.\footnote{1024-dimensional token embeddings. Labeled as \texttt{roberta-large-nil-stsb-mean-tokens} in sBERT.}
All the models are pretrained by \newcite{reimers-2019-sentence-bert} and \newcite{wolf-etal-2020-transformers}.
We used the final layer of source embeddings for creating sentence representations.
We note that the proposed sentence-level meta-embedding method can be used even without fine-tuning the source embeddings on sentence similarity related tasks.
However, the overall performance of meta-embeddings invariably depends on the quality of source embeddings used as the input to the meta-embedding method.
Therefore, it has been common practice in prior work on meta-embedding learning~\cite{poerner-etal-2020-sentence} to use high quality source embeddings as input.

Although we use two source embeddings as a proof-of-concept for the proposed method, note that the proposed method itself is not limited to only two sources.
For example, given $n$ sources, we can consider pairwise combinations between any two of those $n$ sources to compute the four criteria described in \autoref{sec:unsup}.
Moreover, the BERT and RoBERTa sources we use in our evaluations have equal (i.e. 1024) dimensionalities.
However, the proposed method \emph{does not} require the dimensionalities of the source embeddings to be equal.
In fact, the projection matrices will ensure that the averaging operation given by \eqref{eq:meta-word-embedding} can be carried out even if the dimensionalities of the sources are different.
Further analysis on more than two sources and involving unequal dimensionalities are deferred to future work.

\subsection{Data and Evaluation Metrics}
\label{sec:data}
Given that our goal in this paper is to create sentence-level meta-embeddings, we use Semantic Textual Similarity (STS) prediction as an evaluation task.
In STS, we are given two sentences $s$ and $s'$ between which we must predict their semantic similarity.
For example, we can compute the cosine similarity between $s$ and $s'$ using the sentence embeddings computed using a meta sentence embedding method.
There exist several benchmark datasets where human annotators have provided similarity ratings, $r(s,s')$, between $s$ and $s'$, which can be considered as ground truth for evaluating sentence embedding methods.
We can then compute the agreements between human similarity ratings and the predicted cosine similarities using correlation measures such as the Spearman and Pearson correlations.

The SentEval tool~\cite{Conneau:2018ac} provides a unified framework for evaluating sentence embedding methods and reports the class-weighted averages of Pearson and Spearman correlation coefficients as the evaluation metric, which we use in this work.
A high degree of correlation between human similarity ratings and cosine similarity scores computed using the sentence embeddings produced by a meta sentence embedding method is considered as desirable, and accurately capturing the meaning of the sentences.
We use the following STS datasets in our experiments: STS-15~\cite{agirre-etal-2015-semeval}, STS-16~\cite{agirre-etal-2016-semeval}) and STS-B (STS benchmark)~\cite{cer-etal-2017-semeval}.

Note that the purpose of evaluating on STS benchmark dataset is \emph{not} to compare against the state-of-the-art for supervised sentence embedding methods but to compare against the unsupervised sentence-level meta-embedding learning methods.
With this objective in mind, we emphasise that the results reported in this paper must not be compared against general purpose sentence embedding learning methods, which are not combining multiple source embeddings as required by the meta-embedding learning methods we consider in this paper.

\subsection{Baselines Methods}
\label{sec:baselines}

We compare our proposed method against several baseline methods, which we describe next.
\begin{description}[style=unboxed,leftmargin=0cm]

    \item [Single Source Embeddings (SSE):]
    We create sentence embeddings from each source embedding separately by applying either max or mean pooling over the word embeddings produced by that source for the words in a sentence.
    These baselines will demonstrate the level of performance that we can expect to obtain if we had used a single source embedding.
    This can be seen as a lower baseline for meta-embedding because one would expect a meta-embedding to always outperform the individual source embeddings used in it.
    We use both mean and max pooling given respectively by \eqref{eq:mean-pool} and \eqref{eq:max-pool} to create sentence embeddings from the source embeddings in our evaluations.

    \item [Concatenation (CONC):]
    In prior work of meta-embedding, it has been reported that by simply concatenating the different source embeddings of a word, one can obtain a surprisingly accurate meta-embedding for that word~\cite{yin-schutze-2016-learning,coates-bollegala-2018-frustratingly}.
    We extend this method to create sentence-level meta-embeddings by first creating word-level meta-embeddings for each word in a given sentence by the concatenation of each source embedding of that word.
    Next, we apply max or mean pooling over those concatenated word-level meta-embeddings.
    Specifically, for a given sentence $s$, its the sentence-level meta-embeddings, ${\rm conc}_{\rm max}(s)$ and ${\rm conc}_{\rm mean}(s)$ are computed using respectively max and mean pooling methods as follows:
    \begin{align}
        {\rm conc}_{\rm max}(s) = \max_{w \in s} \bigoplus_{i=1}^n \vec{f}_i(w,s) \label{eq:conc-max} \\
        {\rm conc}_{\rm mean}(s) = \frac{1}{|s|}\sum_{w \in s} \bigoplus_{i=1}^n \vec{f}_i(w,s) \label{eq:conc-mean}
    \end{align}
    Here, $\bigoplus$ denotes the vector concatenation.
    Note that token pooling methods are applied \emph{after} the concatenation operation.

    One disadvantage of concatenation as a meta-embedding method is that because it adds up the dimensionalities of the source embeddings, when the number of sources increases or when the dimensionality of the source embeddings increases, thereby creating high-dimensional meta-embeddings.
    High-dimensional meta-embeddings require large storage spaces and are also slower to conduct operations such as computing cosine similarity between words/sentences.
    This is particularly problematic when creating meta-embeddings of contextualised embeddings produced by NLMs such as BERT and RoBERTa, which tend to have larger dimensionalities (e.g. 1024, 2048 etc.).

    \item [Averaging (AVG):]
    Averaging is also a simple method to compute meta-embeddings that has performed surprisingly well in downstream tasks~\cite{coates-bollegala-2018-frustratingly,poerner-etal-2020-sentence}.
    Unlike concatenation, averaging \emph{does not} increases the dimensionality of the meta-embedding when the number of source embeddings increases.
    This overcomes the computational and storage related disadvantages of concatenation described above.
    Specifically, for a given sentence $s$, its the sentence-level meta-embeddings, ${\rm avg}_{\rm max}(s)$ and ${\rm avg}_{\rm mean}(s)$ are computed using respectively max and mean pooling methods as follows:
    \begin{align}
        {\rm avg}_{\rm max}(s) = \max_{w \in s} \sum_{i=1}^n \vec{f}_i(w,s) \label{eq:avg-max} \\
        {\rm avg}_{\rm mean}(s) = \frac{1}{|s|}\sum_{w \in s} \sum_{i=1}^n \vec{f}_i(w,s) \label{eq:avg-mean}
    \end{align}

    Although one cannot simply add (or average) vectors that lie in different vector spaces, \newcite{coates-bollegala-2018-frustratingly} showed that in the case of static word embeddings, such operations can be carried out because the source word embeddings are found to be approximately orthogonal in practice.
    However, this requirement might not be readily satisfied by a set of arbitrary source embeddings and prior work have trained orthogonal projections to explicitly enforce this orthogonality requirement, which has improved the performance of the resultant meta-embeddings~\cite{jawanpuria-etal-2020-learning,he-etal-2020-learning}.
    Moreover, all source embeddings must have equal dimensionalities in order for us to compute their average.
    When this requirement is not satisfied, prior work have padded zeros as necessary prior to computing the average.
    Note that our proposal to learn projection matrices for individual source embeddings as given in \eqref{eq:w-attention} automatically solves both of those issues.
    Specifically, the projection matrices $\mat{A}_i$ convert source embeddings with potentially different dimensionalities to the same meta-embedding space (thereby obviating the need for zero padding), and simultaneously learn orthogonal projections.

    \item [SVD:] Singular Value Decomposition was used as a method for creating sentence-level meta-embeddings by \newcite{yin-schutze-2016-learning}.
    Given a matrix $\mat{A} \in \R^{d_m \times d_n}$, its SVD is given by $\mat{A} = \mat{V}\mat{S}\mat{U}\T$ where $\mat{V}, \mat{S}, \mat{U} \in \R^{d_m \times d_m}, \R^{d_m \times d_n}, \R^{d_n \times d_n}$.
    We follow \newcite{poerner-etal-2020-sentence}  and apply SVD on the matrix $\mat{A}$ formed by row-wise zero-mean concatenation matrix of the source embeddings.
    We then use $\mat{U}\T$  as the meta-sentence embedding.
    SVD overcomes the increasing dimensionality problem associated with CONC because the dimensionality is reduced in the SVD process by limiting to the largest eigenvectors.

    \item [GCCA:] Generalised Canonical Correlation Analysis~\cite{GCCA} (GCCA) was used as a method for creating sentence-level meta-embeddings by \newcite{poerner-etal-2020-sentence}.
    Given two random vectors $\vec{x}$ and $\vec{y}$, the Canonical Correlation Analysis (CCA) finds linear projections $\vec{\lambda}$ and $\vec{\mu}$ such that $\vec{\lambda}\T\vec{x}$ and $\vec{\mu}\T\vec{y}$ are maximally correlated.
    GCCA extends this approach to three or more random vectors.
    We follow \newcite{poerner-etal-2020-sentence} and use the generalised eigenvalue decomposition-based approach proposed by \newcite{Bach:2002} to compute GCCA.
    GCCA is the current state-of-the-art method for creating meta sentence embeddings.

    \item [Supervised Meta-Embedding (SUP):] \label{sec:sup}
    Note that all of the above-mentioned baselines are all unsupervised meta-embedding methods, similar to our proposed method.
    To simulate the level of performance that we can hope to achieve if we had access to some labelled data for the target task, we propose a supervised baseline as follows.
    For this purpose we consider predicting semantic textual similarity between two given sentence as the target task.
    First, given a pair of sentences $(s, s')$ and their human similarity rating $r(s,s')$, we compute a meta sentence embedding separately for $s$ and $s'$ using either mean pooling (given by \eqref{eq:mean-pool}) and max pooling (given by \eqref{eq:max-pool}).
    Let us denote the meta sentence embedding for $s$ and $s'$ respectively by $\vec{m}(s)$ and $\vec{m}(s')$.
    We then compute the cosine similarity, $\cos(\vec{m}(s), \vec{m}(s'))$ using $\vec{m}(s)$ and $\vec{m}(s')$ and minimise the squared loss between the predicted cosine similarity and the human similarity rating, $r(s,s')$, between $s$ and $s'$ as given by \eqref{eq:sup-loss}.
    \begin{align}
        \label{eq:sup-loss}
        &L(\{\mat{A}_i\}_{i=1}^{n}, \{\alpha_i\}_{i=1}^{n}) \nonumber \\
        =& \left(\cos(\vec{m}(s), \vec{m}(s')) - r(s,s')\right)^2
        \end{align}
    Note that projection matrices $\{\mat{A}_i\}_{i=1}^{n}$ and weights $\{\alpha_i\}_{i=1}^{n}$ enter into \eqref{eq:sup-loss} via \eqref{eq:w-attention} and get updated during backpropagation.

    We use the training sentence pairs in the STS-B dataset~\cite{cer-etal-2017-semeval} and minimise the loss given by \eqref{eq:sup-loss} over all such sentence pairs.
    The original human similarity ratings in STS-B are in the range $[0,5]$.
    We linearly transform both cosine similarities and human similarity ratings to $[-1,1]$ by $\frac{x - \min(x)}{\max(x)-\min(x)}$ before computing the loss given in \eqref{eq:sup-loss}.
\end{description}

\subsection{Training Details}
\label{sec:hyperparameters}

To train UNSUP, we use the set of sentences that appear in the sentence pairs in the STS-B training split.
Note that we \emph{do not} use any human similarity ratings for the sentence pairs in the STS-B training split during the training of UNSUP.
In fact, we do not even require pairs of sentences as they appear in STS-B for training UNSUP.
We randomly pick two sentences that appear in (could be the same or different sentence pairs) and compute the loss given by \eqref{eq:unsup-loss},
and update the projection matrices and weights such that the loss is minimised.

To train SUP, we use the sentence pairs and their human similarity ratings from the training sentence pairs in STS-B dataset.

We use SGD optimiser with weight decay $10^{-4}$, learning rate $10^{-2}$ for both UNSUP and SUP methods, and a batch size of 512.
We apply an early stopping as a regularisation technique such that, if the Pearson correlation does not improve for more than five consecutive epochs, computed over the development set in STS-B we stop further training.
All the parameters SUP and UNSUP are randomly initialised in the $[-1,1]$ range.

We use Quadro RTX 8000 with 48GB GPU-RAM for training. On this machine, it takes eight hours to complete the training of the proposed sentence-level meta-embedding method.

\begin{table}[t]
\begin{threeparttable}
\caption{Results of meta-sentence embeddings by using BERT and RoBERTa from sBERT \protect\cite{reimers-2019-sentence-bert}.
We show both Pearson $r$ and Spearman $\rho$ correlation coefficients, separated by /.}
\label{table:main_results}
\small
\begin{tabular}{lrrr}\toprule
Method & STS-15 & STS-16 & STS-B  \\ \midrule
SSE & & & \\
-BERT\tnote{a} & 87.15/87.34 & 81.98/83.05 & 82.40/82.74 \\
-BERT\tnote{b} & 88.29/88.22 & 83.24/84.02 & 83.66/84.05 \\
-RoBERTa\tnote{a} & 75.94/78.69 & 72.96/75.20 & 82.46/83.75 \\
-RoBERTa\tnote{b} & 87.72/87.88 & 84.55/84.93 & 82.46/83.26 \\
\midrule
Meta & & & \\
CONC\tnote{a} & 82.46/83.86 & 78.33/79.88 & 75.12/76.21  \\
CONC\tnote{b} & 88.64/88.59 & 84.27/84.88 & 84.69/85.14  \\
AVG\tnote{a} & 82.23/83.72 & 77.99/79.68 & 83.89/84.53  \\
AVG\tnote{b} & 88.47/88.45 & 83.97/84.63 & 84.21/84.47  \\
SVD & 88.66/88.65 & 84.07/84.63 & 83.98/84.61 \\
GCCA & 88.58/88.58 & 84.00/84.53 & 83.36/84.14 \\
\midrule
SUP & 89.34/89.30 & 85.11/85.72 & 65.21/64.79 \\
UNSUP & \textbf{88.76}/\textbf{88.85} & \textbf{85.06}/\textbf{85.33} & \textbf{85.33}/\textbf{86.08} \\
\bottomrule
\end{tabular}
\begin{tablenotes}
{\small \item[a] max-pooling for word-to-sentence pooling.
\item[b] mean-pooling for word-to-sentence pooling.}
\end{tablenotes}
\end{threeparttable}
\end{table}

\subsection{Results}
\label{sec:results}

\autoref{table:main_results} summarises the main results for the proposed unsupervised method (\textbf{UNSUP}) and the baseline methods described in \autoref{sec:baselines}.
We use 1024-dimensional BERT and RoBERTa from sBERT \cite{reimers-2019-sentence-bert} as the source embeddings in all the experiments.
The dimensionality in all meta-embedding methods compared in \autoref{table:main_results} are 1024, except for CONC, which is 2048 dimensional.
Where applicable, we show the results obtained for the sentence embeddings created by max and mean pooling methods.

From \autoref{table:main_results}, we see that for single source embeddings (SSE) the performance is consistently better when we use mean-pooling instead of max-pooling, except for RoBERTa in STS-B, where there is no significant difference between the two pooling methods.
In particular, we see that mean pooling to produce better sentence embeddings than max pooling for both BERT and RoBERTa source embeddings.
Given that both BERT and RoBERTa source embeddings are pretrained on sentence similarity tasks, one reason for this behaviour could be that mean pooling averages all token-level embeddings in the final layer of the MLM, thus more robust to noise compared to max pooling, which represents the meaning of a sentence using the embedding of a single token.

Next, in \autoref{table:main_results}, we compare the performance of different baseline methods described in \autoref{sec:baselines}.
We evaluate CONC and AVG using both max and mean pooling methods for creating sentence embeddings in \autoref{table:main_results}.
We see that CONC consistently reporting better performances than the best between the two individual source embedding in all datasets.
This shows that CONC is still a strong baseline not only when computing meta embeddings from static word embeddings as reported in prior work~\cite{coates-bollegala-2018-frustratingly}, but also in the case of contextualised word embeddings.

AVG on the other hand closely follows the performance of CONC but never outperforms CONC in any of the datasets.
Here again we see that the mean pooling variants of CONC and AVG outperform their max pooling counterparts.
Although SVD slightly outperforms CONC in terms of Spearman correlation in STS-15, it reports consistently lower performance than CONC in all other settings.
This shows that although SVD can reduce the dimensionality compared to CONC, the performance drops due to possible information loss during the dimensionality reduction process.
GCCA consistently under performs compared to SVD in all datasets.

The proposed \textbf{UNSUP} and the supervised baseline (\textbf{SUP}) are shown at the bottom two rows in \autoref{table:main_results}.
We see that SUP outperforms all baselines including CONC in both STS-15 and STS-16 datasets but not on STS-B.
Therefore, the supervised meta-embedding learning method described in \autoref{sec:baselines} can be considered as a strong baseline for incorporating human similarity ratings into the meta-embedding learning process.
However, UNSUP outperforms all methods compared in \autoref{table:main_results} and obtains the best performance on all datasets in terms of both Spearman and Pearson correlation coefficients.
In particular, UNSUP outperforms the current state-of-the-art meta sentence embedding method, GCCA, proposed by \newcite{poerner-etal-2020-sentence}.
The 95\% confidence intervals for the Pearson's $r$ for \textbf{UNSUP} on STS15, STS16 and STS-B are respectively [88.29, 89.23], [84.45, 85.67], and [83.88, 86.78], which overlap with several other methods indicating that the improvements are not satistically significant.
The supervision considered by \textbf{SUP} is limited to the updating the projection matrices such that the similarity ratings in training datasets could be accurately predicted.
However, there exist alternative methods to provide supervision for sentence-level meta-embeddings such as by fine-tuning the meta-embedding models on sentence pairs in benchmark datasets.
However, we consider this to be beyond the scope of the current work, which focuses on unsupervised sentence-level meta-embedding methods that can be used in the absence of any human supervision.

\subsection{Effect of Weighting and Pooling}
\label{sec:ablation}

\begin{table}[t]
\caption{Ablation of attention and pooling for meta sentence embeddings with BERT and RoBERTa used as the source embeddings.
The column corresponding to Pooling denotes the pooling method used for creating sentence-level meta-embeddings from word-level meta-embeddings as given by \eqref{eq:mean-pool} and \eqref{eq:max-pool}.
Weight column denotes the weights being used. Specifically, w/ represents  using the weights $\alpha_i$ in \eqref{eq:w-attention}, whereas w/o represents not using the weights as in \eqref{eq:wo-attention}.
Correlation coefficients are evaluated on STS-B dataset.}
\label{table:experimental_settings}
\begin{tabular}{l|rrrr}\toprule
Method & Weight & Pooling & Pearson & Spearman  \\
\midrule
UNSUP & w/ & mean & 75.08 & 75.03 \\
UNSUP & w/ & max & \textbf{85.33} & \textbf{86.08} \\
UNSUP & w/o & mean & 80.06 & 80.41 \\
UNSUP & w/o & max & 81.98 & 83.00 \\
\bottomrule
\end{tabular}
\end{table}

We study the effect of using weights $\alpha_i$ with max and mean pooling methods in \autoref{table:experimental_settings}.
Specifically, w/ represents  using the weights $\alpha_i$ in \eqref{eq:w-attention}, whereas w/o represents not using the weights as in \eqref{eq:wo-attention}.
From \autoref{table:experimental_settings}, we see that the best performance is obtained with UNSUP when we use max pooling and with weighting.

\iffalse
\ref{table:experimental_settings} is ablation result of experimental settings among our proposal method.
The column of Method means meta-sentence embedding methods.
Weight column denotes whether the weights $\alpha_{j}$, when the experimental settings are without attention, the attention score $\alpha_{j}$ is not multiplied.
The Pooling means aggregation method from word-level embeddings to sentence-embedding, and max and mean poolings are formulated in \ref{eq:conc-max} and \ref{eq:avg-max}, and \ref{eq:conc-mean} and \ref{eq:avg-mean}.
Interestingly, the ablation results show the max-pooling for words-to-sentence pooling is better than mean method in all cases.
This is a same trend as \newcite{zhelezniak2018dont,chen-etal-2018-enhancing}, which shows max-pooling is better than mean-pooling.
In addition, the usage of attention module and max pooling method for sentence pooling is the best setting for both unsupervised and supervised methods.
\fi

\iffalse
\begin{table}[t]
\caption{Comparison results with previous works. \protect\cite{poerner-etal-2020-sentence} uses BERT from sBERT \protect\cite{reimers-2019-sentence-bert}, ParaNMT \protect\cite{wieting-17-backtrans}, and USE \protect\cite{cer-etal-2018-universal} for source embeddings. We can only use BERT and USE.}
\label{table:comparison_with_previous}
\begin{tabular}{l|rr}
\hline
Method & Pearson & Spearman  \\
\hline
Poerner et al.: concat & 82.8 & 83.4 \\
Poerner et al.: mean & 82.5 & 83.2 \\
Poerner et al.: svd & 83.4 & 83.8 \\
Poerner et al.: gcca & 83.9 & \textbf{84.4} \\
Poerner et al.: ae & 82.1 & 83.3 \\
\hline
Ours: CONC & 83.7 & 83.9 \\
Ours: AVG & \textbf{84.0} & 84.3 \\
Ours: GCCA & 83.3 & 83.5 \\
\hline
\end{tabular}
\end{table}
\fi

\iffalse
\begin{table}[t]
\caption{Results regarding source pooling method by using BERT and RoBERTa for source embeddings. The column of Pooling is source-pooling method. The sentence-pooling method is max.}
\label{table:pooling_results}
\begin{tabular}{l|rrrrr}
\hline
Method & Pooling & Pearson & Spearman  \\
\hline
SUP & mean\tnote{a} & 83.64 & 83.94 \\
SUP & concat\tnote{a} & 83.38 & 84.27 \\
UNSUP & mean & 84.40 & 85.62 \\
UNSUP & concat\tnote{a} & \textbf{85.33} & \textbf{86.08} \\
\hline
\end{tabular}
\end{table}
\fi

\section{Conclusion}
We propose an unsupervised method to create sentence-level meta-embeddings from multiple contextualised word embeddings.
Our proposed method outperformed several competitive baselines and a previous proposed state-of-the-art sentence-level unsupervised meta-embedding method.
In particular, both weighted linear transformation and pooling operations contributed to the optimal performance of the proposed method.

\section*{Bibliographic References}
\bibliographystyle{lrec2022-bib}

redpajama_set_name: RedPajamaArXiv


I had a similar problem with a different birth control pill, when my doctor found out I started suffering from severe migraines she was upset I didn't tell her sooner because I didn't relate the two, but she took me off of them right away and said I should never take anything with estrogen in it as it puts me at a higher risk for a stroke, so the only think she could give me was a pill that you take around the clock without stopping but they gave me severe, severe acne... so, my last resort was the IUD, I had the Mirena inserted which has 'local' estrogen in it but the doctor assured me it's only local, well after years of not having a migraine I got two since it was put in few months ago. I had one yesterday that came with the blurry vision and all and now I'm really worried... Not sure what to do next.
I had a similar problem with a different birth control pill, when my doctor found out I started suffering from severe migraines she was upset I didn't tell her sooner because I didn't relate the two, but she took me off of them right away and said I should never take anything with estrogen in it as it puts me at a higher risk for a stroke, so the only thing she could give me was a pill that you take around the clock without stopping but they gave me severe, severe acne... so, my last resort was the IUD, I had the Mirena inserted which has 'local' estrogen in it but the doctor assured me it's only local, well after years of not having a migraine I got two since it was put in few months ago. I had one yesterday that came with the blurry vision and all and now I'm really worried... Not sure what to do next.
I have had the asme problems with Birth control. I have been on several different brands for the last 8 years on and off but last fall I started getting crazy headaches. I finally realized that these were the beggingin of some crazy migranes. After 6 months, many prescriptions, many crazy migraines that were making me ill for up to 7 days I switched to a birth control with no estrogen only prgesterin. its not as common and a bit more expensive then the others but its cut my headaches back by bout 50%. Give it a try. Otherwise I just really have to watch my diet, stress levels. Wines and beers, chocolate and coffee are major triggers. Still havent found a permanent cure yet and Ive disocered that my 2 aunts and my granmother suffer the same problem.
I have the same issue. The only downside is,its either deal with nearly daily migraines, or live without birth control. My doctor has had me try several forms including an IUD and an implant (Implanon), which use different forms of hormones than do the pills I was taking. I'm still getting migraines, at least I know its my contreceptive causing them.
I am as unlucky as the rest of you. I am unable to take birth control at all. when I was young it was becuase of a heart condition but now any one I have tried have given me horrible migraines. I stopped the birth control over a year ago. Took six months for my migraines to go away again. But now I only get them in waves every 3-4 months.


Thanks to the success of the Get It Done app and web portal, we receive hundreds of reports from residents and visitors every day. Crews are responding across 372 square miles of our world-class city and, in some cases, problems may take a while to fix. With an average of 20,000 reports being filed each month, we work hard to prioritize the most urgent public safety issues. To download the app, visit sandiego.gov/GetItDone.


text:
Admin-on-Demand
Sewell Group
FM Service Provider
24/05/19 | Sally Wotton
Sewell Group is one of the UK's most successful and well-regarded estates services companies. With a pedigree that stretches back to its founding in 1876, it is recognised today as a 'Sunday Times Top 10 Best Company to Work for Business'. Its services range from large-scale project development and delivery to hard and soft Facilities Management and legal compliance services.
Sewell's client portfolio include schools, health and community facilities and offices, which represent more than 60,000m² of buildings and contain some 33,000 assets – everything from boilers and medical lighting equipment down to individual light switches. The company's ethos is to help its clients get the best value from their estate, while ensuring that it meets all their business and organisational needs.
In 2012, it decided to modernise its CAFM strategy, replacing a legacy system with a new platform that would be more adaptable, resilient and could be developed quickly to meet new challenges and business requirements.
As Director Martin Stead explains, the company approached the CAFM market with a substantial wish-list.
"In a nutshell, we had an ageing product in place," he says. "We weren't comfortable that it was being developed quickly enough and our relationship with the supplier was not strong enough. We needed a step-change.
"We reviewed the market place and we reviewed our own requirements, and sent our wish-list out to a long list of providers. We then narrowed them down to six, based on their answers. Finally, we asked our top two for product
demonstrations. This all took place in March 2012 and in August, we chose FSI's Concept Evolution - with the aim of going live in December!"
Concept™ met Sewell's top two requirements head-on: an intuitive and engaging client interface; and the ability to integrate core functions seamlessly.
"As a business, we're completely focused on our people and our customers," says Martin. "The new system had to be easy to use while providing the right levels of reporting. A lot of CAFM packages out there claim to do very similar things to Concept™, so assessing the market essentially became a comparison of user interfaces and the ways in which the different elements linked together.
"An off-the-shelf product won't give you what you want immediately, but you should be able to develop it to suit your requirements – and that is one of Concept's™ biggest assets, particularly with the way the Workflow module helps you to automate business processes. Our wish-list was extensive and some of the requirements were quite specific. FSI responded by showing how the system could meet them, or had the potential to with a certain amount of tweaking."
A tight deadline drove the project. The licence for Sewell's legacy CAFM system was due to expire in December, giving the company and FSI just four months to import the existing data and implement Concept Evolution. It was completed on target.
"We had been through this process with the previous system so we knew the scale of the challenge – the volume of asset information we were dealing with, exporting it, quality-checking and getting it ready for importing to Concept™," says Martin.
"And because we'd done it before, we were probably more informed than many businesses going into similar projects. That made us quite a demanding customer at times! We would present FSI with a request and they would have to interpret it into something that Concept™ could perform."
Martin and his team made a point of visiting FSI as well as inviting FSI's consultants into the business, helping to ensure that the people on both sides knew each other and establishing a strong cultural fit. It was, says Martin, important to know FSI as a supplier and partner. "A close relationship with your implementation consultant is essential," he adds.
The Business Benefits
Sewell's initial Concept Evolution package included the Help Desk, Asset Management, PPM, Resource Planner, Quotes and Workflow Lite™ modules, as well as Concept Reach™, FSI's light front-end web application which gives non FM-specialists the benefits of accessing and using information in the system. Concept Connect™, FSI's web-enabled management reporting tool, soon followed.
The system was well-received. An extensive training programme helped to drill awareness of its features and benefits, and knowledge of how to use it, down through the business. From an efficiency point of view users were able to use it immediately to plan their work and record what they had done.
"I would describe the system as the hub of our estates business," says Martin. "It helps to control everything that we do, and provides a platform for the team to deliver world class customer service.
"The PPM module brings everything under control and helps us to make sure that we are meeting all our clients' statutory obligations. We also use the Resource and Booking modules to provide some internal services, and the Help Desk to provide aftercare for our construction sites."
One of the main benefits has been Concept's™ role in winning Sewell new business.
"This is the big story of our investment in the system," says Martin. "It went a long way to helping us secure two new projects. Along with our business ethos, Concept™ was a winning feature of our bid."
Sewell's clients have also embraced Concept Reach™ - the number who use the portal to access the system and monitor the status of their contracts, tasks and services doubled within eight months after the system went live.
"Concept™ is a very open system. You can always see what you are doing, and when," says Martin. "It makes the Help Desk more efficient, and increased automation means you spend less time on the phone following up requests."
The strong relationship with FSI has been at the heart of the project's success and is already leading to further developments in Sewell's planned use of the system.
"Sometimes you get the 'A' team selling you the system – and you never see them again," says Martin. "The 'B' team turns up to install it. Not here. Our account manager at FSI has stayed with us throughout the project and we're continuously benefiting from that original consultation as we move into the next phases of the deployment."
These will include implementing the Digital Dashboard and SLA modules, and a scoping exercise for FSI GO™, the mobile platform which brings Concept™ to the tablets and smartphones of engineers and contractors in the field, and enables instant communication and updates across the system. Martin expects FSI GO™ to be rolled out by the end of 2013.
Preparation is essential for any business embarking on a similar project. It is vital to have a clear idea of your business processes so that they can be understood by the consultant and implemented in the best way.
"But the old way is not necessarily the right way," counsels Martin. "Concept Evolution could actually help you to be more flexible in your processes, so keep an open mind.
"A system like this is a big commitment, and that should not be under-estimated. Your CAFM consultant can't do it all for you, so you should dedicate serious time to the project internally and appoint a project manager who can rally the rest of the team.
"Finally, look forward to continuous development. Going live is just the start with Concept™. We just implemented the Help Desk initially, but we've steadily added other modules since, and using Concept Reach™ and Connect™ to give clients and contractors portals has added a new dimension to our services. This is very much an on-going process."
Read more about Muheel Facilities Management
Muheel Facilities Management
Read more about Meggitt
Read more about Interserve Group / NHS Nightingale Hospital North West
Healthcare | FM Service Provider
Interserve Group / NHS Nightingale Hospital North West
+971 (0)4 44 95380 | info@fsime.ae

redpajama_set_name: RedPajamaCommonCrawl


text:
J Infect Dis. 1986 Feb;153(2):291-7.
Concurrent circulation of antigenically distinct strains of respiratory syncytial virus during community outbreaks.
Hendry RM, Talis AL, Godfrey E, Anderson LJ, Fernie BF, McIntosh K.
Respiratory syncytial virus (RSV) is considered to be of a single serotype. Antigenic variants are detectable both by neutralization and monoclonal antibodies and have been divided into two broad categories, groups 1 and 2. Group 2 isolates have been considered to be uncommon. We used indirect immunofluorescence with strain-specific monoclonal antibodies to study RSV isolates from hospitalized infants in the greater Boston area. Of 223 RSV isolates recovered over a five-month period in 1983-1984, 125 (56%) were group 1, 92 (41%) were group 2, and 6 (3%) were of an intermediate character. Among 181 community-acquired RSV isolates, both temporal and geographic clustering was observed: group 1 isolates were common from January through March and predominated in central Boston; group 2 isolates were found principally in February and were acquired in outlying, particularly northern, areas. Strain-specific differences were not found with respect to sex, age, or clinical findings. An analysis of 82 RSV isolates from the 1981-1982 season showed 75 (91%) group 1 isolates and 7 (9%) group 2 isolates. We conclude that at least two antigenically distinct groups of RSV isolates may circulate concurrently in the community and that the prevalence of group 2 isolates appears greater than previously suspected.
10.1093/infdis/153.2.291
Antigens, Viral/immunology*
Cross Infection/epidemiology
Cross Infection/microbiology
Epitopes/immunology
Fluorescent Antibody Technique
Respiratory Syncytial Viruses/classification
Respiratory Syncytial Viruses/immunology*
Respiratory Syncytial Viruses/isolation & purification
Respirovirus Infections/epidemiology
Respirovirus Infections/microbiology*
Epitopes
AI-22665/AI/NIAID NIH HHS/United States
Silverchair Information Systems


Featuring some 310 lots, Christie's Geneva auction of important watches on November 12 is estimated to realise a total of more than SFr.15 million (US$16 million).
KOLKATA: Featuring some 310 lots, Christie's Geneva auction of important watches on November 12 is estimated to realise a total of more than SFr.15 million (US$16 million). The top lot of the sale is an ultra-rare platinum perpetual calendar chronograph, from the collection of Eric Clapton (estimate: SFr2,500,000-4,000,000).
Two other extraordinary and historically important platinum Patek Philippe masterpieces will be offered. These include a unique platinum wristwatch, specially made in 1952 for the legendary collector J.B. Champion (estimate: SFr2,000,000-4,000,000), and the superb platinum chronograph wristwatch, manufactured in 1946 (estimate: SFr1,000,000-1,500,000).
Furthermore, the sale will feature one of the finest private collections of antique enamelled watches made for the Chinese market -- previously unrecorded and unpublished in literature. Christie's is also particularly honoured to present A Gentleman's Pursuit for Excellence, Part I, a phenomenal single-owner collection featuring more than twenty complicated Patek Philippe wrist and pocket watches, to be offered in Geneva at various sales in 2012 and 2013.
Aurel Bacs, international head of Christie's watch department, said in an email: "Over the past thirty years, a combined total of over two million collector's watches have changed hands via the international salerooms, leading the worldwide community of scholars, collectors, specialists, museum curators and dealers to form, in an official/unofficial way, a list of the ten most significant wristwatches, in their opinion. To make it onto this 'all-star list' a wristwatch has to standout by its historical importance, mechanical complexity, unmatched beauty, original condition, incredible rarity and superior provenance. This autumn, we are proud to offer for sale examples which deserve to be placed on that list. These include three extraordinary Patek Philippe platinum wristwatches: the world famous 'J.B. Champion' special order platinum observatory wristwatch, the platinum chronograph wristwatch, one of three platinum examples ever made, and the highly important platinum perpetual calendar chronograph from the collection of Eric Clapton."


text:
Toxic Turf at Beaver Pond?
Beaver Pond has become the center of the synthetic turf debate.
Anna Norton, Writer
Have you ever played on the field at Beaver Pond in Franklin, MA? Many kids have. Little do they know, discarded turf from the field has been decomposing in the pond's wetlands for over two years.
Our Varsity Field Hockey team starts the game off strong at the Beaver Pond field. Picture from FHS SmugMug
Artificial turf is made of recycled car tires that carry chemicals such as carbon black, benzothiazole, benzene, cadmium, heavy metals, and a group of man-made chemicals called PFAS.
These chemicals are toxic to the environment and also happen to be cancerous to humans and animals. In addition to causing cancer, PFAS chemicals also cause low infant birth rates, thyroid hormone dysfunction, and negative effects on the immune system.
Those who have heard about this problem are appalled with the situation. says,
" I had no idea that turf was there and it should be fixed. I'm surprised that nothing has been done about it before!" says Franklin town resident Kaileigh Bercume
Having chemicals pollute Beaver Pond is not only bad for the environment, but violates state and federal guidelines. The concentration of toxic chemicals in the water meet neither current federal nor state health guidelines. The rejected turf has violated wetland protections. Currently, the Franklin Town administrator is waiting to hear from the Department of Environmental Protection on what to do.
Boston Globe Article, Toxic Chemicals are Found in Blades of Artifical Turf by David Abel states that, "The findings have raised concerns about the safety of millions of square feet of artificial turf installed in recent years on public fields and playgrounds across the country."
Varsity Field Hockey plays on their home turf against Mansfield. Picture from FHS SmugMug
Unfortunately, the pollution of Beaver Pond directly affects Franklin High school because our field hockey and lacrosse teams play on the field.
"I didn't hear about it, but now that I do, its upsetting to hear that our town is polluting the area; and I hope in the near future there will be a resolution or cure to this problem so our town and the community inside can stay clean", says JV field hockey player Caroline Gannon.
Due to the impact of the turf on our school, it is our job to raise awareness and clean out the polluted wetlands.
For more information, visit Toxic Chemicals are Found in Blades of Artifical Turf.
Top 5 Fall Activities For New England Weather
Diversity Awareness Club "Sweet & Greet" Recap from Club President
Summer 2022 Movie Roundup
The Annual Met Gala Roast
Student Spotlight: Olivia Reagan
Student Spotlight: Ashley Donnelly
Thank You, Mr. O'Donoghue
Student Spotlight: Dan Gurge


text:
There's Plenty of Room for Development in the Industrial Sector
Capitalizing on intense demand for industrial space, Duke Realty Corp. is on a development roll.
As it stands now, the Indianapolis-based industrial REIT owns and operates about 149 million rentable sq. ft. of industrial properties in 20 major U.S. markets. The portfolio is ballooning quickly, though.
This spring, Duke Realty bumped up its forecast for development starts in 2018 from a range of $500 million to $700 million to a range of $650 million to $850 million. As of early April, $337 million in development projects totaling 4.5 million sq. ft. were already underway.
"As a general rule, we've got development going in most all of our markets right now—some more than others," says Jim Connor, chairman and CEO of Duke Realty.
Connor cites Southern California, South Florida, New Jersey, Atlanta, Chicago and Dallas as some of the top-tier markets where Duke Realty is adding the most space. However, he points out that the REIT has developed more than 2 million sq. ft. of industrial space this year in one second-tier market—Columbus, Ohio.
Across the company's 20 markets, e-commerce players drive anywhere from 25 percent to 40 percent of development and leasing activity for what Connor likes to call "modern logistics buildings." Those buildings serve customers such as Amazon, Home Depot, Target, UPS, Walmart and Wayfair.
In a Q&A with NREI, Connor delves into his outlook for the industrial sector, including the role played by e-commerce, and discusses Duke Realty's development and leasing forecast.
This Q&A has been edited for length, style and clarity.
NREI: How are you feeling about the industrial sector these days?
Jim Connor: I'm feeling pretty good. Business is really, really good. Some of my peers jinx us when they say, "It's as good as it gets" or "It's never been better" or anything like that.
NREI: What underlies that "pretty good" feeling?
Jim Connor: You've got 4.5 percent vacancy nationwide. It's the lowest it's been as long as anybody can remember, and they've been keeping records 25-plus years. We keep predicting every year that supply is finally going to catch up with demand, and it still hasn't happened yet. Based on activity, I don't see that trend changing. When you've got that low of a vacancy rate, it allows us to keep the occupancy rate in our portfolios ridiculously high. Our in-service portfolio is 97.5 percent leased [as of the first quarter of 2018], and with that you get great, strong rent growth numbers.
The challenge for us—and there's always a downside with the upside—is we're really in favor right now, so there's a ton of capital chasing industrial. Even though interest rates have gone up 50 to 60 basis points in the last eight or nine months, we've seen cap rates on industrial compress 25 basis points, and maybe more in some cases. So, there's a lot of competition out there.
NREI: The industrial sector has been really hot lately. How long is that streak going to last?
Jim Connor: Absent some global event that obviously nobody anticipates, I would tell you the market dynamics we're seeing look really good for the next 18 to 24 months. That's with the supply-and-demand equation staying in balance; even if we reach equilibrium with supply and demand, equilibrium is not a bad place, particularly when you reach equilibrium with 4.5 percent vacancy. That's still a landlord's market.
Nobody's slowing down at all. E-commerce continues to be a big, big driver in our business. Those guys are not slowing down. Our traditional customers—retail and consumer products companies—are investing in their supply chains to get products to their customers more quickly and more efficiently. You're seeing a tremendous amount of investment and modernization in that sector to help them compete with the e-commerce business.
NREI: How much more industrial activity can be supported by e-commerce?
Jim Connor: I believe, as do most of the industry experts, that e-commerce is in the very, very early innings—arguably the second or third inning. E-commerce today represents about 9.0 or 9.5 percent of total bricks-and-mortar retail sales. Most experts will tell you that's heading to 20 percent.
So, if it heads to 20 percent, that would tell you there's a lot of runway for industrial. It doesn't mean that all of the e-commerce deals that we've done in the last 10 years, we're going to double that square footage. But it means there's a lot more demand for e-commerce facilities, whether it's for pure e-commerce companies like Amazon or Wayfair, or whether it's this new generation of e-commerce food companies like Blue Apron, or whether it's existing retailers and consumer products companies that are investing in the supply chain to better support their e-commerce efforts. The logistics sector looks very, very bright for the next several years.
NREI: In light of the demand you're foreseeing, are you leaning more toward development or acquisitions? How does that play out in terms of adding supply to meet demand?
Jim Connor: We're kind of blessed. We can do either/or, or in some years we do both. The vast majority of our capital and our time and energy today is going toward new development, both build-to-suits and speculative, because it's much more lucrative for us. We're developing projects that are probably 200 to 250 basis points higher in sustained yields than on the acquisition front. We're very, very focused on the development side.
NREI: What are you projecting for leasing?
Jim Connor: In 2017, we did about 23.7 million sq. ft. of leasing, which was a phenomenal year for us, one of our best ever. In the first quarter of 2018, we did just under 7 million sq. ft. Ours is not a perfectly linear business, but based on first-quarter activity, our expectation is to meet or exceed last year's leasing volume, which would put us in the mid-20 million-sq.-ft. range and would be another phenomenal year. With occupancies so high across all of the markets, until you create some softening in some of the markets—either a pretty dramatic fall-off in demand or a bunch of oversupply—I think you're going to continue to see very, very strong numbers in terms of leasing and rent growth that industrial owners are going to put up.
July 24, 2018 /0 Comments/by cellinopropertymanagement_1nq03s
Tags: commercial real estate, real estate, real estate usa
http://www.cellinopropertymanagement.com/wp-content/uploads/2018/07/theres-plenty-of-room-for-development-in-the-industrial-sector.jpg 3840 5760 cellinopropertymanagement_1nq03s http://www.cellinopropertymanagement.com/wp-content/uploads/2017/08/logo.png cellinopropertymanagement_1nq03s2018-07-24 14:15:532018-07-24 14:15:53There's Plenty of Room for Development in the Industrial Sector
10 Must Read Items for CRE Investors on Jan. 15, 2020
Goldman Sachs' Infrastructure Arm Bets $500M on Data Centers
U.S. 20-City Home Price Index Rises the Most Since 2014
New Residential Investment Corp. Is Selling a $6B Debt Portfolio
Colliers International Acquiring Majority Stake in Harrison Street Rea
Manhattan Hotels Pitch Rooms for Office Space With Travel Frozen
New Office and R&D Complex in San Jose Sells for $47.8M
Amazon's Withdraw from NYC HQ2 Plan Angers Local Brokers
© Copyright - CELLINO PROPERTY MANAGEMENT 2017


Truth!! Behind the Curtains of Skull-dan!
(Japanese: 真相!!スカル団の背後! Truth!! Behind the Curtains of Skull-dan!) is the 19th round of the Sun, Moon, Ultra Sun & Ultra Moon chapter of the Pokémon Adventures manga.
Gladion uses Type: Null to battle the Ultra Beasts, which it does with ease as it was created by the Aether Foundation specifically for that purpose.
Gladion reveals that Lusamine is the one behind the Ultra Beasts incidents and dreams of making a paradise just for her and the Ultra Beasts she loves.
Moon arrives at Po Town and asks Gladion if he's telling the truth.
A Xurkitree appears from an Ultra Wormhole and begins attacking everyone while a Nihilego also appears and drags Guzma with it into another Ultra Wormhole.
The other four Ultra Beasts escape Po Town's barrier and begin causing havoc around Ula'ula Island.
The Captains ask Gladion to help them, but he refuses and heads off.
Sun and Moon stay behind while the Captains head off to rescue any people caught in the danger.
The people of Ula'ula are evacuated to Hokulani Observatory for shelter while Molayne and Sophocles keep watch on them.
Sophocles decides to use an invention of his to keep the Ultra Beasts away from the observatory.
Sun collects a Zygarde Cell.
Moon's Grubbin is revealed to have evolved into a Charjabug.
Sun reveals the reason he needs to collect money is to buy back the island the Aether Foundation stole from his great-grandfather.
Anabel and Looker, members of the International Police, arrive to question Sun and Moon.
This page was last edited on 27 February 2019, at 23:19.


A poorly-known species in the British Isles, it is a scarce and local species in southern and south-eastern England, with a few records elsewhere in the country north to Lancashire. It seems probable that some of these are migrants.
The adult resembles C. repandana, but rather redder and with more prominent scale tufts.
It flies between July and early September, and can be attracted to light.
The larvae are reputed to feed on oak (Quercus), amongst spun leaves, though this stage has rarely, if ever, been observed in Britain.


Hand set mosaics are individually created by artisans in the USA. Round plant stand is 18″D x 22″H. Choice of mosaic and/or tumbled stone top. Base is powder coated wrought iron. This piece is similar to the 18″ side table and can be used as a side table. Contact Casual Living for more information.
Bradford Adjustable Chaise Lounge from Woodard maintains the classic diamond mesh pattern of familiar wrought iron yet has an updated style with a curvy arm ending in a swirl motif. "Wears like iron" is an apt phrase for all of the Woodard wrought iron collections; they will last through multiple generations of use.
Briarwood Chaise Lounge from Woodard features the daisy flower pattern that has come to be recognized as synonymous with Woodard. Rear rubber wheels allow for easy maneuverability. Multi position back adjustment provides the perfect fit for all, whether chatting, reading, or napping.
Briarwood High-Back Dining Armchair from Woodard maintains the classic diamond mesh pattern of familiar wrought iron with the classic daisy pattern featured in the back rail. "Wears like iron" is an apt phrase for all of the Woodard wrought iron collections; they will last through multiple generations of use.
Briarwood High-Back Dining Side Chair from Woodard maintains the classic diamond mesh pattern of familiar wrought iron with the classic daisy pattern featured in the back rail. Without arms, the side chair is useful around a table when space is at a premium. "Wears like iron" is an apt phrase for all of the Woodard wrought iron collections; they will last through multiple generations of use.
Briarwood High-Back Lounge Chair from Woodard reflects the classic iron mesh design that was the staple of outdoor furniture for much of the twentieth century. Featuring a floral pattern detail in the back and arm rests, Briarwood has faithfully served both residential and contract markets for many years.
Briarwood High-Back Love Seat from Woodard is a familiar favorite which has endured through generations. Featuring the well-known daisy flower design and diamond patterned mesh, the Briarwood Collection is a classic.


text:
Ikhmas Jaya IPO oversubscribed 8.16 times
Levina Lim
July 10, 2015 15:39 pm +08
KUALA LUMPUR (July 10): The initial public offer (IPO) of Ikhmas Jaya Group Bhd, which is en route to list on the Main Market of Bursa Malaysia on July 27, was oversubscribed 8.16 times.
The Malaysian Issuing House Sdn Bhd said a total of 11,434 applications for 238.19 million public issue shares were received from the public for a total of 26 million public issue shares available for public subscription.
The bumiputera category, of which half or 13 million of the public issue shares have been set aside for, were oversubscribed 6.51 times, with a total of 5,564 applications for 97.58 million public issue shares received.
The private placement of 78 million public issue shares to selected investors has also been placed out.
"All notices of allotment for these IPO shares will be mailed to successful applicants on or before July 24," said the local issuing house in a statement today.
Ikhmas Jaya's IPO entails a public issue of 126 million new shares, of which 26 million shares were available for application by the public.
Another 26 million shares are reserved for eligible directors, employees and business associates of Ikhmas Jaya Group; and 74 million were available for application by way of private placement to selected investors.


text:
Landlocked Laos might be quieter than Southeast Asian neighbors like Vietnam and Thailand, but 2018 could transform the country into the region's next hot spot. Wattay International Airport, in the capital of Vientiane, is set to complete a terminal expansion to accommodate more international links next year, but the biggest changes are foot in the UNESCO World Heritage–inscribed town of Luang Prabang, in central Laos. This serene riverside spot lures travelers to its golden Buddhist temples, French-colonial architecture, hiking trails, nearby elephant sanctuaries — and now, glamorous new digs. Last year, the design-forward Azerai opened inside a century-old French-colonial building that was formerly an officer's quarters. The debut concept from Aman Resorts founder Adrian Zecha has airy, light wood interiors that nod to local culture in their use of batik textiles and Laotian artwork. There's also an 80-foot swimming pool in a tree-shaded central courtyard. The upcoming Rosewood Luang Prabang aims to be a destination in itself. Opening in mid 2018, this highly anticipated resort will feature pavilion-style villas, luxury tents, and a spa that seems to float above the jungle. —Kate Springer
Flight Oakland - Los Angeles (OAK - LAX) $55+ Flight San Francisco - Los Angeles (SFO - LAX) $91+ Flight San José - Los Angeles (SJC - LAX) $91+ Flight Houston - Los Angeles (HOU - LAX) $106+ Flight Houston - Los Angeles (IAH - LAX) $106+ Flight Dallas - Los Angeles (DFW - LAX) $123+ Flight Chicago - Los Angeles (ORD - LAX) $125+ Flight Seattle - Los Angeles (SEA - LAX) $125+ Flight Portland - Los Angeles (PDX - LAX) $137+ Flight New York - Los Angeles (LGA - LAX) $140+ Flight Orlando - Los Angeles (MCO - LAX) $163+ Flight Minneapolis - Los Angeles (MSP - LAX) $165+ Flight Detroit - Los Angeles (DTW - LAX) $166+ Flight Washington - Los Angeles (BWI - LAX) $173+ Flight Atlanta - Los Angeles (ATL - LAX) $178+ Flight Fort Lauderdale - Los Angeles (FLL - LAX) $178+ Flight Philadelphia - Los Angeles (PHL - LAX) $187+ Flight Newark - Los Angeles (EWR - LAX) $191+ Flight Boston - Los Angeles (BOS - LAX) $197+ Flight Raleigh - Los Angeles (RDU - LAX) $199+ Flight Chicago - Los Angeles (MDW - LAX) $201+ Flight Washington - Los Angeles (DCA - LAX) $204+
Known as the Tibet of Japan, the remote Iya Valley is tucked away in the mountainous interior of Shikoku, the least visited of the country's four main islands. Tourism to the region took a leap forward when it hosted the World Rafting Championship in 2017 — putting its turquoise Class Four rapids firmly on the adventure-travel map. New ziplines and hiking trails are sprouting up in the canyons, while upgrades have been made to accommodations in the area's traditional thatched-roof farmhouses, or minkas. Chiiori House is the most luxurious; the property also maintains an excellent portfolio of more affordable options. —Adam Harney Graham
In an interview with Exclaim!, Nick Thorburn explained that the overhauled lineup was a product of his desire to try new things. "That's important to me ― to be able to constantly try new things. That's why this record's largely informed by electronic stuff like drum machines, sequencing, and programming, which really scales it back from the last record." [2] Thompson once again left the band prior to the summer 2010 tour, and was again replaced by Aaron Harris.
Oceanic islands are islands that do not sit on continental shelves. The vast majority are volcanic in origin, such as Saint Helena in the South Atlantic Ocean.[11] The few oceanic islands that are not volcanic are tectonic in origin and arise where plate movements have lifted up the ocean floor above the surface. Examples are Saint Peter and Paul Rocks in the Atlantic Ocean and Macquarie Island in the Pacific.
The Big Easy turns the Big 3-0-0 this year, and in this city where the good times roll, the parties will be epic — think citywide art shows, supersized Mardi Gras parades, and a festival of lights using landmark buildings as backdrops. Thirteen years after Hurricane Katrina, there is much to celebrate: The Central Business District, once a dead zone after dark, now crackles at all hours thanks to four new hotels (the Ace, Troubadour, Catahoula, and NOPSI), each with its own rooftop bar. The neighborhood is also home to new restaurants like Maypop — a Vietnamese-Creole joint from acclaimed chef Michael Gulotta. Another area coming to life is the three-mile riverfront, where a renovated Spanish Plaza will reopen this spring. More riverfront updates, including a new Four Seasons Hotel, will roll out in the next few years. Toast the tricentennial at the Sazerac House, a French Quarter museum dedicated to the official cocktail of New Orleans, opening later this year. —Allison Entrekin
Over the past few years, South Korea's Gangwon Province has shed its sleepy past and come into its own as a prime winter-sports destination — a transformation that will take center stage during the Pyeongchang Winter Olympics (February 9–25). Ahead of the big event, the region will debut a new high-speed rail line that will whisk travelers from Incheon International Airport to the resort town in just 70 minutes, making it easier to access the Taebaek Mountains' panoramic pistes and tourist-friendly attractions. Powder hounds will want to lodge at the InterContinental Alpensia Pyeongchang Resort, which commands a prime location at the foot of the Alpensia ski slopes, steps from two Olympic-class runs, and is one of a handful of hotels built for the games. Break from all the outdoor action at the Ocean 700 indoor water park, complete with wave pools and tubing rides. —Talia Avakian
In 1998, the Travelers Group merged with Citicorp to form Citigroup.[10] However, the synergies between the banking and insurance arms of the company did not work as well as planned, so Citigroup spun off Travelers Property and Casualty into a subsidiary company in 2002,[12] although it kept the red umbrella logo. Three years later, Citigroup sold Travelers Life & Annuity to MetLife.[13] In 2003, Travelers bought renewal rights for Royal & SunAlliance Personal Insurance and Commercial businesses.[14][15]
Flight Edmonton - Vancouver (YEG - YVR) C$ 99+ Flight Calgary - Vancouver (YYC - YVR) C$ 119+ Flight Cranbrook - Vancouver (YXC - YVR) C$ 267+ Flight Nanaimo - Vancouver (YCD - YVR) C$ 270+ Flight Kelowna - Vancouver (YLW - YVR) C$ 275+ Flight London - Vancouver (YXU - YVR) C$ 276+ Flight Winnipeg - Vancouver (YWG - YVR) C$ 278+ Flight Toronto - Vancouver (YHM - YVR) C$ 298+ Flight Penticton - Vancouver (YYF - YVR) C$ 299+ Flight Prince George - Vancouver (YXS - YVR) C$ 334+ Flight Toronto - Vancouver (YYZ - YVR) C$ 342+
As Canada's token middle child, Edmonton has long gotten short shrift amid its glitzier sisters (we're looking at you, Toronto and Vancouver). But no longer — food and museum news is casting a spotlight on Alberta's capital city. After stints at Noma in Copenhagen and Manhattan's Daniel, chef Scott Downey returned to his hometown to open the Butternut Tree in September, with a focus on indigenous foods — grilled bannock with wild mushrooms and winged kelp; bison served with Saskatoon berry jus; maple-butter cake with black-currant jam. We're waiting to make our dinner reservations until the new Royal Alberta Museum opens its doors. Designed by Dialog architects on the site of a former Canada Post distribution center, the 419,000-square-foot space will include Ice Age horse fossils and a dig pit for children. To experience Edmonton's indie side, stay at Crash Hotel, an homage to the Ace, which opened last winter. Its themed Hi-Fi room walls are lined with vintage speakers, and hangover pills are at the ready in the mini-bar. —Kathryn O'Shea-Evans
Resuming their hosts' lives, the team suffer paranoia and hallucinations of their future pre-Travel lives, side effects of the antitoxin. MacLaren's coworkers hold a surprise party for his 15th anniversary with the FBI. Kat confronts him about lying to Forbes; he seduces her, but she is still certain he is cheating. During their lovemaking, MacLaren hallucinates that Kat is a woman with very short hair, with Carly's number (3465) tattooed on her neck. After Rene attempts to shoplift some clothes with Trevor and another friend, Trevor confronts the two outside, in an alley. During a fight between Trevor and the other guy, Trevor possibly experiences a slippage of time, where his friend is able to connect with a punch. This will later develop into Trevor's diagnosis of temporal aphasia. Their intimacy growing, David reluctantly helps Marcy perform surgery on herself to prevent seizures. Carly rebuffs a representative of Child Protective Services. Trevor reins in his host's delinquency, and he and Renee meditate. Ray takes Philip to a twelve-step meeting for drug addicts and gets him a pet turtle to care for. Trevor and Philip learn the team has a new mission.
A third type of volcanic oceanic island is formed over volcanic hotspots. A hotspot is more or less stationary relative to the moving tectonic plate above it, so a chain of islands results as the plate drifts. Over long periods of time, this type of island is eventually "drowned" by isostatic adjustment and eroded, becoming a seamount. Plate movement across a hot-spot produces a line of islands oriented in the direction of the plate movement. An example is the Hawaiian Islands, from Hawaii to Kure, which continue beneath the sea surface in a more northerly direction as the Emperor Seamounts. Another chain with similar orientation is the Tuamotu Archipelago; its older, northerly trend is the Line Islands. The southernmost chain is the Austral Islands, with its northerly trending part the atolls in the nation of Tuvalu. Tristan da Cunha is an example of a hotspot volcano in the Atlantic Ocean. Another hotspot in the Atlantic is the island of Surtsey, which was formed in 1963.
HI Vancouver Central C$ 47+ Budget Inn Patricia Hotel C$ 57+ Barclay Hotel C$ 65+ Kingston Hotel C$ 81+ The Empire Landmark Hotel C$ 83+ Howard Johnson by Wyndham Vancouver Downtown C$ 95+ Ramada by Wyndham Vancouver Downtown C$ 105+ Gec Granville Suites Downtown C$ 105+ Victorian Hotel C$ 115+ Coast Vancouver Airport Hotel C$ 121+ Best Western PLUS Sands C$ 124+
Those with a predilection for high-thread-count sheets will soon be able to luxuriate at the Grand Bohemian Hotel, a ritzy boutique property with Carolina charm. In the meantime, discerning visitors can bunk at the swank Westin Poinsett, a historic hotel that was rescued from the wrecking ball in the late '90s, laying the groundwork for Greenville's great Southern revival. —Rachel Tepper Paley
In a post-apocalyptic future, thousands of special operatives are tasked with preventing the collapse of society. These operatives, known as "travelers", have their consciousnesses sent back in time and transferred into the "host" body of present-day individuals who would otherwise be moments from death, to minimize unexpected impact on the time line. The transfer requires the exact location of the target, made possible by twenty-first century smart phones and GPS providing time, elevation, latitude, and longitude (TELL) coordinates that are archived for use in the future. Trained using social media and public records concerning their hosts, each traveler must maintain the host's pre-existing life as cover for the rest of their lives, while carrying out missions in teams of five. These missions are dictated by the Director in the future who monitors the time line, and are aimed at saving the world from a series of catastrophic events. One method by which the Director communicates with travelers is via pre-pubescent children used as messengers; unlike adults, any child can safely be taken over for a few minutes by the Director and then released from control without risk of killing them. All travelers are required to behave according to certain protocols to protect the time line:
Located at the tip of the Baja Peninsula, the two small colonial towns of Cabo San Lucas and San José del Cabo have become the hottest vacation destinations in Mexico in recent years. With wide, pristine beaches, lively nightclubs, glam resorts, and a farm-to-table food scene, the oasis of Los Cabos is drawing tourists in record numbers. As a result, the hotel scene is booming, with a clutch of new developments and renovations completed this year along the Tourist Corridor, including the sleek Chileno Bay, an Auberge Resort, and a stunning beachfront Solaz resort.


A 39 year old woman came to us for help with fertility concurrent with plans to begin IVF. She and her husband had been trying to conceive for four years, but she had experienced two miscarriages at 7 and 9 weeks respectively. At the time of her initial consultation, she was in the ovulation tracking phase of IVF. Her AMH levels were good, and all of her tests were normal, though her husband had poor sperm morphology. She was experiencing some anxiety as a result of her experiences. We began weekly treatments for both her and her husband using acupuncture, moxibustion, and herbs. He was treated for sperm morphology, and she was treated to improve circulation to the uterus and strengthen her body's ability to hold a pregnancy. She fell pregnant naturally within a few weeks of beginning treatment–without the use of IVF. We treated her weekly through her first trimester. She is now in the early days of her second trimester and scans are showing a healthy baby.
A 34 year old mother of two came to us after trying to conceive a third child for 6 months. She had used the contraceptive pill for 2 years after giving birth to her second child, and her periods had not resumed to a normal schedule after discontinuing its use. She had used IVF assistance for ovulation induction and been able to fall pregnant, but she had miscarried three times, each at 4 weeks. We treated her weekly to restore her period and promote fertility. After 5 weeks of treatment, her period resumed. She decided to begin IVF and continued with weekly acupuncture treatment. Two months into her treatment, she took the pill for 10 days to prepare for IVF. When her period did not come at the end of the 10 days, she discovered that she had fallen pregnant naturally. She experienced some spotting at 7-8 weeks gestation which stopped after 2 weeks. Scans have shown a normal and healthy pregnancy to date. She is now in her second trimester. Specialists are still surprised as they did not think that she would be able to fall pregnant naturally due to her very irregular period.
A 36 year old woman with a history of stage 2-3 endometriosis came to us for natural fertility treatment after trying to conceive for close to 2 years. She had consulted an IVF specialist, and all tests came back with normal results. We commenced treatment with acupuncture, herbs, and moxibustion to improve and normalize her blood circulation, especially to her uterus. Within one month of treatment, she had fallen pregnant naturally. We continued to treat her to preserve the pregnancy, but the fetus was not viable, and she miscarried at 9 weeks. She continued to come for treatment, and in three months' time, she fell pregnant again. This time, her body and the baby were stronger, and she was able to carry the baby to term. She and her partner decided to try for a second child, and 10 months after her first child was born, she conceived again. She continued treatment through her second pregnancy and gave birth to another healthy child.
A 32 year old mother-of-one came to us for help after having a large number of miscarriages in close succession. She had no problem falling pregnant, but at the time of the initial consult, she had experienced 5 miscarriages in the past year alone with others in the year previous. There was no known reason for this problem. We treated her weekly with acupuncture, herbs, and moxibustion to strengthen her body's ability to hold a pregnancy. We advised that she receive treatment for three months before attempting to fall pregnant again as this is the normal time it takes to create a good egg; however, she fell pregnant within the first two weeks of acupuncture treatment. She came weekly for treatment during her first trimester, and we focused heavily on maintaining her pregnancy. Treatment was successful, and she is now approaching her third trimester without any complications.
Introducing our February 2018 special!


text:
<?php

namespace App;

use Illuminate\Notifications\Notifiable;
use Illuminate\Foundation\Auth\User as Authenticatable;
use Spatie\Permission\Traits\HasRoles;
use OwenIt\Auditing\Auditable;
use OwenIt\Auditing\Contracts\Auditable as AuditableContract;
use Illuminate\Database\Eloquent\SoftDeletes;

class User extends Authenticatable implements AuditableContract //Auditable
{
	use Notifiable, HasRoles, Auditable, SoftDeletes;

	public $incrementing = false; //Necesario para usar id alfanumerico
	public $timestamps = true; //usa campos timestamp create_at, updated_at, deleted_at
	protected $dates = ['deleted_at']; //Soft Delete
	protected $table = 'users'; //nombre de la tabla
	protected $primaryKey = 'id'; //llave primaria

    /**
     * The attributes that are mass assignable.
     *
     * @var array
     */
    protected $fillable = [
    		'email',
            'password',
            'personas_idpersonas',
            'firma',
            // oAuth
            'avatar',
            'provider_id',
            'provider',
            'access_token'
    ];

    /**
     * Campos que deben estar ocultos en los listados
     *
     * @var array
     */
    protected $hidden = [
        'password', 'remember_token',
    ];

    public function posts()
    {
        return $this->hasMany(Post::class);
    }

    public static function boot()
    {
    	parent::boot();

    	static::creating(function($table)
    	{
    		$table->id = str_random(36); //crear id ramdom de tipo char
    	});
    }
}

redpajama_set_name: RedPajamaGithub


text:
Q: Проверка поля на изображение в Laravel 5 Стоит правило на проверка что файл должен быть изображение
class CommentStoreRequest extends FormRequest
{
    public function rules()
    {
        return [
            'bot' => 'accepted',
            'message' => 'accepted',
            'product_id' => 'integer|nullable',
            'rate' => 'integer',
            'name' => ['required','string','min:2','max:255','regex:/[А-Яа-яЁё]/u'],
            'city' => 'string|min:3|max:125',
            'description' => 'string|min:2|max:2048',
            'user_id' => 'integer|exists:users,id|nullable',
            'created_at' => 'date|date_format:Y-m-d H:i:s|nullable',
            'file_1' => 'image',
            'file_2' => 'image',
            'file_3' => 'image',
        ];
    }
}

class CommentController extends BaseController
{
 final public function storeProduct(
        CommentStoreRequest $request,
        ProductRepository $productRepository
    )
    {
$comment = Comment::create($request->all());
            if( $request->file('file_1')) {
                $comment->addMediaFromRequest('file_1')->toMediaCollection('comment');
            }
            if( $request->file('file_2')) {
                $comment->addMediaFromRequest('file_2')->toMediaCollection('comment');
            }
            if( $request->file('file_3')) {
                $comment->addMediaFromRequest('file_3')->toMediaCollection('comment');
            }

            $product->comments()->save($comment);
     }
}

Но при обработке формы возвращается ошибка
message: "The given data was invalid.",…}
errors: {file_1: ["Поле file 1 должно быть изображением."], file_2: ["Поле file 2 должно быть изображением."]}
file_1: ["Поле file 1 должно быть изображением."]
0: "Поле file 1 должно быть изображением."
file_2: ["Поле file 2 должно быть изображением."]
0: "Поле file 2 должно быть изображением."
message: "The given data was invalid."

Загружаю jpeg картинки
Что может быть не так ?

redpajama_set_name: RedPajamaStackExchange