text:
\section{Introduction}

\setlength{\intextsep}{0pt}%
\setlength{\columnsep}{0.1in}%
\begin{wrapfigure}{r}{1.6in} \label{fig: illust}
\vspace{-0.06in}
\centering
\includegraphics[width=1.6in]{./imgs/top1.eps}
\includegraphics[width=1.6in]{./imgs/top2.eps}
\vspace{-0.2in}
\caption{Neither pair of texts has common words. WMD can choose a similar sentence appropriately, whereas the BOW distance cannot distinguish these cases.}
\vspace{-0.1in}
\end{wrapfigure}

The optimal transport (OT) distance is an effective tool for comparing probabilistic distributions. Applications of OT include image processing \cite{ni2009local, rabin2011wasserstein, de2012blue}, natural language processing (NLP) \cite{kusner2015word, rolet2016fast}, biology \cite{schiebinger2019optimal, lozupone2005unifrac, evans2012phylogenetic}, and generative models \cite{arjovsky2017wasserstein, salimans2018improving}.

A prominent OT application is the word mover's distance (WMD) \cite{kusner2015word} for document comparison. WMD treats a document as a probabilistic distribution of words, defines the underlying word geometry using pre-trained word embeddings, and computes the distance using the optimal transport distance between two word distributions (i.e., documents). WMD is preferable because it considers the underlying geometry. For example, bag-of-words (BOW) will conclude that two documents are dissimilar if they have no common words, whereas WMD will determine that they are similar if the words are semantically similar (even if they are not exactly the same), as illustrated in Figure \ref{fig: illust}.

WMD has been widely used in NLP owing to this preferred property. For example, \citet{kusner2015word} and others \cite{huang2016supervised, li2019classifying} used WMD for document classification, \citet{wu2018word} used WMD for computing document embeddings, \citet{xu2018distilled} used WMD for topic modeling, \citet{kilickaya2017reevaluating} and others \cite{clark2019sentence, zhao2019mover, zhao2020limitations, wang2020diversity, gao2020supert, lu2019cot, chen2020reinforcement} used WMD for evaluating text generation. Many extensions have been proposed including supervised \cite{huang2016supervised, takezawa2021supervised} and fast \cite{le2019tree, backurs2020scalable, genevay2016stochastic, dong2020study, sato2020fast} variants. WMD is one of the fundamental tools used in NLP, and understanding the deep mechanism of WMD is crucial for further applications. \textbf{The main goal of this study is dissecting the mechanism of WMD}.

The most fundamental task of WMD is document classification. The original study on WMD \cite{kusner2015word} conducted extensive experiments using kNN classifiers. Figure \ref{fig: orig} shows the classification errors reported in \cite{kusner2015word}. This figure clearly shows that WMD is superior to BOW and TF-IDF.

Figure \ref{fig: orig} is surprising in the following senses. First, WMD outperforms the classical baselines by excessively large margins. BOW and TF-IDF have long been recognized as effective tools for document classification. Although it is reasonable for WMD to outperform them, the improvements are surprisingly large. In particular, the performance is ten times better on the classic dataset and five times better on the bbcsport dataset. Such results are excessively impressive. Second, although TF-IDF is known to be more effective than BOW, it performs worse than BOW on the ohsumed, reuters, and amazon datasets. In fact, the number of misclassification doubles on the reuters datasets.

\begin{figure}[t]
\begin{center}
\includegraphics[width=\hsize]{./imgs/original_eval.eps}
\end{center}
\vspace{-0.1in}
\caption{kNN classification errors reported in the original study on WMD \cite{kusner2015word} (Figure 3 in the original paper). Lower is better. WMD outperformed the naive baselines by significant margins.}
\vspace{-0.1in}
\label{fig: orig}
\end{figure}

In this paper, we point out the possibility that the evaluations conducted in the original study on WMD \cite{kusner2015word} could be misleading. In fact, the main improvements of WMD were due to a subtle preprocessing. Using the same normalization, WMD is comparable to BOW and TF-IDF, or WMD achieves improvements of only two to eight percent at the price of heavy computations. Subsequently, We investigate the cause of these results and find that WMD is more similar to BOW than we previously believed owing to the curse of dimensionality.

The contributions of this paper are summarized as follows.

\begin{itemize}
    \item We point out that the performance of WMD is not as high as we previously believed and comparable to classical baselines in document classification with the same normalization.
    \item We introduce an analogy between WMD and L1-normalized BOW and find that WMD resembles BOW owing to the curse of dimensionality.
    \item We point out several confusing aspects in the evaluations conducted in the original study on WMD. We suspect that many readers and researchers are unaware of these issues. Clarifying them is crucial for a solid evaluation and analysis in this field\end{itemize}

\uline{\textbf{Reproducibility.}} Code is available in \url{https://github.com/joisino/reeval-wmd}. It contains a script to download datasets and pre-computed results, algorithm implementations, and evaluation code.

\section{Related Work}

\uline{\textbf{Word Mover's Distance (WMD) and Optimal Transport (OT) in NLP.}}
WMD \cite{kusner2015word} is one of the most thriving applications of OT. WMD can take the underlying word geometry into account and inherit many elegant theoretical properties from OT. The success of WMD has facilitated many applications of OT in NLP. EmbDist \cite{kobayashi2015summarization} is a method concurrent with WMD. It also regards a document as a distribution of word embeddings but uses greedy matching instead of OT. \citet{kumer2017earth} apply WMD to hidden representations of words instead of raw word embeddings. \citet{yurochkin2019hierarchical} consider a document as a distribution of topics and compute the document similarities using OT. They also use the OT distance for computing the ground distance of the topics. \citet{melis2018structured} proposed a structured OT and applied it to a document comparison to take the positional consistency into account. \citet{singh2020context} consider a word as a probabilistic distribution of surrounding words and compute the similarity of the words using the OT distance of the distributions. \citet{muzellec2018generalizing} and others \cite{deudon2018learning, frogner2019learning, sun2018gaussian} embed words or sentences into distributions instead of vectors and compute the distance between embeddings using OT. \citet{chen2019improving} and others \cite{li2020improving, chen2020sequence} regularize the text generation models based on the OT distance between the generated texts and the ground truth texts to improve the generation. Nested Wasserstein \cite{zhang2020nested} compares the distributions of sequences and is successfully used in imitation learning for text generation. \citet{lei2019discrete} use WMD to generate paraphrase texts for creating adversarial examples. \citet{zhang2020semantic} introduced partial OT to drop meaningless words. \citet{michel2017does} use a Gromov Wasserstein-like distance instead of the standard OT to compare documents. \citet{zhang2016building} and others \cite{zhang2017bilingual, zhang2017earth, grave2019unsupervised, dou2021word} use OT to align word embeddings of different languages. \citet{trapp2017retrieving} use WMD to compare compositional documents by weighting each document. \citet{kilickaya2017reevaluating} and others \cite{clark2019sentence, zhao2019mover, zhao2020limitations, wang2020diversity, gao2020supert, lu2019cot, chen2020reinforcement} used WMD for evaluating the text generation. BERTScore \cite{zhang2020bertscore} is a relevant method, but it uses greedy matching instead of OT. To summarize, OT and WMD have been used in many NLP tasks. It is important to understand the underlying mechanism of WMD for further advancements in this field.

\uline{\textbf{Re-evaluation of Existing Methods.}}
\citet{armstrong2009improvements} found that, although many studies have claimed statistically significant improvements against the baselines, most have used excessively weak baselines, and the performance did not improve from the classical baselines. \citet{dacrema2019are} recently found that many of the deep learning-based recommender systems are extremely difficult to reproduce, and for the methods whose results authors could reproduce, the performance was not as high as people had believed, and the deep approaches were actually comparable to classical baselines with an appropriate hyperparameter tuning. Their paper was awarded a best paper prize at RecSys 2019. Similar observations have also been made in sentence embeddings \cite{arora2017simple, shen2018baseline}, session-based recommendations \cite{ludewig2019performance}, and graph neural networks \cite{errica2020fair} as well.
In general, science communication suffers from publication and confirmation biases. The importance of reproducing existing experiments by third-party groups has been widely recognized in science \cite{lin2018neural, sculley2018winner, munafo2017manifesto, collins2014policy, goodman2016does}.

\section{Backgrounds}

\subsection{Problem Formulation} In this paper, we consider document classification tasks. Each document is represented by a bag-of-word vector ${\boldsymbol{x}} \in \mathbb{R}^m$, where $m$ is the number of unique words in the dataset. The $i$-th component of ${\boldsymbol{x}}$ represents the number of occurrences of the $i$-th word in the document.
We focus on the kNN classification following the original paper. The kNN classification gathers $k$ samples of the smallest distances (with respect to a certain distance) to a test sample from the training dataset and classifies the sample to the majority class of the gathered labels. The design of the distance function is crucial for the performance of kNN classification.

\subsection{Word Mover's Distance (WMD)}

WMD provides an effective distance function utilizing pre-trained word embeddings. Let ${\boldsymbol{z}}_i$ be the embedding of the $i$-th word. To utilize OT, WMD first regards a document as a discrete probabilistic distribution of words by normalizing the bag-of-word vector:
\begin{align}
    {\boldsymbol{n}}_{\text{L1}}({\boldsymbol{x}}) = {\boldsymbol{x}} / \sum_i {\boldsymbol{x}}_i, \label{eq: normalization}
\end{align}
WMD defines the cost matrix ${\boldsymbol{C}} \in \mathbb{R}^m \times \mathbb{R}^m$ as the distance of the embeddings, i.e., ${\boldsymbol{C}}_{ij} = \|{\boldsymbol{z}}_i - {\boldsymbol{z}}_j\|_2$. The distance between the two documents ${\boldsymbol{x}}$ and ${\boldsymbol{x}}'$ is the optimal value of the following problem:
\begin{align}
    &\minimize_{{\boldsymbol{P}} \in \mathbb{R}^{m \times m}} \quad \sum_{ij} {\boldsymbol{C}}_{ij} {\boldsymbol{P}}_{ij} \label{eq: wmd} \\
    &\text{s.t.} \qquad \qquad {\boldsymbol{P}}_{ij} \ge 0, {\boldsymbol{P}} \mathbbm{1} = {\boldsymbol{n}}_{\text{L1}}({\boldsymbol{x}}), {\boldsymbol{P}}^\top \mathbbm{1} = {\boldsymbol{n}}_{\text{L1}}({\boldsymbol{x}}'), \notag
\end{align}
where ${\boldsymbol{P}}^\top$ denotes the transpose of ${\boldsymbol{P}}$, $\mathbbm{1} \in \mathbb{R}^m$ is the vector of ones. Intuitively, ${\boldsymbol{P}}_{ij}$ represents the amount of word $i$ that is transported to word $j$. WMD is defined as the minimum total distance to convert one document to another document. Let $\text{OT}({\boldsymbol{x}}, {\boldsymbol{x}}', {\boldsymbol{C}}) \in \mathbb{R}$ be the optimal value of eq. (\ref{eq: wmd}).

\subsection{Experimental Setups} \label{sec: setup}

\begin{table}[t]
    \centering
    \caption{Dataset statistics.}
    \scalebox{0.78}{
    \begin{tabular}{ccccccccc} \toprule
         & bbcsport & twitter & recipe & ohsumed & classic & reuters & amazon & 20news \\ \midrule
        Number of documents & 737 & 3108 & 4370 & 9152 & 7093 & 7674 & 8000 & 18821 \\
        Number of training documents & 517 & 2176 & 3059 & 3999 & 4965 & 5485 & 5600 & 11293 \\
        Number of test documents & 220 & 932 & 1311 & 5153 & 2128 & 2189 & 2400 & 7528 \\
        Size of the vocabulary & 13243 & 6344 & 5708 & 31789 & 24277 & 22425 & 42063 & 29671 \\
        Unique words in a document & 116.5 & 9.9 & 48.3 & 60.2 & 38.7 & 36.0 & 44.6 & 69.3 \\
        Number of classes & 5 & 3 & 15 & 10 & 4 & 8 & 4 & 20 \\
        Split type & five-fold & five-fold & five-fold & one-fold & five-fold & one-fold & five-fold & one-fold \\ \midrule
        Duplicate pairs & 15 & 976 & 48 & 1873 & 2588 & 143 & 159 & 59 \\
        Duplicate samples & 30 & 474 & 66 & 3116 & 600 & 197 & 285 & 88 \\ \bottomrule
    \end{tabular}
    }
    \label{tab: statistics}
\end{table}

\uline{\textbf{Datasets.}} We use the same datasets \cite{greene2006practical,sanders2011sanders,joachims1998text,sebastiani2002machine,lang1995news} as in the original paper \cite{kusner2015word} and use the same train/test splits as in the original paper. Three of the eight datasets have the standard train/test splits (e.g., based on timestamps), and the other datasets do not. Thus, the original paper used five random splits for such datasets. We refer to the former type as one-fold datasets and the latter as five-fold datasets. Table \ref{tab: statistics} shows the statistics.

We should point out that there are many duplicate samples in the datasets used in the original study. The last two rows in Table \ref{eq: wmd} report the numbers of duplicate samples. Some of these samples cross the train/test splits, and some of them have different labels despite having the same contents. This causes problems in the evaluations. If the pairs have different labels, it is impossible to classify both of them correctly. If the pairs have the same label, a kNN classifier places more emphasis on this class for no reason. We discuss this issue in more detail in Appendix \ref{sec: duplicate}.

In the following, we first use the same dataset as the WMD paper to highlight the essential differences with the original evaluation. We then evaluate using clean datasets to further corroborate the findings.

\uline{\textbf{Embeddings.}} We use the same word embeddings as in the original paper \cite{kusner2015word}. Namely, we use the $300$-dimentional word2vec embeddings trained on the Google News corpus. We found that the official code of WMD \footnote{\url{https://github.com/mkusner/wmd}} normalized the embeddings by the L2 norm, although this was not explicitly stated in the paper. Following this setting, we normalize the embeddings using the L2 norm. Note that it was reported that normalizing the word embedding in WMD improves the performance \cite{yokoi2020word}.

\uline{\textbf{Preprocessing.}} For WMD, we use the same preprocessing as in the original paper \cite{kusner2015word}, whereas for BOW and TF-IDF, we use a different preprocessing to clearly contrast the results. In WMD, we have to discard certain words because the word embeddings do not contain all words. In the original study, the authors discard out-of-vocabulary words for WMD but maintain them for BOW and TF-IDF. In this paper, we discard out-of-vocabulary words for BOW and TF-IDF as well and use the same vocabulary for all of WMD, BOW, and TF-IDF. This setting is slightly advantageous for WMD. We show that WMD is comparable to BOW and TF-IDF even under this setting. We include the results with out-of-vocabulary words at the end of Section \ref{sec: normalization} for completeness. Note that we do not apply stemming but use raw words for BOW and TF-IDF following the original study.

\uline{\textbf{Evaluation Protocol.}} WMD, BOW, and TF-IDF all have only one hyperparameter, i.e., size $k$ of the neighborhood in kNN classification. We split the training set into an 80/20 train/validation set uniformly and randomly and select the neighborhood size from $\{1, 2, \cdots, 19\}$ using the validation data. During the process of our re-evaluations, we found that the kNN classifiers were much less capable than we previously thought and that kNN classification may underestimate the performances of distance-based methods. Nevertheless, we adopt kNN evaluations in the main part to clearly contrast our results with the original evaluation. This is justified because we only use kNN classification for all methods. We report this issue in more detail in Appendix \ref{sec: wknn}.

\uline{\textbf{Implementation.}} We use the POT library \cite{flamary2017pot} to solve the OT problem in eq. (\ref{eq: wmd}). We use the gensim library \cite{rehurek_lrec} to compute the TF-IDF vector.

\uline{\textbf{Environment.}} We use a server cluster to compute WMD. Each node has two 2.4GHz Intel Xeon Gold 6148 CPUs. We use a Linux server with Intel Xeon E7-4830 v4 CPUs to evaluate the performances.

\begin{figure}[t]
\begin{center}
\includegraphics[width=\hsize]{./imgs/our_eval.eps}
\end{center}
\caption{kNN classification errors in our re-evaluation. Lower is better. The shaded bars are the performance without normalization. WMD is comparable to classical baselines with normalization.}
\label{fig: ours}
\end{figure}

\section{Normalization is Crucial} \label{sec: normalization}
\uline{\textbf{Importance of Normalization.}}
In the original paper, raw BOW and TF-IDF vectors are used for the nearest neighbor classification. However, this is problematic because the length of these vectors varies based on the length of the documents. Even if two documents share the same topic, the BOW vectors are distant if their lengths differ. We need to normalize these vectors to effectively use them in nearest neighbor classification.

To make direct comparisons to WMD, we employ L1 normalization (Eq. (\ref{eq: normalization})) for BOW and TF-IDF vectors. We use the L1 distance to compare the vectors, i.e., $d_{\text{BOW/L1/L1}}({\boldsymbol{x}}, {\boldsymbol{x}}') = \|{\boldsymbol{n}}_{\text{L1}}({\boldsymbol{x}}) - {\boldsymbol{n}}_{\text{L1}}({\boldsymbol{x}}')\|_1$. With this normalization, BOW is a special case of WMD that does not use the underlying geometry. Let ${\boldsymbol{C}}^\text{unif} = 2(\mathbbm{1} \mathbbm{1}^\top - \text{diag}(\mathbbm{1}))$, i.e., ${\boldsymbol{C}}^\text{unif}_{ij} = 0$ if $i = j$ and ${\boldsymbol{C}}^\text{unif}_{ij} = 2$ if $i \neq j$. This matrix corresponds to the cost matrix of WMD when we use one-hot vectors as word embeddings. Then,
\begin{proposition} \label{prop: analogy}
$d_{\textup{BOW/L1/L1}}({\boldsymbol{x}}, {\boldsymbol{x}}') = \textup{OT}({\boldsymbol{x}}, {\boldsymbol{x}}', {\boldsymbol{C}}^\textup{unif})$
\end{proposition}
The proof is in Appendix \ref{sec: proof}. Therefore, the difference in the performances between WMD and BOW indicates the benefit of the underlying geometry.

Figure \ref{fig: ours} shows the classification errors.
First, we can see that the errors of BOW and TF-IDF decrease drastically. Even BOW performs better than WMD in bbcsport and ohsumed. TF-IDF outperforms WMD in five out of eight datasets. Even in the other datasets where WMD outperforms the classical baselines, the improvements are far less significant than what was reported in the original evaluation. This indicates that \textbf{the majority of the performance improvement of WMD was from the normalization (Eq. (\ref{eq: normalization})), not from the optimal transport formulation (Eq. (\ref{eq: wmd}))}. We can also observe that TF-IDF always performs better than BOW in contrast to the original evaluation.

\begin{table}[t]
    \centering
    \caption{kNN classification errors. Lower is better. Here, (x/y) uses x as the normalization and y as the metric. The last column reports the average relative performance to the normalized BOW. These values correspond to Figure 4 in the original paper, but we use BOW (L1/L1) as the base performances while the original paper used BOW (None/L2) as the base performances. Figure 4 in \cite{yurochkin2019hierarchical} also reports the relative performances, but it uses BOW (L1/L2) as the base performances. The standard deviations are reported for five-fold datasets. The first three rows are the same as in Figure \ref{fig: ours}. For BOW and TF-IDF, a cell is highlighted with \textbf{bold} if the mean score is better than that of WMD. For WMD, a cell is highlighted with \textbf{bold} if the mean score is better than those of both BOW and TF-IDF. The fourth row reports the performances with WMD with TF-IDF weighting. The following rows report the performance with different normalization and metrics. }
    \scalebox{0.72}{
    \begin{tabular}{lccccccccc} \toprule
& bbcsport & twitter & recipe & ohsumed & classic & reuters & amazon & 20news & rel. \\ \midrule
BOW (L1/L1) & \textbf{3.9 $\pm$ 1.1} & 30.0 $\pm$ 1.1 & 43.4 $\pm$ 0.8 & \textbf{44.1} & 4.1 $\pm$ 0.5 & 5.7 & 10.4 $\pm$ 0.5 & 29.1 & 1.000 \\
TF-IDF (L1/L1) & \textbf{2.8 $\pm$ 1.1} & \textbf{28.9 $\pm$ 0.8} & \textbf{40.1 $\pm$ 0.7} & \textbf{37.8} & 3.3 $\pm$ 0.4 & 5.5 & 8.0 $\pm$ 0.3 & \textbf{25.9} & 0.861 \\
WMD & 5.1 $\pm$ 1.2 & 29.6 $\pm$ 1.5 & 42.9 $\pm$ 0.8 & 44.5 & \textbf{2.9 $\pm$ 0.4} & \textbf{4.0} & \textbf{7.4 $\pm$ 0.5} & 26.8 & 0.917 \\ \midrule
WMD-TF-IDF & 3.3 $\pm$ 0.9 & 28.3 $\pm$ 2.3 & 39.9 $\pm$ 1.1 & 39.7 & 2.7 $\pm$ 0.3 & 4.0 & 6.6 $\pm$ 0.2 & 24.1 & 0.804 \\ \midrule
BOW (None/L2) \cite{kusner2015word} & 19.4 $\pm$ 3.0 & 34.2 $\pm$ 0.6 & 60.0 $\pm$ 2.3 & 61.6 & 35.0 $\pm$ 0.9 & 11.8 & 28.2 $\pm$ 1.0 & 57.7 & 3.024 \\
BOW (None/L1) & 25.4 $\pm$ 1.5 & 32.7 $\pm$ 1.6 & 65.8 $\pm$ 2.5 & 69.3 & 52.1 $\pm$ 0.5 & 14.2 & 31.4 $\pm$ 1.2 & 73.9 & 3.931 \\
TF-IDF (None/L2) \cite{kusner2015word} & 24.5 $\pm$ 1.3 & 38.2 $\pm$ 4.6 & 65.0 $\pm$ 1.9 & 65.3 & 38.8 $\pm$ 1.0 & 28.0 & 41.2 $\pm$ 3.2 & 60.0 & 3.867 \\
TF-IDF (None/L1) & 30.6 $\pm$ 1.3 & 37.8 $\pm$ 4.8 & 70.3 $\pm$ 1.3 & 70.6 & 52.6 $\pm$ 0.2 & 29.1 & 41.5 $\pm$ 4.9 & 74.6 & 4.602 \\ \midrule
BOW (L1/L2) \cite{yurochkin2019hierarchical} & 11.4 $\pm$ 3.6 & 37.0 $\pm$ 1.4 & 50.8 $\pm$ 1.1 & 56.7 & 17.3 $\pm$ 1.5 & 12.3 & 35.7 $\pm$ 1.3 & 46.5 & 2.253 \\
BOW (L2/L1) & 15.2 $\pm$ 1.5 & 33.3 $\pm$ 1.1 & 61.1 $\pm$ 1.1 & 65.7 & 51.1 $\pm$ 0.4 & 16.2 & 32.2 $\pm$ 1.3 & 77.6 & 3.622 \\
BOW (L2/L2) \cite{werner2020speeding, wrzalik2019balanced} & 5.5 $\pm$ 0.7 & 31.0 $\pm$ 0.8 & 46.1 $\pm$ 0.6 & 46.2 & 6.3 $\pm$ 0.7 & 8.8 & 13.1 $\pm$ 0.5 & 33.2 & 1.254 \\
TF-IDF (L1/L2) & 25.5 $\pm$ 11.2 & 35.7 $\pm$ 1.4 & 54.2 $\pm$ 2.7 & 61.4 & 22.6 $\pm$ 4.2 & 24.7 & 41.9 $\pm$ 2.0 & 45.6 & 3.226 \\
TF-IDF (L2/L1) & 27.5 $\pm$ 7.2 & 33.4 $\pm$ 1.7 & 64.9 $\pm$ 3.8 & 69.7 & 52.0 $\pm$ 0.2 & 19.5 & 40.8 $\pm$ 6.6 & 78.3 & 4.245 \\
TF-IDF (L2/L2) \cite{yurochkin2019hierarchical, li2019classifying} & 4.0 $\pm$ 0.7 & 29.8 $\pm$ 1.5 & 43.7 $\pm$ 1.2 & 38.4 & 5.2 $\pm$ 0.3 & 10.5 & 11.1 $\pm$ 0.9 & 31.6 & 1.145 \\ \bottomrule
    \end{tabular}
    }
    \label{tab: re-eval}
\end{table}

\begin{table}[t]
    \centering
    \caption{kNN classification errors with clean data. Lower is better. The same notations as in Table \ref{tab: re-eval}. }
    \scalebox{0.72}{
    \begin{tabular}{lccccccccc} \toprule
& bbcsport & twitter & recipe & ohsumed & classic & reuters & amazon & 20news & rel. \\ \midrule
BOW (L1/L1) & \textbf{3.7 $\pm$ 1.0} & \textbf{30.6 $\pm$ 1.1} & \textbf{42.9 $\pm$ 0.6} & \textbf{39.7} & 4.2 $\pm$ 0.5 & 5.5 & 10.6 $\pm$ 0.6 & 29.2 & 1.000 \\
TF-IDF (L1/L1) & \textbf{2.3 $\pm$ 1.4} & \textbf{30.2 $\pm$ 0.7} & \textbf{40.0 $\pm$ 1.1} & \textbf{33.4} & 3.5 $\pm$ 0.2 & 5.9 & 8.0 $\pm$ 0.6 & \textbf{25.9} & 0.866 \\
WMD & 5.5 $\pm$ 1.2 & 30.6 $\pm$ 1.2 & 42.9 $\pm$ 0.9 & 40.6 & \textbf{3.4 $\pm$ 0.6} & \textbf{3.8} & \textbf{7.3 $\pm$ 0.4} & 26.9 & 0.952 \\ \midrule
WMD-TF-IDF & 4.1 $\pm$ 1.5 & 28.8 $\pm$ 1.6 & 40.2 $\pm$ 0.9 & 35.7 & 2.8 $\pm$ 0.3 & 4.3 & 6.6 $\pm$ 0.3 & 24.2 & 0.848 \\ \bottomrule
    \end{tabular}
    }
    \label{tab: clean}
    \vspace{-0.1in}
\end{table}

\uline{\textbf{Comparison with other Normalization.}}
Although normalized BOW and TF-IDF have been employed in other studies \cite{yurochkin2019hierarchical, li2019classifying, werner2020speeding, wrzalik2019balanced}, it was reported that normalized BOW is still far worse than WMD, which is incompatible with our observations above. We found out that this occurred because of the normalization methods and metrics used to compare the vectors. For example, \citet{yurochkin2019hierarchical} used L1 normalization and the L2 metric for BOW, i.e., $\|{\boldsymbol{n}}_\text{L1}({\boldsymbol{x}}) - {\boldsymbol{n}}_\text{L1}({\boldsymbol{x}}')\|_2$, and L2 normalization and the L2 metric for TF-IDF, i.e., $\|{\boldsymbol{n}}_\text{L2}({\boldsymbol{x}}) - {\boldsymbol{n}}_\text{L2}({\boldsymbol{x}}')\|_2$, where ${\boldsymbol{n}}_\text{L2}({\boldsymbol{x}}) = {\boldsymbol{x}} / \|{\boldsymbol{x}}\|_2$. Note that the L2/L2 scheme corresponds to the cosine similarity. We investigate the performance with different normalization and metrics using the same protocol as in the previous experiments. Table \ref{tab: re-eval} shows the classification errors. First, it is easy to see in the fifth through eighth rows that the performances degrade without normalization. In addition, even with normalization, the performances are still poor if the normalization and metric use different norms. Although the L2/L2 scheme is better than these schemes, the L1/L1 scheme is the best for all datasets. This result is natural if we adopt the stance that a document is a distribution of words, and based on the analogy between the L1/L1 BOW and WMD, i.e., Proposition \ref{prop: analogy}. To summarize, the normalization method significantly affects the performance. The benefit of L1 normalization over L2 normalization (i.e., the first versus the eleventh row of Table \ref{tab: re-eval}) is more significant than the benefit of WMD over L1/L1 BOW (i.e., the third versus the first row of Table \ref{tab: re-eval}). This ``ablation study'' indicates that L1 normalization is the most significant factor of improvement in WMD. We stress that the improvements brought about by L1 normalization are obtained almost for free, whereas the improvements by WMD are at the price of substantial computational costs. We should also point out that many papers have not clarified the normalization scheme. Because the normalization method and metrics used to compare the vectors significantly affect the performance, we recommend clarifying them.

To better illustrate the benefit of WMD, we also use TF-IDF weighting for the marginal measures of WMD. Specifically, let ${\boldsymbol{x}}_\text{tfidf} \in \mathbb{R}^d$ denote a TF-IDF vector. We use ${\boldsymbol{n}}_\text{L1}({\boldsymbol{x}}_\text{tfidf})$ instead of ${\boldsymbol{n}}_\text{L1}({\boldsymbol{x}})$ in the marginal constraints in Eq. (2). The TF-IDF weighting makes WMD as robust to noise as TF-IDF. The fourth row in Table \ref{tab: re-eval} reports the classification errors. WMD with TF-IDF weighting performs the best, with approximately a six percent improvement from the normalized TF-IDF, which corresponds to the WMD with the uniform distance matrix ${\boldsymbol{C}}^\text{unif}$. Investigating the (BOW, WMD) and (TF-IDF, WMD-TF-IDF) pairs better clarifies the benefit of the optimal transport formulation. We can observe six to eight percent improvements in the optimal transport formulation from the naive baselines. This is in contrast to the \emph{sixty} percent improvement claimed in the original study.

\uline{\textbf{Evaluation on Clean Data.}} We evaluate the methods using clean datasets as well. We adopt the same protocol as in the previous experiments except for the removal of duplicate documents. Table \ref{tab: clean} reports the classification errors. The tendency is the same as that in the previous experiments. The differences between (BOW, WMD) and (TF-IDF, WMD-TF-IDF) become even narrower. Namely, an approximately five percent improvement in WMD and two percent improvement in WMD-TF-IDF are found. The results for other normalization and metrics are reported in Table \ref{tab: clean-full} in the appendix.

\uline{\textbf{Evaluation with Out of Vocabulary Words.}}
We hypothesized that the two to eight percent improvements of WMD are due to the discarded vocabularies, which made the evaluations slightly advantageous to WMD, as we stated in Section \ref{sec: setup}. We evaluate BOW and TF-IDF, including words not in the word2vec vocabulary. We use all but the recipe dataset because the raw texts of the recipe dataset have not been released by the authors. We include stopwords for the Twitter dataset and remove them for the other datasets following the original paper. Tables \ref{tab: oov} and \ref{tab: oov-clean} report the classification errors. Against expectations, these results show that the use of all words does not improve the performance. We hypothesize that this is because the word2vec vocabulary implicitly helps the classification by removing noisy words.

\begin{table}[t]
    \centering
    \caption{kNN classification errors with all words and the original datasets. Lower is better. The last column reports the average relative performance to the normalized BOW (the first row in Table \ref{tab: re-eval}). }
    \scalebox{0.72}{
    \begin{tabular}{lcccccccc} \toprule
& bbcsport & twitter & ohsumed & classic & reuters & amazon & 20news & rel. \\ \midrule
BOW (L1/L1) & 3.0 $\pm$ 0.8 & 29.5 $\pm$ 0.7 & 46.0 & 3.9 $\pm$ 0.4 & 6.1 & 12.3 $\pm$ 0.5 & 32.1 & 1.015 \\
TF-IDF (L1/L1) & 2.8 $\pm$ 0.8 & 29.4 $\pm$ 0.9 & 38.7 & 3.1 $\pm$ 0.4 & 6.8 & 7.9 $\pm$ 0.3 & 24.8 & 0.877 \\ \bottomrule
    \end{tabular}
    }
    \label{tab: oov}
\end{table}

\begin{table}[t]
    \centering
    \caption{kNN classification errors with all words and clean data. Lower is better. The last column reports the average relative performance to the normalized BOW (the first row in Table \ref{tab: clean}). }
    \scalebox{0.72}{
    \begin{tabular}{lcccccccc} \toprule
& bbcsport & twitter & ohsumed & classic & reuters & amazon & 20news & rel. \\ \midrule
BOW (L1/L1) & 2.7 $\pm$ 0.6 & 31.0 $\pm$ 2.0 & 41.0 & 4.1 $\pm$ 0.4 & 6.3 & 12.3 $\pm$ 0.3 & 32.1 & 1.022 \\
TF-IDF (L1/L1) & 1.8 $\pm$ 0.8 & 30.3 $\pm$ 1.5 & 34.9 & 3.4 $\pm$ 0.5 & 6.4 & 8.1 $\pm$ 0.2 & 24.8 & 0.849 \\ \bottomrule
    \end{tabular}
    }
    \label{tab: oov-clean}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=0.9\hsize]{./imgs/scatter.eps}
\caption{Scatter plot between WMD in five-dimensional spaces and L1/L1 BOW. WMD is determined almost by L1/L1 BOW.}
\label{fig: scatter}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=\hsize]{./imgs/histogram.eps}
\caption{Histogram of distances between matched word embeddings.}
\label{fig: transport}
\end{figure}

\uline{\textbf{Summary.}} We should emphasize that we have not concluded that the improvements brought by WMD are spurious. Based on our careful evaluations, we conclude that the two to eight percent improvements from the naive baselines reported in Tables \ref{tab: re-eval} and \ref{tab: clean} are genuine. These improvements are less sensational than those reported in the original paper, i.e., \emph{sixty} percent improvement. However, we believe that our results indicate the true performance of WMD. It is important to realize the true value of WMD. Our results indicate that if the speed is important, WMD may not be worth trying, whereas if the performance is crucial at any cost, WMD may be a worthwhile candidate.

\section{Curse of Dimensionality} \label{sec: curse}

The improvements of WMD from the naive baselines reported in Tables \ref{tab: re-eval} and \ref{tab: clean} are surprisingly subtle. Intuitively, WMD should outperform the baselines by larger margins as reported in the original paper because WMD exploits the external information about the underlying word geometry, which naive baselines do not have access to. We reveal the reason for the comparable performance of WMD.

We first found that not only did the performances of WMD and L1/L1 BOW resemble, the distances themselves were also similar. Figure \ref{fig: scatter} shows scatter plots of WMD and L1/L1 BOW and reports the Person's correlation coefficients. WMD and L1/L1 BOW are surprisingly similar to each other. This is partially due to the correspondence of WMD and L1/L1 BOW, i.e., Proposition \ref{prop: analogy}. We find that they are all more similar owing to the high-dimensional nature of the underlying metric\footnote{See Appendix \ref{sec: additional-related-work} for the discussion on the curse of the dimensionality for the sample complexity of OT.}.

WMD uses the $300$-dimensional word2vec as the ground metric. It is well known that two random vectors in high-dimensional spaces are orthogonal with high probability. Besides, word embeddings are known to be distributed uniformly within the embedding space \cite{arora2016latent, arora2017simple}. Although OT matches similar (non-random) vectors, it suffers from the curse of dimensionality. We confirm this numerically.

For each document, we compute the nearest document with respect to WMD. For the one-fold datasets, the source documents are in the training data, and the target documents are in the test data. For the five-fold datasets, the source documents are the full set of documents, and the target documents are all but the source document (i.e., leave-one-out). For each nearest neighbor pair, we solve the OT problem (Eq. (\ref{eq: wmd})), and we place a fraction of a mass with weight ${\boldsymbol{P}}_{ij}$ at position ${\boldsymbol{C}}_{ij}$. The resulting histogram represents the probability distribution of the transport distances.

\setlength{\intextsep}{0pt}%
\setlength{\columnsep}{0.1in}%
\begin{wrapfigure}{r}{3.1in}
\centering
\includegraphics[width=\hsize]{./imgs/illust_problem.eps}
\caption{We retrieve a similar document as document A. Each box represents a dimension of BOW. A filled box represents one in a BOW vector. Documents B and C have three and two common words with A, respectively. Suppose the (fourth, fifth) and (eighth, ninth) words are paraphrases, but the (fourth, seventh) are not. Thus, documents A and C are paraphrases, whereas documents A and B are not. The philosophy of WMD may expect C to be the better candidate. However, the distances between words are approximately $1.0$. Thus, even paraphrase words are as distant as $0.9$, and non-paraphrase words are as close as $1.4$. Therefore, WMD between A and B is $1.4$, and WMD between A and C is $0.9 + 0.9 = 1.8$. As a result, WMD chooses document B as BOW does.}
\label{fig: illust_problem}
\end{wrapfigure}

Figure \ref{fig: transport} shows the histograms. The x-axis represents the transport distances, and the y-axis represents the frequency. All of the histograms have acute peaks at $x = 0$. These peaks indicate that the source and target masses exactly match. Other pairs of matched masses are approximately $x \approx 1$, and are approximately equally distant. Therefore, using similar (but not exactly the same) words reduces the transportation cost only slightly, e.g., from $1.1$ to $1.0$. This indicates that non-overlap words are used just for tie-breaking because using more overlap words further decreases the transportation cost, e.g., from $1.0$ to $0.0$. In other words, \textbf{WMD in high-dimensional spaces is almost the same as BOW except for a tie-breaking}. Figure \ref{fig: illust_problem} illustrates this phenomenon. This is not what we imagined with the two-dimensional illustrations in Figure \ref{fig: illust}.

\uline{\textbf{Dimensionality Matters.}} To clarify the difference between low- and high-dimensional spaces, we compute WMD in low-dimensional spaces as well. Specifically, we project the $300$-dimensional word2vec to a five-dimensional space using principal component analysis and compute the scatter plots and histograms as in the previous experiment. Figures \ref{fig: scatter_lowdim} and \ref{fig: transport_lowdim} show the scatter plots and histograms, respectively. First, WMD in low-dimensional spaces is not as similar to L1/L1 BOW as in high-dimensional spaces. For example, two sentences with no common words (i.e., L1/L1 BOW $= 2.0$) can have a low WMD if they have similar embeddings. This is in contrast to the high-dimensional cases in Figure \ref{fig: scatter}, where WMD is almost determined by L1/L1 BOW. Secondly, all of the histograms are almost unimodal and have peaks at only $x = 0$ in Figure \ref{fig: transport_lowdim}. They are compatible with our intuitions in Figure \ref{fig: illust}. Comparing Figures \ref{fig: scatter} and \ref{fig: scatter_lowdim} and Figures \ref{fig: transport} and \ref{fig: transport_lowdim} indicate that the counter-intuitive behaviour of WMD is due to the dimensionality. It should be noted that the performance of WMD degrades in the five-dimensional space because it loses a significant amount of information, despite no curses of dimensionality. (See Appendix \ref{sec: low-dim-performance} for the quantitative evaluations.)

\uline{\textbf{Document embeddings.}} Some document embedding methods such as SIF \cite{arora2017simple} use a weighted sum of word embeddings as a document embedding. If the word embeddings are sufficiently high-dimensional, our discussion above indicates that such document embeddings also resemble BOW. However, an inner product of such document embeddings has $n^2$ terms of inner products of word embeddings, where $n$ is the number of words in a document. By contrast, WMD has only $n$ terms. Therefore, WMD has fewer chances to deviate from BOW than document embeddings.

\begin{figure}[t]
\centering
\includegraphics[width=0.9\hsize]{./imgs/scatter_5dim.eps}
\vspace{-0.1in}
\caption{Scatter plot between WMD and L1/L1 BOW in the $5$-dimensional space. WMD in low-dimensional spaces can take a low value even if two documents do not have common words.}
\label{fig: scatter_lowdim}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=\hsize]{./imgs/histogram_5dim.eps}
\vspace{-0.1in}
\caption{Histogram of distances between matched word embeddings in the $5$-dimensional space.}
\label{fig: transport_lowdim}

\vspace{-0.1in}
\end{figure}

\section{Discussion} \label{sec: discussion}

\uline{\textbf{Limitation.}} The main limitation of this paper is that we do not provide any solutions to overcome the curse of dimensionality or how to boost the performance of WMD. We found it considerably challenging to solve. We consider we should not jump to a quick conclusion. Thus, we focused on stating what the problem is so that other researchers can start with our findings.

\uline{\textbf{Value of WMD.}} We stress that WMD has qualitative advantages in addition to the two to eight percent improvements observed in Section \ref{sec: normalization}. The most prominent property of WMD is the differentiability, which makes WMD compatible with deep learning methods \cite{kumer2017earth}. WMD can provide gradient signals to embeddings, whereas BOW cannot. This property has been successfully used in training word embedding \cite{xu2018distilled} and cross-lingual word alignment \cite{zhang2016building, zhang2017bilingual, zhang2017earth, grave2019unsupervised}.

\uline{\textbf{Implication of Our Findings.}} Although we focused on re-evaluating WMD, our findings are beneficial for other topics as well. First, the datasets released by the WMD paper have been used in non-OT studies as well \cite{skianis2020rep, mollaysa2017regularizing, gupta2020psif}. Our findings regarding the duplicate documents are important for proper evaluations of other methods as well. Secondly, we have found that L1/L1 BOW and L1/L1 TF-IDF performed well regardless of their simplicity and efficiency. Practitioners can use them as an initial choice. Besides, using such simple yet effective methods can clarify the essential components of the methods, as shown in Section \ref{sec: curse}.

\section{Conclusion}

In this paper, we pointed out that the major improvement in WMD against classical baselines is not from the inherent feature of WMD but mainly from the normalization. We re-evaluated the performance of WMD and classical baselines and found that classical baselines are competitive with WMD if we normalize the vectors. We showed that this is due to the high-dimensional nature of the underlying metric. We found that WMD resembled BOW much more than
we had thought in two-dimensional illustrations owing to the curse of dimensionality. In the process of our re-evaluation, we found several confusing aspects in the original evaluations of the original paper on WMD. We pointed them out in this paper to help the following researchers conduct solid evaluations.

\section*{Acknowledgments and Funding Disclosure}

This work was supported by the JSPS KAKENHI GrantNumber 20H04243 and 20H04244.

redpajama_set_name: RedPajamaArXiv


text:
Two New Stakes Winners for BROKEN VOW
CAN'T BE WRONG made it three in a row after closing with a rush to go by the front runner and win the blacktype Texas Horse Racing Hall of Fame S. by 1 1/2 lengths. The 3-year-old son of BROKEN VOW was still 10 lengths back after the first half of this 1 1/16-mile turf event at Retama, but charged down the stretch to overhaul the leader in the final 1/16th and get the victory. CAN'T BE WRONG was coming off a nearly seven-length allowance win here two weeks ago, and also was a 9 1/4-length Maiden Special Weight winner on the dirt at Lone Star in July. He has now won or placed in eight of 11 starts this year, including four wins. Lifetime record: 15-4-5-1, $83,511. O/B-Macassar Corporation. T-Danny Pish.
Also, EMERGING TALENT (ARG) won his third straight after taking the blacktype Jockey Club de Venezuela at La Plata. This 4-year-old colt also finished on the board in the G2 Polla de Potrillos de La Plata in August of 2015 following his five-length maiden win. His dam Truth Seeking, by Seeking the Gold, was purchased out of the 2010 Keeneland November Sale. This is the family of champions Refuse To Bend (IRE), Go and Go (IRE), and Munaaji, and G1 SWs and millionaires Twilight Agenda and Media Puzzle.
SWEAR BY IT Rallies to Place in Keeneland's G3 Rood and Riddle Dowager	CHAMPAGNE ROOM Wins Breeders' Cup Juvenile Fillies-G1

redpajama_set_name: RedPajamaCommonCrawl


text:
/*
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.facebook.presto.jdbc;

import com.facebook.presto.server.testing.TestingPrestoServer;
import io.airlift.log.Logging;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.math.BigDecimal;
import java.sql.Connection;
import java.sql.Date;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Time;
import java.sql.Timestamp;
import java.sql.Types;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.LocalTime;

import static com.facebook.presto.jdbc.TestPrestoDriver.closeQuietly;
import static java.lang.String.format;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertThrows;
import static org.testng.Assert.assertTrue;

@Test(singleThreaded = true)
public class TestJdbcResultSet
{
    private TestingPrestoServer server;

    private Connection connection;
    private Statement statement;

    @BeforeClass
    public void setupServer()
            throws Exception
    {
        Logging.initialize();
        server = new TestingPrestoServer();
    }

    @AfterClass(alwaysRun = true)
    public void teardownServer()
    {
        closeQuietly(server);
    }

    @SuppressWarnings("JDBCResourceOpenedButNotSafelyClosed")
    @BeforeMethod
    public void setup()
            throws Exception
    {
        connection = createConnection();
        statement = connection.createStatement();
    }

    @AfterMethod
    public void teardown()
    {
        closeQuietly(statement);
        closeQuietly(connection);
    }

    @Test
    public void testDuplicateColumnLabels()
            throws Exception
    {
        try (ResultSet rs = statement.executeQuery("SELECT 123 x, 456 x")) {
            ResultSetMetaData metadata = rs.getMetaData();
            assertEquals(metadata.getColumnCount(), 2);
            assertEquals(metadata.getColumnName(1), "x");
            assertEquals(metadata.getColumnName(2), "x");

            assertTrue(rs.next());
            assertEquals(rs.getLong(1), 123L);
            assertEquals(rs.getLong(2), 456L);
            assertEquals(rs.getLong("x"), 123L);
        }
    }

    @SuppressWarnings("UnnecessaryBoxing")
    @Test
    public void testObjectTypes()
            throws Exception
    {
        checkRepresentation("123", Types.INTEGER, 123);
        checkRepresentation("12300000000", Types.BIGINT, 12300000000L);
        checkRepresentation("REAL '123.45'", Types.REAL, 123.45f);
        checkRepresentation("1e-1", Types.DOUBLE, 0.1);
        checkRepresentation("1.0E0 / 0.0E0", Types.DOUBLE, Double.POSITIVE_INFINITY);
        checkRepresentation("0.0E0 / 0.0E0", Types.DOUBLE, Double.NaN);
        checkRepresentation("0.1", Types.DECIMAL, new BigDecimal("0.1"));
        checkRepresentation("true", Types.BOOLEAN, true);
        checkRepresentation("'hello'", Types.LONGNVARCHAR, "hello");
        checkRepresentation("cast('foo' as char(5))", Types.CHAR, "foo  ");
        checkRepresentation("ARRAY[1, 2]", Types.ARRAY, (rs, column) -> assertEquals(rs.getArray(column).getArray(), new int[] {1, 2}));
        checkRepresentation("DECIMAL '0.1'", Types.DECIMAL, new BigDecimal("0.1"));

        checkRepresentation("DATE '2018-02-13'", Types.DATE, (rs, column) -> {
            assertEquals(rs.getObject(column), Date.valueOf(LocalDate.of(2018, 2, 13)));
            assertEquals(rs.getDate(column), Date.valueOf(LocalDate.of(2018, 2, 13)));
            assertThrows(IllegalArgumentException.class, () -> rs.getTime(column));
            assertThrows(IllegalArgumentException.class, () -> rs.getTimestamp(column));
        });

        checkRepresentation("TIME '09:39:05'", Types.TIME, (rs, column) -> {
            assertEquals(rs.getObject(column), Time.valueOf(LocalTime.of(9, 39, 5)));
            assertThrows(() -> rs.getDate(column));
            assertEquals(rs.getTime(column), Time.valueOf(LocalTime.of(9, 39, 5)));
            assertThrows(() -> rs.getTimestamp(column));
        });

        checkRepresentation("TIME '09:39:07 +01:00'", Types.TIME /* TODO TIME_WITH_TIMEZONE */, (rs, column) -> {
            assertEquals(rs.getObject(column), Time.valueOf(LocalTime.of(14, 9, 7))); // TODO this should represent TIME '09:39:07 +01:00'
            assertThrows(() -> rs.getDate(column));
            assertEquals(rs.getTime(column), Time.valueOf(LocalTime.of(14, 9, 7))); // TODO this should fail, or represent TIME '09:39:07 +01:00'
            assertThrows(() -> rs.getTimestamp(column));
        });

        checkRepresentation("TIMESTAMP '2018-02-13 13:14:15.123'", Types.TIMESTAMP, (rs, column) -> {
            assertEquals(rs.getObject(column), Timestamp.valueOf(LocalDateTime.of(2018, 2, 13, 13, 14, 15, 123_000_000)));
            assertThrows(() -> rs.getDate(column));
            assertThrows(() -> rs.getTime(column));
            assertEquals(rs.getTimestamp(column), Timestamp.valueOf(LocalDateTime.of(2018, 2, 13, 13, 14, 15, 123_000_000)));
        });

        checkRepresentation("TIMESTAMP '2018-02-13 13:14:15.227 Europe/Warsaw'", Types.TIMESTAMP /* TODO TIMESTAMP_WITH_TIMEZONE */, (rs, column) -> {
            assertEquals(rs.getObject(column), Timestamp.valueOf(LocalDateTime.of(2018, 2, 13, 17, 59, 15, 227_000_000))); // TODO this should represent TIMESTAMP '2018-02-13 13:14:15.227 Europe/Warsaw'
            assertThrows(() -> rs.getDate(column));
            assertThrows(() -> rs.getTime(column));
            assertEquals(rs.getTimestamp(column), Timestamp.valueOf(LocalDateTime.of(2018, 2, 13, 17, 59, 15, 227_000_000))); // TODO this should fail or represent TIMESTAMP '2018-02-13 13:14:15.227 Europe/Warsaw'
        });
    }

    private void checkRepresentation(String expression, int expectedSqlType, Object expectedRepresentation)
            throws Exception
    {
        checkRepresentation(expression, expectedSqlType, (rs, column) -> assertEquals(rs.getObject(column), expectedRepresentation));
    }

    private void checkRepresentation(String expression, int expectedSqlType, ResultAssertion assertion)
            throws Exception
    {
        try (ResultSet rs = statement.executeQuery("SELECT " + expression)) {
            ResultSetMetaData metadata = rs.getMetaData();
            assertEquals(metadata.getColumnCount(), 1);
            assertEquals(metadata.getColumnType(1), expectedSqlType);
            assertTrue(rs.next());
            assertion.accept(rs, 1);
            assertFalse(rs.next());
        }
    }

    @Test
    public void testStatsExtraction()
            throws Exception
    {
        try (PrestoResultSet rs = (PrestoResultSet) statement.executeQuery("SELECT 123 x, 456 x")) {
            assertNotNull(rs.getStats());
            assertTrue(rs.next());
            assertNotNull(rs.getStats());
            assertFalse(rs.next());
            assertNotNull(rs.getStats());
        }
    }

    @Test
    public void testMaxRowsUnset()
            throws SQLException
    {
        assertMaxRowsLimit(0);
        assertMaxRowsResult(7);
    }

    @Test
    public void testMaxRowsUnlimited()
            throws SQLException
    {
        assertMaxRowsLimit(0);
        statement.setMaxRows(0);
        assertMaxRowsLimit(0);
        assertMaxRowsResult(7);
    }

    @Test
    public void testMaxRowsLimited()
            throws SQLException
    {
        assertMaxRowsLimit(0);
        statement.setMaxRows(4);
        assertMaxRowsLimit(4);
        assertMaxRowsResult(4);
    }

    @Test
    public void testMaxRowsLimitLargerThanResult()
            throws SQLException
    {
        assertMaxRowsLimit(0);
        statement.setMaxRows(10);
        assertMaxRowsLimit(10);
        assertMaxRowsResult(7);
    }

    @Test
    public void testLargeMaxRowsUnlimited()
            throws SQLException
    {
        assertMaxRowsLimit(0);
        statement.setLargeMaxRows(0);
        assertMaxRowsLimit(0);
        assertMaxRowsResult(7);
    }

    @Test
    public void testLargeMaxRowsLimited()
            throws SQLException
    {
        assertMaxRowsLimit(0);
        statement.setLargeMaxRows(4);
        assertMaxRowsLimit(4);
        assertMaxRowsResult(4);
    }

    @Test
    public void testLargeMaxRowsLimitLargerThanResult()
            throws SQLException
    {
        long limit = Integer.MAX_VALUE * 10L;
        statement.setLargeMaxRows(limit);
        assertEquals(statement.getLargeMaxRows(), limit);
        assertMaxRowsResult(7);
    }

    private void assertMaxRowsLimit(int expectedLimit)
            throws SQLException
    {
        assertEquals(statement.getMaxRows(), expectedLimit);
        assertEquals(statement.getLargeMaxRows(), expectedLimit);
    }

    private void assertMaxRowsResult(long expectedCount)
            throws SQLException
    {
        try (ResultSet rs = statement.executeQuery("SELECT * FROM (VALUES (1), (2), (3), (4), (5), (6), (7)) AS x (a)")) {
            assertEquals(countRows(rs), expectedCount);
        }
    }

    @Test(expectedExceptions = SQLException.class, expectedExceptionsMessageRegExp = "Max rows exceeds limit of 2147483647")
    public void testMaxRowsExceedsLimit()
            throws SQLException
    {
        statement.setLargeMaxRows(Integer.MAX_VALUE * 10L);
        statement.getMaxRows();
    }

    private Connection createConnection()
            throws SQLException
    {
        String url = format("jdbc:presto://%s", server.getAddress());
        return DriverManager.getConnection(url, "test", null);
    }

    private static long countRows(ResultSet rs)
            throws SQLException
    {
        long count = 0;
        while (rs.next()) {
            count++;
        }
        return count;
    }

    @FunctionalInterface
    private interface ResultAssertion
    {
        void accept(ResultSet rs, int column)
                throws Exception;
    }
}

redpajama_set_name: RedPajamaGithub


text:
Top 100 MSSPs (2017, 2018)
Top 200 MSSPs (2019)
MSSP Alert
Managed Security Services Providers (MSSP) News, Analysis and Cybersecurity Research
Top 250 MSSPs: Cybersecurity Company List and Research for 2020
Continue to next page for companies ranked 60 to 51
60. Cysiv Inc.
2019: NR 2018: NR
Las Colinas, Texas, USA
Cysiv SOC-as-a-Service combines all the elements of a modern 24/7 SOC—including a cloud-native next-gen SIEM platform, a team of experts, actionable threat intel, SLAs and runbooks, and an optional managed security stack—and delivers them as a service, with monthly, consumption-based billing. Cysiv was incubated within Trend Micro, a global leader in cybersecurity solutions with a proud 30+ year history of innovation, and was spun out as an independent company in 2018. With a rich heritage in developing market-leading security software for enterprises, and a strong conviction that a cloud-native, "all-in-one" next-gen SIEM was essential to delivering an effective and valuable SOC service, Cysiv has invested heavily in developing a unique, robust and data science-driven, modern SaaS platform.
59. Cofense Inc.
Leesburg, Virginia, USA
Cofense, the leading provider of intelligent phishing defense solutions worldwide, is uniting humanity against phishing. The Cofense suite of products combines timely attack intelligence on phishing threats that have evaded perimeter controls and were reported by employees, with best-in-class security operations technologies to stop attacks faster and stay ahead of breaches. Cofense customers include Global 1000 organizations in defense, energy, financial services, healthcare and manufacturing sectors that understand how changing user behavior will improve security, aid incident response and reduce the risk of compromise.
58. Fujitsu
2019: 47 2018: 37
Kawasaki, Japan
Fujitsu's Managed Security Services deliver all this and more. Comprehensive threat protection identifies potential risks, mitigates any immediate impact and prevents subsequent attacks from happening. As a leading security service provider, our intelligence-led, enterprise-grade solutions minimize disruption and maintain business continuity across your entire organization – leaving you to focus on other priorities. Our services include identity and access management, infrastructure protection services, data protection, threat and vulnerability management.
57. RadarServices Smart IT-Security GmbH
Wien, Vienna, Austria
Radar Cyber Security is a European supplier of Managed Detection & Response Solutions that provides its services based on inhouse developed technology. More than 10 years of research and development have resulted in the evolution of a comprehensive cybersecurity and IT risk detection platform we use every day to monitor the IT security of global market leaders of all sectors, as well as public authorities. Through our research we are a global leader in machine learning for IT monitoring, continuously incorporating the latest findings to the services we provide to our customers. Headquartered in Vienna (Austria), we run the largest Cyber Defense Center in Europe for managed services, attracting the best IT security professionals in the world.
56. Digital Defense
San Antonio, Texas, USA
Since 1999, Digital Defense has created patented, leading-edge SaaS information security technology that helps organizations small and large safeguard sensitive data and make information security easier. As a pioneer in the development of SaaS technology, these organizations trust Digital Defense to deliver unparalleled solutions to bolster their security posture and ease the burden of vulnerability and threat management. Frontline.Cloud solutions offer ease of use and consistent, accurate results accompanied by the highest level of support and service.
55. Digital Hands
2019: 99 2018: NR
Tampa, Florida, USA
Digital Hands is an award-winning managed security service provider focused on providing world-class security operations to customers and partners. The company's adaptive security solutions are designed to easily integrate into customers' environments and processes while providing comprehensive protection against the latest security threats.
54. Booz Allen Hamilton
McLean, Virginia, USA
Booz Allen Managed Threat Services (MTS) combines threat intelligence with managed detection and response to provide actionable insights, 24/7 threat detection, investigation, and response—all delivered by analysts with the noted National Security Agency Cyber Incident Response Assistance (NSA CIRA) accreditation.
53. Braintrace
Salt Lake City, Utah, USA
Braintrace is a privately held, sole-source security provider. In addition to providing IT security services, Braintrace has vast experience in advising customers in broad areas of security-related business risks, including govermment compliance and regulatory issues. Braintrace operates five unique business units: MSSP, Advisory Services, VAR, BraintraceLABs and MSP.
52. Optiv
Denver, Colorado, USA
Optiv Security is the world's leading security solutions integrator (SSI). With years of experience supporting thousands of enterprise level clients, we know the solutions landscape better than anyone. We understand the challenges our clients face every day – because many of our people have occupied a seat on the other side of the table. Our unique team of former CISOs, security technologists and operations experts leverage an "I've experienced that, and solved that" perspective to deliver solutions that help our clients maximize and communicate the effectiveness of their cybersecurity programs.
51. LMNTRIX
2019: 198 2018: NR
Orange, California, USA
LMNTRIX Active Defense positions itself as a best-in-class Managed Detection & Response (MDR) service that detects and responds to advanced threats that bypass perimeter controls. We combine deep expertise with cutting-edge technology, leading intelligence, and advanced analytics to detect and investigate threats with great speed, accuracy, and focus. Threats detected are generally missed by existing security controls including the SIEM and MSSP. The outcomes we deliver clients are validated breaches that are investigated, contained and remediated. All incidents are aligned to the kill chain and Mitre ATT&CK frameworks and contain detailed investigative actions and recommendations that your organisation follows to protect against the unknown, insider threat and malicious attacker.
All contents © 2021 MSSP Alert and After Nines Inc.


text:
By Shae Russell
The Worst Financial Decision in History: Selling Gold
Today is a journey back in time.
To a time when teens wore their pants below their buttocks.
When you could fill up your Commodore or Falcon for less than 74 cents a litre.
When roughly one third of adult Australians had a mobile phone…
A moment in history when connecting to the internet sounded like ET phoning home.
The digital age was here. The new economic age was coming.
And we made sure we ditched all old assets to ready ourselves for the future.
Swapping hard money for paper money
In November 1997, then Treasurer Peter Costello announced to the Australian public that he was about to sell two thirds of our gold. He told us gold 'no longer plays a significant role in the international financial system.'
In one day, our government sold 167 tonnes of the yellow metal, netting about $400 per ounce.
Overall, the sale scored Australia about $2.4 billion, which was used to pay off some debts.
The move effectively suppressed the gold price. Hundreds of tonnes of gold for sale drove the price down over the next few months.
Incidentally, it seems China decided to buy up our sloppy seconds. It's rumoured the Chinese picked up some of the gold we so carelessly sold to the lowest bidder.
The thing is, we weren't the only ones selling our gold.
A new industrial age was taking over the US.
Tech stocks were hot. Precious metals were not. The euro was confirmed and would start circulating in a few more years.
Plus, there was Greenspan.
A belief coursed through the monetary system that US Federal Reserve Chairman Alan Greenspan, along with the US dollar, was the backbone of modern money practices.
So much so that The New York Times ran a piece titled 'Who Needs Gold When We Have Greenspan?':
'Is gold on its way to becoming just another commodity? The people who run the world's financial system are doing their best to secure that fate for the metal that once was viewed as the only "real" money.
'…The argument against retaining gold is that its day is past. Once it was useful as a hedge against inflation that would hold its value when paper currencies did not. Now financial markets have their own sophisticated ways, using exotic derivative securities, to hedge against inflation.'
Globally, central banks were dumping gold.
Germany, France, Austria, Sweden and Argentina all sold their gold into the late 90s.
Switzerland held a contentious referendum about selling its gold. It went ahead, and some 1,300 tonnes of its total 2,590-tonne gold hoard were dumped.
Belgium, Canada and the Netherlands sold a collective 1,590 tonnes in the late 90s. By 2002, Canada didn't have an ounce of gold in its vaults.
The future was here, and gold was nothing more than an attachment to old-world money.
And then came the most famous of all gold sales…
Brown's Bottom
May 2019 marked 20 years since Brown's Bottom.
A time when the UK decided to sell 401 tonnes of its total 715-tonne gold stash.
History now considers it the worst investment decision the UK ever made…
On 7 May 1999, the UK Chancellor of the Exchequer, Gordon Brown, announced that he would sell half of the UK's gold.
Why on Earth would the world's gold centre opt to sell more than half of its supply of the precious metal?
The official line is that Brown sold the gold to move into securities that would actually pay interest to the UK's Treasury.
Gold, which pays no yearly return and costs money to store, was a financial drain on the country at the time.
The yellow metal was nothing more than a legacy asset from a bygone era.
Another argument put forward is that the new millennium was approaching, and there was excitement about the future. As stated earlier, a new currency — the euro — would begin circulating in a little over a year.
Plus, there was Greenspan at the top of the Fed — and an unwavering belief that Greenspan could do no wrong.
There are also rumours that Brown sold all the gold to bail out the UK banks.
There is a persistent whisper that a major US bank was short selling gold, and, had someone tried to redeem it, it would technically make the bank insolvent.
As the tale goes, Brown was faced with either a global collapse in the banking system, or selling the gold to drive the price down just enough to allow this major bank to buy back the gold at lower prices.
Whether the rumour is true or not doesn't matter.
The point is, while Brown's decision to sell some 401 tonnes of gold was ill-fated, he wasn't the only one.
By the time Brown sold more than half of his country's gold, most of the other central banks had already done it.
The difference? Brown just happened to be selling into the bottom of the gold price.
No Greenspan and no gold
Two decades later, there is no Greenspan running the Fed.
The US dollar gold price per ounce is now four and a half times higher than it was in the late 90s.
Western central banks — in their rush to dump assets and support paper money — essentially cost their countries billions in foreign reserve assets.
Only a handful of central banks in that time have been increasing their gold stores.
The big buyers of gold now are the central banks in India, Russia, Poland and China.
These central banks are paying billions of dollars to secure gold, despite the perceived high prices.
That tells us they understand the importance of the yellow metal and Western central banks don't.
Shae Russell,
Editor, The Daily Reckoning Australia
PS: Discover how some investors are preserving their wealth and even making a profit, as the economy tanks. Download your FREE report by clicking here.
Home » Australian Economy » The Worst Financial Decision in History: Selling Gold
About Shae Russell:
Shae Russell's first foray into the wide world of financial markets began more than a decade ago. As a consultant at a derivatives brokering firm, she helped clients navigate the complex world of derivatives and technical analysis.
Over the past eight years, Shae's worked as an analyst on some of Australia's most widely-read financial investment newsletters, including Gold Stock Trader and Markets and Money. Now she joins the team at Fat Tail Media as co-editor of The Daily Reckoning Australia.
Shae, an avid gold investor, also works alongside Jim Rickards as his Australian analyst for Strategic Intelligence, an advisory dedicated to showing Aussie investors how to read and take advantage of global macro trends.
Shae Russell's Investment Services:
Rock Stock Insider
Hard Money Trader
View All Articles Sign Up Today
Venus Metals' Share Price Rebounds on High-Grade Gold Intercepts
New BY Lachlann Tierney POSTED 1 day ago
The share price of small-cap gold explorer Venus Metal Corp Ltd [ASX:VMC] has rebounded nicely today after a sharp drop on Thursday. […]
All advice is general in nature and has not taken into account your personal circumstances. Please seek independent financial advice regarding your own situation, or if in doubt about the suitability of an investment. Read more..
"The value of any investment and the income derived from it can go down as well as up. Never invest more than you can afford to lose and keep in mind the ultimate risk is that you can lose whatever you've invested. While useful for detecting patterns, the past is not a guide to future performance. Some figures contained in our reports are forecasts and may not be a reliable indicator of future results. Any actual or potential gains in these reports may not include taxes, brokerage commissions, or associated fees.
All content is © 2021 Fat Tail Media Pty Ltd All Rights Reserved. (ACN: 617 997 547)
FAQ| Terms & Conditions| Privacy Policy| Financial Services Guide| Contact| Sign up free


Did you know carbon comes in blue? Blue carbon refers to the carbon in oceans and coastal areas. These ecosystems are excellent carbon sinks - they can efficiently absorb and store carbon from the atmosphere.
The Food and Agriculture Organization of the United Nations estimates that around 45 million tons of phosphorus fertilizers will be used around the world in 2018.
Take a trip down into the soil beneath a field of crops. You won't find just dirt, water, and creepy-crawlies. You'll also find reactions that remind you of high school chemistry lab.


text:
*** DISCONTINUED *** CHARLIE CHAPLIN "The Little Tramp" Cardboard Cutout Standup / Standee
Little Tramp
"The Little Tramp"
Actress Mary Pickford once told a story that Charlie Chaplin entered a "Charlie Chaplin Look-Alike" contest and came in 20th place, though that may or may not be a myth. But we can guarantee that the "Little Tramp" on this lifesize (6-foot-tall) black and white cardboard cutout is the genuine article!
Please note that this image was sourced from old analog film stock and may not be as crisp and clear as modern images obtained from digital media. While every effort has been made to "clean up" the image, analog images like this one may have minor quality issues when enlarged to lifesize proportions, particularly when viewed up close.


text:
Mercy Hospital Oklahoma City Wins National Consumer Choice Award
OKLAHOMA CITY – Mercy Hospital Oklahoma City received the 2014/2015 Consumer Choice Award this month by National Research Corporation. The annual award identifies hospitals across the United States that health care consumers choose as having the highest quality and image.
"It is truly one of the biggest honors to receive an award based on the feedback of our patients," said Jim Gebhart, president of Mercy Hospital Oklahoma City. "Every day, our talented doctors and co-workers dedicate themselves to providing the very best care and service to patients and their loved ones. We are so thrilled to know that patients recognize and value that high level of service and care."
Mercy Hospital Oklahoma is one of four Mercy hospitals and one of just 273 hospitals nationwide to receive the national award this year. Consumers in the Oklahoma City area have recognized Mercy Hospital Oklahoma City as a top hospital based on four key metrics:
Best overall quality
Best image/reputation
Best doctors
Best nurses
Winners are determined by consumer perceptions on multiple quality and image ratings collected in the company's Market Insights survey, the largest online consumer health care survey in the country.
"The complexity of health care has urged consumers to play much more of an integral role in their care decisions, which in turn, is prompting hospitals and health care systems to lead a proactive approach to brand awareness," said Brian Wynne, vice president of sales at National Research Corporation. "This year's winners have done an exceptional job of representing their organizations in terms of high-quality care, improvement initiatives, and positive consumer perceptions and experiences."
Mercy is the fifth largest Catholic health care system in the U.S. and serves millions annually. Mercy includes 34 acute care hospitals, four heart hospitals, two children's hospitals, three rehab hospitals and one orthopedic hospital, nearly 700 clinic and outpatient facilities, 40,000 co-workers and more than 2,000 Mercy Clinic physicians in Arkansas, Kansas, Missouri and Oklahoma. Mercy also has outreach ministries in Louisiana, Mississippi and Texas.
About National Research Corporation
For more than 30 years, National Research Corporation (NASDAQ: NRCIA and NRCIB) has been at the forefront of patient-centered care. Today, the company's focus on empowering customer-centric healthcare across the continuum extends patient-centered care to incorporate families, communities, employees, senior housing residents, and other stakeholders.
Recognized by Modern Healthcare as the largest patient satisfaction firm in the U.S., National Research is dedicated to representing the true voice of patients and other healthcare stakeholders. This integration of cross-continuum metrics and analytics uncovers insights for effective performance improvement, quality measurement, care transitions, and many other factors that impact population health management.
National Research is headquartered in Lincoln, Nebraska. For more information, call 800-388-4264, write to info@nationalresearch.com, or visit www.nationalresearch.com.


text:
Embargo EEZ na uvoz oružja u SFRJ je bila odluka EEZ usvojena 5. srpnja 1991. godine.

Opis
JNA i oružje koje je JNA bespravno uzela od TO Hrvatske koje je JNA podijelila pobunjenim hrvatskim Srbima, velikosrpskim dobrovoljcima i paravojnim skupinama, davali su velikosrpskom agresoru i njegovim pomagačima nesrazmjerno veću prednost u odnosu na operetno, gotovo nikako naoružane i opremljene hrvatske postrojbe.

Odluka je bila još jedna od sramotnog niza odluka međunarodne zajednice (moratorij na odluke o razdruženju, izbjegavanje imenovanja agresora, javne potpore velikih sila bilo kakvoj Jugoslaviji, Rezolucija Vijeća sigurnosti UN-a 713) kojom su odmagali bespomoćnoj žrtvi velikosrpske agresije Hrvatskoj i opstruirali njen otpor, a pomagali srpsko-crnogorskoj agresiji. Odlukom je uveden embargo na uvoz svih vrsta oružja i vojne opreme u SFRJ.Osim što je odluka bila protiv svake logike međunarodnog prava i humanosti, donesena je za žrtvu u najgorem vremenu. Agresija na Hrvatsku koju su izvršile JNA, Srbija, Crna Gora, uz pomoć pobunjenih hrvatskih Srba i elemenata iz BiH, bila je pred trenutkom pojačanja zamaha, kako je tijek događaja poslije pokazao. Embargo je išao na ruku velikosrpskim snagama. Uz to je sprovodio prljave politike velikih sila koje su stvorile Jugoslaviju, koja je bila njihova miljenica, a za čijeg su čuvara izabrale Srbiju, kojoj su cijelo vrijeme Jugoslavije davale odriješene ruke za pacificiranje nezadovoljnika koji su prisilno uvedeni u Jugoslaviju. S obzirom na to da je embargo uveden u vrijeme kad je sa SFRJ otvoreno spadala krabulja "zajednice ravnopravnih naroda i narodnosti" i kad su velikosrpske snage otvoreno krenule u nasilno pretvaranje Jugoslavije u Veliku Srbiju, etnički i vjerski očišćenu od nepodobnih elemenata, čin embarga bio je otvorena potpora nasilju nad međunarodnim pravom, svim idejama prava naroda na svoju državu i slobodu. Embargo je vrlo naštetio borbi hrvatskog naroda koji je poslije svih embargâ bio prisiljen na crnom tržištu po neusporedivo većim cijenama nabavljati prijeko potrebno oružje. Odluka je bila ponižavanje Hrvatske i Hrvata, koje je moćna Europa kao mnogo puta prije praktički ostavila osamljene i ostavljene.

Uz to isti dan donesena je odluka čije vremensko podudaranje s odlukom EEZ-a nije slučajno. Sastali su se dužnosnici Srbije i JNA (Slobodan Milošević – Borisav Jović – Veljko Kadijević) i na tom su sastanku dogovorili vojnu osvetu prema Sloveniji te razmještanje postrojba JNA u Hrvatskoj na područja koja su Srbi svakako željeli izdvojiti iz Hrvatske, uz osvajanje dodatnih područja od onih koje su u tome trenutku držali.

Vidi još
 Embargo na uvoz svih vrsta oružja i vojne opreme u Jugoslaviju
 Raspad Jugoslavije

Bilješke

Vanjske poveznice

Europska unija
Domovinski rat
Povijest neovisne Hrvatske

redpajama_set_name: RedPajamaWikipedia


text:
Photo: OTP
OTP studio visit: Cecilia Fiona
@cecilia_fiona
An interview with the Danish artist at her studio in Copenhagen.
We talk Picasso, public-transport and paintings that 'irritate the eye'.
I arrive at Cecilia Fiona's studio in Nordvest (The northwest region of Copenhagen) on an overcast Thursday afternoon. It's quiet, even for the capital where the majority cycle and sirens rarely sound. Standing next to a huge brickwork façade, that reflects the cold wind, I take in my surroundings; across the road is a park full of fir trees. I ring the buzzer and watch a group of boys guarding their bikes, as they smoke a joint in a clearing by the park gate.
A window swings open above my head and Cecilia Fiona leans out shouting. 'Oscar? Hej!'. The door buzzes, and shortly afterwards I'm welcomed into a warm apartment. There are hardwood floors, the smell of coffee sits in the air, and Ahmad Jamal's 'Stomping at the Savoy' plays through a large hifi system, sprawled in the middle of the room. We position ourselves on a pair of mustard-yellow chairs surrounded by new and half-finished works.
Combining a fauvist palette with primitivist style, Cecilia Fiona's paintings are nostalgic and disarming.
OTP Do you always paint to music?
CF Yes. Not always Jazz, it changes. If I don't know what to listen to, Leonard Cohen is a go-to. I really enjoy his music when painting. It's beautiful and poetic. That atmosphere of melancholy, I think its very inspiring. You like him right?
OTP I don't know his music very well, but I like what I've heard.
CF Ok good. Just checking.
OTP Would you describe your paintings as melancholic?
CF Sometimes. What I'm interested in, in my paintings, is human relations and conflicts more generally. I'm trying to show the moments where people are most vulnerable, where they have to make decisions. This point where things can go one of two ways, and in either case that person's world is about to change. And they know it.
OTP There seem to be some figures who appear more than once in your work, are there recurring 'characters'?
CF Umm yeah, I think that's fair to say. The woman with a child and the elderly man with the hat for example. I have to say it's not on purpose that they keep popping up, its more their relationship that I'm trying to figure out. With each other and the space they're in in. Somehow that's also what I'm doing in my paintings, creating a space or a room for them to occupy or react to. Relationships change in different environments. I am interested in the theme of entrapment, especially in relationships and how rooms can mirror that. That's what I was focusing on in the paintings I did for the exhibition at Virum.
OTP When was that?
CF At the end of last summer, September 2018.
OTP How long have you been painting for?
CF Since 2016. That's when I started painting - before then I did a lot of drawing - I illustrated some children's books, and a novel. All my paintings are trying to frame some sort of story, in a way words aren't able to.
OTP Is that why you started painting? Instead of writing and illustrating?
CF It was more a question of having time and space to paint. My drawings were always closer to painting than drawing somehow, there were a lot of layers and colours, and a lot of blending. I started painting when I finished high school because I had more time, and I moved out of my parents' house.
OTP How do you decide when you're going to paint, now that you have more time? Is it a spontaneous thing or do you plan in advance?
CF I wish that I could just wake up in the morning and be like 'I want to paint today!' And just start painting! But unfortunately that's not possible, I have a job and I study art history as well. To be able to paint you have to plan your time to make it happen. You have to decide 'today I'm going to paint.' And whether you want to do it or not, you have to. Some days it's a mess and you shouldn't have done anything today, but other days you force yourself to do it and you create something exciting. Even though you started out with a mindset saying you didn't want to do anything.
OTP I notice you have quite a lot of sketchbooks around. Do you sketch every work before you paint it?
CF Oh yeah my sketching books, yes I do that. Often the sketches are made on public transport - to do them I just have to sit and be really bored and then my imagination starts to drive. It clicks and then I'll sketch something down. A lot of the sketches I don't use and most of the ones I paint, I sketch a long time before the paintings are made. After they're done I look at the sketches and picture how the painting would be structured and the colours would look, and then if I can feel it in my body, I know that's the one. It's not just a mental process its about body feeling too.
OTP And do the paintings always follow the sketches when it comes to putting paint on canvas?
CF No, sometimes they're quite different. Sometimes I finish a painting, or not finish, but finish the 'first-round' of painting. That usually takes around a week. And then sometimes I'll look at it again after a month and I'll love it even more and sometimes I'll hate it and paint over it. Not the whole thing, usually just a section. How you feel about a painting really changes over time. You're not able to decide if it's good when you've just finished it.
OTP Why do you think that is?
CF Because you are so mixed in to the painting, you're part of it when you're making it and either you're too self-critical or you are too generous and you can't accept your mistakes. You need time so you can look at it with a clear mind. Time gives you new eyes.
OTP You mentioned earlier that you are studying History of Art. Does it effect your work at all coming from that theoretical background?
CF I wouldn't say that I come from a theoretical background. I'm self taught and then the theory comes second, like an after-thought. Its been very important for me to look at other people's art and to visit exhibitions, I think that's what really made me grow as an artist, surrounding myself with other artist's work.
OTP Are there any exhibitions you have seen recently that have had a particularly strong effect?
CF Yes sure, last month I went to an exhibition at V1 in Kødbyen (Meatpacking District) of Emma Kohlmann's work. I was definitely inspired by it afterwards. Sometimes when you paint, you think there are certain things you're not allowed to do in the painting. Formal rules with figuration or colours that you have to follow. It can be unconscious! But you're marked by it. After seeing her exhibition I felt more free in my style, she creates these fantastic dream-like figures, they're original.
When you're on instagram or you go to exhibitions at galleries you can feel pressured to follow the style that everyone is showing. You have to do it your own way instead. That's what people really appreciate. When I was a child I got so bored visiting galleries and I think that was lucky. My parents took me to see Picasso at Louisiana when I was seven but I hated it. I wanted to make my own. I don't want that to sound arrogant, I really enjoy Picasso today.
OTP Do you think you inherited that? Are your parents creative too?
CF I think I did yes, they're both writers. My Dad used to do satirical illustrations for the newspapers. He did cartoons when he was younger. Humour is important in my work too, it's not so obvious but there's definitely an ironic element. It was more pronounced in my early paintings, they were really sarcastic. In my new paintings there's not so much humour but there is definitely irony. The paintings have got darker, more serious.
OTP Do you think this trend will continue with your paintings into the future?
CF The most important thing for me is not get stuck - in the process or the style. What I'm trying to do all the time when I paint is to develop, the ideas and the relationships and the narrative. I think its easy for the artist, when they've found their style or way of painting, its easy to just make the same painting again and again. Copying yourself, its just as bad as copying someone else. When I look on social media, as soon as an artist becomes known for a style they just replicate it over and over again. Of course some themes or ideas recur in my work but it's important for me not to stagnate. To get trapped in a routine. It get's boring.
Every time I look at some older paintings I did, I feel like I can do better, or I can do something else. That's what I want to do with my paintings. I want you to find something new every time you look at them, for something else to irritate the eye. I don't know if you can say that I want to get closer to some kind of truth, because I don't want to talk about truth, but I want my paintings to get closer, maybe to some kind of essence. It's difficult to explain. Look at the paintings.
See more of Cecilia Fiona's work:
All works featured are available to purchase internationally.
Contact OTP or the artist for more information.


In perhaps the final installment of the BetBomb Podcast before the apocalypse, Matt and Steve breakdown the upcoming Bowl games between now and Christmas Day.
Two former conference foes renew a rivalry that is much more important to San Diego State.
Ball State began the season 3-3 and has won six straight games since, while Central Florida was victorious in two of its first four games before finishing 7-2.
Louisiana-Lafayette won four of its last five games, scoring 31 or more points in each of the wins in claiming second place in the Sun Belt, and makes its second consecutive trip to The Big Easy after winning the bowl last year.
It is a testament to how well Chris Petersen and his staff have done at Boise State that a 10-2 season feels like a down year.
The lone Christmas Eve bowl game figures to be gift-wrapped with plenty of offense. Southern Methodist will try to keep pace with Fresno State, which is ranked 14th nationally in total offense at 488.5 yards per game while averaging 40.2 points.
LITTLE CAESARS BOWL - Western Kentucky at Central Michigan Western Kentucky makes its first bowl appearance in history as a program in transition. Coach Willie Taggart left to take the South Florida job, disgraced former Arkansas coach Bobby Petrino takes over the Hilltoppers, but defensive coordinator Lance Guidry will guide the team in the bowl game.
Coach Mike Tomlin dismissed Ben Roethlisberger's post-game comments about Todd Haley's play-calling and game plan as frustration, but said he met with the Steelers quarterback anyway to make sure "he's on board with what it is we're doing."
The Cowboys were able to get by in pass coverage last week without Morris Claiborne against the Pittsburgh Steelers.They'll need him this week against the New Orleans Saints.
The Ravens take on the defending Super Bowl champion New York Giants this Sunday in a game that the visitors need just as badly.Home games had been an advantage for coach John Harbaugh's team for two years until a few weeks ago.