text:
\section{Introduction}
\label{intro}
For many machine learning systems, being able to detect data that is anomalous or significantly different from that used in training can be critical to maintaining safe and reliable predictions.  This is particularly important for deep neural network classifiers which have been shown to %
incorrectly classify such \emph{out-of-distribution} (OOD) inputs into in-distribution classes with high confidence~\citep{goodfellow6572explaining,nguyen2015deep}.
This behaviour can have serious consequences when the predictions inform real-world decisions such as medical diagnosis, e.g.\ falsely classifying a healthy sample as pathogenic or vice versa can have extremely high cost. The importance of dealing with OOD inputs, also referred to as distributional shift, has been recognized as an important problem for AI safety \citep{amodei2016concrete}.
The majority of recent work on OOD detection for neural networks is evaluated on image datasets where the neural network is trained on one benchmark dataset (e.g. CIFAR-10) and tested on another (e.g. SVHN). While these benchmarks are important, there is a need for more realistic datasets which reflect the challenges of dealing with OOD inputs in practical applications.

Bacterial identification is one of the most important sub-problems of many types of medical diagnosis.
For example, diagnosis and treatment of infectious diseases, such as sepsis, relies on the accurate detection of bacterial infections in blood \citep{blauwkamp2019analytical}.
Several machine learning methods have been developed to perform bacteria identification by classifying existing known genomic sequences \citep{patil2011taxonomic,rosen2010nbc}, including deep learning methods \citep{busia2018deep} which are state-of-the-art.
 Even if neural network classifiers achieve high accuracy  as measured through cross-validation, deploying them is challenging as real data is highly likely to contain genomes from unseen classes not present in the training data.
{Different bacterial classes continue to be discovered gradually over the years (see Figure~\ref{fig:accumu} in Appendix~\ref{dataset}) and it is estimated that 60\%-80\% of genomic sequences belong to as yet unknown} bacteria~\citep{zhu2018metagenomic, eckburg2005diversity, nayfach2019new}.
Training a classifier on existing bacterial classes and deploying it may result in
OOD inputs being wrongly classified as one of the classes from the training data with high confidence.
 In addition, OOD inputs can also be the contaminations from the bacteria's host genomes such as human, plant, fungi, etc., which also need to be detected and excluded from predictions \citep{ponsero2019promises}. %
Thus having a method for accurately detecting OOD inputs is critical to enable the practical application of machine learning methods to this important problem.

A popular and intuitive
strategy for detecting OOD inputs is to train a generative model (or a hybrid model cf.~\cite{nalisnick2019hybrid}) on training data and use that to detect OOD inputs at test time  \citep{bishop1994novelty}. However, \citet{nalisnick2018deep} and \citet{choi2018waic} recently showed that deep generative models trained on image datasets can assign higher likelihood to OOD inputs.
We report a similar failure mode for likelihood based OOD detection using deep generative models of genomic sequences. We investigate this phenomenon and find that the likelihood can be confounded by general population level background statistics.
We propose a likelihood ratio method which uses a background model to correct for the background %
statistics and enhances the %
in-distribution specific features for OOD detection. While our investigation was motivated by the genomics problem, we found our methodology to be more general and it shows positive results on image datasets as well.
In summary, our contributions are:
\begin{itemize}
    \item We create a realistic benchmark for OOD detection, that is motivated by challenges faced in applying deep learning models on genomics data. The sequential nature of genetic sequences provides a new modality and hopefully encourages the OOD research community to contribute to ``machine learning that matters''~\citep{wagstaff-2012}.
    \item {We show that likelihood from deep generative models can be confounded by background statistics.}
    \item We propose a %
    likelihood ratio method for OOD detection, which significantly outperforms the raw likelihood on OOD detection for deep generative models on image datasets. %
    \item We evaluate existing OOD methods on the proposed genomics benchmark and demonstrate that our method achieves  state-of-the-art (SOTA) performance on this challenging problem.
    %
    %
    %
    %
\end{itemize}

\section{Background}

Suppose we have an in-distribution dataset $\mathcal{D}$  of $({\bm{x}}, y)$ pairs sampled from the distribution $p^*({\bm{x}}, y)$, where ${\bm{x}}$ is the extracted feature vector or raw input and $y \in \mathcal{Y} := \{1, \dots, k, \dots, K\}$ is the label assigning membership to one of $K$ in-distribution classes. For simplicity, we assume inputs to be discrete, i.e.~$x_d\in\{A,C,G,T\}$ for genomic sequences and $x_d\in\{0,\dots,255\}$ for images. In general, OOD inputs are samples $({\bm{x}}, y)$ generated from an underlying distribution other than $p^*({\bm{x}}, y)$. %
In this paper, we consider an input $({\bm{x}}, y)$ to be OOD if $y \not\in \mathcal{Y}$: that is, the class $y$ does not belong to one of the $K$ in-distribution classes. Our goal is to accurately detect if an input ${\bm{x}}$ is OOD or not.
 %

Many existing methods involve computing statistics using the predictions of (ensembles of) discriminative classifiers trained on in-distribution data, e.g.\ taking the confidence or entropy of the predictive distribution $p(y|{\bm{x}})$~\citep{hendrycks2016baseline,lakshminarayanan2017simple}. %
An alternative is to use generative model-based methods, which %
are appealing as they do not require labeled data and directly model the input distribution.  These methods fit a  generative model $p({\bm{x}})$ to the input data,
and then evaluate the likelihood of new inputs under that model.
However, recent work has highlighted significant issues with this approach for OOD detection on images, showing that deep generative models such as Glow~\citep{kingma2018glow} and PixelCNN~\citep{oord2016pixel,salimans2017pixelcnn++} sometimes assign higher likelihoods to OOD than in-distribution inputs.
For example, \citet{nalisnick2018deep} and \citet{choi2018waic} %
show that Glow models trained on the CIFAR-10 image dataset assign higher likelihood to OOD inputs from the SVHN dataset than they do to in-distribution CIFAR-10 inputs;   \citet{nalisnick2018deep}, \citet{shafaei2018does} and \citet{hendrycks2018deep} show failure modes of PixelCNN and PixelCNN++ for OOD detection. %

\textbf{Failure of density estimation for OOD detection}
We investigate whether density estimation-based methods work well for OOD detection in genomics.
As a motivating observation, we train a deep generative model, more precisely  LSTM \citep{lstm}, on in-distribution genomic sequences (composed by \{A, C, G, T\}), and plot the log-likelihoods of both in-distribution and OOD inputs (See Section~\ref{sec:genomics} for the dataset and the full experimental details).
Figure~\ref{fig:gc_before}a shows that the histogram of the log-likelihood for OOD sequences largely overlaps with that of in-distribution sequences with AUROC of 0.626, making it unsuitable for OOD detection.
Our observations show a failure mode of deep generative models for OOD detection on genomic sequences
and are complementary to earlier work which showed similar results for deep generative models on images \citep{nalisnick2018deep, choi2018waic}.
\begin{figure}[ht]
\begin{center}
    %
     \includegraphics[width=0.325\textwidth]{figs/gene_hist_likelihood_before.pdf}
     \includegraphics[width=0.325\textwidth]{figs/gene_gc_vs_likelihood.pdf}
     %
    \caption{(a) Log-likelihood hardly separates in-distribution and OOD inputs with AUROC of 0.626. (b) The log-likelihood is heavily affected by the GC-content of a sequence. %
    }
    %
    \label{fig:gc_before}
\end{center}
\end{figure}

When investigating this failure mode, %
we discovered that the log-likelihood under the model is heavily affected by a sequence's \emph{GC-content}, see Figure~\ref{fig:gc_before}b.
GC-content is defined as the percentage of bases that are either G or C,  %
and is used widely in genomic studies as a basic statistic for describing overall genomic composition \citep{sueoka1962genetic}, and studies have shown that bacteria have an astonishing diversity of genomic GC-content, from 16.5\% to 75\% \citep{hildebrand2010evidence}.
Bacteria from similar groups tend to have similar GC-content at the population level, but they also have characteristic biological patterns that can distinguish them well from each other.
The confounding effect of GC-content in Figure~\ref{fig:gc_before}b makes the likelihood less reliable as a score for OOD detection, because an
OOD input may result in a higher likelihood than an in-distribution input, because it has high GC-content (cf.~the bottom right part of Figure~\ref{fig:gc_before}b) and not necessarily because it contains characteristic  patterns specific to the in-distribution bacterial classes.

\section{Likelihood Ratio for OOD detection}
We first describe the high level idea and then describe how to adapt it to deep generative models.

\textbf{High level idea} Assume that an input ${\bm{x}}$ is composed of two components, (1) a \textit{background} component characterized by population level background statistics, and (2) a \textit{semantic} component characterized by  patterns specific to the in-distribution data.
For example, images can be modeled as backgrounds plus objects; text can be considered as a combination of high frequency stop words plus semantic words \citep{luhn1960key}; genomes can be modeled as background sequences plus motifs \citep{bailey1995value, reinert2009alignment}.
More formally, for a $D$-dimensional input ${\bm{x}}=x_1, \ldots, x_D$, we assume that there exists an unobserved variable ${\bm{z}}=z_1,\ldots, z_D$, where $z_d \in \{B, S\}$ indicates if the $d$th dimension of the input $x_d$ is generated from the \textit{B}ackground model or the \textit{S}emantic model. Grouping the semantic and background parts, the input can be factored as ${\bm{x}}=\{{\bm{x}}_B,{\bm{x}}_S\}$ where ${\bm{x}}_B = \{x_d \mid z_d = B, d=1, \ldots, D \}$.
For simplicity, assume that the background and semantic components are generated independently.
The likelihood can be then decomposed as follows,
\begin{align}\label{eq:factorization}
p({\bm{x}}) = p({\bm{x}}_B)  p({\bm{x}}_S).
\end{align}
When training and evaluating deep generative models, we typically do not distinguish between these two terms in the likelihood. However, we may want to use just the semantic likelihood $p({\bm{x}}_S)$ to avoid the likelihood term being dominated by the background term (e.g. OOD input with the same background but different semantic component). %
In practice, we only observe ${\bm{x}}$, and it is not always easy to split an input into background and semantic parts $\{{\bm{x}}_B,{\bm{x}}_S\}$. %
As a practical alternative, we propose training a background model by perturbing inputs. Adding the right amount of perturbations to inputs can corrupt the semantic structure in the data, and hence the model trained on perturbed inputs captures only the population level background statistics.

Assume that $p_{\bm{\theta}}(\cdot)$ is a  model trained using in-distribution data, and $p_{{\bm{\theta}}_0}(\cdot)$ is a background model that captures general background statistics.
We propose a likelihood ratio statistic that is defined as
\begin{align}\label{eq:lr}
\mathsf{LLR}({\bm{x}}) = \log \frac{p_{\bm{\theta}}({\bm{x}})}{p_{{\bm{\theta}}_0}({\bm{x}})}
= \log \frac{p_{\bm{\theta}}({\bm{x}}_{B}) \ p_{\bm{\theta}}({\bm{x}}_{S})}{p_{{\bm{\theta}}_0}({\bm{x}}_{B})\ p_{{\bm{\theta}}_0}({\bm{x}}_{S})},
\end{align}
where we use the factorization from \Eqref{eq:factorization}.
Assume that (i) both models capture the background information equally well, that is $p_{{\bm{\theta}}}({\bm{x}}_{B}) \approx p_{{\bm{\theta}}_0}({\bm{x}}_{B})$ and (ii)
   $p_{\bm{\theta}}({\bm{x}}_{S})$ is more peaky than $p_{{\bm{\theta}}_0}({\bm{x}}_{S})$ as the former is trained on data containing semantic information, while the latter model ${\bm{\theta}}_0$ is trained using data with noise perturbations.
Then, the likelihood ratio can be approximated as
\begin{align}
\mathsf{LLR}({\bm{x}}) \approx \log p_{\bm{\theta}}({\bm{x}}_{S}) - \log p_{{\bm{\theta}}_0}({\bm{x}}_{S}).
\end{align}

After taking the ratio, the likelihood for the background component ${\bm{x}}_{B}$ is cancelled out, and only the likelihood %
 for the semantic component ${\bm{x}}_{S}$ remains.  %
Our method produces a \textit{background contrastive} score that captures the significance of the semantics compared with the background model.

\textbf{Likelihood ratio for auto-regressive models}
Auto-regressive models are one of the popular choices for generating images \citep{oord2016pixel, van2016conditional,salimans2017pixelcnn++} and sequence data such as genomics \citep{zou2018primer, killoran2017generating} and drug molecules \citep{olivecrona2017molecular, gupta2018generative}, and text \citep{jozefowicz2016exploring}.
In auto-regressive models, the log-likelihood of an input can be expressed as  $\log p_{{\bm{\theta}}}({\bm{x}})=\sum_{d=1}^{D} \log p_{{\bm{\theta}}} (x_d|{\bm{x}}_{<d})$,
where ${\bm{x}}_{<d}=x_1\ldots x_{d-1}$. Decomposing the log-likelihood into background and semantic parts, we have
\begin{align}
    \log p_{{\bm{\theta}}}({\bm{x}})=\sum_{d: x_d \in {\bm{x}}_{B}} \log p_{{\bm{\theta}}} (x_d|{\bm{x}}_{<d}) + \sum_{d:x_d \in {\bm{x}}_{S}} \log p_{{\bm{\theta}}} (x_d|{\bm{x}}_{<d}).
\end{align}
We can use a similar auto-regressive decomposition for the background model $p_{{\bm{\theta}}_0}({\bm{x}})$ as well. Assuming that both the models capture the background information equally well, $\sum_{d:x_d \in {\bm{x}}_{B}} \log p_{{\bm{\theta}}} (x_d|{\bm{x}}_{<d}) \approx \sum_{d:x_d \in {\bm{x}}_{B}} \log p_{{\bm{\theta}}_0} (x_d|{\bm{x}}_{<d})$, the likelihood ratio is approximated as
\begin{align}
\mathsf{LLR}({\bm{x}}) \approx \sum_{d:x_d \in {\bm{x}}_{S}} \log p_{{\bm{\theta}}} (x_d|{\bm{x}}_{<d}) - \sum_{d:x_d \in {\bm{x}}_{S}}  \log p_{{\bm{\theta}}_0} (x_d|{\bm{x}}_{<d}) = \sum_{d:x_d \in {\bm{x}}_{S}} \log \frac{p_{{\bm{\theta}}} (x_d|{\bm{x}}_{<d})}{p_{{\bm{\theta}}_0} (x_d|{\bm{x}}_{<d})}.
\end{align}

\textbf{Training the Background Model} In practice, we add perturbations to the input data by randomly selecting positions in $x_1\ldots x_D$ following an independent and identical Bernoulli distribution with rate $\mu$ and substituting the original character with one of the other characters with equal probability. %
The procedure is inspired by genetic mutations.
See Algorithm~\ref{alg:noise} in Appendix~\ref{sec:pseudocode} for the pseudocode for generating input perturbations. %
The rate $\mu$
is a hyperparameter and can be easily tuned using a small amount of validation OOD dataset (different from the actual OOD dataset of interest).
In the case where validation OOD dataset is not available, we show that $\mu$ can also be tuned using simulated OOD data.
In practice, we observe that $\mu \in [0.1,0.2]$ achieves good performance empirically for most of the experiments in our paper.
Besides adding perturbations to the input data, we found other techniques that can improve model generalization and prevent model memorization, such as adding $L_2$ regularization with coefficient $\lambda$
to model weights, can help to train a good background model. In fact, it has been shown that adding noise to the input is equivalent to adding $L_2$ regularization to the model weights under some conditions \citep{bishop1995regularization, bishop1995training}.
Besides the methods above, we expect adding other types of noise or regularization methods would show a similar effect.
The pseudocode for our proposed OOD detection algorithm can be found in Algorithm \ref{alg:llr} in Appendix~\ref{sec:pseudocode}.

\section{Experimental setup}
We design  experiments on multiple data modalities (images, genomic sequences) to evaluate our method and compare with other baseline methods.
For each of the datasets, we build an auto-regressive model for computing the log-likelihood $\log p_{\bm{\theta}}({\bm{x}})=\sum_{d=1}^{D} \log p_{{\bm{\theta}}}(x_d|{\bm{x}}_{<d})$.
For training the background model $p_{{\bm{\theta}}_0}({\bm{x}})$, we use the exact same architecture as  $p_{{\bm{\theta}}}({\bm{x}})$, and the only differences are that it is trained on perturbed inputs and (optionally) we apply $L_2$ regularization to model weights.

\textbf{Baseline methods for comparison}
We compare our approach to several existing methods. %
\begin{enumerate}
    \item The maximum class probability, $p(\hat{y}|{\bm{x}})=\max_k p(y=k|{\bm{x}})$.  %
    OOD inputs tend to have lower scores than in-distribution data~\citep{hendrycks2016baseline}.
    \item The entropy of the predicted class distribution, $-\sum_{k} p(y=k|{\bm{x}}) \log p(y=k|{\bm{x}})$. High entropy
    of the predicted class distribution, and therefore a
    %
    high predictive uncertainty, which suggests that the input may be OOD.
    \item The ODIN method proposed by \citet{liang2017enhancing}. ODIN uses temperature scaling \citep{guo2017calibration}, adds small perturbations to the input, and applies a threshold to the resulting predicted class to distinguish in- and out-of- distribution inputs. This method was designed for continuous inputs and cannot be directly applied to discrete genomic sequences. We propose instead to add perturbations to the input of the last layer that is closest to the output of the neural network.
    \item The Mahalanobis distance of the input to the nearest class-conditional Gaussian distribution estimated from the in-distribution data. \citet{lee2018simple} fit  class-conditional Gaussian distributions
    %
    to the activations from the last layer of the neural network. %
    \item The classifier-based ensemble method that uses the average of the predictions from multiple independently trained models with random initialization of network parameters and random shuffling of training inputs \citep{lakshminarayanan2017simple}.
   %
    \item The log-odds of a binary classifier trained to distinguish between in-distribution inputs from all classes as one class and randomly perturbed in-distribution inputs as the other.
    \item The maximum class probability over $K$ in-distribution classes of a $(K+1)$-class classifier where the additional class is perturbed in-distribution.
    \item The maximum class probability of a $K$-class classifier for in-distribution classes but the predicted class distribution is explicitly trained to output %
    uniform distribution on perturbed in-distribution inputs. This is similar to  using simulated OOD inputs from GAN \citep{lee2017training} or using auxiliary datasets of outliers \citep{hendrycks2018deep} %
    for calibration purpose. %
    \item The generative model-based ensemble method that measures
    $\mathbb{E}[\log p_{\bm{\theta}}({\bm{x}})]-\mathrm{Var}[\log p_{\bm{\theta}}({\bm{x}})]$
    of multiple likelihood estimations from independently trained model with random initialization and random shuffling of the inputs. \citep{choi2018waic}.
    %
    %
\end{enumerate}

Baseline methods 1-8 are classifier-based and method 9 is generative model-based.
For classifier-based methods, we choose the commonly used model architecture, convolutional neural networks (CNNs). %
Methods 6-8 are based on perturbed inputs which aims to mimic OOD inputs. Perturbations are added to the input in the same way as that we use for training background models.
Our method and methods 3, 6, 7, and 8 involve hyperparameter tuning; %
we follow the protocol of \citet{hendrycks2018deep} where optimal hyperparameters are picked on a different OOD validation set than the final OOD dataset it is tested on. %
For Fashion-MNIST vs. MNIST experiment, we use the  NotMNIST \cite{notmnist} dataset for hyperparameter tuning. For CIFAR-10 vs SVHN, we used gray-scaled CIFAR-10 for hyperparameter tuning. For genomics, we use the OOD bacteria classes discovered between 2011-2016, which are disjoint from the final OOD classes discovered after 2016.
While this set of baselines is not exhaustive, it is broadly representative of the range of existing methods.
{Note that since our method does not rely on OOD inputs for training, we do not compare it with other %
methods that do utilize OOD inputs in training.}

\textbf{Evaluation metrics for OOD detection}
We trained the model using only in-distribution inputs, and we tuned the hyperparameters using validation datasets that include both in-distribution and OOD inputs. The test dataset is used for the final evaluation of the method.
For the final evaluation, we randomly selected the same number of in-distribution and OOD inputs from the test dataset, and for each example ${\bm{x}}$ we computed the log likelihood-ratio statistic $\mathsf{LLR}({\bm{x}})$ as the score.
A small value of the score suggests a high likelihood of being OOD. We use the area under the ROC curve (AUROC$\uparrow$), the area under the precision-recall curve (AUPRC$\uparrow$), and the false positive rate at 80\% true positive rate (FPR80$\downarrow$), as the metrics for evaluation. These three metrics are commonly used for evaluating OOD detection methods \citep{hendrycks2016baseline, hendrycks2018deep, alemi2018uncertainty}.

\section{Results}
We first present results on image datasets as they are easier to visualize, and then present results on our proposed genomic dataset. For image experiments, our goal is not to achieve  state-of-the-art performance but to show that our likelihood ratio effectively corrects for background statistics and significantly outperforms the likelihood. While previous work has shown the failure of PixelCNN for OOD detection, we believe ours is the first to provide an explanation for why this phenomenon happens for PixelCNN, through the lens of background statistics.

 %
\subsection{Likelihood ratio for detecting OOD images}

Following existing literature \citep{nalisnick2018deep, hendrycks2018deep}, we evaluate our method using two experiments for detecting OOD images: (a) Fashion-MNIST as in-distribution and MNIST as OOD, (b) CIFAR-10 as in-distribution and SVHN as OOD.
For each experiment, we train a PixelCNN++ \citep{salimans2017pixelcnn++, van2016conditional} model using in-distribution data. %
We train a background model by adding perturbations to the training data. %
To compare with classifier-based baseline methods, we use CNN-based classifiers.
See Appendix \ref{sec:image_model} for model details.
Based on the likelihood from the PixelCNN++ model, we confirm that the model assigns a higher likelihood to MNIST than Fashion-MNIST, as previously  reported by \citet{nalisnick2018deep}, and the AUROC for  OOD detection is only 0.091, even worse than  random (Figure~\ref{fig:image_hist}a). %
We discover that the proportion of zeros i.e.~\emph{number of pixels belonging to the background in an image is a confounding factor to the likelihood score} (Pearson Correlation Coefficient 0.85, see Figure~\ref{fig:image_hist}b, Figure~\ref{fig:showimage_likelihood}).
Taking the likelihood ratio between the original and the background models, we see that the AUROC improves significantly from 0.091 to 0.996 (Figure~\ref{fig:image_hist}d).
The log likelihood-ratio for OOD images are highly concentrated around value 0, while that for in-distribution images are mostly positive (Figure~\ref{fig:image_hist}c).

\begin{figure}[h]%
    \centering
    \myvspace{-1em}
    \begin{subfigure}[]{
    \includegraphics[width=0.325\textwidth]{figs/image_hist_likelihood_before.pdf}
    }\end{subfigure}
    \begin{subfigure}[]{
    \includegraphics[width=0.325\textwidth]{figs/image_freq0_vs_likelihood.pdf}
    }\end{subfigure}
    \begin{subfigure}[]{
    \includegraphics[width=0.325\textwidth]{figs/image_hist_likelihood-ratio_after.pdf}
    }\end{subfigure}
    %
    \begin{subfigure}[]{
    \includegraphics[width=0.325\textwidth]{figs/image_auc_before_and_after_emilycl_no_transform.pdf}
    }\end{subfigure}
    %
    \myvspace{-1em}
    \caption{(a) Log-likelihood of MNIST images (OOD) is higher than that of Fashion-MNIST images (in-distribution). (b) Log-likelihood is highly correlated with the background (proportion of zeros in an image). (c) Log-likelihood ratio is higher for Fashion-MNIST (in-dist) than MNIST (OOD). %
    %
    (d) Likelihood ratio significantly improves the AUROC of OOD detection from 0.091 to 0.996.
    %
    }
    %
    \label{fig:image_hist}
   \myvspace{-1em}
\end{figure}

\textbf{Which pixels contribute the most to the likelihood (ratio)?} To qualitatively evaluate the difference between the  likelihood and the likelihood ratio, we plot their values for each pixel for  Fashion-MNIST and MNIST images. This allows us to visualize which pixels contribute the most to the two terms respectively.
Figure~\ref{fig:image_heat:mnist} shows a heatmap, with lighter (darker) gray colors indicating higher (lower) values.
Figures~\ref{fig:image_heat:mnist}(a,b) show that the likelihood value is dominated by the ``background'' pixels, whereas likelihood ratio focuses on the ``semantic'' pixels. Figures~\ref{fig:image_heat:mnist}(c,d) %
confirm that the background pixels cause MNIST images to be assigned high likelihood, whereas likelihood ratio focuses on the semantic pixels.
We present additional qualitative results in Appendix~\ref{appendix:image}. For instance, Figure~\ref{fig:showimage_ratio} shows that images with the highest likelihood-ratios are those with prototypical Fashion-MNIST icons, e.g. ``shirts'' and ``bags'', highly contrastive with the background, while images with the lowest likelihood-ratios are those with rare patterns, e.g.~dress with stripes or sandals with high ropes.

\begin{figure}[ht]%
    \centering
    \begin{subfigure}[Likelihood]{
        \includegraphics[width=0.22\textwidth]{figs/image_fashimnist_heatmap_likelihood_grey.png}
    }\end{subfigure}
    \begin{subfigure}[Likelihood-Ratio]{
    \includegraphics[width=0.22\textwidth]{figs/image_fashimnist_heatmap_ratio_grey.png}
    }\end{subfigure}
    \begin{subfigure}[Likelihood]{
    \includegraphics[width=0.22\textwidth]{figs/image_mnist_heatmap_likelihood_grey.png}
    }\end{subfigure}
    \begin{subfigure}[Likelihood-Ratio]{
    \includegraphics[width=0.27\textwidth]{figs/image_mnist_heatmap_ratio_grey.png}
    }\end{subfigure}
    %
    \vspace{-1em}
    \caption{The log-likelihood of each pixel in an image $\log p_{{\bm{\theta}}}(x_d | {\bm{x}}_{<d})$, and the log likelihood-ratio of each pixel $\log p_{{\bm{\theta}}}(x_d | {\bm{x}}_{<d}) - \log p_{{\bm{\theta}}_0}(x_d | {\bm{x}}_{<d}), d=1\dots, 784$., for 16 Fashion-MNIST images (a, b) and MNIST images (c, d).  Lighter gray color indicates larger value (see colorbar). Note that the range of log-likelihood (negative value) is different from that of log likelihood-ratio (mostly positive value). For the ease of visualization, we unify the colorbar by adding a constant to the log-likelihood score. The images are randomly sampled from the test dataset and sorted by their likelihood $p_{\theta}(\mathbf{x})$. Looking at which pixels contribute the most to each quantity, we observe that the likelihood value is dominated by the ``background'' pixels on both Fashion-MNIST and MNIST, whereas likelihood ratio focuses on the ``semantic'' pixels.}
    %
    \label{fig:image_heat:mnist}
   %
\end{figure}

We compare our method with other baselines. The classifier-based baseline methods are built using LeNet architecture. %
Table \ref{tab:both}a
shows that our method achieves the highest AUROC$\uparrow$, AUPRC$\uparrow$, and the lowest FPR80$\downarrow$.
The method using Mahalanobis distance performs better than other baselines.
Note that the binary classifier between in-distribution and perturbed in-distribution does not perform as well as our method, possibly due to the fact that while the features learned by the discriminator can be good for detecting perturbed inputs, they
may not generalize well for OOD detection.
The generative model approach based on $p({\bm{x}})$ captures more fundamental features of the data generation process than the discriminative approach.

\begin{table}
\caption{AUROC$\uparrow$, AUPRC$\uparrow$, and FPR80$\downarrow$ for detecting OOD inputs using likelihood and likelihood-ratio method and other baselines on (a) Fashion-MNIST vs. MNIST datasets and (b) genomic dataset. The up and down arrows on the metric names indicate whether greater or smaller is better.
$\mu$ in the parentheses indicates the background model is tuned only using noise perturbed input, and ($\mu$ and $\lambda$) indicates the background model is tuned by both perturbation and $L_2$ regularization. Numbers in front and inside of the brackets are mean and standard error respectively based on 10 independent runs with random initialization of network parameters and random shuffling of training inputs. For ensemble models, the mean and standard error are estimated based on 10 bootstrap samples from 30 independent runs, which can be underestimations of the true standard errors.}
\vspace{-0.5em}
\begin{minipage}[b]{0.50\linewidth}
\myvspace{-2em}
\begin{adjustbox}{max width=\textwidth}
\centering
\begin{tabular}{lccc}%
 &AUROC$\uparrow$&AUPRC$\uparrow$&FPR80$\downarrow$  \\ \hline
Likelihood                       & 0.089 (0.002) & 0.320 (0.000) & 1.000 (0.001)  \\
Likelihood Ratio (ours, $\mu$) &  0.973 (0.031) & 0.951 (0.063) & 0.005 (0.008) \\
Likelihood Ratio (ours, $\mu, \lambda$) & \textbf{0.994 (0.001)} & \textbf{0.993 (0.002)} & \textbf{0.001 (0.000)} \\
$p(\hat{y}|{\bm{x}})$               & 0.734 (0.028) & 0.702 (0.026) & 0.506 (0.046)    \\
Entropy of $p(y|{\bm{x}})$          & 0.746 (0.027) & 0.726 (0.026) & 0.448 (0.049)    \\
ODIN                                  & 0.752 (0.069) & 0.763 (0.062) & 0.432 (0.116) \\
Mahalanobis distance                  & 0.942 (0.017) & 0.928 (0.021) & 0.088 (0.028)  \\
Ensemble, 5 classifiers & 0.839 (0.010) & 0.833 (0.009) & 0.275 (0.019) \\
Ensemble, 10 classifiers & 0.851 (0.007) & 0.844 (0.006) & 0.241 (0.014) \\
Ensemble, 20 classifiers & 0.857 (0.005) & 0.849 (0.004) & 0.240 (0.011)  \\
Binary classifier                     & 0.455 (0.105) & 0.505 (0.064) & 0.886 (0.126) \\
$p(\hat{y}|{\bm{x}})$ with noise class & 0.877 (0.050) & 0.871 (0.054) & 0.195 (0.101) \\
$p(\hat{y}|{\bm{x}})$ with calibrations & 0.904 (0.023) & 0.895 (0.023) & 0.139 (0.044) \\
WAIC, 5 models                  & 0.221 (0.013) & 0.401 (0.008) & 0.911 (0.008) \\ \hline \\
\end{tabular}
\end{adjustbox}
\centerline{(a)}
\end{minipage}
\hfill
\begin{minipage}[b]{0.50\linewidth}
\centering
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{lccc}%
 & AUROC$\uparrow$ & AUPRC$\uparrow$ & FPR80$\downarrow$ \\ \hline
Likelihood                & 0.626 (0.001) & 0.613 (0.001) & 0.661 (0.002) \\
Likelihood Ratio (ours, $\mu$)  & 0.732 (0.015) & 0.685 (0.017) & 0.534 (0.031) \\
Likelihood Ratio (ours, $\mu, \lambda$)  & \textbf{0.755 (0.005)} & \textbf{0.719 (0.006)} & \textbf{0.474 (0.011)} \\
 $p(\hat{y}|{\bm{x}})$                 & 0.634 (0.003) & 0.599 (0.003) & 0.669 (0.007) \\
Entropy of $p(y|{\bm{x}})$             & 0.634 (0.003) & 0.599 (0.003) & 0.617 (0.007) \\
Adjusted ODIN                     & 0.697 (0.010) & 0.671 (0.012) & 0.550 (0.021) \\
Mahalanobis distance              & 0.525 (0.010) & 0.503 (0.007) & 0.747 (0.014) \\
Ensemble, 5 classifiers           & 0.682 (0.002) & 0.647 (0.002) & 0.589 (0.004) \\
Ensemble, 10 classifiers          & 0.690 (0.001) & 0.655 (0.002) & 0.574 (0.004) \\
Ensemble, 20 classifiers          & 0.695 (0.001) & 0.659 (0.001) & 0.570 (0.004) \\
Binary classifier                 & 0.635 (0.016) & 0.634 (0.015) & 0.619 (0.025) \\
$p(\hat{y}|{\bm{x}})$ with noise class & 0.652 (0.004) & 0.627 (0.005) & 0.643 (0.008) \\
$p(\hat{y}|{\bm{x}})$ with calibration & 0.669 (0.005) & 0.635 (0.004) & 0.627 (0.006) \\
WAIC, 5 models                    & 0.628 (0.001) & 0.616 (0.001) & 0.657 (0.002) \\ \hline \\
\end{tabular}
\end{adjustbox}
\centerline{(b)}
\end{minipage}
\myvspace{-1em}
\label{tab:both}
\end{table}

For the experiment using CIFAR-10 as in-distribution and SVHN as OOD, we apply the same training procedure using the PixelCNN++ model and choose hyperparameters using grayscaled CIFAR-10 which was shown to be OOD by \citet{nalisnick2018deep}. See Appendix~\ref{sec:image_model} for model details.
Looking at the results in Table~\ref{tab:cifar-llr},
we observe that the OOD images from  SVHN  have  higher likelihood than the in-distribution images from CIFAR-10, confirming the observations of \citet{nalisnick2018deep}, with AUROC of 0.095.
Our likelihood-ratio method  significantly improves the  AUROC to 0.931. Figure~\ref{fig:image_heat:cifar} in Appendix~\ref{appendix:image} shows additonal qualitative results.
For detailed results including other baseline methods, see Table \ref{tab:cifar} in Appendix~\ref{sec:image_table}.
\begin{table}
\centering
\caption{
CIFAR-10 vs SVHN results:
AUROC$\uparrow$, AUPRC$\uparrow$, FPR80$\downarrow$ for detecting OOD inputs using likelihood and our likelihood-ratio method. %
}
\vspace{-0.5em}
\begin{adjustbox}{max width=0.6\textwidth}
\begin{tabular}{llll}
& AUROC$\uparrow$ & AUPRC$\uparrow$ & FPR80$\downarrow$       \\ \hline
Likelihood & 0.095 (0.003) &	0.320 (0.001) & 1.000 (0.000) \\
Likelihood Ratio (ours, $\mu$) & 0.931 (0.032)	& 0.888 (0.049) & 0.062 (0.073) \\
Likelihood Ratio (ours, $\mu$, $\lambda$) & 0.930	(0.042)	& 0.881	(0.064)	& 0.066	(0.123) \\ \hline
\end{tabular}
\end{adjustbox}
\myvspace{-2em}
\label{tab:cifar-llr}
\end{table}

\subsection{OOD detection for genomic sequences}\label{sec:genomics}

\textbf{Dataset for detecting OOD genomic sequences}
\begin{figure}[ht]
\begin{center}
    \centering
     %
    %
    \includegraphics[width=0.48\textwidth]{figs/ood_dataset.pdf}
    \caption{The design of the training, validation, and test datasets for genomic sequence classification including in and OOD data.
    %
    }
    \label{fig:dataset}
   %
\end{center}
\end{figure}
We design a new dataset for evaluating OOD methods.
As bacterial classes are discovered gradually over time, in- and out-of-distribution data can be naturally separated by the time of discovery.
Classes discovered before a cutoff time can be regarded as in-distribution classes, and those discovered afterward, which were unidentified at the cutoff time, can be regarded as OOD.
We choose two cutoff years, 2011 and 2016, to define the training, validation, and test splits (Figure~\ref{fig:dataset}).
Our dataset contains of 10 in-distribution classes, 60 OOD classes for validation, and 60 OOD classes for testing.
Note that the validation OOD dataset is only used for hyperparameter tuning, {and the validation OOD classes are disjoint from the test OOD classes}.
To mimic sequencing data, we fragmented genomes in each class into short sequences of 250 base pairs, which is a common length that current sequencing technology generates.
Among all the short sequences, we randomly choose 100,000 sequences for each class for training, validation, and test.
Additional details about the dataset, including pre-processing and the information for the in- and out-of-distribution classes, can be found in Appendix \ref{dataset}.

\textbf{Likelihood ratio method for detecting OOD sequences}
We build an LSTM model for estimating the likelihood $p({\bm{x}})$ based on the transition probabilities $p(x_d|{\bm{x}}_{<d})$, $d=1,\dots, D$.
In particular, we feed the one-hot encoded DNA sequences into an LSTM layer, followed by a dense layer and a softmax function to predict the probability distribution over the 4 letters of $\{A, C, G, T\}$, and train the model using only the in-distribution training data.
We evaluate the likelihood for sequences in the OOD test dataset under the trained model, and compare those with the likelihood for sequences in the in-distribution test dataset.
The AUROC$\uparrow$, AUPRC$\uparrow$, and FPR80$\downarrow$ scores are 0.626, 0.613, and 0.661 respectively
(Table
\ref{tab:both}b).

We  train a background model by using the perturbed in-distribution data and optionally adding $L_2$ regularization to the model weights. Hyperparameters are tuned using validation dataset which contains in-distribution and validation OOD classes, and the validation OOD classes are disjoint from test OOD classes.
Contrasting against the background model, the AUROC$\uparrow$, AUPRC$\uparrow$, and FPR80$\downarrow$ for the likelihood-ratio significantly improve to 0.755, 0.719, and 0.474, respectively
(Table \ref{tab:both}b,
Figure~\ref{fig:genomics}b).
Compared with the likelihood, the AUROC and AUPRC for likelihood-ratio increased 20\% and 17\% respectively, and the FPR80 decreased 28\%.
Furthermore, Figure~\ref{fig:genomics}a shows that the likelihood ratio is less sensitive to GC-content, and the separation between in-distribution and OOD distribution becomes clearer. %
We evaluate other baseline methods on the test dataset as well.
For classifier-based baselines, we construct CNNs with one convolutional layer, one max-pooling layer, one dense layer, and a final dense layer with softmax activation for predicting class probabilities, as in \citet{alipanahi2015predicting, busia2018deep, ren2018identifying}.
Comparing our method to the baselines in
Table~\ref{tab:both}b,
our method achieves the highest AUROC, AUPRC, and the lowest FPR80 scores on the test dataset.
Ensemble method and ODIN perform better than other baseline methods.
Comparing with the Fashion-MNIST and MNIST experiment, the Mahalanobis distance performs worse for detecting genomic OOD possibly due to the fact that Fashion-MNIST and MNIST images are quite distinct while in-distribution and OOD bacteria classes are interlaced under the same taxonomy (See Figure \ref{fig:phylo} for the phylogenetic tree of the in-distribution and OOD classes).

\begin{figure}[ht]
     \centering
    %
    %
    \begin{subfigure}[]{
   %
        \includegraphics[width=0.31\textwidth]{figs/gene_gc_vs_likelihood-ratio.pdf} %
    }\end{subfigure}
    \begin{subfigure}[]{
  %
        \includegraphics[width=0.31\textwidth]{figs/gene_auc_before_and_after_0528.pdf} %
    }\end{subfigure}
    %
    %
    \begin{subfigure}[]{
    %
        \includegraphics[width=0.31\textwidth]{figs/gene_mindist_auc_llr_ensemble_0528.pdf} %
    }\end{subfigure}
    \myvspace{-1em}
    \caption{
    (a) The  likelihood-ratio score is roughly independent of the  GC-content  which makes it less susceptible to background statistics and better suited for OOD detection. %
    (b) ROCs and AUROCs for OOD detection using likelihood and likelihood-ratio. (c) Correlation between the AUROC of OOD detection and distance to in-distribution classes using Likelihood Ratio and the Ensemble method. %
    %
    %
    }
        \myvspace{-0.5em}
    \label{fig:genomics}
\end{figure}

\textbf{OOD detection correlates with its distance to in-distribution} %
We investigate the effect of the distance between the OOD class to the in-distribution classes, %
on the performance of OOD detection.
To measure the distance between the OOD class to the in-distribution, we randomly select  representative genome from each of the in-distribution classes and OOD classes.
We use the state-of-the-art alignment-free method for genome comparison, $d_2^S$ \citep{ren2018alignment, reinert2009alignment}, to compute the genetic distance between each pair of the genomes in the set.
This genetic distance is calculated based on the similarity between the normalized nucleotide word frequencies ($k$-tuples) of the two genomes, and studies have shown that this genetic distance reflects true evolutionary distances between genomes \citep{chan2014inferring, bernard2016alignment, lu2017cafe}.
For each of the OOD classes, we use the minimum distance between the genome in that class to all genomes in the in-distribution classes as the measure of the genetic distance between this OOD class and the in-distribution.
Not surprisingly, the the AUROC for OOD detection is positively  correlated with the genetic distance (Figure~\ref{fig:genomics}c), and an OOD class far away from in-distribution is easier to be detected.
Comparing our likelihood ratio method and one of the best classifier-based methods, ensemble method, we observe that our likelihood ratio method has higher AUROC for different OOD classes than ensemble method in general.
Furthermore, our method has a higher Pearson correlation coefficient (PCC) of 0.570 between the minimum distance and AUROC for Likelihood Ratio method, than the classifier-based ensemble method with 20 models which has PCC of 0.277.
The dataset and code for the genomics study is available at \url{https://github.com/google-research/google-research/tree/master/genomics_ood}.

  \myvspace{-0.5em}
\section{Discussion and Conclusion}
  \myvspace{-0.5em}

We investigate deep generative model-based methods for OOD detection and show that the likelihood of auto-regressive models can be confounded by %
background statistics, providing an explanation to the failure of PixelCNN for OOD detection observed by recent work  \citep{nalisnick2018deep,hendrycks2018deep,shafaei2018does}. %
We propose a likelihood ratio method that alleviates this issue by contrasting the likelihood against a background model.
We show that our method effectively corrects %
for the background components, and significantly improves the accuracy of OOD detection on both image datasets and genomic datasets. %
Finally, we create and
release a realistic genomic sequence dataset for OOD detection which highlights an important real-world problem, and
hope that this serves as a valuable OOD detection benchmark for the research community.

\subsubsection*{Acknowledgments}
We thank
Alexander A. Alemi,
Andreea Gane,
Brian Lee,
D. Sculley,
Eric Jang,
Jacob Burnim,
Katherine Lee,
Matthew D. Hoffman,
Noah Fiedel,
Rif A. Saurous,
Suman Ravuri,
Thomas Colthurst,
Yaniv Ovadia,
the Google Brain Genomics team, and Google TensorFlow Probability team
for helpful feedback and discussions.

redpajama_set_name: RedPajamaArXiv


The form Picture books for children represents a specific category or genre of resources found in District of Columbia Public Library System.
Bear & Hare : snow!
Cómo dicen estoy enojado los dinosaurios?
Does a fiddler crab fiddle?
First Dog of 1600 Pooch'lvania Avenue : My First Year in Arf, Arf Office!!
Has anyone seen my Emily Greene?
Help! We need a title!
How long is a whale?
Hugo & Miles in : I've painted everything!


text:
Lyndon Johnson and Vietnam
He wasn't a dove and he exiled those—including his Vice President—who wanted to end the war.
Written by Norman Sherman
| Published on May 21, 2014
Vice President Hubert H. Humphrey in 1968 with his press secretary, Norman Sherman, aboard Air Force II.
Fifty years ago, Lyndon Baines Johnson was sworn in as President and for the next five years lived with a war far away and protests close to home. Today his family and former aides are trying to remake him as a tormented dove who tried his best to end the Vietnam war.
His daughter Luci Baines Johnson recently told the New York Times: "Nobody wanted the war less than Lyndon Johnson," adding, "No matter how hard he tried, he didn't seem to be able to get out of that quagmire. Not only did he not get out of it in his lifetime, but his legacy indeed has the weight of the world on it."
The Johnson effort is understandable, but it is built on fantasy. When LBJ took the oath of office, there were 19,000 American military in Vietnam. When he left office there were 500,000.
Someone ordered them there.
One person who really "wanted the war less" than LBJ was his Vice President, Hubert H. Humphrey. At Humphrey's first National Security Council meeting, he made the case for getting out of Vietnam. That was his last opportunity for a year. Because the Vice President is by statute a member of the NSC, Lyndon Johnson limited Vietnam discussion to other meetings—those to which Humphrey was not invited.
Humphrey became a Vice President without portfolio. He was silenced and demeaned in absentia. Johnson soon began delivering in Texan poetry an off-repeated line, "Ah got Hubert's pecker in mah pocket." Johnson found petty ways to show his anger, and his example was not lost on White House aides. Joseph Califano, one of today's revisionist leaders but back then only in his mid-thirties and prematurely arrogant, once kept the Vice President waiting for 45 minutes after Humphrey arrived for a scheduled White House meeting.
Ultimately Humphrey was begrudgingly resurrected through the efforts of Lady Bird Johnson, who maintained her long friendship with Muriel Humphrey. Johnson sent Humphrey to Vietnam, and the Vice President became a quietly skeptical public cheerleader for Johnson policies.
When Johnson announced on March 31, 1968, that he would not seek a second term, whatever dove-like tendencies he held should have made him an advocate of Humphrey and at least accepting of some deviation from his hardline Vietnam policies. Without that, the mood of the country that drove him out of office would haunt Humphrey. But Johnson invited no private discussion with Humphrey, no wiggle room for any policy shift.
Humphrey was not alone in this treatment. Mike Mansfield, the Senate majority leader and a dove, told a friend that he never discussed Vietnam with Johnson because "LBJ doesn't think it is useful to spend time on an issue where he is not going to change his mind."
As the presidential nominating convention was about to begin at the end of August 1968, chaos was inevitable in Chicago. We knew what television would show. Senator Walter Mondale, close friend of Humphrey for 20 years and co-chair of the Humphrey for President Committee, called Mayor Richard Daley and suggested that the convention be moved to Miami—where the Republicans had held their convention—as the city would be prepared for another. When that suggestion was rejected, Mondale urged that some space be made available where protesters could gather and make their speeches. Daley said in dismissal, "We will give them every courtesy."
A second imperative was a Democratic party peace plank. It did not require total repudiation of the war or an insult to Johnson. Humphrey set out to get one. Accepting suggestions and reservations, he found common ground with hawks and doves. He cleared his ideas with Secretary of State Dean Rusk, White House foreign policy advisor Walt Rostow, Charlie Murphy, a close outside advisor to the President, and with the Eugene McCarthy forces—although McCarthy remained not quite totally satisfied or cooperative.
Congressman Hale Boggs of Louisiana, a close friend of Johnson, showed up in the Humphrey convention suite to say that no deviation from the Johnson Vietnam policy would come out of the platform committee he chaired. Boggs dismissed what Humphrey reported about finding consensus. Boggs clearly spoke for the President.
What none of us knew was that President Johnson had decided he was ready to challenge Humphrey for the nomination. Johnson called Mayor Daley, a reluctant Humphrey supporter, asking two things: Could his helicopter land safely on the Hilton Hotel roof and did Daley think he could get enough votes to beat Humphrey? Daley, no structural engineer or national vote counter, said yes to both. The Secret Service said no. They couldn't assure LBJ's safety on the roof or on the street. He finally gave it up.
But his minions were on the ground, put in place when Johnson still was likely to be the candidate. They ran the convention—from where we all would stay to what would be in the platform and who would be in the audience. Humphrey's son-in-law stood in line with the public to get tickets for the family to attend the nominating session.
The Chicago convention, my fourth, was worse than we expected. Daley's courtesy was later called a "police riot." Protestors were beaten, jailed, and threatened by police swinging their horses' back ends into crowds. Some protesters brought their anger into the hotels, often making a ride on an elevator into a misadventure of tension and fear.
All of that made my own job as Humphrey press secretary close to impossible. Press briefings could not be held in a normal way: in a room with podium, microphone, and seats. I briefed the press standing in a hallway surrounded by elevator doors opening and closing. On nomination day, I stood and shouted my information, answered shouted questions, taking a final question from a reporter I could not see. "Will Hubert take Lyndon as his vice presidential running mate tonight?" There was laughter and I said, "Lyndon who? And that's off the record." One reporter decided that, since I had instructed that my quip be kept off the record after I'd said it, he could file it. Unfortunately he was from the Associated Press.
When I got to Humphrey's suite, he asked, "Did some son-of-a-bitch just ask 'Whose Lyndon?'" I said, "You've got the right son-of-a-bitch but the wrong line." He had just received a call from the LBJ Ranch from Arthur Krim, Hollywood mogul, New York financier, and our main fundraiser. He said if Humphrey had guys like me on the staff, he would not be raising another dollar. Humphrey told me he could hear the President breathing into a separate telephone.
The air was leaving our campaign as well. Humphrey finally decided he had to speak out, that if he remained silent on Vietnam, the protests would continue with shouts of "Dump the Hump" and signs that said "Killer of Babies." No national campaign is easy, but this one was unrelentingly hard and almost hopeless. By the fall, Humphrey was 18 percentage points behind Richard Nixon in the polls, and there seemed no way up without some new policy on Vietnam.
At the end of September, Humphrey phoned Johnson as a courtesy from a television studio in Salt Lake City. He told the President he was going to call for stopping the bombing of North Vietnam. Johnson had few words. He said first: "I take it you are not asking for my advice." He listened for a bit as Humphrey tried to explain what he intended to do then icily said, "You're going to give the speech anyway. Thanks for calling, Hubert." And he hung up.
Humphrey did not appease Johnson, but he did persuade many Americans. The signs along our campaign motorcades changed. They read, "If you mean it, we're for you." Shouts of anger turned into cheers of support. Humphrey said he could feel the difference. We came to the final week, ahead in one poll, almost even in others.
As we approached election day, Vietnam peace talks went on in Paris. President Johnson had softened his hard-line attitude, and there seemed hope for a change in policy leading to peace. But Anna Chennault, a Chinese charmer who had married an American World War II hero, was a major fundraiser for Nixon and assumed, or more likely was assigned, the task of preventing any peace agreement before the election.
Theodore White, the writer-chronicler of presidential elections, later wrote that Anna Chennault "had undertaken most energetically to sabotage (the peace talks). In contact with the Formosan, the South Korean and the South Vietnamese governments, she had begun early, by cable and telephone, to mobilize their resistance to the agreement—apparently implying that she spoke for the Nixon campaign." The South Vietnamese President repudiated any peace agreements.
Humphrey, although he knew what was happening, would not say anything about Chennault's activities because the information was based on intelligence sources. Had Johnson informed him, given Humphrey a heads-up, we might have been able to speak out just enough to make a difference. But we heard nothing. I begged Humphrey to let me tell all of this to the press. I was certain that Americans of both parties would be outraged at what was a treasonous act by Nixon and that we would get the final boost we needed. I told him that if it rebounded against us, he could fire me as the unauthorized leaker.
Teddy White concluded: "Fully informed of the sabotage of the negotiations by our negotiators (secretly and without White House knowledge) and the recalcitrance of the Saigon government, Humphrey might have won the Presidency of the United States by making it the prime story of the last four days of the campaign. He was urged by several members of his staff to do so. And I know of no more essentially decent story in American politics than Humphrey's reluctance to do so." It was the final moment when Johnson could have helped elect Humphrey, and probably seen the Vietnam war end. He chose not to.
Lyndon Johnson was a good President, a great one in the eyes of some historians. But he could be arbitrary and mean. His treatment of Humphrey, on Vietnam and other matters, was not the mark of civility, much less greatness. Humphrey was not the first to suffer his wrath.
In 1960, Johnson's close friend Jim Rowe sent him a letter. "I have not seen you pay one compliment, thank one person. I have seen you do nothing but yell at them….Maybe you do not know it—I do—the morale of your staff is awful. They are in tears, all of them, they are beginning to dislike you intensely. They cannot do anything right, they don't dare make a decision about where to hang your clothes even, and they bend their heads and wait for the blows to fall—like obdurate mules who know the blow is coming."
Rowe had a solution and suggestion: "One day a week, go up and down that plane and tell George Reedy and Bill Moyers and the stenographers…that you appreciate what they are doing for you….I have a feeling that at present you are caught between vanity…and a curious lack of self-confidence about your judgment of men." Then he laments, "This is probably the end of an old friendship. But somebody has to say these things. And I will say one more thing I didn't mean to say—lay off that booze."
Shutting up Vice President Humphrey after that first NSC meeting may well have resulted in thousands of American and millions of Vietnamese lives lost. The war dragged on, and the country endured turmoil and protest on the streets, on campuses, even in churches. It brought us Richard Nixon and Watergate. Had Humphrey been permitted to speak, the course of the war might well have been different. Had his voice been heard and his ideas considered, there would be no need today to remake Johnson into something he wasn't. Lyndon Johnson never flew on the wings of a tormented dove.
Norman Sherman was Vice President Hubert H. Humphrey's press secretary and editor of Humphrey's autobiography, The Education of a Public Man. He spent many years living in Chevy Chase.
More: Washington Voices
Norman Sherman
City Kids Wilderness Project Turns 20, Looks Forward to Its Next Decade
25 Years Ago: The Fall of the Berlin Wall
La Belle Indifference
Ruby: The Rest of the Story

redpajama_set_name: RedPajamaCommonCrawl


Jefferson Montero has returned to Emelec, in Ecuador, on a sub-loan from Getafe.
Swansea City winger Jefferson Montero has joined former club Emelec on a sub-loan from Spanish side Getafe for the remainder of the season.
The Ecuador international moved to Getafe on a season-long loan in the summer but has been limited to just 55 minutes in four substitute appearances.
Swansea permitted Getafe to complete the deal, subject to international clearance, though Montero will remain on Getafe's books, according to a statement from Swansea.
FC Dallas have loaned Anibal Chala to LDU Quito for the remainder in 2017, hoping to have him healthy and ready for 2018.
Walker Zimmerman talks about FC Dallas' unbeaten start in MLS and what to expect against NYCFC's potent attack.
Oscar Pareja breaks down the challenges FC Dallas face when they host New York City FC at the weekend.
FC Dallas loaned out one of its new designated players on Friday, before Anibal Chala ever played a game for the MLS club.
Chala joined Dallas as a young DP from Ecuador's Nacional in December, but the 21-year-old has yet to debut after suffering a sports hernia injury in the preseason.
And Dallas has now put off the defender's first MLS appearance until 2018 by loaning him back to another Ecuador club, LDU Quito, for the rest of the season.
Fernando Zampedri scored as Atletico Tucuman used kits borrowed from the Argentina U20 side after they arrived at their Copa Libertadores match without their luggage.
Atletico Tucuman on Tuesday beat El Nacional of Ecuador, 1-0, in the return leg of the second round of Copa Libertadores on Tuesday despite being more than an hour-and-a-half late for the kickoff in Quito and wearing kits and boots borrowed from the Argentina under-20 team.
The Argentina side saw their flight delayed nearly three hours in Guayaquil because their chartered flight to Quito did not have clearance to fly with passengers.
Jose Angulo will not play for Granada after failing a drugs test.
Granada are in the process of terminating the contract of summer signing Jose Angulo because of a failed drugs test.
The player had been suspended since Aug. 25, when Granada learned from the South American Football Confederation (CONMEBOL) that the Ecuador international striker had tested positive for a banned substance during the Copa Libertadores.
The Spanish club have announced that the player's B sample was also positive.
Brazil legend Ronaldinho is in advanced talks to play a Jan. 30 friendly match for Ecuador side Barcelona Sporting Club and could also be making a return to former club Fluminense.
Ronaldinho's brother and agent, Roberto de Assis, said that the Guayaquil club's front-office directors are near a deal for the club's "Noche Amarilla," the event at which the Ecuador Primera A side present their players for the 2016 season and play a friendly match against another league club.


Enlarge / Comcast's mobile offerings come by reselling Verizon service.
The short version is that videos will be throttled to 480p (DVD quality) on all Comcast mobile plans unless you pay extra, while Comcast's "unlimited" plan will limit mobile hotspot speeds to 600kbps. Only customers who pay by the gigabyte will get full-speed tethering, but the cost would add up quickly as Comcast charges $12 for each gigabyte.
Comcast last year began selling mobile plans with data, voice, and texting. Comcast doesn't operate its own cellular network, so it resells Verizon Wireless service.
The new speed limits could help Comcast save money on the reselling fees it pays Verizon. In a statement to Ars, Comcast said it's making the changes "to help us maintain the low price point of Xfinity Mobile." While Comcast doesn't control the Verizon network and thus can't impose speed limits directly, Comcast told Ars that Verizon offers resellers an option to limit video resolution.
Comcast said that 480p "is consistent with standard unlimited plans across carriers," and that it's making the changes to bring Comcast's offerings in line with the rest of the industry. Verizon began throttling mobile video to 480p or 720p on smartphones last year; the system limits the amount of bandwidth available to anything identified as a video. Carriers argue that many consumers can't tell the difference between 480p and higher quality on smartphone screens.
Comcast has been telling Xfinity Mobile customers about the changes in an email titled, "Update on cellular video resolution and personal hotspots." One Comcast customer forwarded the email to Ars, and there's also a Reddit thread discussing the email. All Xfinity Mobile customers are receiving this email, Comcast confirmed to Ars.
But after the interim period is over, getting HD-quality video will cost extra. "Later this year, 720p video over cellular data will be available as a fee-based option with your service," Comcast told customers. Comcast hasn't mentioned any plans to make 1080p or any resolution higher than 720p available over cellular, and there's no word from Comcast yet on what enabling 720p video will cost. Comcast previously imposed no speed limits on video streaming on the Xfinity Mobile service.
The Comcast email also directs customers to an FAQ describing the video changes.
While the video throttling applies both to Comcast's limited and unlimited service plans, the new tethering speed limit only applies to the unlimited plan.
"If you have the Unlimited data option, your speeds on any device connected to a personal hotspot will not exceed 600kbps," Comcast told customers. Xfinity Mobile users can get 4G hotspot speeds by switching to Comcast's "By the Gig" pricing.
By the Gig customers pay $12 for each gigabyte of usage. The unlimited plan costs $45 a line each month, and it's unlimited in the sense that there are no overage fees for using too much data. But the unlimited plan only provides 20GB of full-speed data each month—after 20GB, the plan is limited to 1.5Mbps downstream and 750kbps upstream at all times. This limitation was already in place when Comcast started offering mobile service last year.
Comcast only sells mobile service to customers who subscribe to Comcast home Internet.
Comcast: Speed limits are good for you!
While Comcast is imposing the video and hotspot changes shortly after the repeal of net neutrality rules, the rules may not have stopped this form of throttling. The rules banned throttling, but they had an exception for "reasonable network management." The major wireless carriers have imposed various video and hotspot limits on their mobile networks without being punished by the Federal Communications Commission.
Comcast is trying to convince customers that the new speed limits will be good for them. The 480p video limit "can help you save money if you pay By the Gig," and it could help unlimited data customers stay under the 20GB throttling threshold, Comcast told customers in the email.
On the 600kbps tethering limit, Comcast told customers, "At this speed, you'll conserve data so that it takes longer to reach the 20GB threshold, but you'll still be able to do many of the online activities you enjoy."
I love the doublethink required to act like helping you conserve data on a supposedly-unlimited plan is doing you a favor.
Data is data is data. Why the hell should your ISP care how the hell your data is being used when you have a contract spelling out how much data you're getting?
I'm not American but this is rage inducing nonetheless.


text:
The Greenwich Village Apartments Created by Jane Jacobs Are Threatened With Demolition
By Roberta Brandes Gratz
The West Village Houses are threatened with demolition. This is the Greenwich Village complex developed by the community group led by Jane Jacobs in the 1960s as an alternative to the proposed Urban Renewal plan the group defeated. It is the seminal Jacobs physical mark on this or any city, a landmark of greater significance than many buildings designated as such. Demolition would be no less a loss than demolition of an important designated landmark.
In 1989, my first book, The Living City: Thinking Small in a Big Way, was published and included the full story of the West Village Houses development, as told to me by Jacobs, who had become a friend. This was one of several examples in the book that I cited of real community engagement, genuine debate and effective planning. The following is an excerpt from the book, with occasional bracketed current comments.
The West Village Houses grew out of a community-led battle in the 1960s against Robert Moses' urban renewal plan. This was one of the first times—if not the first—a New York community triumphed over an imposed public-private project that would have destroyed the neighborhood. The high-rise development plan, between Christopher and Eleventh Streets on the West Side Highway, called for clearance of fourteen blocks of apartment houses, older brick and brownstone houses and loft buildings deemed deteriorated, marginal and "a no man's land" by a City Planning study. A community-initiated study, however, showed that the area contained several hundred families and more than 80 businesses employing hundreds of people. After all the demolition and relocation there would have been a net increase of only about 300 dwelling units. A diverse group of neighbors formed the West Village Committee with Jane Jacobs as president. Death and Life of Great American Cities had just been published. The Committee hired its own planning and design team [Perkins & Will] to come up with an alternative. That alternative, the West Village Houses, consists of forty-two five-story walk-ups that required no demolition or relocation and provided a net increase of 420 apartments. "Not a single person—not a single sparrow—shall be displaced" was their slogan.
The Houses went through a tortuous history, including a mortgage foreclosure that gave its critics ammunition with which to minimize its accomplishments and gave its defenders ammunition with which to accuse public officials of subtle sabotage. Few publicly built projects ever came in on time and on budget. This was no exception. It had so much going against it because of its unconventional nature and official disapproval that it is nothing but a miracle that it was built at all. (Its community sponsorship alone was enough to disturb establishment circles afraid that the idea might spread.) Official thinking then: Demolition, replacement with new high rises surrounded by acres of "keep off the grass" open space and parking lots and no mixing with commercial uses (the antithesis of the threatened neighborhood).
The community design provided for low-rise row houses with private courtyards. The goal was to avoid disruptions and displacements and to enhance the area by filling in of weed-strewn, odd-sized empty lots left when the old freight-railroad spur [connected to what is now the High Line] demolished years before. Respect for existing scale and flexibility of design were basic goals. The plan, furthermore, was achievable without total site assemblage, so it could be contracted or expanded according to available land. No condemnation of private property would be necessary.
Pedestrian-oriented public spaces added to the area's traditional lively street life. Car space was provided in a nearby garage. Three basic floor layouts were designed—adjustable in width and depth and thus usable on different lot sizes—for five-story walk-ups. Elevators were expensive and space-consuming. Brownstone and brick walk-ups had long been a rentable Village commodity. As in the older houses, the first floor was raised and a basement level was a few steps down. Wherever space permitted, top floors contained duplexes reachable by four stair flights. Each floor had either two 2-bedroom apartments or one 3-bedroom and one 1-bedroom, guaranteeing a variety in family sizes. The orientation of apartments alternated on each floor, with a living room facing the front and a kitchen facing the back on one floor and the reverse on the next, "the idea being," Jacobs said, "that at all times there would be eyes and ears of the neighborhood" on the street and backyard. One of the schemes provided that the ground floor could be either a store or an apartment, providing flexibility for changing times and needs.
This was the first in New York City of what planners now advocate as "infill" housing, new construction filling in existing open spaces without causing demolition. On every point, however, the West Village Housing plan was at loggerheads with the "experts" of the day. Today, it seems so simple and basic but would still find official resistance. "The plan is deceptive," Jacobs said in retrospect. "A great deal of innovative and creative planning went into this. It was done so it could fit into all kinds of empty holes in the Village or elsewhere. We could see the holes coming in the city. This plan could be used on a lot as small as fifty feet deep. It alarmed city officials that we had something for which you didn't have to knock anything down. They were frightened enough about a neighborhood doing it themselves."
This plan was too radical for its time. Opposition came from hostile city officials threatened by the unheard-of community demand for meaningful participation, as well as from private real-estate interests who objected to developable land being turned over to nonprofit community-design housing.
This plan was too radical for its time. Opposition came from hostile city officials threatened by the unheard-of community demand for meaningful participation, as well as from private real-estate interests who objected to developable land being turned over to nonprofit community-designed housing.
Rachelle Wall, like many inexperienced-neighbors-turned-activist, said: "We could have established a community pattern that others would learn from, but the opposition made sure that didn't happen. They created delays, inflated estimates, kept construction people away from bidding, nibbled at the design. One roadblock after another created unnecessary but costly delays. We never knew who was working against us."
Two mayors came and went before the city's Board of Estimate finally approved construction in 1969 under the city's middle-income housing program. Another three-year bureaucratic bog-down ensued, doubling costs by the time construction began in 1972, almost 10 years after the community proposal was made. Complications didn't end there. The city plunged into fiscal chaos in 1975. The builder walked off the job, leaving the project incomplete and near bankruptcy. Federal subsidies were used to bail out two big privately sponsored faltering high-rise projects built elsewhere in the city under the same program. City officials said, however, that the same federal subsidies were unavailable to similarly rescue the West Village House, because nearby truck traffic and pollution violated federal environmental standards. The city foreclosed on the mortgage, canceled the original plans as a co-operative and sold the complex to the builder who had first abandoned it. The buildings were finished as rental units.
After years of cost-cutting sacrifices of architectural niceties, the completed structures had a decidedly bleak exterior. One could make a long list of the design flaws. Yet, when finally on the market, rental was immediate, with an unending waiting list and a low turnover—clearly signs of success. If anything, the West Village Houses, spread over six sites and slipped in and around other buildings, make an acceptable background for the infinite variety of noteworthy and luxuriously renovated old buildings, many of which would have been demolished under the urban renewal scheme [and now in the heart of the Historic District—Jane had hoped that ivy would grow on them.]
In its planning and design flexibility and its potential cost-saving efficiency, West Village Houses was a trailblazer. Actually, it fulfilled the "two basic aims" set forth in the West Village Community's 1963 published plan:"(a) to fulfill its promise both to Robert Wagner and to its own neighbor-members to improve and further develop the community once the threat of dictatorial redevelopment was lifted; and (b) to devise, as a public example, a practical means of adding harmonious planned housing into an existing community without any sacrifice of the people already there."
It was a Pyrrhic victory at best. Under the best of circumstances, citizen-led change is nearly impossible. The almost seventeen years of controversy and delay guaranteed that few lessons would be learned or transmitted. Nowadays, it is referred to as a failed experiment. The "experts" made sure they were proved right, says Wall with a wistful smile. "How do you say to people that there is no truth in what the city said? We weren't trying to be entrepreneurs. We were simply spearheading concepts and ideas we thought could work and should be tried." An October 14, 1974, New York Times editorial applauded the aim of the West Village Houses "to prove that new development could reinforce the Village's specific attractions and lifestyle rather than destroy them." It further noted that "although design was reduced to the lowest common denominator to keep the project alive, there is still much to commend in terms of appropriate scale, neighborhood conservation and well-planned apartments." But, the editorial concluded, the city bureaucracy's handling of the project was obstructionist and the West Village Houses are "unplanned monuments to a system geared to sabotage and a city determined to red-tape itself to death."
By the standards of many, the West Village Houses are not architecturally attractive and, in fact, may be viewed as plain at best. This only demonstrates that what constitutes neighborhood is often enhanced by aesthetics, but not necessarily. Aesthetics are too often used as a standard that in the context of neighborhood rebirth or stabilization is not the highest priority. Most of all, the West Village project fits the fabric of the city and its neighborhood. It did not disrupt the institutions and social networks of the community. The larger area remains diverse and exciting. Stability and reinforcement were the result, not accelerated change and alterations. Those are the characteristics that give this project value, aesthetics notwithstanding.
Featured image via Peter Olive.
democratic design
Roberta Brandes Gratz is an award-winning journalist and author of The Battle For Gotham: New York in the Shadow of Robert Moses and Jane Jacobs. She is also president of the Center For the Living City.


The chart shows the China ETF and it is one of indecision. In the last month, the fund has entered what looks to be a symmetrical triangle formed by two converging blue trendlines.
I drew the green line along the bottom of the triangle, backward in time to the congestion area in July. Along the tops, I drew a red line. This line represents overhead resistance.
If the ETF does break out upward, then look for it to throwback at the red line especially since it coincides with the three valleys in November and December (numbered).
The breakout from a symmetrical triangle can be in any direction and when it does breakout, price sometimes retraces its move, dropping out the opposite side of the original breakout.
That is what happened here. Price closed above the top trendline and then reversed, moving below the bottom of the pattern. On a happier note, price rebounded and when it touched overhead resistance shown by the red line, the ETF reversed, just as I predicted.


Editors of the 90s like Sound Designer and later Peak were designed to pull samples from a sampler so you could edit them adding and editing markers and loop points etc. graphically instead of having to do it from the front panel of the sampler.
Ah, the good old days! All I really need is an editor that will allow me to do basic audio editing to a sample clip, set and embed seamless loop points and then export that as wave (or AIFF etc depending) with the edited info intact for import to whatever sampler I want to use. I am a bit disappointed that for all the other great audio editing features in PT, you can set and embed loop points in a clip.
Not sure what you mean? Before you can play anything live, you have to create the patch. To do that, I need to be able to prep and edit the sample I want to use to create the patch.
Or do you mean playing a live acoustic instrument? But I'm mostly not creating live instrument type patches, I'm going for more original type sounds.
Which is what a lot of folks do, and there's a lot you can do with that for sure. But, to do pads and other sounds that could be sustained over a few measures, you really need to do the loop thing. Structure works fairly well in that regard. The drawback is that it is not always easy to find the sweet spot using the Structure wave editor. Also, Structure doesn't have any sort of "Duplicate Part" function. I find this very annoying as quite often I'll import a set of multisamples to the first part layer in Structure. Edit the samples with loops, etc. But, now if I want to duplicate all that to have, say, 2 or 3 of the same part layered up for depth or whatever, there's no option to do that. So, I have to use all sorts of workarounds to accomplish that. I find that very annoying!
With a dedicated sample editor, I could take a set of multisamples, do all the edits at once. Then import those to as many part layers as I wish with the editing embedded.


text:
Pittsburgh Flicks and Nightlife in August
This month: Wigle Whiskey's Shot and a Beer Summer Sundays.
Sean Collier,
photo courtesy wigle whiskey
In the year or so since the bygone, Strip District gathering place Bayardstown closed its metaphorical doors, no comparable entity has risen to take its place. The onetime social club — which I sadly only began to appreciate in its dying hours — served as an all-ages backyard for the city's apartment-dwellers, inviting neighbors from near and far to bring drinks and gather around an outdoor fire on summer evenings.
Wigle Whiskey's Shot and a Beer Summer Sundays, however, provide a pretty close approximation of the experience — albeit on a somewhat smaller scale. Located at Wigle's barrelhouse in Spring Garden, these afternoons do have one pretty big advantage over comparable hangouts: Spirits and cocktails from Wigle (wiglewhiskey.com) for sale at the barrelhouse's bar.
There's also a weekly beer-tasting component; for $10, you'll receive a pour of one of Wigle's offerings and a draft from a local brewery, the titular shot-and-beer special. It's ideal for those looking to sample regional offerings; one recent visit featured sour beers from nearby Draai Laag Brewing Company, well suited to pair with Wigle's spirits.
Like Bayardstown, the environment is a large part of the appeal. Having received two drinks for your $10, you'll want to grab one of the picnic tables or chairs scattered around Wigle's lot-sized yard and relax. There may be a stray guitarist strumming somewhere; there inevitably will be small groups broken out playing board games (a selection are available).
You'd be better off arriving by a Lyft, and not just because of the alcohol; parking is a bit of a challenge. The location (and, frequently, the heat) remove some of the charm — frankly, I'd rather have these sessions on heady summer nights than sweaty summer afternoons — but that's quibbling. With a beer and a whiskey in your hand, you'll be glad you came.
Summer Sundays, which last from 2-7 p.m. and frequently feature food truck visits, are scheduled to continue through Oct. 22.
We've had plenty of superhero movies over the past quarter-century, but no single year has offered as many great comic-book flicks as 2017. "Logan," "Wonder Woman," "Spider-Man: Homecoming" and "Guardians of the Galaxy Vol. 2" are all exemplary — and we're not done yet.
If this is the year of the superhero movie, then, what finer time to look back? Films featuring capes and capers weren't always as ubiquitous as they are in the modern era, but that scarcity only made those older adventures more memorable. From Aug. 25-31, Row House Cinema hosts a week of superhero movies — and at least three of the films will be from the 20th century.
"The older films carry a great amount of weight in plot — the emotional struggle of being a superhero," says Brian Mendelssohn, owner of Row House. "And nostalgia that cannot be matched by the newer movies."
Row House will spotlight the 1978 "Superman," starring Christopher Reeve, as well as Tim Burton's "Batman Returns" — wherein Michael Keaton squares off against Danny DeVito's Penguin and Michelle Pfeiffer's Catwoman — and "Blade." (A fourth film is unconfirmed as of press time but will probably be a modern choice.)
This article appears in the August 2017 issue of Pittsburgh Magazine.
Categories: Things To Do


LIVERPOOL will not be able to land Aleix Vidal for any less than £45million because he is expected to replace Dani Alves, according to reports.
Vidal joined Barcelona only last summer in a deal that could be worth up to £17m depending on add-on conditions met.
He had attracted the attentions of the Spanish giants in his first and only season at Sevilla as he helped them to the second of three consecutive Europa League titles.
The right-back had previously been on the books of no fewer than 15 clubs including both Barca and Real Madrid as a youth player.
Vidal has struggled to break onto the first team at the Nou Camp this season with only 14 appearances in all competitions.
And according to Spanish newspaper Mundo Deportivo, Jurgen Klopp is planning on offering Vidal a way out of Barcelona this summer.
However, the report also claims that the Liverpool will have a hard time convincing Barcelona to sell.
Luis Enrique is said to see Vidal as the perfect replacement for 33-year-old Dani Alves who is expected to leave this summer with Juventus the most likely destination.
It means that the only way Barca could be convinced to sell is if the Reds match his minimum fee release clause which is reportedly around the £45m mark.
And while Klopp is desperate to reinforce his squad in a big way this summer, spending nearly £50m on a right-back is unlikely to be at the top of his list.


Many veterans of the Gulf War have expressed concern that their unexplained illnesses may result from their experiences in that war. In response to veterans' concerns, the Department of Defense established a task force in June 1995 to investigate incidents and circumstances relating to possible causes. The Office of the Special Assistant to the Deputy Secretary of Defense for Gulf War Illnesses assumed responsibility for these investigations on November 12, 1996, and has continued to investigate topics related to reports of chemical warfare agent incidents.
To inform the public about the progress of these efforts, the Department of Defense is publishing on the Internet and elsewhere accounts that may contribute to the discussion of possible causes of illnesses of Gulf War veterans, along with documentary evidence or personal testimony used in compiling the accounts. This information paper will aid in understanding incidents involving Iraq's use of Scud missiles.


text:
Speaker Series: August 15
Lauridsen Performing Arts Theatre
Okoboji Magazine presents the last of 2019's Pearson Lakes Art Center Speaker Series: "Booze, Books, & Business"
Thursday, August 15th at 7:30 pm
Featuring Jonathan Reed, historian and author of "Lost Resorts of the Iowa Great Lakes"
Closing the regular four program season will be Jonathan Reed, Arnolds Park resident, author and historian who has just completed his latest volume, Lost Resorts of the Iowa Great Lakes.
Jonathan's book should be back from the printer by August and with the help of some power point images Reed promises, Thursday, August 15, to amaze readers with long forgotten stories about various lodges, hotels and entertainment centers that helped make Okoboji the destination it is today.
Become a Friend of the Arts and Enjoy Discounts on Classes and Workshops
Become a member to receive discounts, be the first to learn about upcoming events and enjoy the satisfaction of knowing you're supporting the Lakes Art Center.
Being in the Know Feels Great
Be the first to know about new exhibitions, show tickets and classes.
September — May
Tues. — Sat. 10 am — 4 pm
Free Thursdays 10 am — 9 pm
June — August
Mon. — Sat. 10 am — 4 pm
Sundays 12 pm — 3 pm
Okoboji, IA 51355
© 2019 All Rights Reserved. Pearson Lakes Art Center


With just the page name the test has always only worked for html files and that is still true. The test used to work with php files if you included the .php extension on the file name. Now even that doesn't work.
It is not necessary to set php in the remaining html files as the default value for php is false.


text:
Burlington police seek to debunk video accusing officers of wrongdoing
YouTube video wrongly accuses officers of watching illegal drug injection
Updated: 11:35 PM EDT Aug 17, 2018
Tom Garris
Anchor/Reporter
WEBVTT DRUGS. NBC5'S TOM GARRIS IS LIVE OUTSIDE OF THE BURLINGTON POLICE DEPARTMENT. TOM, WHAT DID POLICE HAVE TO SAY TODAY? TOM: AND THEY SAY THINGS LIKE THIS HAPPEN EVERY SO OFTEN. WHEN THEY TAKE VIDEO, SOMETIMES IT GETS TAKEN OUT OF CONTEXT. THE VIDEO ACCUSES OFFICERS OF WRONGDOING WHEN REALLY THEY WERE JUST DOING THEIR JOB. >> THEY ARE SHOOTING UP IN FRONT OF TWO COPS. DIDN'T KNOW THAT WAS LEGAL. TOM: THIS YOUTUBE VIDEO POPPED UP AGAIN, CLAIMING TO POLICE OFFICERS WATCHED SOMEONE INJECT HEROIN. POLICE SAY THAT ISN'T WHAT HAPPENED. >> JUST BECAUSE SOMETHING WAS POSTED ON SOCIAL MEDIA DOESN'T MEAN IT'S ACCURATELY TRYING WHAT HAPPENED. TOM: NO ILLEGAL DRUGS WERE INVOLVED. THE MAN WAS GIVING HIM SELF-MEDICATION TO TREAT DIABETES. >> WE CAN LOOK AT VIDEO FOOTAGE AND THERE ARE INCIDENT REPORTS TO FIND OUT WHAT IS GOING ON. TOM: VIDEOS CAN RESURFACE. THIS IS A-YEAR-OLD. >> EVERYBODY NEEDS TO TAKE A STEP BACK. TOM: HE'S A PROFESSOR OF JOURNALISM FOCUSING ON SOCIAL MEDIA POLICIES. HE SAYS EVERYBODY IS FOCUSED ON ATTACKS LIKE THIS. >> RESPONDING QUICKLY AND A CASE LIKE THIS SO YOU CAN DIFFUSE THE VILE NATURE OF IT RIGHT AWAY. I THINK THAT'S CRITICAL. TOM: BURLINGTON POLICE KNOW THEY ARE DEALING WITH THE PUBLIC EYE AND DEAL WITH THIS ON A CASE-BY-CASE ACES. >> UNLESS YOU KNOW THE FACTS, YOU SHOULD JUST BE PATIENT. TOM: THE VIDEO RESURFACING AFTER A YEAR. THEY SAID THIS IS WHAT HAPPENED AND WE STAND BY OUR OFFICERS. IT'S IMPORTANT TO SEE IF IT'S COMING FROM A VERIFIED SOURCE AND POLICE WILL BE KEEPING AN
Burlington police say a YouTube video from a year ago has resurfaced on social media sites.The video's title accuses two BPD officers of watching someone inject heroin on a public street."Just because something is posted online, doesn't mean it's true," Sgt. James Trieb said.Police reports from the day in question say that the man was injecting himself with saline solution as a response to a diabetic condition. He let police search him, and they found insulin and other related materials."Everybody needs to take a step back, because we know there are so many dupe-sters out there," said Anthony Adornato, a journalism professor at Ithaca College who studies social media habits and policies.He said that everyone is vulnerable to attacks like this on social media platforms, whether intentional or not."Responding quickly in a case like this so you can diffuse the viral nature of this, I think that is kind of critical," he said.The Burlington Police Department said they deal with incidents like these on a case by case basis.
BURLINGTON, Vt. —
Burlington police say a YouTube video from a year ago has resurfaced on social media sites.
The video's title accuses two BPD officers of watching someone inject heroin on a public street.
"Just because something is posted online, doesn't mean it's true," Sgt. James Trieb said.
Police reports from the day in question say that the man was injecting himself with saline solution as a response to a diabetic condition.
He let police search him, and they found insulin and other related materials.
"Everybody needs to take a step back, because we know there are so many dupe-sters out there," said Anthony Adornato, a journalism professor at Ithaca College who studies social media habits and policies.
He said that everyone is vulnerable to attacks like this on social media platforms, whether intentional or not.
"Responding quickly in a case like this so you can diffuse the viral nature of this, I think that is kind of critical," he said.
The Burlington Police Department said they deal with incidents like these on a case by case basis.


text:
Bidding System
Plates List
VEG9
https://www.carplates.my/offer/veg-9
See Price.
More Plates From this Agent.
Alphabet:
Digit:
Car Plate
Contact Car plate Owner
Send a contact e-mail to car plate owner
This page informs you of our policies regarding the collection, use and disclosure of Personal Information we receive from users of the Site.
While using our Site, we may ask you to provide us with certain personally identifiable information that can be used to contact or identify you. Personally identifiable information may include, but is not limited to your name (Personal Information).
Cookies are files with small amount of data, which may include an anonymous unique identifier. Cookies are sent to your browser from a web site and stored on your computer's hard drive. Like many sites, we use cookies to collect information. You can instruct your browser to refuse all cookies or to indicate when a cookie is being sent. However, if you do not accept cookies, you may not be able to use some portions of our Site.
This Privacy Policy is effective as of (1/10/2018) and will remain in effect except with respect to any changes in its provisions in the future, which will be in effect immediately after being posted on this page.
An item has been added to the cart
View Shopping cart	Close
No 1A, Jalan Molek 2/1, Taman Molek, 81100 Johor Bahru, Johor.
BRANCH :
E-03-03A , Pusat Komersial Starparc Point, Jalan Taman Ibu Kota, Setapak, 53300 Kuala Lumpur.
Returns, Refund & Cancellation Policy
HOURS: Mon - Fri : 9.00am - 6.00pm
EMAIL: admin@carplates.my
CONTACT: +60 19-788 9999
Copyright © 2022 CarPlates.My
Don't have an account yet?	Register Now! Or Become an Agent!


text:
Here's why one pollster believes Michelle Obama has the best chance to win in 2020
Across the U.S.A.
Caught on video
Energy future
Race in America
Rare exclusive
What's wrong with people
How President Trump has talked about 9/11, then and now
Patrick McMahon, September 11, 2017 8:27 am
Man in Coma After a Bulldog Ripped Off His Penis and Testicles
Rob Fox, July 18, 2019 4:43 pm
Public Schools Will Require Mental Health Classes Starting in Sixth Grade
Man Tries to Escape Cops on Bike, Faceplants Hard, Gets Tasered
President Donald Trump and the first lady are marking the 16th anniversary of the Sept. 11 attacks on Monday by observing a moment of silence and attending a Pentagon ceremony with Vice President Mike Pence.
It's his first 9/11 anniversary in office, and Trump's honoring the day in keeping with his predecessors. In a statement last week, his language was appropriately presidential:
[O]ur Nation recalls the nearly 3,000 innocent people murdered on September 11, 2001. As we reflect on our sorrow and our grief, we come together to pray for those who lost loved ones. As a Nation, we pray that the love of God and the comfort of knowing that those who perished are forever remembered brings them peace and gives them courage.
But how did Trump talk about the attacks before coming to the White House?
On Sept. 11, 2001: Size still matters
On the day of the attacks, Trump also made a jaw-dropping comment that, after the collapse of the Twin Towers, a property of his at 40 Wall Street was then the "tallest in Manhattan," a dubious claim; 40 Wall Street was shorter than the tallest post-9/11 building in New York City by 25 feet.
RELATED: Donald Trump pulled this insensitive 9/11 tweet from 2013 after sharing it again
Two days later: "New York is very strong and resilient and will rebuild very quickly"
Trump is a native New Yorker, meaning the events of Sept. 11 have an additional significance to him. The former real estate developer was interviewed in Lower Manhattan near the site of the attack by German news network N24 on Sept. 13, 2001.
"I've never seen anything like it, the devastation … New York is very strong and resilient and will rebuild very quickly," he told the network. He also advocated for a "quick" and "effective" United States response to the perpetrators of the World Trade Center attacks and vowed that he and his businesses would "be involved" in clearing and reconstructing the area around Ground Zero.
On Twitter: "Best wishes" on this "special date"
In 2013, Trump tweeted his "best wishes" to everyone, even his oft-disparaged "haters and losers," on this "special date," Sept. 11.
"@realDonaldTrump: I would like to extend my best wishes to all, even the haters and losers, on this special date, September 11th."
On the campaign trail:
A missing donation
On the campaign trail in 2015, Trump boasted that he'd given enormous amounts of money to charity in the days after the attacks, including a $10,000 donation to the Twin Towers Fund, one of the two foremost charities handling charitable giving and support for the 9/11 attacks. The Twin Towers Fund was dedicated to first responders and their families, especially families who lost a first responder in the attack; they disbursed $216 million before closing.
Reviews of charity records, however, showed "no evidence" that Trump or his foundation gave any money to the Twin Towers fund or its counterpart, New York City Public/Private Initiatives Inc. While he received praise and credit for that donation on Howard Stern's radio show, the money never showed up in charity records or IRS forms.
Furthermore, while reviews of Trump Foundation giving have found hundreds of thousands of dollars in charitable donations over the years, there are no donations to 9/11 charities — except one, a $1,000 donation to a Tom Cruise-founded "Workers Detoxification Fund" that used Scientology-approved methods to "detox" rescue workers.
The mystery of the missing Muslims
Trump also claimed to have seen "thousands" of "Muslims" on "rooftops" cheering the 9/11 attacks as they went on, a claim that has yet to be verified or even substantially corroborated by anyone beyond a few fringe internet sources.
The July 11 attacks
Last year, in a campaign speech in Buffalo, N.Y., Trump said he "watched our police and our firemen down on 7/11, down at the World Trade Center, right after it came down." He likely meant 9/11, not July 11 or the convenience store that carries the Slurpee.
Bush to blame?
During the campaign, Trump also blamed former president George W. Bush for the terror attacks, claiming Bush failed to listen to the CIA.
Better ratings than 9/11
This year, during an interview with the Associated Press, he claimed to have given CBS' "Face the Nation" its highest ratings since the show's coverage of the 9/11 attacks, with "5.2 million" people.
"It's the highest for 'Face the Nation,' or as I call it, 'Deface the Nation.' The highest for 'Deface the Nation' since the World Trade Center, since the World Trade Center came down. It's a tremendous advantage," said the President of the United States.
About the author: Patrick	McMahon, Rare Staff
Patrick is a content editor for Rare.


text:
Q: Time tracking app arbtt case insensitive match in rules is it possible to use modifiers on regex? Like:
m/some.*title/i ==> tag...

A: Yes: arbtt uses perl-compatible regexes, and they allow you to set modifiers inside the pattern:
m/(?i)some.*title/ ==> tag...

See https://www.pcre.org/original/doc/html/pcrepattern.html#SEC13 for more details.

redpajama_set_name: RedPajamaStackExchange


The year 2016 has been full of harrowing images of migrants on the move from the Middle East, Africa and Asia to Europe. Many have braved a precarious sea crossing, in boats that are hardly seaworthy, navigated by those who have no other motivation than profit.
The horror of these refugees was perhaps most poignantly underlined by the death of three-year-old Kurdish boy Alan Kurdi from Syria. The photo of him face down, drowned in the foaming Mediterranean has become a defining moment in the public perception of the crisis. Patrick Johnstone, of Operation World fame has produced a very helpful, simple book to help Christians think about how we should respond.
This is done in three sections: What's Going On – What To Know – What To Do. This well-researched book provides a useful snapshot of what is going on, it seeks to adjust our perceptions about refugees (this section includes the reminder that, Jesus was a refugee), and gives a number of practical suggestions on how we can do something.
One very simple idea is: putting together a welcome pack for a migrant family moving into our area. Another is the wise suggestion that we might want to listen to the new arrivals and learn from them before we try to do anything else.
Not every church or Christian will be in a community that attracts lots of refugees but with so many false perceptions continuing around this pressing issue, it's a useful book for everyone to read.


Dr. Davis earned his DDS degree in 1970 from St. Louis University School of Dentistry, St. Louis, Missouri. He graduated from University of Oklahoma, Norman, Oklahoma, in 1967 with a bachelor degree in Zoology.
Dr. Davis served in the U.S. Army and Army Reserves as Captain of Dental Corps (1968-1972). He has been practicing Dentistry in Muskogee since 1972.
Although he provides all facets of general and restorative services, he has a special interest in the specialties of root canal treatment and cosmetic dentistry.
Dr. Davis has served on dental missions to Haiti, Guatemala, Peru, Ecuador, Israel (Gaza, Jerusalem, Bethlehem, West Bank), Indonesia – Bali, France, Germany, Austria, Italy, Canada, Panama and Mexico.
Dr. Davis attends St. Joseph Catholic Church in Muskogee, and he has three sons, Andrew, Jamie and Ethan.
Dr. Davis is licensed by the State of Oklahoma; Fellow, American College of Forensic Examiners; life member, Oklahoma Dental Association (1972 – present); life member, American Dental Association (1972-present); and member, Eastern District Dental Society (1972-present). He is also a Fellow of the American Board of Forensic Dentistry, and serves on Muskogee Regional Medical Center's Active Medical Staff.
Dr. Davis is a life member of Lions Club International, and Muskogee Noon Lions Club (a founding club of Lions Club International). He is also a Melvin Jones Fellow of Lions International.
He is Past President of the Muskogee Noon Lions Club, Eastern District Dental Society, Goodwill Industries of Muskogee, Lake Country United Way of Muskogee, and the Muskogee Chapter of the American Cancer Society.
He is a 35 year member of the National Federation of Independent Businesspersons, a life member of the National Rifle Association (NRA), and has been a Bronze Sponsor for Ducks Unlimited for 35 years. His practice is a member of the Greater Muskogee Area Chamber of Commerce.