text:
\section{Introduction}\label{sec:introduction}

Deep neural networks can achieve good performance on the condition that the training and testing data are drawn from the same distribution. However, this condition might not hold true in practice. Data shifts caused by mismatches between training and testing domain~\cite{balaji2018metareg,li2019episodic,sinha2018certifying,volpi2018generalizing,zhao2019semantic}, small corruptions to data distributions~\cite{hendrycks2019benchmarking,zhao2020knowledge}, or adversarial attacks~\cite{goodfellow2015explaining,kurakin2017adversarial} are
often inevitable in real-world applications, and lead to significant performance degradation of deep learning models. Recently, adversarial data augmentation~\cite{ganin2016domain,sinha2018certifying,volpi2018generalizing} emerges as a strong baseline where fictitious target distributions are generated by an adversarial loss to resemble unforseen data shifts, and used to improve model robustness through training. The adversarial loss is leveraged to produce perturbations that fool the current model. However, as shown in~\cite{qiao2020learning}, this heuristic loss function is insufficient to synthesize large data shifts, i.e., ``hard'' adversarial perturbations from the source domain, which makes the model still vulnerable to severely shifted or corrupted testing data.

To mitigate this issue, we propose a regularization technique for adversarial data augmentation from an information theory perspective using the \emph{Information Bottleneck} (IB)~\cite{tishby1999information} principle. The IB principle encourages the model to learn an optimal representation by diminishing the irrelevant parts of the input variable that do not contribute to the prediction. Recently, there has been a surge of interest in combining the IB method with training of deep neural networks~\cite{achille2018information,amjad2019learning,elad2019direct,kolchinsky2019caveats,ozair2019wasserstein,tschannen2020mutual}, while its effectiveness for adversarial data augmentation still remains unclear.

In the IB context, a neural network does not generalize well on out-of-domain data often when the information of the input cannot be well-compressed by the model, i.e., the mutual information of the input and its associated latent representation is high~\cite{shamir2010learning,tishby2015deep}. Motivated by this conceptual observation, we aim to regularize adversarial data augmentation through maximizing the IB function. Specifically, we produce ``hard'' fictitious target domains that are largely shifted from the source domain by enlarging the mutual information of the input and latent distribution within the current model. However, mutual information is shown to be intractable in the literature~\cite{belghazi2018mutual,paninski2003estimation,song2020understanding}, and therefore directly optimizing this objective is challenging.

In this paper, we develop an efficient \emph{maximum-entropy} regularizer to achieve the same goal by making the following contributions: (i) to the best of our knowledge, we are the first work to investigate adversarial data argumentation from an information theory perspective, and address the problem of generating ``hard'' adversarial perturbations from the IB principle which has not been studied yet; (ii) we theoretically show that the IB principle can be bounded by a maximum-entropy regularization term in the maximization phase of adversarial data argumentation, which results in a notable improvement over~\cite{volpi2018generalizing}; (iii) we also show that our formulation holds in an approximate sense under certain non-deterministic conditions (e.g., when the neural network is stochastic or contains Dropout~\cite{srivastava2014dropout} layers). Note that our maximum-entropy regularizer can be implemented by one line of code with minor computational cost, while it consistently and statistically significantly improves the existing state of the art on three standard benchmarks.

\section{Background and Related Work}

\p{Information Bottleneck Principle} We begin by summarizing the concept of information bottleneck and, along the way, introduce the notations. The \textit{Information Bottleneck} (IB)~\cite{tishby1999information} is a principled way to seek a latent representation $Z$ that an input variable $X$ contains about an output $Y$. Let $I(X; Z)$ be the \textit{mutual information} of $X$ and $Z$, i.e., $I(X; Z) = \mathbb{D}_\mathsf{KL} \left(p_{XZ} \middle\| p_X p_Z \right)$, where $\mathbb{D}_\mathsf{KL}$ denotes the \textit{KL-divergence}~\cite{kullback1951information}. Intuitively, $I(X; Z)$ measures the uncertainty in $X$ given $Z$. The representation $Z$ can be quantified by two terms: $I(X; Z)$ which reflects how much $Z$ compresses $X$, and $I(Y; Z)$ which reflects how well $Z$ predicts $Y$. In practice, this IB principle is explored by minimizing the following \textit{IB Lagrangian}:
\begin{equation}
\label{eq:ib}
\minimize \left\{ I(X; Z) - \lambda I(Y; Z) \right\},
\end{equation}
where $\lambda$ is a positive parameter that controls the trade-off between compression and prediction. By controlling the amount of compression within the representation $Z$ via the compression term $I(X; Z)$, we can tune desired characteristics of trained models such as robustness to adversarial samples~\cite{alemi2017deep}, generalization error~\cite{amjad2019learning,cheng2019utilizing,kolchinsky2019caveats,shamir2010learning,tishby2015deep,vera2018role}, and detection of out-of-distribution data~\cite{alemi2018uncertainty}.

\p{Domain Generalization} Domain adaptation~\cite{ganin2015unsupervised,tzeng2017adversarial} transfers models in source domains to related target domains with different distributions during the training procedure. On the other hand, domain generalization~\cite{balaji2018metareg,bousmalis2016domain,li2017deeper,li2018learning,li2019episodic,mancini2018best} aims to learn features that perform well when transferred to \emph{unseen} domains during evaluation. This paper further studies a more challenging setting named single domain generalization~\cite{qiao2020learning}, where networks are learned using \emph{one single} source domain compared with conventional domain generalization that requires \emph{multiple} training source domains. Recently, adversarial data augmentation~\cite{volpi2018generalizing} is proven to be a promising solution which synthesizes virtual target domains during training so that the generalization and robustness of the learned networks to unseen domains can be improved. Our approach improves it by proposing an efficient regularizer.

\p{Adversarial Data Augmentation} We are interested in the problems of training deep neural networks in a single source domain $P_0$ and deploying it to unforeseen domains $P$ following different underlying distributions. Let $X \in \mathcal{X}$ be random data points with associated labels $Y \in \mathcal{Y}$ ($|\mathcal{Y}|$ is finite) drawn from the source distribution $P_0$. We consider the following worst-case problem around $P_0$:
\begin{equation}
\label{eq:worst}
\minimize_{\theta \in \Theta} \left\{\sup_P \left\{\mathbb{E}[\mathcal{L}(\theta; X, Y)] : D_\theta(P, P_0) \leq \rho \right\}\right\},
\end{equation}
where $\theta \in \Theta$ is the network parameters, $\mathcal{L}: \Theta \times (\mathcal{X} \times \mathcal{Y}) \rightarrow \mathbb{R}$ is the loss function, and $D_\theta$ measures the distance between two distributions $P$ and $P_0$. We denote $\theta = \{\theta_f, \mathbf{w}\}$, where $\mathbf{w}$ represents the parameters of the final prediction layer and $\theta_f$ represents the parameters of the rest of the network. Letting $f(\theta_f; x)$ be the latent representation of input $x$, we feed it into a $|\mathcal{Y}|$-way classifier such that using the \textit{softmax activation}, the probability $p_{(i)}$ of the $i$-th class is:
\begin{equation}
p_{(i)}(\theta; x) = \frac{\exp \left(\mathbf{w}_i^\top f(\theta_f; x)\right)}{\sum_{j = 1}^{|\mathcal{Y}|}\exp \left(\mathbf{w}_j^\top f(\theta_f; x)\right)},
\end{equation}
where $\mathbf{w}_i$ is the parameters for the $i$-th class. In the classification setting, we minimize the \textit{cross-entropy loss} over each sample $(x, y)$ in the training domain: $\mathcal{L}_\mathsf{CE} (\theta; x, y) \coloneqq -\log p_{(y)}(\theta; x)$. Moreover, in order to preserve the semantics of the input samples, the metric $D_\theta$ is defined in the latent space $\mathcal{Z}$. Let $c_\theta: \mathcal{Z} \times \mathcal{Z} \rightarrow \mathbb{R}_+ \cup \{\infty\}$ denote the transportation cost of moving mass from $(x_0, y_0)$ to $(x, y)$: $c_\theta \left((x_0, y_0), (x, y)\right) \coloneqq {\|z_0 - z\|}_2^2 + \infty \cdot \mathbf{1}\{y_0 \neq y\}$, where $z = f(\theta_f; x)$. For probability measures $P$ and $P_0$ supported on $\mathcal{Z}$, let $\Pi(P, P_0)$ be their couplings. Then, we use the \emph{Wasserstein metric} $D_\theta$ defined by $D_\theta (P, P_0) \coloneqq \inf_{M \in \Pi(P, P_0)} \left\{\mathbb{E}_M \left[ c_\theta\left((X_0, Y_0), (X, Y)\right) \right]\right\}$. The solution to the worst-case problem~\eqref{eq:worst} ensures good performance (robustness) against any data distribution $P$ that is $\rho$ distance away from the source domain $P_0$. However, for deep neural networks, this formulation is intractable with arbitrary $\rho$. Instead, following the reformulation of~\cite{sinha2018certifying,volpi2018generalizing}, we consider its \textit{Lagrangian relaxation} $\mathcal{F}$ for a fixed penalty parameter $\gamma \geq 0$:
\begin{equation}
\label{eq:lagworst}
\minimize_{\theta \in \Theta} \left\{\mathcal{F}(\theta) \coloneqq \sup_P \left\{ \mathbb{E}[\mathcal{L}(\theta; X, Y)] - \gamma D_\theta (P, P_0) \right\} \right\}.
\end{equation}

\section{Methodology}\label{sec:method}

In this paper, our main idea is to incorporate the IB principle into adversarial data augmentation so as to improve model robustness to large domain shifts. We start by adapting the IB Lagrangian~\eqref{eq:ib} to supervised-learning scenarios so that the latent representation $Z$ can be leveraged for classification purposes. To this end, we modify the IB Lagrangian~\eqref{eq:ib} following~\cite{achille2018emergence,achille2018information,amjad2019learning} to $\mathcal{L}_\mathsf{IB}(\theta; X, Y) \coloneqq \mathcal{L}_\mathsf{CE} (\theta; X, Y) + \beta I(X; Z)$,
where the constraint on $I(Y; Z)$ is replaced with the risk associated to the prediction according to the loss function $\mathcal{L}_\mathsf{CE}$. We can see that $\mathcal{L}_\mathsf{IB}$ appears as a standard cross-entropy loss augmented with a regularizer $I(X; Z)$ promoting minimality of the representation. Then, we rewrite Eq.~\eqref{eq:lagworst} to leverage the newly defined $\mathcal{L}_\mathsf{IB}$ loss function:
\begin{equation}
\label{eq:ibworst}
\minimize_{\theta \in \Theta} \left\{\mathcal{F}_\mathsf{IB}(\theta) \coloneqq \sup_P \left\{ \mathbb{E}[\mathcal{L}_\mathsf{IB}(\theta; X, Y)] - \gamma D_\theta (P, P_0) \right\}\right\}.
\end{equation}

As discussed in~\cite{sinha2018certifying,volpi2018generalizing}, the worst-case setting of Eq.~\eqref{eq:ibworst} can be formalized as a \emph{minimax} optimization problem. It is solved by an iterative training procedure where two phases are alternated in $K$  iterations. In the \emph{maximization} phase, new data points are produced by computing the inner maximization problem $\mathcal{F}_\mathsf{IB}$ to mimic fictitious target distributions $P$ that satisfy the constraint $D_\theta(P, P_0) \leq \rho$. In the \emph{minimization} phase, the network parameters are updated by the loss function $\mathcal{L}_\mathsf{IB}$ evaluated on the adversarial examples generated from the maximization phase.

The main challenge in optimizing Eq.~\eqref{eq:ibworst} is that exact computation of the compression term $I(X; Z)$ in $\mathcal{L}_\mathsf{IB}$ is almost impossible due to the high dimensionality of the data. The way of approximating this term in the minimization phase has been widely studied in recent years, and we follow~\cite{elad2019direct,gal2015bayesian} to express $I(X; Z)$ by $\ell_2$ penalty (also known as weight decay~\cite{krogh1992simple}). Below, we discuss how to effectively implement $I(X; Z)$ in the maximization phase for adversarial data augmentation. The full algorithm is summarized in Algorithm~\ref{alg:full}.

\begin{algorithm}[t]
\begin{algorithmic}[1]
\Require Source dataset $\mathcal{D}_0 = \{X_i, Y_i\}_{1\leq i \leq N}$ and initialized network weights $\theta_0$
\Ensure Learned network weights $\theta$
\State Initialize $\theta \leftarrow \theta_0$, $\mathcal{D} \leftarrow \mathcal{D}_0$
\For{$k = 1, \dots, K$} \Comment{Run the minimax procedure $K$ times}
    \For{$t = 1, \dots, T_\mathsf{MIN}$} \Comment{Run the minimization phase $T_\mathsf{MIN}$ times}
        \State Sample $(X_t, Y_t)$ uniformly from dataset $\mathcal{D}$
        \State $\theta \leftarrow \theta - \alpha \nabla_\theta \mathcal{L}_\mathsf{IB}(\theta;X_t, Y_t)$
    \EndFor
    \ForAll{$(X_i, Y_i) \in \mathcal{D}$}
        \State $X_i^k \leftarrow X_i$
        \For{$t = 1, \dots, T_\mathsf{MAX}$} \Comment{Run the maximization phase $T_\mathsf{MAX}$ times}
            \State $X_i^k \leftarrow X_i^k + \eta \nabla_{X_i^k} \left\{\mathcal{L}_\mathsf{CE}(\theta; X_i^k, Y_i) + \beta h(\theta; X_i^k) - \gamma c_\theta((X_i^k, Y_i), (X_i, Y_i))\right\}$
        \EndFor
        \State Append $(X_i^k, Y_i^k)$ to dataset $\mathcal{D}$
    \EndFor
\EndFor
\While{\textit{not reach maximum steps}}
    \State Sample $(X_i, Y_i)$ uniformly from dataset $\mathcal{D}$
    \State $\theta \leftarrow \theta - \alpha \nabla_\theta \mathcal{L}_\mathsf{IB}(\theta;X_i, Y_i)$
\EndWhile
\end{algorithmic}
\caption{Max-Entropy Adversarial Data Augmentation (ME-ADA)}
\label{alg:full}
\end{algorithm}

\subsection{Regularizing Maximization Phase via Maximum Entropy}

Intuitively, regularizing the mutual information $I(X; Z)$ in the maximization phase encourages adversarial perturbations that cannot be effectively ``compressed'' by the current model. From the information theory perspective, these perturbations usually imply large domain shifts, and thus can potentially benefit model generalization and robustness. However, since $Z$ is high dimensional, maximizing $I(X; Z)$ is intractable. One of our key results is that, when we restrict to classification scenarios, we can efficiently approximate and maximize $I(X; Z)$ during adversarial data augmentation. As we will show, this process can be effectively implemented through maximizing the entropy of network predictions, which is a tractable lower bound of $I(X; Z)$.

To set the stage, we let $\hat{Y} \in \mathcal{Y}$ denote the predicted class label given the input $X$. As described in~\cite{amjad2019learning,tishby1999information}, deep neural networks can be considered as a Markov chain of successive representations of the input where information flows obeying the structure: $X \rightarrow Z \rightarrow \hat{Y}$. By the \emph{Data Processing Inequality}~\cite{cover2012elements}, we have $I(X; Z) \geq I(X; \hat{Y})$. On the other hand, when performing data augmentation during each maximization phase, the model parameters $\theta$ are fixed; and $\hat{Y}$ is a \emph{deterministic} function of $X$, i.e., any given input is mapped to a single class. Consequently, it holds that $H(\hat{Y}|X) = 0$, where $H(\cdot)$ is the \emph{Shannon entropy}. After combining all these together, then we have Proposition~\ref{prop:epy}.

\begin{proposition}
\label{prop:epy}
Consider a deterministic neural network, the parameters $\theta$ of which are fixed. Given the input $X$, let $\hat{Y}$ be the network prediction and $Z$ be the latent representation of $X$. Then, the mutual information $I(X; Z)$ is lower bounded by $H(\hat{Y})$, i.e., we have that,
\begin{equation}
\label{eq:epy}
I(X; Z) \geq I(X; \hat{Y}) = H(\hat{Y}) - H(\hat{Y}|X) = H(\hat{Y}).
\end{equation}
\end{proposition}

Note that Eq.~\eqref{eq:epy} does not mean that calculating $H(\hat{Y})$ does not need $X$, since $\hat{Y}$ is generated by inputting $X$ into the network. There are two important benefits of our formulation~\eqref{eq:epy} to be discussed. First, it provides a method to maximize $I(X; Z)$ that is not related to the input dimensionality. Thus, for high dimensional images, we can still maximize mutual information this way. Second, our formulation is closely related to the \emph{Deterministic Information Bottleneck}~\cite{strouse2017deterministic}, where $I(X; Z)$ is approximated by $H(Z)$. However, $H(Z)$ is still intractable in general. Instead, $H(\hat{Y})$ can be directly computed from the softmax output of a classification network as we will show later. Next, we modify Eq.~\eqref{eq:ibworst} by replacing $I(X; Z)$ with $H(\hat{Y})$, which becomes a relaxed worst-case problem:
\begin{equation}
\label{eq:dibworst}
\mathcal{F}_\mathsf{DIB}(\theta) \coloneqq \sup_P \left\{ \mathbb{E}[\mathcal{L}_\mathsf{CE} (\theta; X, Y) + \beta H(\hat{Y})] - \gamma D_\theta (P, P_0) \right\}.
\end{equation}
From a Bayesian perspective, the prediction entropy $H(\hat{Y})$ can be viewed as the predictive uncertainty~\cite{kendall2017uncertainties,snoek2019can} of a model. Therefore, our maximum-entropy formulation~\eqref{eq:dibworst} is equivalent to perturbing the underlying data distribution so that the predictive uncertainty of the current model is enlarged in the maximization phase. This motivates us to extend our approach to \emph{stochastic} neural networks for better capturing the model uncertainty as we will show in the experiment.

\p{Empirical Estimation} Now, $\mathcal{F}_\mathsf{DIB}$ involves the expected prediction entropy $H(\hat{Y})$ over the data distribution. However, during training we only have sample access to the data distribution, which we can use as a surrogate for empirical estimation. Given an observed input $x$ sampled from the source distribution $P_0$, we start from defining the prediction entropy of its corresponding output by:
\begin{equation}
h(\theta; x) \coloneqq -\sum_{i = 1}^{|\mathcal{Y}|} p_{(i)}(\theta; x) \log p_{(i)}(\theta; x).
\end{equation}
Then, through calculating the expectation over the prediction entropies of all possible observations $\{x_i\}_{1\leq i \leq N}$ contained in the source dataset $P_0$, we can obtain the empirical estimation of $H(\hat{Y})$:
\begin{equation}
\label{eq:epyemp}
\hat{H}(\hat{Y}) \coloneqq -\sum_{\mathcal{Y}} \left( \sum_i p_{\hat{Y}|X}(\cdot|x_i)\hat{p}(x_i) \right) \log \left( \sum_i p_{\hat{Y}|X}(\cdot|x_i)\hat{p}(x_i) \right) \approxeq \frac{1}{N}\sum_{i = 1}^{N} h(\theta; x_i),
\end{equation}
where $\hat{H}(\cdot)$ denotes the empirical entropy, $\hat{p}$ is the empirical distribution of $x_i$, and the approximation is achieved by Jensen's inequality. After combing Eq.~\eqref{eq:epyemp} with the relaxed worst-case problem~\eqref{eq:dibworst}, we will have the empirical counterpart of $\mathcal{F}_\mathsf{DIB}$ which is defined by $\hat{\mathcal{F}}_\mathsf{DIB}(\theta) \coloneqq \sup_P \left\{ \mathbb{E}_{x \sim P}[\mathcal{L}_\mathsf{CE}(\theta; x, y) + \beta h(\theta; x)] - \gamma D_\theta (P, P_0) \right\}$. Taking the dual reformulation of the penalty problem $\hat{\mathcal{F}}_\mathsf{DIB}$, we can obtain an efficient solution procedure. The following result is a minor adaptation of~\cite{volpi2018generalizing}~(Lemma~1):

\begin{proposition}
Let $\mathcal{L}: \Theta \times (\mathcal{X} \times \mathcal{Y}) \rightarrow \mathbb{R}$ and $h: \Theta \times \mathcal{X} \rightarrow \mathbb{R}$ be continuous. Let $\phi_\gamma$ denote the robust surrogate loss. Then, for any distribution $P_0$ and any $\gamma \geq 0$, we have that,
\begin{gather}
\sup_P \left\{ \mathbb{E}_{x \sim P}[\mathcal{L}(\theta; x, y) + \beta h(\theta; x)] - \gamma D_\theta (P, P_0) \right\} = \mathbb{E}_{x \sim P_0}\left[ \phi_\gamma(\theta; x, y) \right], \\
\label{eq:sgloss}
\textrm{where} \; \phi_\gamma(\theta; x_0, y_0) \coloneqq \sup_{x \in \mathcal{X}} \left\{ \mathcal{L}(\theta; x, y_0) + \beta h(\theta; x) - \gamma c_\theta((x, y_0), (x_0, y_0)) \right\}.
\end{gather}
\end{proposition}

To solve the penalty problem of Eq.~\eqref{eq:ibworst}, in the minimization phase of the iterative training procedure, we can perform \emph{Stochastic Gradient Descent} (SGD) on the robust surrogate loss $\phi_\gamma$. To be specific, under suitable conditions~\cite{boyd2004convex}, we have that $\nabla_\theta \phi_\gamma(\theta; x_0, y_0) = \nabla_\theta \mathcal{L}_\mathsf{IB}(\theta;x^\star_\gamma, y_0)$, where $x^\star_\gamma = \argmax_{x \in \mathcal{X}} \left\{ \mathcal{L}_\mathsf{CE}(\theta; x, y_0) + \beta h(\theta; x) - \gamma c_\theta((x, y_0), (x_0, y_0)) \right\}$ is an adversarial perturbation of $x_0$ at the current model $\theta$. On the other hand, in the maximization phase, we solve the maximization problem~\eqref{eq:sgloss} by \emph{Maximum-Entropy Adversarial Data Augmentation}~(ME-ADA) in this work. Concretely, in the $k$-th \emph{maximization} phase, we compute $N$ adversarially perturbed samples at the current model $\theta$:
\begin{equation}
\label{eq:xmax}
X_i^k \in \argmax_{x \in \mathcal{X}} \left\{ \mathcal{L}_\mathsf{CE}(\theta; x, Y_i) + \beta h(\theta; x) - \gamma c_\theta((x, Y_i), (X_i^{k-1}, Y_i)) \right\}.
\end{equation}
Note that the entropy term $h(\theta; x)$ is efficient to be calculated from the softmax output of a model, which can be implemented with one line of code in modern deep learning frameworks, and substantial performance improvement can be achieved by it as we will show in the experiments.

\p{Theoretic Bound} It is essential to guarantee that the empirical estimate of the entropy $\hat{H}$ (from a training set $\mathcal{S}$ containing $N$ samples) is an accurate estimate of the true expected entropy $H$. The next proposition ensures that for large $N$, in a classification problem, the sample estimate of average entropy is close to the expected entropy.

\begin{proposition}
\label{prop:epybnd}
Let $Y$ be a fixed probabilistic function of $X$ into an arbitrary finite target space $\mathcal{Y}$, determined by a fixed and known conditional probability distribution $p_{Y|X}$, and $\mathcal{S}$ be a sample set of size $N$ drawn from the joint probability distribution $p_{XY}$. For any $\delta \in (0, 1)$, with probability of at least $1 - \delta$ over the sample set $\mathcal{S}$, we have,
\begin{equation}
|H(Y) - \hat{H}(Y)| \leq \frac{|\mathcal{Y}|\log(N)\sqrt{\log(2/\delta)}}{\sqrt{2N}} + \frac{|\mathcal{Y}| - 1}{N}.
\end{equation}
\end{proposition}

We prove Proposition~\ref{prop:epybnd} in the supplementary material. The proof adapts the setting in~\cite{shamir2010learning}, where we bound the deviations of the information estimations from their expectation and then use the bound on the expected bias of entropy estimation. Here, it is also worth discussing two important properties of this bound. First, we note that Proposition~\ref{prop:epybnd} holds for any fixed probabilistic function. Compared with prior studies on the plug-in estimate of discrete entropy over a finite size alphabet~\cite{valiant2011estimating,wu2016minimax}, we focus on the bound of non-optimal estimators. In particular, this proposition holds for any $\hat{Y}$, even if $\hat{Y}$ is not a globally optimal solution for $\mathcal{F}_\mathsf{DIB}$ in Eq.~\eqref{eq:dibworst}. This is the case of models in the maximization phase, which thus ensures the effectiveness of our formulation across the whole iterative training procedure. Second, the bound does not depend on $|\mathcal{X}|$. In addition, the complexity of the bound is mainly controlled by $|\mathcal{Y}|$. By constraining $|\mathcal{Y}|$ to be small, a tight bound can be achieved. This assumption usually holds for the setting of training classification models, i.e., $|\mathcal{Y}| \ll N$.

\subsection{Maximum Entropy in Non-Deterministic Conditions}

It is important to note that not all models are deterministic, e.g., when deep neural networks are stochastic~\cite{florensa2017stochastic,tang2013learning} or contain Dropout layers~\cite{gal2016dropout,srivastava2014dropout}. The mapping from $X$ to $\hat{Y}$ may be intrinsically noisy or non-deterministic. Here, we show that when $\hat{Y}$ is a small perturbation away from being a deterministic function of $X$, our maximum-entropy formulation~\eqref{eq:dibworst} still applies in an approximate sense. We now consider the case when the joint distribution of $X$ and $\hat{Y}$ is $\epsilon$-close to having $\hat{Y}$ be a deterministic function of $X$. The next result is a minor adaptation of~\cite{kolchinsky2019caveats} (Theorem~1) and it shows that the conditional entropy $H(\hat{Y}|X)$ is $\mathcal{O}(-\epsilon \log \epsilon)$ away from being zero.

\begin{corollary}
\label{cor:nondet}
Let $X$ be a random variable and $Y$ be a random variable with a finite set of outcomes $\mathcal{Y}$. Let $p_{XY}$ be a joint distribution over $X$ and $Y$ under which $Y = f(X)$. Let $\tilde{p}_{XY}$ be a joint distribution over $X$ and $Y$ which has the same marginal over $X$ as $p_{XY}$, i.e., $\tilde{p}_X = p_X$, and obey ${|\tilde{p}_{XY} - p_{XY}|}_1 \leq \epsilon \leq \frac{1}{2}$. Then, we have that,
\begin{equation}
\left| H(\tilde{p}(Y|X)) \right| \leq -\epsilon \log \frac{\epsilon}{{|\mathcal{Y}|}^3}.
\end{equation}
\end{corollary}

As we show in this corollary, even if the relationship between $X$ and $\hat{Y}$ is not perfectly deterministic but close to being so, i.e., it is $\epsilon$-close to a deterministic function, then we have $H(\hat{Y}|X) \approx 0$. Hence, in this case, the proposed Proposition~\ref{prop:epy} and our maximum-entropy adversarial data augmentation formulation~\eqref{eq:dibworst} still hold in an approximate sense.

\section{Experiments}\label{sec:experiments}

In this section, we evaluate our approach over a variety of settings. We first test with MNIST under the setting of large domain shifts, and then test on a more challenging dataset, with PACS data under the domain generalization setting. Further, we test on CIFAR-10-C and CIFAR-100-C which are standard benchmarks for evaluating model robustness to common corruptions. We compare the proposed \emph{Maximum-Entropy Adversarial Data Augmentation}~(ME-ADA) with previous state of the art when available. We note that \emph{Adversarial Data Augmentation}~(ADA)~\cite{volpi2018generalizing} is our main competitor, since our method downgrades to \cite{volpi2018generalizing} when the maximum-entropy term is discarded.

\p{Datasets} MNIST dataset~\cite{lecun1998gradient} consists of handwritten digits with 60,000 training examples and 10,000 testing examples. Other digit datasets, including SVHN~\cite{netzer2011reading}, MNIST-M~\cite{ganin2015unsupervised}, SYN~\cite{ganin2015unsupervised} and USPS~\cite{denker1989neural}, are leveraged for evaluating model performance. These four datasets contain large domain shifts from MNIST in terms of backgrounds, shapes and textures. PACS~\cite{li2017deeper} is a recent dataset with different object style depictions and a more challenging domain shift than the MNIST experiment. This dataset contains four domains (art, cartoon, photo and sketch), and shares seven common object categories (dog, elephant, giraffe, guitar, house, horse and person) across these domains. It is made up of 9,991 images with the resolution of $224 \times 224$. For fair comparison, we follow the protocol in \cite{li2017deeper} including the recommended train, validation and test split.

CIFAR-10 and CIFAR-100 are two datasets~\cite{krizhevsky2009learning} containing small $32 \times 32$ natural RGB images, both with 50,000 training images and 10,000 testing images. CIFAR-10 has 10 categories, and CIFAR-100 has 100 object classes. In order to measure the resilience of a model to common corruptions, we evaluate on CIFAR-10-C and CIFAR-100-C datasets~\cite{hendrycks2019benchmarking}. These two datasets are constructed by corrupting the original CIFAR test sets. For each dataset, there are a total of fifteen noise, including blur, weather, and digital corruption types, and each of them appears at five severity levels or intensities. We do not tune on the validation corruptions, so we report the average performance over all corruptions and intensities.

\subsection{MNIST with Domain Shifts}

\p{Experiment Setup} We follow the setup of~\cite{volpi2018generalizing} in experimenting with MNIST dataset. We use 10,000 samples from MNIST for training and evaluate prediction accuracy on the respective test sets of SVHN, MNIST-M, SYN and USPS. In order to work with comparable datasets, we resize all the images to $32 \times 32$, and treat images from MNIST and USPS as RGB images. We use LeNet~\cite{lecun1989backpropagation} as a base model and the batch size is 32. We use Adam~\cite{kingma2014adam} with $\alpha = 0.0001$ for minimization and SGD with $\eta = 1.0$ for maximization. We set $T_\mathsf{MIN} = 100$, $T_\mathsf{MAX} = 15$, $\gamma = 1.0$, $\beta = 10.0$ and $K = 3$. We compare our method against ERM~\cite{vapnik1998statistical}, ADA~\cite{volpi2018generalizing}, and PAR~\cite{wang2019robust}.

We also implement a variant of our method through \emph{Bayesian Neural Networks} (BNNs)~\cite{blundell2015weight,gal2016dropout,lakshminarayanan2017simple} to demonstrate our compatibility with stochastic neural networks. BNNs learn a distribution over network parameters and are currently the state of the art for estimating predictive uncertainty~\cite{ebrahimi2020uncertainty,neal2012bayesian}. We follow \cite{blundell2015weight} to implement the BNN via variational inference. During the training procedure, in each maximization phase, a set of network parameters are drawn from the variational posterior $q(\theta|\cdot)$, and then the predictive uncertainty is redefined by the expectation of all prediction entropies: $\bar{h}(q; x) = \mathbb{E}_q[h(\theta; x)]$. We refer to the supplementary material for more details of this BNN variant.

\p{Results} Table~\ref{tbl:mnist} shows the classification accuracy and standard deviation of each model averaged over ten runs. We can see that our model with the maximum-entropy formulation achieves the best performance, while the improvement on USPS
is not as significant as those on other domains due to its high similarity with MNIST. We then notice that, after engaging the BNN, our performance is further improved. Intuitively, we believe this is because the BNN provides a better estimation of the predictive uncertainty in the maximization phase. We are also interested in analyzing the behavior of our method when $K$ is increased. Figure~\ref{fig:mnist} shows the results of our method and other baselines by varying the number of iterations $K$ while fixing $\gamma$ and $\beta$. We observe that our method improves performances on SVHN, MNIST-M and SYN, outperforming both ERM and \cite{volpi2018generalizing} statistically significantly in different iterations. This demonstrates that the improvements obtained by our method are consistent.

\begin{table}[t]
\caption{Average classification accuracy (\%) and standard deviation of models trained on MNIST~\cite{lecun1998gradient} and evaluated on SVHN~\cite{netzer2011reading}, MNIST-M~\cite{ganin2015unsupervised}, SYN~\cite{ganin2015unsupervised} and USPS~\cite{denker1989neural}. The results are averaged over ten runs. Best performances are highlighted in bold. The results of PAR are obtained from~\cite{xu2020robust}.}\label{tbl:mnist}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}l|cccc|c@{}}
\toprule
& SVHN~\cite{netzer2011reading} & MNIST-M~\cite{ganin2015unsupervised} & SYN~\cite{ganin2015unsupervised} & USPS~\cite{denker1989neural} & Average\\
\midrule
Standard (ERM~\cite{vapnik1998statistical}) & 31.95 $\pm$ 1.91 & 55.96 $\pm$ 1.39 & 43.85 $\pm$ 1.27 & 79.92 $\pm$ 0.98 & 52.92 $\pm$ 0.98\\
PAR~\cite{wang2019robust} & 36.08 $\pm$ 1.27 & 61.16 $\pm$ 0.21 & 45.48 $\pm$ 0.35 & 79.95 $\pm$ 1.18 & 55.67 $\pm$ 0.33\\
\midrule
Adv.~Augment (ADA)~\cite{volpi2018generalizing} & 35.70 $\pm$ 2.00 & 58.65 $\pm$ 1.72 & 47.18 $\pm$ 0.61 & 80.40 $\pm$ 1.70 & 55.48 $\pm$ 0.74\\
+ \emph{Max Entropy} (ME-ADA) & 42.00 $\pm$ 1.74 & \textbf{63.98} $\pm$ 1.82 & 49.80 $\pm$ 1.74 & 79.10 $\pm$ 1.03 & 58.72 $\pm$ 1.12\\
+ \emph{Max Entropy} w/ BNN & \textbf{42.56} $\pm$ 1.45 & 63.27 $\pm$ 2.09 & \textbf{50.39} $\pm$ 1.29 & \textbf{81.04} $\pm$ 0.98 & \textbf{59.32} $\pm$ 0.82\\
\bottomrule
\end{tabular}}
\end{table}

\begin{figure}[t]
  \centering
  \includegraphics[width=.245\textwidth]{fig_mnist_k_1}
  \includegraphics[width=.245\textwidth]{fig_mnist_k_2}
  \includegraphics[width=.245\textwidth]{fig_mnist_k_3}
  \includegraphics[width=.245\textwidth]{fig_mnist_k_4}
  \vspace{-1.2em}
  \caption{Test accuracy associated with models trained using 10,000 MNIST~\cite{lecun1998gradient} samples and tested on SVHN~\cite{netzer2011reading}, MNIST-M~\cite{ganin2015unsupervised}, SYN~\cite{ganin2015unsupervised} and USPS~\cite{denker1989neural}. We compare our method (\emph{blue}) to \cite{volpi2018generalizing} (\emph{orange}) with different number of iterations $K$, and Empirical Risk Minimization (ERM)~\cite{vapnik1998statistical} (\emph{red line}). The results are averaged over ten runs; and black bars indicate the range of accuracy spanned.}
  \label{fig:mnist}
\end{figure}

\subsection{PACS}

\begin{table}[t]
\caption{Classification accuracy (\%) of our approach on PACS dataset~\cite{li2017deeper} in comparison with the previously
reported state-of-the-art results. Bold numbers indicate the best performance (two sets, one for each scenario engaging or forgoing domain identifications, respectively).}\label{tbl:pacs}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}c|cccccc|ccccc@{}}
\toprule
 & DSN & L-CNN & MLDG & Fusion & MetaReg & Epi-FCR & AGG & HEX & PAR & ADA & ME-ADA\\
\midrule
Domain ID & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark & \xmark & \xmark & \xmark & \xmark & \xmark\\
\midrule
Art & 61.1 & 62.9 & 66.2 & 64.1 & \textbf{69.8} & 64.7 & 63.4 & 66.8 & 66.9 & 64.3 & \textbf{67.1}\\
Cartoon & 66.5 & 67.0 & 66.9 & 66.8 & 70.4 & \textbf{72.3} & 66.1 & 69.7 & 67.1 & 69.8 & \textbf{69.9}\\
Photo & 83.3 & 89.5 & 88.0 & 90.2 & \textbf{91.1} & 86.1 & 88.5 & 87.9 & \textbf{88.6} & 85.1 & \textbf{88.6}\\
Sketch & 58.6 & 57.5 & 59.0 & 60.1 & 59.2 & \textbf{65.0} & 56.6 & 56.3 & 62.6 & 60.4 & \textbf{63.0}\\
\midrule
Average & 67.4 & 69.2 & 70.0 & 70.3 & \textbf{72.6} & 72.0 & 68.7 & 70.2 & 71.3 & 69.9 & \textbf{72.2}\\
\bottomrule
\end{tabular}}
\end{table}

\p{Experiment Setup} We continue to experiment on PACS dataset, which consists of collections of images over four domains. Each time, one domain is selected as the test domain, and the rest three are used for training. Following~\cite{li2017deeper}, we use the ImageNet pretrained AlexNet~\cite{krizhevsky2012imagenet} as a base network. We compare with recently reported state of the art engaging domain identifications, including DSN~\cite{bousmalis2016domain}, L-CNN~\cite{li2017deeper}, MLDG~\cite{li2018learning}, Fusion~\cite{mancini2018best}, MetaReg~\cite{balaji2018metareg} and Epi-FCR~\cite{li2019episodic}, as well as methods forgoing domain identifications, including AGG~\cite{li2019episodic}, HEX~\cite{wang2019learning}, and PAR~\cite{wang2019robust}. Former methods often obtain better results because they utilize domain identifications. Our method belongs to the latter category. Other training details are provided in the supplementary material.

\p{Results} We report the results in Table~\ref{tbl:pacs}. We note that our method achieves the best performance among techniques forgoing domain identifications. More impressively, our method, without using domain identifications, is only slightly shy of MetaReg~\cite{balaji2018metareg} in terms of overall performance, which takes advantage domain identifications. Interestingly, it is also worth mentioning that our method improves previous methods with a relatively large margin when ``sketch'' is the testing domain. This is notable because ``sketch'' is the only colorless domain which owns the largest domain shift out of the four domains in PACS. Our method handles this extreme case by producing larger data shifts from the source domain with the proposed maximum-entropy term during data augmentation.

\subsection{CIFAR-10 and CIFAR-100 with Corruptions}

\p{Experiment Setup} In the following experiments, we show that our approach endows robustness to various architectures including All Convolutional Network (AllConvNet)~\cite{salimans2016weight,springenberg2014striving}, DenseNet-BC~\cite{huang2017densely} (with $k = 12$ and $d = 100$), WideResNet (40-2)~\cite{zagoruyko2016wide}, and ResNeXt-29 ($32 \times 4$)~\cite{xie2017aggregated}. We train all networks with an initial learning rate of 0.1 optimized by SGD using Nesterov momentum, and the learning rate decays following a cosine annealing schedule~\cite{loshchilov2016sgdr}. All input images are pre-processed with standard random left-right flipping and cropping in the minimization phase. We train AllConvNet and WideResNet for 100 epochs; DenseNet and ResNeXt require 200 epochs for convergence. Following the setting of~\cite{hendrycks2020augmix}, we use a weight decay of 0.0001 for DenseNet and 0.0005 otherwise. Due to the space limitation, we ask the readers to refer to the supplementary material for detailed settings of our training parameters for different architectures.

\p{Baselines} To demonstrate the utility of our approach, we compare to many state-of-the-art techniques designed for robustness to image corruptions. These baseline techniques include (i) the standard data augmentation baseline and Mixup~\cite{zhang2018mixup}; (ii) two regional regularization strategies for images, i.e., Cutout~\cite{devries2017improved} and Cutmix~\cite{yun2019cutmix}; (iii) AutoAugment~\cite{cubuk2019autoaugment}, which searches over data augmentation policies to find a high-performing data augmentation policy via reinforcement learning; (iv) Adversarial Training~\cite{kang2019testing} for model robustness against unforeseen adversaries, and Adversarial Data Augmentation~\cite{volpi2018generalizing} which generates adversarial perturbations using Wasserstein distances.

\p{Results} The results are shown in Table~\ref{tbl:cifar}. Our method enjoys the best performance and improves previous state of the art by a large margin (5\% of accuracy on CIFAR-10-C and 4\% on CIFAR-100-C). More importantly, these gains are achieved across different architectures and on both datasets. Figure~\ref{fig:corruptions} shows more detailed comparisons over all corruptions. We find that our substantial gains in robustness are spread across a wide variety of corruptions, with a small drop of performance in only three corruption types: fog, brightness and contrast. Especially, for glass blur, Gaussian, shot and impulse noises, accuracies are significantly improved by 25\%. From the Fourier perspective~\cite{yin2019fourier}, the performance gains from our adversarial perturbations lie primarily in high frequency domains, which are commonly occurring image corruptions. These results demonstrate that the maximum-entropy term can regularize networks to be more robust to common image corruptions.

\begin{table}[t]
\caption{Average classification accuracy (\%). Across several architectures, our approach obtains CIFAR-10-C and CIFAR-100-C corruption robustness that exceeds the previous state of the art by a large margin. Best performances are highlighted in bold.}\label{tbl:cifar}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}ll|cccccccc@{}}
\toprule
& & Standard & Cutout & CutMix & AutoDA & Mixup & AdvTrain & ADA & ME-ADA\\
\midrule
\multirow{4}{*}{CIFAR-10-C} & AllConvNet & 69.2 & 67.1 & 68.7 & 70.8 & 75.4 & 71.9 & 73.0 & \textbf{78.2}\\
& DenseNet & 69.3 & 67.9 & 66.5 & 73.4 & 75.4 & 72.4 & 69.8 & \textbf{76.9}\\
& WideResNet & 73.1 & 73.2 & 72.9 & 76.1 & 77.7 & 73.8 & 79.7 & \textbf{83.3}\\
& ResNeXt & 72.5 & 71.1 & 70.5 & 75.8 & 77.4 & 73.0 & 78.0 & \textbf{83.4}\\
\midrule
\multicolumn{2}{c|}{Average} & 71.0 & 69.8 & 69.7 & 74.0 & 76.5 & 72.8 & 75.1 & \textbf{80.5}\\
\midrule
\midrule
\multirow{4}{*}{CIFAR-100-C} & AllConvNet & 43.6 & 43.2 & 44.0 & 44.9 & 46.6 & 44.0 & 45.3 & \textbf{51.2}\\
& DenseNet & 40.7 & 40.4 & 40.8 & 46.1 & 44.6 & 44.8 & 45.2 & \textbf{47.8}\\
& WideResNet & 46.7 & 46.5 & 47.1 & 50.4 & 49.6 & 44.9 & 50.4 & \textbf{52.8}\\
& ResNeXt & 46.6 & 45.4 & 45.9 & 48.7 & 48.6 & 45.6 & 53.4 & \textbf{57.3}\\
\midrule
\multicolumn{2}{c|}{Average} & 44.4 & 43.9 & 44.5 & 47.5 & 47.4 & 44.8 & 48.6 & \textbf{52.3}\\
\bottomrule
\end{tabular}}
\end{table}

\begin{figure}[t]
  \centering
  \includegraphics[width=\textwidth]{fig_cifar10}
  \vspace{-2.2em}
  \caption{Test accuracy of the Empirical Risk Minimization (ERM)~\cite{vapnik1998statistical} principle compared to our approach on the fifteen CIFAR-10-C~\cite{hendrycks2019benchmarking} corruptions using WideResNet (40-2)~\cite{zagoruyko2016wide}. Each bar represents an average over all five corruption strengths for a given corruption type.}
  \label{fig:corruptions}
\end{figure}

\section{Conclusion}

In this work, we introduced a \emph{maximum-entropy} technique that regularizes adversarial data augmentation. It encourages the model to learn with fictitious target distributions by producing ``hard'' adversarial perturbations that enlarge predictive uncertainty of the current model. As a result, the learned model is able to achieve improved robustness to large domain shifts or corruptions encountered during deployment. We demonstrate that our technique obtains state-of-the-art performance on MNIST, PACS, and CIFAR-10/100-C, and is extremely simple to implement. One major limitation of our method is that it cannot be directly applied to regression problems since the maximum-entropy lower bound is still difficult to compute in this case. Our future work might consider alternative measurements of information~\cite{ozair2019wasserstein,tschannen2020mutual} that are more suited for general machine learning applications.

\section*{Broader Impact}

The proposed method will be used to train a perception system that can robustly and reliably classify object instances. For example, this system can be used in many fundamental real-world applications in which a user desires to classify object instances from a product database, such as products found on local supermarkets or online stores. Similar to most deep learning applications learning from data which run the risk of producing biased or offensive content reflecting the training data, our work that learns a data-driven classification model is no exception. Our method moderates this issue by producing efficient fictitious target domains that are largely shifted from the source training dataset, so that the trained model on these adversarial domains are less biased. However, a downside of this moderation is the introduction of new hyper-parameters to be tuned for different tasks. Compared with other methods that obtain the same robustness but have to be trained on larger datasets, the proposed research can significantly reduce the data collection from different domains to train classification models, thereby reducing the system development time and lower related costs.

\begin{ack}
This research was funded based on partial funding to Dimitris Metaxas from NSF: IIS 1703883, CNS-1747778, CCF-1733843, IIS-1763523, IIS-1849238-825536 and MURI-Z8424104-440149.
\end{ack}

\begin{appendices}

\section{Supplementary Materials}

\subsection{Proofs}

\subsubsection{Proof of Proposition 3}

Here, we follow the guidance of~\cite{shamir2010learning} to prove Proposition~\ref{prop:epybnd}. Let $\mathcal{S}$ be a sample set of size $m$, and let $T$ be a probabilistic function of $X$ into an arbitrary finite target space, defined by $p(t|x)$ for all $x \in \mathcal{X}$ and $t \in \mathcal{T}$. To prove Proposition~\ref{prop:epybnd}, we bound the deviations of the entropy estimations from its expectation: $|\hat{H}(T)| - \mathbb{E}[\hat{H}(T)]|$, and then use a bound on the expected bias of entropy estimation.

To bound the deviation of the entropy estimates, we use McDiarmid's inequality~\cite{mcdiarmid1989}, in a manner similar to~\cite{antos2001convergence}. For this, we must bound the change in value of each of the entropy estimations when a single instance in $\mathcal{S}$ is arbitrarily changed. A useful and easily proven inequality in that regard is the following: for any natural $m$ and for any $a \in [0, 1 - 1/m]$ and $\Delta \leq 1/m$,
\begin{equation}
\label{eq:supp:delta}
\left|(a + \Delta)\log (a + \Delta) - a \log(a) \right| \leq \frac{\log(m)}{m}.
\end{equation}

With this in equality, a careful application of McDiarmid's inequality leads to the following lemma.

\begin{lemma}
\label{lem:supp:epybnd1}
For any $\delta \in (0, 1)$, with probability of at least $1 - \delta$ over the sample set, we have that,
\begin{equation}
\label{eq:supp:lemma1}
|\hat{H}(T) - \mathbb{E}[\hat{H}(T)]| \leq \frac{|\mathcal{T}|\log(m)\sqrt{\log(2/\delta)}}{\sqrt{2m}}.
\end{equation}
\end{lemma}

\begin{proof}
First, we bound the change caused by a single replacement in $\hat{H}(T)$. We have that,
\begin{equation}
\hat{H}(T) = -\sum_t \left( \sum_x p(t|x)\hat{p}(x) \right) \log \left( \sum_x p(t|x)\hat{p}(x) \right).
\end{equation}

If we change a single instance in $\mathcal{S}$, then there exist two pairs $(x,y)$ and $(x', y')$ such that $\hat{p}(x,y)$ increases by $1/m$, and $\hat{p}(x', y')$ decreases by $1/m$. This means that $\hat{p}(x)$ and $\hat{p}(x')$ also change by at most $1/m$, while all other values in the distribution remain the same. Therefore, for each $t \in \mathcal{T}$, $\sum_x p(t|x)\hat{p}(x)$ changes by at most $1/m$.

Based on this and Eq.~\eqref{eq:supp:delta}, $\hat{H}(T)$ changes by at most $|\mathcal{T}|\log (m)/m$. Applying McDiarmid's inequality, we get Eq.~\eqref{eq:supp:lemma1}. We have thus proven Lemma~\ref{lem:supp:epybnd1}.
\end{proof}

Lemma~\ref{lem:supp:epybnd1} provides bounds on the deviation of the $\hat{H}(T)$ from their expected values. In order to relate these to the true values of the entropy $H(T)$, we use the following bias bound from~\cite{paninski2003estimation} and \cite{shamir2010learning}.

\begin{lemma}[Paninski~\cite{paninski2003estimation}; Shamir~et~al.~\cite{shamir2010learning}, Lemma~9]
\label{lem:supp:epybnd2}
For a random variable $T \in \mathcal{T}$, with the plug-in estimation $\hat{H}(\cdot)$ on its entropy, based on an i.i.d.~sample set of size $m$, we have that,
\begin{equation}
|\mathbb{E}[H(T) - \hat{H}(T)]| \leq \log \left( 1 + \frac{|\mathcal{T}| - 1}{m} \right) \leq \frac{|\mathcal{T}| - 1}{m}.
\end{equation}
\end{lemma}

From this lemma, the quantity $|\mathbb{E}[H(T) - \hat{H}(T)]|$ is upper bounded by $(|\mathcal{T}| - 1)/m$. Combining it with Eq.~\eqref{eq:supp:lemma1}, we get the bound in Proposition~\ref{prop:epybnd}.

\begin{proposition}[Proposition~\ref{prop:epybnd} restated]
Let $Y$ be a fixed probabilistic function of $X$ into an arbitrary finite target space $\mathcal{Y}$, determined by a fixed and known conditional probability distribution $p_{Y|X}$, and $\mathcal{S}$ be a sample set of size $N$ drawn from the joint probability distribution $p_{XY}$. For any $\delta \in (0, 1)$, with probability of at least $1 - \delta$ over the sample set $\mathcal{S}$, we have,
\begin{equation}
|H(Y) - \hat{H}(Y)| \leq \frac{|\mathcal{Y}|\log(N)\sqrt{\log(2/\delta)}}{\sqrt{2N}} + \frac{|\mathcal{Y}| - 1}{N}.
\end{equation}
\end{proposition}

\begin{proof}
To prove the proposition, we start by using the triangular inequality to write,
\begin{equation}
|H(Y) - \hat{H}(Y)| \leq |H(Y) - \mathbb{E}[\hat{H}(Y)]| + |\hat{H}(Y) - \mathbb{E}[\hat{H}(Y)]|.
\end{equation}

Because $H(Y)$ is constant, we have:
\begin{equation}
|H(Y) - \mathbb{E}[\hat{H}(Y)]| = |\mathbb{E}[H(Y)] - \mathbb{E}[\hat{H}(Y)]|.
\end{equation}
By the linearity of expectation, we have:
\begin{equation}
|\mathbb{E}[H(Y)] - \mathbb{E}[\hat{H}(Y)]| = |\mathbb{E}[H(Y) - \hat{H}(Y)]|.
\end{equation}
Combining these with Lemmas~\ref{lem:supp:epybnd1} and \ref{lem:supp:epybnd2}, we get the bound in Proposition~\ref{prop:epybnd}.
\end{proof}

\subsubsection{Proof of Corollary 1}

The proof of Corollary~\ref{cor:nondet} is based on the following bound proposed by~\cite{kolchinsky2019caveats}.

\begin{lemma}[Kolchinsky~et~al.~\cite{kolchinsky2019caveats}, Theorem~1]
\label{lem:supp:nondet}
Let $Z$ be a random variable (continuous or discrete), and $Y$ be a random variable with a finite set of outcomes $\mathcal{Y}$. Consider two joint distributions over $Z$ and $Y$, $p_{ZY}$ and $\tilde{p}_{ZY}$, which have the same marginal over $Z$, $p(z) = \tilde{p}(z)$, and obey ${|p_{ZY} - \tilde{p}_{ZY}|}_1 \leq \epsilon \leq \frac{1}{2}$. Then,
\begin{equation}
\label{eq:supp:nondet}
\left| H(p(Y|X)) - H(\tilde{p}(Y|X)) \right| \leq -\epsilon \log \frac{\epsilon}{{|\mathcal{Y}|}^3}.
\end{equation}
\end{lemma}

This lemma upper bounds the quantity $|H(p(Y|X)) - H(\tilde{p}(Y|X))|$ by $-\epsilon \log \log (\epsilon / {|\mathcal{Y}|}^3)$. After extending it to the case when $Y$ is a deterministic function of $X$, we get the bound in Corollary~\ref{cor:nondet}.

\begin{corollary}[Corollary~\ref{cor:nondet} restated]
Let $X$ be a random variable and $Y$ be a random variable with a finite set of outcomes $\mathcal{Y}$. Let $p_{XY}$ be a joint distribution over $X$ and $Y$ under which $Y = f(X)$. Let $\tilde{p}_{XY}$ be a joint distribution over $X$ and $Y$ which has the same marginal over $X$ as $p_{XY}$, i.e., $\tilde{p}_X = p_X$, and obey ${|\tilde{p}_{XY} - p_{XY}|}_1 \leq \epsilon \leq \frac{1}{2}$. Then, we have that,
\begin{equation}
\left| H(\tilde{p}(Y|X)) \right| \leq -\epsilon \log \frac{\epsilon}{{|\mathcal{Y}|}^3}.
\end{equation}
\end{corollary}

\begin{proof}
Since $Y$ is a deterministic function of $X$, i.e., $Y = f(X)$, we then have $H(p(Y|X)) = 0$. Combining this with Eq.~\eqref{eq:supp:nondet}, we prove the bound in Corollary~\ref{cor:nondet}.
\end{proof}

\subsection{Implementation Details}

\subsubsection{BNN Variant}

We follow~\cite{blundell2015weight} to implement the BNN variant of our method. Let $x$ be the observed input variable and $\theta$ be a set of latent variables. Deep neural networks can be viewed as a probabilistic model $p(y|x, \theta)$, where $\mathcal{D} = {\{x_i, y_i\}}_i$ is a set of training examples and $y$ is the network output which belongs to a set of object categories by using the network parameters $\theta$. The variational inference aims to calculate this conditional probability distribution over the latent variables (network parameters) by finding the closest proxy to the exact posterior by solving an optimization problem.

Following the guidance of~\cite{blundell2015weight}, we first assume a family of probability densities over the latent variables $\theta$ parameterized by $\psi$, i.e., $q(\theta|\psi)$. We then find the closest member of this family to the true conditional probability $p(\theta|\mathcal{D})$ by minimizing the KL-divergence between $q(\theta|\psi)$ and $p(\theta|\mathcal{D})$, which is equivalent to minimizing the following variational free energy:
\begin{equation}
\mathcal{L}_\mathsf{BNN}(\theta, \mathcal{D}) = \mathbb{D}_\mathsf{KL} \left[ q(\theta|\psi) \middle\| p(\theta) \right] - \mathbb{E}_{q(\theta|\psi)} \left[ \log (p(\mathcal{D}|\theta)) \right].
\end{equation}
This objective function can be approximated using $T$ Monte Carlo samples $\theta_i$ from the variational posterior~\cite{blundell2015weight}:
\begin{equation}
\mathcal{L}_\mathsf{BNN}(\theta, \mathcal{D}) \approx \sum_{i=1}^{T} \log (q(\theta_i|\psi)) - \log (p(\theta_i)) - \log (p(\mathcal{D}|\theta_i)).
\end{equation}

We assume $q(\theta|\psi)$ have a Gaussian probability density function with diagonal covariance and parameterized by $\psi = (\mu, \sigma)$. A sample weight of the variational posterior can be obtained by the reparameterization trick~\cite{diederik2014auto}: we sample it from a unit Gaussian and parameterized by $\theta = \mu + \sigma \circ \epsilon$, where $\epsilon$ is the noise drawn from the unit Gaussian $\mathcal{N}(0, 1)$ and $\circ$ is the point-wise multiplication. For the prior, as suggested by~\cite{blundell2015weight}, a scale mixture of two Gaussian probability density functions are chosen: they are zero-centered but have two different variances of $\sigma_1^2$ and $\sigma_2^2$ with the ratio of $\pi$. In this work, we let $-\log \sigma_1 = 0$, $-\log \sigma_2 = 6$, and $\pi=0.25$. Then, the optimizing objective of adversarial perturbations in the maximization phase of our method is redefined by:
\begin{equation}
X_i^k \in \argmax_{x \in \mathcal{X}} \left\{ \mathcal{L}_\mathsf{CE}(\theta; x, Y_i) + \beta \sum_{j=1}^{T} \frac{1}{T} h(\theta_j; x) - \gamma c_\theta((x, Y_i), (X_i^{k-1}, Y_i)) \right\},
\end{equation}
where $\theta_j$ is sampled $T$ times from the learned variational posterior.

\subsubsection{PACS}

The learning principle of the previous state-of-the-art method on this dataset follows two streams. The first stream of methods, including DSN~\cite{bousmalis2016domain}, L-CNN~\cite{li2017deeper}, MLDG~\cite{li2018learning}, Fusion~\cite{mancini2018best}, MetaReg~\cite{balaji2018metareg} and Epi-FCR~\cite{li2019episodic}, engages domain identifications, which means that when training the model, each source domain is regarded as a separate domain. The second stream of methods, containing AGG~\cite{li2019episodic}, HEX~\cite{wang2019learning}, and PAR~\cite{wang2019robust}, does not leverage domain identifications and combines all source domains into a single one during the training procedure. We can find that the first stream leverages more information, i.e., the domain identifications, during the network training, and thus often yields better performance than the second stream. Our work belongs to the latter stream.

\begin{table}[h]
\caption{The settings of different target domains on PACS.}\label{tbl:supp:pacs}
\centering
\resizebox{.86\linewidth}{!}{
\begin{tabular}{@{}ccccccccc@{}}
\toprule
Target Domain & $K$ & $T$ (loop) & $T_\mathsf{MIN}$ (loop) & $T_\mathsf{MAX}$ (loop) & $\alpha$ & $\eta$ & $\gamma$ & $\beta$\\
\midrule
Art & 1 & 45,000 & 100 & 50 & 0.001 & 50.0 & 10.0 & 1.0\\
Cartoon & 1 & 45,000 & 100 & 50 & 0.001 & 50.0 & 10.0 & 100.0\\
Photo & 1 & 45,000 & 100 & 50 & 0.001 & 50.0 & 10.0 & 1.0\\
Sketch & 1 & 45,000 & 100 & 50 & 0.001 & 50.0 & 10.0 & 100.0\\
\bottomrule
\end{tabular}}
\end{table}

We follow the setup of~\cite{li2017deeper} for network training. To align with the previous methods, the ImageNet pretrained AlexNet~\cite{krizhevsky2012imagenet} is employed as the baseline network. In the network training, we set the batch size to 32. We use SGD with the learning rate of 0.001 (the learning rate decays following a cosine annealing schedule~\cite{zagoruyko2016wide}), the momentum of 0.9, and weight decay of 0.00005 for minimization, while we use the SGD with the learning rate of 50.0 for maximization. Table~\ref{tbl:supp:pacs} shows more detailed setting of all parameters under four different target domains.

\subsubsection{CIFAR-10 and CIFAR-100}

\begin{table}[h]
\caption{The settings of different network architectures on CIFAR-10-C and CIFAR-100-C.}\label{tbl:supp:cifar}
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}llcccccccc@{}}
\toprule
& & $K$ & $T$ (epoch) & $T_\mathsf{MIN}$ (epoch) & $T_\mathsf{MAX}$ (loop) & $\alpha$ & $\eta$ & $\gamma$ & $\beta$\\
\midrule
\multirow{4}{*}{CIFAR-10-C} & AllConvNet & 2 & 100 & 10 & 15 & 0.1 & 20.0 & 0.1 & 10.0\\
& DenseNet & 2 & 200 & 10 & 15 & 0.1 & 20.0 & 1.0 & 100.0\\
& WideResNet & 2 & 100 & 10 & 15 & 0.1 & 20.0 & 1.0 & 10.0\\
& ResNeXt & 2 & 200 & 10 & 15 & 0.1 & 20.0 & 1.0 & 10.0\\
\midrule
\multirow{4}{*}{CIFAR-100-C} & AllConvNet & 2 & 100 & 10 & 15 & 0.1 & 20.0 & 0.1 & 10.0\\
& DenseNet & 2 & 200 & 10 & 15 & 0.1 & 20.0 & 10.0 & 10.0\\
& WideResNet & 2 & 100 & 10 & 15 & 0.1 & 20.0 & 1.0 & 10.0\\
& ResNeXt & 2 & 200 & 10 & 15 & 0.1 & 20.0 & 10.0 & 10.0\\
\bottomrule
\end{tabular}}
\end{table}

The experimental settings follow the setups in~\cite{hendrycks2020augmix}. We use SGD for both minimization and maximization. In Table~\ref{tbl:supp:cifar}, we report the detailed settings of all parameters under different network architectures on CIFAR-10-C and CIFAR-100-C. Note that $T$ and $T_\mathsf{MIN}$ are measured by number of training epoches, while $T_\mathsf{MAX}$ is measured by number of iterations. In this work, we do not compare our method with \cite{hendrycks2020augmix}, since the design of \cite{hendrycks2020augmix} depends on a set of pre-defined image corruptions which is with a different research target compared to our method.

\end{appendices}

\bibliographystyle{plain}
\small

redpajama_set_name: RedPajamaArXiv


text:
Click on titles below to read the entire post, access the archive, and make comments.
Pope Francis adding frequent flyer miles at record pace
When he was elected in 2013 at the age of 76, Pope Francis was not expected to be a globe-trotting pope. After all, this was a man who, as archbishop of Buenos Aires, traveled infrequently to Rome because he considered such trips an extravagance.
But in Panama last week, the pope logged his 26th foreign trip – and he shows no sign of slowing down. He plans to make four more trips before the middle of this year, including a February visit to the United Arab Emirates, the first by a pope to the Arab peninsula.
In the second half of 2019, the pope is said to be considering trips to Uganda, Mozambique and Japan.
Francis is putting on miles at a rate faster than any pope in history. Even Pope John Paul II, whose international travels turned the papacy into a global ministry, didn't reach his 30th foreign trip until nearly eight years in office.
Most of Pope Francis' trips have been outside Europe – meaning long-distance flights, jet lag and plenty of time for airborne press conferences.
Speaking to reporters on his first such trip in 2013, Francis seemed to accept international travel as part of the modern papal job description. He said John Paul II had made it clear that the pope must first of all be a "great missionary."
"He was a missionary, a man who carried the Gospel everywhere, as you know better than I. How many trips did he make? But he went! He felt this fire of carrying forth the Word of the Lord. He was like Paul, like Saint Paul, he was such a man; for me this is something great," Francis said.
Pope says no to wholesale change on priestly celibacy, but sees room for exceptions
Pope Francis has offered some clarity about how far he'll go on the issue of married priests.
Speaking to reporters aboard his flight from Panama to Rome Jan. 27, the pope said he did not agree with those who want to make priestly celibacy "optional" throughout the Latin-rite church.
The pope was asked whether he could envision the Catholic Church adopting the practice of Orthodox Churches, in which priestly candidates can choose whether to marry or remain celibate – a choice made before they are ordained as deacons.
"My decision is: optional celibacy before the diaconate, no," the pope said. "That is what I think. I will not do it and this remains clear. Am I closed minded? Perhaps, but I don't feel that I can make this decision before God."
But he left the door open to ordaining married men as an exception to the celibacy rule, in local areas of great pastoral need.
"I think the question should be open in places where there is a pastoral problem because of the lack of priests. I'm not saying that we have to do it, because I have not reflected and prayed about this sufficiently, but theologians have to study this," he said.
He said that for local church communities, the essential question is access to Mass and the sacraments. "Where there is not the Eucharist … who will make the Eucharist?" he said.
The issue is expected to be raised during a regional Synod of Bishops from the Amazon region, to be held at the Vatican in October. The region faces a severe shortage of priests, and a preparatory document for the synod has called for "courageous, daring and fearless" proposals to deal with pastoral challenges.
If such an exception is granted in the Amazon region, many believe it will place the church on a path of change regarding the overall celibacy rule.
The Catholic Church already has married priests. Generally, Catholic Eastern churches allow married men to be ordained. Celibacy in the Latin-rite church – to which the vast majority of Catholics belong – has been a tradition for many centuries, and a topic of increasing theological debate for decades.
New role of laity is key issue in sex abuse summit
There are signs that Vatican officials preparing the February summit on sex abuse hope the event will launch a new role for Catholic laity.
That would be a significant development in a scandal that, as Pope Francis himself has said, has been perpetuated by clericalism.
At first glance, the Feb. 21-24 summit looks like another "bishops monitoring the bishops" event. But planners have quietly taken steps to involve lay experts, and are signaling a more open approach to lay authority and supervision when it comes to dealing with accusations of clerical sex abuse.
In a recent interview with America magazine, Maltese Archbishop Charles Scicluna, one of the meeting's organizers, said the need for transparency and accountability on sex abuse requires bishops to "empower the lay people":
"We bishops need to approach the issue of the sexual abuse of minors together as churches, and we also need to adopt what Pope Francis is calling 'a synodal approach,' that is we cannot do it alone in our community, we need also to empower the lay people, the laity, in order to help us be good stewards."
Scicluna repeated the point for emphasis: "Synodality means that we appreciate the different charisms and gifts of the laity, their expertise, and that we empower them to join bishops in the role of stewardship."
As if to anticipate the objections of conservatives, the archbishop added: "It's not a question of (the laity) having control over the hierarchy, it is the hierarchy empowering and facilitating the sharing of charisms which the Spirit also gives to the laity, because there are gifts there that will help issues of prevention and safeguarding that we need to bring on board, and we need to facilitate as bishops."
What that means in terms of practical responsibility and authority remains to be seen. Pope Francis did name two Vatican lay women to help prepare the summit: Dr. Gabriella Gambino and Dr. Linda Ghisoni, both undersecretaries in the Vatican's office for Laity, Family and Life.
A key question is whether the pope's "synodality" vision will trump the more legalistic approach of Vatican canon lawyers when it comes to lay responsibility regarding bishops' decision-making and accountability.
It was Archbishop Scicluna who, in an address to canon lawyers in 2013, pointed out that under church law bishops can lose their office for abuse or negligence in ministry, and in this sense are seen as accountable to their faithful.
Over the last year, "more involvement by the laity" has been a popular phrase in the church's discussion of sexual abuse, and it's come from all quarters – the pope, U.S. bishops, victims' advocacy groups and leading Catholics. Now it needs to be translated into meaningful measures.

redpajama_set_name: RedPajamaCommonCrawl


text:
The Wood Word • March 19, 2020 • https://www.thewoodword.org/sports/2020/03/19/2020-mlb-power-rankings-13-washington-nationals/
2020 MLB Power Rankings: 13. Washington Nationals
Photo via Wikimedia Commons under Creative Commons license
Brendan Murphy, Assistant Sports Editor
Last week, Major League Baseball (MLB) announced that it was pushing back the start of the 2020 season to May, so we will be without baseball for a while. These rankings haven't been coming out as frequently as usual due to everything going on in the world with coronavirus and trying to stay safe and healthy.
We return to the power rankings with the 2019 World Series Champions, the Washington Nationals. After losing arguably their best player in free agency, I am ranking them surprisingly low at number 13.
The 2019 Nationals showed that hard work pays off. Not only did the team overcome enormous odds to win their first World Series title in franchise history, but they beat the Houston Astros to do it. The Astros were rumored to be cheating in 2019 by using buzzers to indicate what pitch was coming, but the claim could not be proven as there was no hard evidence. The Nationals became the first team in MLB history to win the championship by winning every game on the road. It's a feat that we will likely never see again or at least for a long while.
As for the 2020 Nationals, they will be without superstar third basemen Anthony Rendon, who departed during the offseason. Even though Rendon is the only big piece they lost, I still think the Nationals will take a significant step back. When the team lost Bryce Harper to the Phillies last offseason, they were equipped with a guy who would immediately take his place in Juan Soto. With Rendon gone, there is no clear replacement option. Carter Kieboom is the logical choice since he is their top prospect and already has some big league experience, but his experience is almost exclusively at shortstop and second base. Kieboom will have to adjust to a new position as well as work on hitting big league pitching.
The rest of the lineup hasn't chanced much. Juan Soto is still a superstar left fielder at only 21 years old, Trea Turner is still electric in all facets of the game and Victor Robles will look to build on a very successful 2019 campaign and the team signed Starlin Castro to play second base after spending the last few seasons with the Miami Marlins. It will be interesting to see how this team performs without Rendon in the middle of the order.
The Nationals starting rotation is still arguably the best in baseball. Three-time Cy Young Award winner Max Scherzer and 2019 World Series MVP Stephan Strasburg come in at the top and Patrick Corbin is no slouch as a number three starter as he was a bulldog in the postseason for the Nationals both in the rotation and out of the bullpen. Anibal Sanchez is a good number four starter and Joe Ross and Erick Fedde will be competing for the fifth starter spot.
The Nationals have the worst farm system in baseball according to Bleacher Report, especially with Carter Kieboom already reaching the majors; however, Jackson Rutledge is a name who isn't high on many prospect boards but was a 2019 first round pick and has the potential to impact the Nationals rotation as early as this season.
Rutledge is a tall righty out of a Texas Junior College who rocked up draft boards after his impressive 2019 season. He is only 20 years old but already possesses electric stuff with a fastball in the upper 90s and a wicked slider. Rutledge could be a guy that shoots up prospect boards all year as he has already flown through the Nationals farm system, making it all the way to single-A baseball in just his first pro season.
The Nationals are a team that interest me because I think there is a chance they could still be really good without Anthony Rendon. However, the fact is that Rendon is a huge loss and there questions surrounding a possible World Series hangover for the team. I expect the Nationals to fall back a bit and be a team that is in contention for a Wild Card spot, but will ultimately come up just a bit short.
Top Acquisition – Starlin Castro
Biggest loss – Anthony Rendon
MVP – Juan Soto
Projected Finish – 4th in NL East
Projected Lineup:
Trea Turner, SS
Adam Eaton, RF
Starlin Castro, 2B
Juan Soto, LF
Victor Robles, CF
Eric Thames, 1B
Kurt Suzuki, C
Carter Keiboom, 3B
Projected Rotation:
Stephen Strasburg
Joe Ross
Twitter: @B_Murphy_MU


text:
NG GENIUS
Graphene use in composites and polymers
Home/Articles Idealab/Graphene use in composites and polymers/Graphene use in composites and polymers
The biggest problem confronting products made from composites or polymer materials is static electricity. For example, self-leveling epoxy flooring increases in popularity all the time, but its propensity to build up static can result in electrical discharges and fires.
Similarly, the generation of static electricity in the blades of wind turbines can harm the generators and their wiring.
Static may be a simple phenomenon of everyday life that commonly goes unnoticed, but it can still have serious consequences.
However, the problem of static electricity can now be solved very simply: The addition of nanographene to epoxy flooring and polyester resins and polymers in the proportion of 0.05% to 1.5% provides composites and resins with anti-static properties.
Meanwhile, the expanded use of composites and polymers also depends on increasing materials' strength and durability. Nanographene use in polymers has attracted considerable attention in the automobile, aerospace, yacht and shipbuilding industries. The addition of graphene in the proportion of a few hundredths makes polymers and composites 90-100% stronger!
Also of note in scientific and tech circles is the prospect of adding graphene to conductive polymers. Currently, graphene is used in polystyrene, polyethylene, polypropylene and in various electrical products (electrical plugs and sockets, various switches, alarms, circuit breakers, etc). In addition, graphene conductive polymers are used in the manufacture of shielding layers for electrical cables.
Moreover, graphene can be introduced into any production process with virtually no change in technology, lending resins and polymers a significant advantage in quality and cost.
By admin| 2017-10-12T21:50:30+00:00 October 1st, 2017|Graphene use in composites and polymers|0 Comments
Share The New Reality Post!
NANOGRAPHENE RECEIVES Notification of Passing the Preliminary Examination of Patent Application for invention IN CHINA
Ultrapure Affordable and Environmentally Friendly Graphene
Graphene: the water purifier for a thirsty world in the next 50 years
Graphene use in transmission fluids and oils
Copyright 2016 - 2017 |
Email: info@nanographene.net
NanoGraphene Inc Signs a 5 Year Binding Off-Take Agreement with the Australian Based Graphite Exploration Company, Volt Resources Ltd.


Clear Male 3/4 Torso Form make of unbreakable plastic. Available with hanging loop to display on hanging base or from face out. Also you can add a leg fitting for use on base. Made in the USA of unbreakable polycarbonate.
Chest: 38.5" Waist: 31" Hip: 38.5" Height: 37"
See Hanging loop below to use with Hanging Base. Also offset torso bases with pole heights of 32" and 42". Select in options above as needed.


text:
Jamie Hawkesworth
Beirut Re-Store is selling art to help the Lebanese people
With the help of Jamie Hawkesworth, JW Anderson, Ashley Williams, Ronan McKenzie (and loads more), online marketplace Beirut Re-Store hopes to raise money following the city's devastating explosion earlier this year.
Beirut's devastating explosion left more than 200 dead, around 5,000 injured and 300,000 homeless, when a warehouse storing ammonium nitrate caused an explosion in the Lebanese capital on 4th August this year.
Like many, French-Lebanese creative director Laura Tabet was shocked by the events unfolding and determined to do something productive. Inspired by her bustling network of photographers, designers and artists – all of whom offered unwavering support to both her and her sister, Sophie – she decided to set up Beirut Re-Store: stocking the best in art, fashion and design.
"Creating Beirut Re-Store is our way of helping Lebanon," she says of an online marketplace featuring work by the likes of Jamie Hawkesworth, Harley Weir, Ashley Williams, JW Anderson, Frank LeBon, IDEA Books and Ronan McKenzie. "There was only so much we could give as individuals, but this will hopefully allow us to give back to the country on a bigger scale."
Alongside friend Ulysse Sabbagh, and in partnership with fellow Lebanese creatives Flavie Audi and Emilie Kareh, Beirut Re-Store will have new stock added weekly – priced anywhere between £10 – 15,000 – with 100% of its proceeds disbursed to well-vetted non-profit organisations aiding shelter, rehabilitation for households, and medical and mental health.
"The funds raised will be disbursed with full transparency to well-vetted non-profit organisations, [aiding] shelter and rehabilitation for households, as well as medical and mental health."
With an eclectic mix of work being sold – a sunflower print by Hawkesworth, lime green shades from Dior's archive, and an oil pastel work of art by artist Faye Wei Wei – Tabet hopes there will be something for everyone. The end goal is, after all, to raise as much money as possible.
"Stay tuned for some unique and limited edition products – and lucky dip experiences!" Tabet says. "I hope everyone is able to find something that delights them and, in turn, help the people of Lebanon."
Shop art, fashion, jewellery and more, while helping a vital cause, at beirutrestore.com


text:
Plans for 30-minute Covid testing in England halted amid accuracy fears
Exclusive: government shelves Christmas rollout of lateral flow test centres
Josh Halliday and Helen Pidd
Tue 22 Dec 2020 10.25 EST Last modified on Tue 22 Dec 2020 23.37 EST
A swab being processed in Grangetown, Redcar. Lateral flow tests provide results in 30 minutes, but a pilot in Liverpool showed they missed 30% of cases with a high viral load. Photograph: Ian Forsyth/Getty Images
The government has shelved plans to open rapid-turnaround coronavirus test centres across England over Christmas amid concerns from public health experts about the accuracy of their results, the Guardian has learned.
Ministers had planned to convert a number of existing testing sites into centres for lateral flow tests, which provide results in 30 minutes, to help cope with an anticipated surge in demand.
However, the scheme was halted last week after concerns were raised by directors of public health about the accuracy of the tests and the potential false reassurance given to people who test negative. A government source said the planned rollout "proved unnecessary".
The development is a blow to the UK government's £100bn "Operation Moonshot" mass-testing plan, which aims to increase the number of tests carried out each day from 430,000 to 10m by early next year.
It also raises questions about the plan to rapidly test lorry drivers taking goods across the Channel, which is one of the UK government's proposed solutions to bring an end to the disruption at Britain's ports.
The Guardian has learned that the Department of Health and Social Care (DHSC) invited local health directors to convert their existing coronavirus testing sites into lateral flow test centres to help ease a possible surge in demand over Christmas.
But the plans were shelved within days of a phone call last week between the government and directors of public health, who raised concerns over accuracy.
It is understood that the government is instead planning to convert one testing centre and analyse the results before potentially expanding the scheme more widely. This process is not expected to conclude before the end of January.
Government figures from the mass testing programme in Liverpool revealed earlier this month that the tests missed 30% of cases with a high viral load and half of positive cases that were detected by standard coronavirus tests.
Lateral flow devices were used to mass-test university students across the UK before they returned home for Christmas earlier this month. However, a study of the results of more than 7,000 students at the University of Birmingham suggested about 60 positive cases were missed.
Jon Deeks, a professor of biostatistics and the head of the test evaluation research group at the University of Birmingham, said the tests were "not fit for purpose" unless they were used on highly infectious people and that even then, a follow-up test was required using a swab.
"There's a big risk this test will give a lot of false reassurance which will inevitably lead to more Covid disease," he said.
Deeks said the best thing the government could do with the 20m tests it had ordered was donate them to a poor country that did not have laboratory capacity to carry out the gold-standard swab tests. Their use as proposed by the government was "dangerous" and an "enormous waste of time and money", he said.
A DHSC spokeswoman said the tests were "accurate, reliable and successfully identify those with Covid-19 who don't show symptoms and could pass on the virus without realising".
She added: "The country's leading scientists rigorously evaluated the lateral flow test and confirmed the accuracy of the tests using a sample of over 8,500. Latest figures for similar settings showing sensitivity of 57.5% generally and 84.3% in people with high viral loads.
"With up to a third of individuals with Covid-19 not displaying symptoms, broadening testing to identify those showing no symptoms will mean finding positive cases more quickly and break chains of transmission. Anyone who tested positive with a lateral flow test during the university testing earlier this month would have been asked to get a confirmatory PCR test."


This is "Focusing, Developing, and Synthesizing", section 5.1 from the book A Guide to Perspective Analysis (v. 1.0). For details on it (including licensing), click here.
Discuss how to focus and develop the essay.
Reveal how to convey an effective organization through transitions.
Suggest strategies for producing effective opening and closing paragraphs.
After discussing general strategies for analysis and applying these strategies to specific examples in class, I inevitably encounter a student asking, "This has all been well and good, but when are we going to actually learn how to write?" The student's confusion most likely emerges from how he was taught in the past. In most school assignments, writing does not require thinking so much as the stuffing of obvious considerations or memorized material into formulated structures, like a five-paragraph essay or a short answer exam. However, in less restrictive writing situations the specific way we articulate our analysis emerges from what we think of it, and thus our best writing comes through our most careful considerations. The good news, then, is that if you have been following the advice I've given throughout this book about coming up with your analysis, then you will have already finished most of the work on your essay. The bad news is that there is no easy formula for putting it all together. However, we still can examine general strategies that successful analytical writers tend to use, though the specific way you enact these strategies will depend on the ideas that you have already discovered.
If you have taken the time to examine your subject thoroughly and read what others have written about it, then you might have so much to say that you will not be able to cover your perspective adequately without turning your essay into a book. In such a case you would have two options: briefly cover all the aspects of your subject or focus on a few key elements. If you take the first option, then your essay may seem too general or too disjointed. A good maxim to keep in mind is that it is better to say a lot about a little rather than a little about a lot; when writers try to cover too many ideas, they often end up reiterating the obvious as opposed to coming up with new insights. The second option leads to more intriguing perspectives because it focuses your gaze on the most relevant parts of your subject, allowing you to discern shades of meaning that others might have missed.
Once you have focused the scope of your thesis, revise your essay to reflect it. This will require you to engage in what is usually the most painful part of the writing process—cutting. If something does not fit in with your perspective, it has to go, no matter how brilliantly considered or eloquently stated. In the course of writing this book, I've had to cut several sections simply because they no longer corresponded with the main perspective I wanted to convey. But do not throw away the parts you cut. You never know when you might find a use for them again. Just because a particular section does not fit well with the focus of one essay does not mean that you won't be able to use it in another essay down the road.
Are there clear assertions throughout the essay that reveal your perspectives on the subject?
Do you provide the specific examples that inspired these assertions?
Do you explain how you derived your assertions from a careful reading of these examples?
Do you explore the significance of these assertions as they relate to personal and broader concerns?
If any long sections seem lacking in any of these areas of AXES, you might explore them further by taking time out from your more formal writing to play with one of the heuristics recommended in various sections throughout this book (freewriting, metaphor extension, issue dialogue, the Pentad, brainstorming, and clustering). You can then incorporate the best ideas you discover into your essay to make each section seem more thoughtful and more thorough.
As we further consider the implications, we might be reminded of past teachers who made reading Shakespeare feel less like a celebration and more like a task, as something to be respected but not enjoyed. We could then explain how the word "revels" serves as a reminder to enjoy his plays, and not because they are "good for us" like a nasty tasting vitamin pill, but because if we're willing to take the effort to understand the language, the plays become deeply entertaining. Looking back over the passage and seeing how plays are equated to our lives outside the theater leads to an even more significant insight. We should try to see life as a celebration, as something to be enjoyed before we too disappear into "thin air." In discussing the significance of this, we wouldn't simply wrap it up in a cliché like "I intend to live only for today," but explore more responsible ways we can balance fulfilling our obligations with enjoying the moments that make up our lives.
Now we can go back and expand the main assertion. Instead of simply writing, In The Tempest, Shakespeare connects plays, lives and dreams by showing that while each contains an illusion of permanence, they're all only temporary, we might also add, But this does not mean that we should waste the time we have on earth or in the theater lamenting that it will all soon be over. Instead we should celebrate, in a responsible manner, our remaining moments. And because all of these insights came about from examining the implications of only one word, "revels," the essay will continue to expand as we consider more details of the passage and consult related research. Eventually, however, we will need to stop expanding our analysis and consider how to present it more deliberately.
When revising your essay, you do not have to write it in the exact order that it will be read, as any section you work on in a given moment may appear anywhere in your final draft. In fact, many times it's best to write the first paragraph last because we may not know how to introduce the essay until we've discovered and articulated the main perspectives. However, eventually you will need to consider not only what your analysis consists of, but also the effect you want it to have. An essay that commands attention seems like a discussion between intelligent and aware people, in which ideas are not thrown out randomly but in a deliberate manner with each thought leading logically to the next.
For this reason, the opening paragraphThe part of your essay where you entice the reader to want to continue by leading her into the main perspective (as opposed to spelling out the main details in a rigid manner). should be the place where you invite your readers into this discussion, making them want to read what will follow without delineating the main content in a rigid manner. Again, imagine being at a party, but this time instead of meeting someone who bores you by reciting irrelevant details of the past, he tells you exactly what will follow in the near future: "Over the next ten minutes we will discuss three things: work, politics, and leisure activities. During the course of our discussion, we will raise relevant personal experiences, draw from a bevy of beliefs and morals, and reflect on the current state of international affairs." Again, most likely you and everyone else this person approaches will find an excuse to move to the other side of the room as quickly as possible. Similarly when writers begin their essays with a step-by-step announcement of what will follow, we don't feel the sense of anticipation that we do when the perspective unfolds more organically. Successful analytical essay writers do not begin by blatantly spelling out the main points that they will cover, but rather create "leads," openings that hook the reader into wanting to read further.
One way to capture the reader's attention is to share a story or anecdote that directly relates to the main perspective. For instance, in the first chapter, I created a story about a hypothetical student named Jeff who was having difficulty writing an analytical paper on The Tempest in order to reveal a situation that not only was widely familiar but also allowed me to introduce the various components of analysis.
You can also capture your reader's attention with a quote: "Oh what fools these mortals be" has become one of my favorite Shakespeare quotes since I began working in a restaurant. I am always amazed by the litany of ridiculous questions and requests I have to entertain during each of my shifts.
Or you might try a joke: Once there was a small boy who lost the key to his house. Though he dropped it in the front yard, he chose to look for it near the sidewalk because they light was much better there. Like him, many people look for the easiest solutions to their problems even when they know the truth is far more complicated and obscure..
Or perhaps you can startle the reader with an unexpected twist:The best day of my life occurred last summer. First, I was fired from my job, next my girlfriend dumped me, and finally I was kicked out of my parent's house. All this motivated me to find a better job, a better girlfriend, and a better place to live. History is full of days like this, ones that seem tragic yet turn out to have positive consequences in the long run.
Finally, you might begin with an analogy: Trying to write a perfect essay all at once is like attempting to ride a bike while juggling and singing opera. You are likely to crash unless you take on each task separately: invention, drafting, revising, and editing.
There's an old Sufi joke that points out that "the moon is more valuable than the sun because at night we need the light more." Of course the joke's humor arises from the fact that without the sun, it would be night all the time, and yet it does seem to be human nature to take advantage of that which is constant in our lives, the people and things that add warmth and light on a daily basis. In applying this to the television show, Mad Men, it's easy to see how Donald Draper, the main character, undervalues his wife Betty in order to chase after other women. Though these other women are as inconstant as the moon, disappearing and reappearing in new forms, they give him light during the dark times in his life when he needs it the most. His affairs, however, do not provide lasting satisfaction, but only a fleeting illusion of happiness, much like the advertisements he creates for a living.
Notice how this paragraph leads the reader from the hook to the main focus of the essay without spelling out what will follow in a rigid manner. The Sufi joke is not simply thrown out for a chuckle, but to set up the thesis that the main character of the show prefers illusions to reality in both his personal life and his work. As a result, this paragraph is likely to engage our attention and make us want to read further.
Once you've led your readers into your essay, you can keep their attention by making certain that your ideas continue to connect with each other by writing transitionsAssertions that link the main perspective of one paragraph to the paragraph that follows. between your paragraphs and the main sections within them. At the beginning of a paragraph, a transition functions as a better kind of assertion than a topic sentence because it not only reveals what the paragraph will be about but also shows how it connects to the one that came before it. Take this paragraph you are currently reading as an example. Had I begun by simply writing a topic sentence like "A second strategy for effective writing is to develop effective transitions," I would not only have ignored my own advice, but also would have missed an important point about how transitions, like opening paragraphs, function to lead readers through various aspects of our perspectives.
Before you can write effective transitions, you need to make certain that your paper is organized deliberately throughout. To insure this, you might try the oldest writing trick in the composition teacher's handbook, the outline. But wait until after you have already come up with most of your analysis. To begin a paper with an outline requires that you know the content before you have a chance to consider it. Writing, as I've argued throughout this book, is a process of discovery—so how can you possibly put an order to ideas that you have not yet articulated? After you have written several paragraphs, you should read them again and write down the main points you conveyed in each of them on a separate piece of paper. Then consider how these points connect with each other and determine the best order for articulating them, creating a reverse outlineA method of organizing a paper in which you list the main points of a draft, organize these points into an outline, and reorganize the draft to reflect the outline. from the content that you've already developed. Using this outline as a guide, you can then reorganize the paper and write transitions between the paragraphs to make certain that they connect and flow for the reader.
An excellent method for producing effective transitions is to underline the key words in one paragraph and the key words in the one that follows and then to write a sentence that contains all of these words. Try to show the relationship by adding linking words that reveal a causal connection (however, therefore, alternatively) as opposed to ones that simply announce a new idea (another, in addition to, also). For example, if I were to write about how I feel about having to pay taxes, the main idea of one paragraph could be: Like everyone else, I hate to see so much of my paycheck disappear in taxes. And the main idea of the paragraph that follows could be: Without taxes we wouldn't have any public services. My transition could be: Despite the fact that I hate to pay taxes, I understand why they are necessary because without them, we wouldn't be able to have a police force, fire department, public schools and a host of other essential services. If you cannot find a way to link one paragraph to the next, then you should go back to your reverse outline to consider a better place to put it. And if you cannot find any other place where it fits, then you may need to cut the paragraph from your paper (but remember to save it for potential use in a future essay).
I love my two pets. My cat, Clyde is very independent. My dog, Mac, barks if I leave him alone for very long. I can leave Clyde alone for four days. I'm only taking Clyde with me to college. I have to come home twice a day to feed Mac. Mac does a lot of tricks. Clyde loves to purr on my lap.
I have two pets that I love for very different reasons. For instance, I love when my cat, Clyde, sits on my lap and purrs, and I also love when my dog Mac performs many of the tricks I've taught him. But when I leave for college, I plan to take only Clyde with me. Unfortunately I can only leave Mac at home for a few hours before he starts to bark; however, Clyde is very independent and can be left in my dorm for days without needing my attention.
This revision not only is much easier to read and recall but also gives a sense of coherence to what previously seemed liked scattered, random thoughts.
Once you've led your readers all the way through to the closing paragraphThe part of your essay where you stress the significance of your analysis by calling attention to what you hope the reader has learned from it or by challenging him to action or further thought (as opposed to simply summarizing what you've already covered)., try not to sink their enthusiasm by beginning it with the words "in conclusion." Not only is this phrase overused and cliché, but it also sends the wrong message. The phrase implies that you have wrapped up all the loose ends on the subject and neither you nor your readers should have any need to think about it further. Rather than close off the discussion, the last paragraph should encourage it to continue by stressing how your analysis opens up new avenues for thinking about your subject (as long as these thoughts emerge from your essay and are not completely unrelated to what you wrote about before). This is the place where you should stress the significance of your analysis, underscoring the most important insights you discovered and the implications for further thought and action.
Scattered amid the broken bumper cars and ferris wheel seats are nostalgic bits and pieces of Southern California's famous extinct amusement parks (in the pre-Disney days when admission was free or $1); the Pike, Belmont Shores, Pacific Ocean Park, and so on. Suddenly rearing up from the back of a flatbed trailer are the fabled stone elephants and pouncing lions that once stood at the gates of Selig Zoo in Eastlake (Lincoln) Park, where they had enthralled generations of Eastlake kids. I tried to imagine how a native of Manhattan would feel, suddenly discovering the New York Public Library's stone lions discarded in a New Jersey wrecking yard. I suppose the Selig lions might be Southern California's summary, unsentimental judgment on the value of its lost childhood. The past generations are like so much debris to be swept away by the developers' bulldozers.Mike Davis, City of Quartz (New York: Vintage Books, 1990), 435.
Imagine, if instead of this paragraph, he had written: In conclusion I have shown many instances in which Southern Californians try to erase their past. First I showed how they do so by constructing new buildings, concentrating especially on the Fontana region. Second I showed… Can't you just feel the air leaving your sails?
In light of this advice, you have probably already discerned that certain parts of your essay will emphasize various aspects of analysis. The beginning of the paper will announce your main assertion or thesis and the transitions in subsequent paragraphs will present corollary assertions. The bulk of your paper will most likely center on your examples and explanations, and the end will focus more on the significance. However, try to make certain that all of these elements are present to some degree throughout your essay. A long section without any significance may cause your readers to feel bored, a section without assertions may cause them to feel confused, and a section without examples or explanations may cause them to feel skeptical.
Write a lead paragraph for a potential essay on a subject that you've already developed a strong perspective on. Begin with one of the strategies I mention in this section—an anecdote, a quote, an analogy, a story, an unexpected twist—and connect your lead to your thesis or question that you wish to explore. Consider, too, how you might end this essay. Think further about what you find to be the most significant aspect of your subject and what key images or thoughts you want to leave lingering in the minds of your readers.
The writing process begins when we first start to consider a subject because we form, develop, and articulate our thoughts recursively.
It is important to focus your analysis on the essential features of the subject and to make sure that each of these features receives adequate development.
Effective essays subtly lead us into the key perspective, provide transitions. between the main sections, and leave us with something important to consider.


LA BORRIQUITA (The Small donkey). HERMANDAD DE LA SAGRADA ENTRADA EN JERUSALEN (Seville). Float vid + text.
I'VE HAD BAD THOUGHTS. HAVE I OFFENDED GOD?
POPE FRANCIS: CONTEMPLATE ON THE HEART OF THE FATHER OF THE PRODIGAL SON. 4th Sunday of Lent (C).


As a manufacturer of quality plastic-based medical equipment and accessibility devices in South Bend, Indiana, we set our standards high when it comes to quality and customer service. Our products are constructed of the highest quality polyvinyl chloride resin structural tubing which will not chip, dent, crack, peel, fade, stain or sustain bacterial growth. We customize and modify our products to fit nearly any bathroom and client situation.