Antiques galore! Furniture, from antique to fun, chippy and funky! Large selection of Antique tools, beautiful vintage jewelry, art, glassware, barware and so very much more! Come by and visit!
Antiques & furniture fill our 6000 square feet of shopping, all on one floor. Lots of free parking right in front of our store. Several of our vendors re-purpose items and have fun, funky painted furniture so popular now. Also several booths are focused on items geared toward men...antique tools and Man-tiques!
Fantastic collection of antiques and antique furniture. Friendly staff, 3 large rooms of useful items and fond memories.
Display case and booth rental available. Furniture consignments taken. Give us a call! Stop by and see us!


A light, crisp starter of baby carrots and beetroot puree, finished with parmesan crisps and fresh lemon balm.
Grate the parmesan and place small mounds on a baking sheet covered in parchment paper.
Bake until browned, approx. 5 minutes.
Brown the carrots in a pan with olive oil and a pinch of salt.
layers. Add a few drops of olive oil, lemon balm leaves and serve with the beet leaves.


text:
\section*{Appendices}

\section{Discussion and Conclusion}

We suggest an easy substitution of polynomiality for exponentiality in several scenarios, showing it leads to conservative behavior regarding samples in the tail of the distribution.
For our polynomial softRmax, this behavior also leads to increased robustness against adversarial attacks.  We show that the robustness of softRmax also comes from an enlarged margin and link this to the inherent gradient regularization of softRmax, which demonstrates that its success does not stem from gradient obfuscation.  Our softRmax can be combined readily with many other adversarial defense strategies and it would be of interest to study their combined strength.

Given the type of conservative behavior polynomiality induces, it seems worthwhile to study its usage in OOD detection and other problems related to non-i.i.d. sampling, domain adaptation, etc.

As for softmax, good DNN weight initialization is important to avoid gradient vanishing for softRmax. Apart from that, considering the current level of understanding and the experimental evidence provided, we see no restrictions to its usage.  In conclusion: why not give softRmax a try?

\section{Experiments}\label{section: Experiments}
We present experimental results on conservativeness and robustness when using standard exponential terms and polynomial substitutes respectively.
First, we use covariate shift adaptation by importance weighting with outliers as an example to demonstrate that the conservativeness brought by polynomiality is necessary in an LDA-like setting. A next experiment shows that, even under attack, softRmax gives conservative posteriors.
We also perform standard adversarial attacks on public datasets
to compare the robustness of softmax and softRmax. To better understand the behavior of softRmax, we introduce a new, so-called, average-sample attack and the magnitude-margin ratio.
\subsection{Conservativeness}\label{section: covariate shift}
\subsubsection{Covariate Shift}
\begin{figure*}[h]
  \centering
  \begin{subfigure}{0.19\linewidth}
    \includegraphics[width = 1 \textwidth]{cov_st_pdf.pdf}

    \caption{Marginal distribution}
     \label{marginal}
  \end{subfigure}
  \begin{subfigure}{0.19\linewidth}
    \includegraphics[width = 1 \textwidth]{cov_target_function.pdf}
    \caption{Target function}
    \label{target_function}
  \end{subfigure}
  \begin{subfigure}{0.19\linewidth}
    \includegraphics[width = 1 \textwidth]{cov_only_source.pdf}
    \caption{Wihtout outliers}
    \label{no_noise}
  \end{subfigure}
  \begin{subfigure}{0.19\linewidth}
    \includegraphics[width = 1 \textwidth]{cov_Gaussian_source_noise.pdf}
    \caption{Gaussian}
    \label{noise_Gaussian}
  \end{subfigure}
  \begin{subfigure}{0.19\linewidth}
    \includegraphics[width = 1 \textwidth]{cov_t_source_noise.pdf}
    \caption{Student-t}
    \label{noise_t}
  \end{subfigure}
  \caption{(a) visualizes the marginal distribution of the source and target domain. The target function and the output value for the regression task are shown in (b). Figure (c) visualizes the fitted lines in the original scenario with Gaussian density estimation without outliers added. Figures (d) and (e) present the adaptation results of Gaussian density estimation and using Student's $t$-distribution with outliers separately. Lambda in the legend refers to the power $\lambda$ in Equation \eqref{eq: covariate weight}. Gaussian density estimation overfits to the outliers. }
\label{covariate_shift_regression}
\end{figure*}

Under covariate shift between a source domain $D_s$ and a target domain $D_t$, a fixed labelling function is assumed. We consider a standard weighting approach \cite{sugiyama2007covariate} to make the source domain distribution $p_{D_s}$ approximate the target domain distribution $p_{D_t}$:
\begin{equation}\label{eq: covariate weight}
    w_{cov} = \bigg(\frac{p_{D_t}(x)}{p_{D_s}(x)}\bigg)^\lambda.
\end{equation}
Here, $\lambda$ controls the strength of the weighting scheme. Similar to Section \ref{section: Bayes classifier}, Gaussian distributions $N(x|\mu; \sigma^2)$ with mean $\mu$ and variance $\sigma^2$ are estimated for $p_{D_s}$ and $p_{D_t}$. When outliers occur in the tail of the source distribution, extreme weights $w_{cov}$ are assigned to those outliers if the target distribution $p_{D_t}$ has a larger variance $\sigma^2_{D_t}$ than $\sigma^2_{D_s}$ of the source domain. This will lead to overfitting to these outliers only. With the use of a $t$-distribution, a weight of 1 is obtained, resulting in improved estimator behavior.

A domain adaptation regression setting is considered similar to the original work \cite{sugiyama2007covariate}.
The target function is $f(x) = \text{sinc}(x)$, shown in Figure \ref{target_function}. The source and the target densities are
$p_{D_s}(x) = N(x| 1.1,(1/2)^2)$ and $p_{D_t}(x) = N(x| 2.1, (10/17)^2)$, respectively.
We add noise $\epsilon_{s_i}$ to the target function to create the output values for the source domain $y_{s_i} = f(x_{s_i}) + \epsilon_{s_i}$ with $p(\epsilon_{s})=N(\epsilon_{s}|0,(1/4)^2).$ We set the source sample size to $n_s=150$ and the target sample size to $n_t=100$. To approximate the target domain, each source sample is assigned a weight $w_{cov}$ according to Eq. \eqref{eq: covariate weight}. We randomly sample 5 outliers in the range $[-5, -4]$ and add them to the source domain after density estimation. All parameters are estimated by maximum likelihood.

The sensitivities to outliers with Gaussian density estimation and using the $t$-distribution are compared in Figure \ref{covariate_shift_regression}. With Gaussian density estimation, noisy samples receive large weights due to the larger variance of the target domain, therefore the regressor overfits to the noisy samples when $\lambda$ is not 0. Student's $t$-distribution leads to small weights for the noisy samples because of its heavy tail.

\subsubsection{Conservative Prediction}\label{section: conservative prediction}

We show that conservativeness leads to further desirable behavior under adversarial attacks. For networks trained with softmax, adversarial attacks make the network misclassify samples with high confidence. But when a model with softRmax is attacked, the sample is misclassified but with low confidence due to the conservativeness.

We show the confidence of misclassified sample is low with softRmax by examining the posteriors of misclassified samples from the public dataset MNIST \cite{lecun1998mnist} under different levels of adversarial FGSM attacks. The network has four convolutional layers and one fully connected layer. We set a batchsize of $32$ and optimize the network by Adam with a learning rate of $1\text{e}-3$. We use the same architecture for the softmax and softRmax setting, with the only difference being the activation function after the final layer.

As shown in Figure \ref{fig: posteriors}, for the network trained with the standard softmax, posteriors on the predicted class of misclassified samples are high on average. Specifically, using softmax, under large scale attacks with $\epsilon=100$, all samples are misclassified with a posterior of 1. Due to the conservativeness in the tail and the soft posterior change, our softRmax leads to posteriors around random-guess level.
\begin{figure}[ht]
  \centering
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{max_posterior_softmax_eps0.pdf}

    \caption{Softmax no attack}
     \label{subfig: pos_softmax_eps0}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{max_posterior_marco_eps0.pdf}

    \caption{Ours no attack}
     \label{subfig: pos_marco_eps0}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{max_posterior_softmax_eps0.3.pdf}

    \caption{Softmax $\epsilon$=0.3}
     \label{subfig: pos_softmax_eps0.3}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{max_posterior_marco_eps0.3.pdf}

    \caption{Ours  $\epsilon$=0.3}
     \label{subfig: pos_marco_eps0.3}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{max_posterior_softmax_eps100.pdf}

    \caption{Softmax $\epsilon$=100}
     \label{subfig: pos_softmax_eps100}
  \end{subfigure}
   \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{max_posterior_marco_eps100.pdf}

    \caption{Ours  $\epsilon$=100}
     \label{subfig: pos_marco_eps100}
  \end{subfigure}
  \caption{Posteriors of predicted class for misclassified MNIST test samples under different levels of FGSM attacks. The legends give the number of misclassified samples. Misclassified samples in the setting of softRmax receive less confident prediction. Even under extreme attacks with $\epsilon=100$, our softRmax gives non-saturated posteriors at random-guess level.}
\label{fig: posteriors}
\end{figure}

\subsection{Robustness}

\subsubsection{Adversarial Defense}\label{subsubsection: public datasets}

We perform experiments on public datasets MNIST \cite{lecun1998mnist}, CIFAR10 \cite{Krizhevsky09learningmultiple}, and CIFAR100 with standard softmax and our softRmax.
The setting of MNIST is the same as in Section \ref{section: conservative prediction}. A randomly initialized VGG16 network is used for CIFAR10 classification. We optimize VGG16 by SGD with a learning rate of $5\text{e}-3$, batchsize $256$ and weight decay $5\text{e}-6$. For CIFAR100, we adopt ResNet50 pretrained on ImageNet and finetune it with Adam. We set the learning rate to be $1\text{e}-4$, batchsize $512$ and weight decay $5\text{e}-6$. Note that no extra data augmentation is used in any experiment. The only difference between the baseline with softmax and our approach is the activation function after the final fully connected layer. By simply substituting the softmax activation function with the polynomial softRmax activation function, the network develops strong adversarial defense ability (see Table \ref{tab: adversarial attack}). Without being combined with other approaches, the naive softRmax model can outperform state of the art adversarial defense approaches based on attention mechanism on CIFAR datasets \cite{agrawal2021impact}.
\begin{table}[h]
    \caption{Adversarial defense results. We compare networks with softmax and our softRmax activation under FGSM and BIM attacks ($T$=10). `Clean' refers to the classification accuracy on the testset without any attack. We consider binary classification for class 3 and 7 from MNIST, MNIST, CIFAR10, and CIFAR100 under different attack levels $\epsilon$. The results show a clear improvement of the robustness to adversarial attacks with softRmax.}
    \label{tab: adversarial attack}
    \centering
    \resizebox{0.47\textwidth}{!}{\begin{tabular}{lllllll}
        \toprule
        \multirow{2}{*}{Dataset} & \multirow{2}{*}{Method} &\multirow{2}{*}{Clean}
       &  \multicolumn{2}{c}{FGSM} &   \multicolumn{2}{c}{BIM} \\
        \cmidrule{4-7}
        {} & {} &{} &$\epsilon$=0.1 &$\epsilon$=0.3 &$\epsilon$=0.1 &$\epsilon$=0.3 \\
        \midrule
        \multirow{2}{*}{MNIST 3$\&$7} &softmax &99.75 &77.18&18.69  &71.64& 0.24 \\
        {}&ours &\textbf{99.95} &\textbf{95.88}& \textbf{88.71}  &\textbf{94.90}& \textbf{66.24}\\
        \cmidrule{2-7}
        \multirow{2}{*}{MNIST} &softmax &96.8 &48.3&0.41  &53.49&0.02 \\
        {}&ours &\textbf{97.78} &\textbf{75.55}&\textbf{49.30} &\textbf{69.73}&\textbf{33.94}\\
        \cmidrule{2-7}
        \multirow{2}{*}{CIFAR10} &softmax &\textbf{80.31} &17.62&13.95 &10.39&4.83 \\
        {}&ours &80.28 & \textbf{49.93}&\textbf{41.03} & \textbf{44.25}&\textbf{18.11}\\
        \cmidrule{2-7}
        \multirow{2}{*}{CIFAR100} &softmax &\textbf{61.43} &11.23&6.78 &1.94&0.05 \\
        {}&ours &61.04 & \textbf{19.31}&\textbf{11.04} & \textbf{9.06}&\textbf{2.16}\\
        \bottomrule
    \end{tabular}}
\end{table}

\subsubsection{Gradient Obfuscation}\label{section: exp_gradient_obfuscation}
As we noted in Section \ref{section: robustness from the tail}, different from gradient obfuscation, our loss landscape is not rough but simply has a different structure in the tail of the distribution. Leading a sample to the tail is different from blocking the attack by local minimum due to a rough loss landscape. The tail is the right direction to perturb the sample from the point of view of the gradient based attacks because the overall loss monotonically increases towards the tail.

Nevertheless, we also rule out the possibility of gradient obfuscation by performing the iterative BIM attack at very large iteration numbers. In fact, softRmax shows stronger robustness after the accuracy stabilises with increased iterations, as shown in Figure \ref{figure: BIM iterations}. The experiment in the next section shows that the black-box attack is a weaker attack than the white-box attack, which further diminishes the possibility that the softRmax robustness can be explained by gradient obfuscation.
\begin{figure}[h]
  \centering
  \begin{subfigure}{0.49\linewidth}
    \includegraphics[width = 1 \textwidth]{BIM_01.pdf}

    \caption{$\epsilon$=0.1}
     \label{subfigure: BIM_0.1}
  \end{subfigure}
  \begin{subfigure}{0.49\linewidth}
    \includegraphics[width = 1 \textwidth]{BIM_03.pdf}
    \caption{$\epsilon$=0.3}
    \label{subfigure: BIM_0.3}
  \end{subfigure}
  \caption{BIM attack on CIFAR10 with different iteration numbers. For both attack levels $\epsilon=0.1$ and $\epsilon=0.3$, the stabilized accuracy of softRmax is significantly higher than that of softmax.}
\label{figure: BIM iterations}
\end{figure}
\subsubsection{Robustness from Conservativeness}
Existing gradient-based attacks can only examine the robustness of a model as a whole but cannot show whether the robustness comes from the enlarged margin or from the conservativeness or both. To check the effect of the enlarged margin and the conservative tail on robustness, we propose a semi-black-box attack, coined the average-sample attack. It does not rely on the gradient but simply perturbs a sample to the direction of a selected target class based on a precomputed average, so the sample is guaranteed to not be pushed towards the tail. We precompute the average sample $\mathbf{Avg}_y= \frac{1}{n}\sum^n \mathbf{x}_{ny}$ for each class $y$. With $t$ the target class, the adversarial input $\mathbf{x}'$ then becomes
\begin{equation}
\mathbf{x}'_y = \mathbf{x}_y + \epsilon\text{sign}(\mathbf{Avg}_t - \mathbf{Avg}_y).
\end{equation}

In general, our average-sample attack should not be stronger than the gradient-based ones because the attacking direction found by the former attack is not optimized. It prevents the sample from going to the tail, so if it becomes a stronger attack for the softRmax, it indicates that the conservativeness in the tail indeed gives added robustness. Otherwise the gradient based white-box attack should be the worst attack for our softRmax model as well.

\begin{table}[hbt]
    \caption{Results of targeted attack on MNIST dataset. White refers to the targeted attack with the network available for generating adversarial samples. Black is the black-box version of the targeted attack, where a substitute network is first learned to approximate the decision boundary of the original model. Avg is our average-sample attack. The column Clean is the original per class accuracy of different models without any adversarial attack. The rest results are the accuracy of all the 10 classes under adversarial attacks. We highlight the \textbf{worst} performance of each model among all attacks.}
    \label{tab: targeted attack}
    \centering
    \resizebox{0.47\textwidth}{!}{\begin{tabular}{lllllllll}
        \toprule
        \multirow{2}{*}{Classes} & \multicolumn{2}{c}{Clean}
       & \multicolumn{2}{c}{White} &   \multicolumn{2}{c}{Black} & \multicolumn{2}{c}{Avg}\\
        \cmidrule{2-9}
        {} &softmax &Ours &softmax &Ours &softmax &Ours &softmax &Ours \\
        \midrule
        0 &98.88 &99.18 &\textbf{11.14}  &\textbf{53.2} &33.55 &74.3 &30.69 &58.13\\
        \cmidrule{2-9}
        1 &98.50 &99.21 &\textbf{15.05} &\textbf{46.73} &53.41 &60.73 &48.77 &63.44\\
        \cmidrule{2-9}
        2 &98.26 &98.16 &\textbf{10.50} &54.00 &25.93 &50.42 &16.53 &\textbf{42.26}\\
        \cmidrule{2-9}
        3 &96.34 &98.12 &\textbf{10.31} &51.73 &27.16 &58.99 &15.54 &\textbf{37.77}\\
        \cmidrule{2-9}
        4 &96.84 &97.35 &\textbf{13.28} &51.84 &37.21 &63.03 &26.97 &\textbf{47.94}\\
        \cmidrule{2-9}
        5 &97.87 &98.09 &\textbf{9.33} &52.24 &32.84 &64.52 &16.68 &\textbf{46.76}\\
        \cmidrule{2-9}
        6 &97.91 &98.64 &\textbf{12.00} &51.52 &35.46 &52.14 &24.43 &\textbf{44.56}\\
        \cmidrule{2-9}
        7 &96.60 &96.69 &\textbf{12.11} &52.88 &32.46 &60.10 &22.03 &\textbf{45.50}\\
        \cmidrule{2-9}
        8 &92.61 &96.61 &\textbf{9.36} &54.38 &24.96 &73.04 &18.90 &\textbf{41.93}\\
        \cmidrule{2-9}
        9 &94.05 &95.64 &\textbf{6.33} &51.72 &30.65 &68.58 &29.31 &\textbf{49.67}\\

        \bottomrule
    \end{tabular}}
\end{table}

To check whether the average-sample attack is a stronger attack on the softRmax model, we also perform a gradient based targeted attack in both the white-box setting and the black-box setting. The latter one checks whether gradient obfuscation happens. In the black-box attack \cite{papernot2017practical}, a substitute model that is used to generate adversarial samples is first learned to mimic the decision boundary of the original model. We show that the white-box attack is the strongest attack for softmax while our customized average-sample attack is more effective on the softRmax (see Table \ref{tab: targeted attack}). This indicates that the conservative tail of softRmax indeed leads to robustness. The fact that the black-box attack is weaker than the white-box attack eliminates the possibility that the weaker performance of average-sample attack is brought by gradient obfuscation. Also, even when the average-sample attack gives the lowest accuracy on softRmax, its performance is still significantly better than that of softmax.

\subsubsection{Robustness from Enlarged Margin}

We further demonstrate that the robustness of softRmax also comes from the enlarged margin. If all samples are pushed to the decision boundary instead of the tail, then the model with a larger margin is more robust. It is hard to measure the margin in the input space
so we use the prediction margin $M_z$, as specified in Equation \eqref{eq: prediction margin}, as alternative. If the perturbation in $\mathbf{x}$ can push the sample across the margin, then it means the corresponding change in $M_z$ is also larger than the original prediction margin $M_z$.
However, $M_z$ cannot be used to measure the margin directly due to different mappings from $\mathbf{x}$ to $\mathbf{z}$ of different models.  A larger $M_z$ does not imply a larger margin in the input space. So we introduce a new metric, the magnitude-margin ratio, to measure the change in $M_z$ caused by an attack with respect to the original prediction margin. If the change is larger than the original prediction margin for a sample, it indicates that this sample can be successfully attacked.

To derive the ratio for an $\mathbf{x}$, we assume the index for $\text{max}_{i\neq y}\{\mathbf{z}_i\}$ is $j$. We denote the gradient of $\mathbf{z}_y$ and $\mathbf{z}_j$ of the input $\mathbf{x}$ by $\mathbf{w}_y$ and $\mathbf{w}_j$, respectively. After adding a perturbation $\mathbf{\eta}$ in the input $\mathbf{x}$, $\mathbf{z}_y$ and $\mathbf{z}_j$ change to $\Tilde{\mathbf{z}_y}$ and $ \Tilde{\mathbf{z}_j}$, where $ \Tilde{\mathbf{z}_y} = \mathbf{w}_y^T\mathbf{x} + \mathbf{w}_y^T\mathbf{\eta}$. The new prediction margin $\Tilde{M_z} = \Tilde{\mathbf{z}_y} - \Tilde{\mathbf{z}_j}$. According to \cite{goodfellow2015explaining}, $\mathbf{w}^T\mathbf{\eta}$ can be approximated by the magnitude $m$ of gradients, the attacking level $\epsilon$ of $\mathbf{\eta}$, and the dimension $n$ of input as $\epsilon mn$ and so
\begin{equation}
   r =\frac{|\Tilde{M_z} - M_z|}{M_z}

    = \frac{|(\mathbf{w}_y^T - \mathbf{w}_j^T)\mathbf{\eta}|}{M_z}\approx \frac{\epsilon mn}{M_z}.
\end{equation}
Given the same input dimension $n$ and attacking level $\epsilon$, the simplified ratio $R = \tfrac{m}{M_z}$
can serve as the metric. A model with a distribution of lower ratio $R$ means that, with the same level of attack, it is harder to change the prediction margin $M_z$, which indicates a larger margin in the input space and a higher robustness of this model. Figure \ref{fig: ratio} shows that the model with softRmax has ratios $R$ lower than softmax has on MNIST, CIFAR10, and CIFAR100.

\begin{figure}[hbt]
  \centering
     \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{ori_l2_ratio.pdf}

    \caption{MNIST}
     \label{subfig: softmax_mnist}
    \end{subfigure}
    \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{marco_l2_ratio.pdf}

    \caption{MNIST}
     \label{subfig: marco_mnist}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{Softmax_cifar_ratio.pdf}

    \caption{CIFAR10}
     \label{subfig: softmax_cifar10}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{Ours_CIFAR_ratio.pdf}

    \caption{CIFAR10}
     \label{subfig: marco_cifar10}
    \end{subfigure}
    \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{Softmax_cifar100_ratio.pdf}

    \caption{CIFAR100}
     \label{subfig: softmax_cifar100}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{Ours_cifar100_ratio.pdf}

    \caption{CIFAR100}
     \label{subfig: marco_cifar100}
    \end{subfigure}
  \caption{Histogram of the magnitude-margin ratio $R$ of softmax and softRmax on all test data from MNIST, CIFAR10, and CIFAR100. The ratio of softRmax is significantly smaller, which indicates a higher robustness against adversarial attacks.}
\label{fig: ratio}
\end{figure}

\section{Introduction}\label{section: Introduction}
Models
that show some form of exponential behavior are ubiquitous in machine learning: from the Gaussian class conditional distribution in linear discriminant analysis (LDA) \cite{fisher1936use,hastie1996discriminant} to sigmoid activation for logistic regression \cite{kleinbaum2002logistic, menard2002applied}, and the softmax activation function in deep neural networks \cite{krizhevsky2012imagenet, simonyan2014very}.
Models with such use of exponentiality can, however, have unwanted behavior. We describe and illustrate such behavior, examine its reason, and propose a partial remedy by switching to models that behave polynomially.  Like \cite{hill1975simple, chen1978remark}, we consider the distribution tail and show that samples in the tails receive overconfident posterior predictions \cite{kristiadi2020being}. This renders the model sensitive to outliers and causes overfitting, especially in the case of distribution shift. Moreover, we link overconfident predictions to the lack of robustness against gradient based adversarial attacks.

A model should not be certain about a sample that deviates too much from the training data. Overconfident predictions on samples in the distribution tails should often be avoided, e.g.\ an atypical patient may otherwise be classified to be healthy or diseased with strong confidence.  We want what we call \emph{conservativeness}, which expresses the fact that we are uncertain.  Specifically, we define it to be random guess-level prediction for samples in the tail of the distribution and show that this can be achieved by moving from exponential to polynomial behavior both in LDA and logistic regression.
In addition, for logistic regression and deep learning, studies into the standard softmax activation have shown that it is not necessarily the best choice in many settings \cite{de2016exploration, kanai2018sigsoftmax, titsias2016one}. We propose a polynomial form of softmax posterior estimation that we coin \emph{softRmax}. For this, we exploit the connection between the standard softmax function and LDA \cite{bishop1995neural} and adopt a modified Cauchy distribution as the substitute for the (super)exponential Gaussian term.

Besides overconfident predictions, the use of exponentiality is also linked to vulnerability to adversarial attacks. Such attacks aim to cause malicious prediction changes by adding an unnoticeable perturbation to the original input. \emph{Robustness} is the ability to maintain performance under adversarial attacks \cite{carlini2019evaluating}.
We demonstrate that a higher robustness of neural networks can be obtained by simply substituting the standard softmax with our softRmax. We show that the robustness can be linked back to the conservativeness of softRmax and  inherent gradient regularization. The first factor, conservativeness, mainly brings robustness against gradient based attacks.
The second leads to an enlarged margin between samples and the decision boundary, thereby boosting robustness against attacks as well. The effectiveness of various strategies countering adversarial attacks can be attributed to gradient obfuscation \cite{finlay2021scaleable, athalye2018obfuscated}. We show that our inherent gradient regularization does not rely on such obfuscation.

We sketch the benefits of conservativeness under covariate shift \cite{sugiyama2007covariate,sugiyama2008direct, gretton2009covariate} and show it when a model is under attack. We verify the robustness of our polynomial substitutes empirically on toy and public datasets.
We further propose a semi-black-box attack, which we call an average-sample attack, to confirm that the robustness of our softRmax indeed comes from the above two factors.
We also introduce a scale-invariant metric, the magnitude-margin ratio, for comparing the robustness of different models under the same level of attack.

\section{Exponentiality vs Polynomiality}\label{section: Method}
We first demonstrate the presence of overconfident prediction in the distribution tail and sensitivity to adversarial attacks with classical LDA, logistic regression, and deep learning. We then replace the exponential terms in each scenario by polynomial ones and show that this substitution is a simple yet effective approach to deliver conservativeness and improved robustness. Notably, for the latter, no adversarial training or extra regularization is required
\subsection{Conservativeness}

Conservativeness is defined as estimating the posterior class probabilities $p(y_i|\mathbf{x})$ at random-guess level for $\mathbf{x}$ in the tail, away from the bulk of the data.
To study such tail behavior, we basically study $\mathbf{x}$ for which the norm grows indefinitely, i.e., $\|\mathbf{x}\|\to\infty$. Assuming $k$ classes and ignoring class priors, conservativeness comes down to the requirement that we can informally state as:
\begin{equation}\label{eq: conservative definition}
    \lim_{\|\mathbf{x}\|\to \infty}p(y_i|\mathbf{x}) \approx \frac{1}{k}.
\end{equation}

\subsubsection{LDA}\label{section: Bayes classifier}

We consider $k$-class classification using LDA.
We elaborate upon the link between the overconfident prediction and exponentiality.  Following Bayes' rule, the posterior of class $y_i$, under equal priors, is
\begin{equation}\label{eq:bayes}
    p(y_i|\mathbf{x}) = \frac{p(y_i)p(\mathbf{x}|y_i)}{\sum_k p(y_k)p(\mathbf{x}|y_k)} = \frac{p(\mathbf{x}|y_i)}{\sum_k p(\mathbf{x}|y_k)}.
\end{equation}
Consider $\mathbf{x}$ to be 1D for simplicity.
The class conditional distribution $p(x|y)$ is estimated by fitting a Gaussian  $N(x|\mu_k,\sigma^2)$ with $\mu_k$ and $\sigma^2$ being the mean and variance of class $k$.
When $x$ goes to $\pm$infinity, the posterior saturates to one-hot encoding due to the (faster than) exponential rate of decrease of the Gaussian distribution.  Specifically, we have
\begin{align}
& p(y_i|x) = \label{eq:other} \\
& \left( 1 + \sum_{k\neq i} \exp \left( -\frac{1}{2 \sigma^2}(2x(\mu_i-\mu_k)-\mu_i^2+\mu_k^2) \right) \right)^{-1}\nonumber
\end{align}
from which we see that $\lim_{x\to\pm\infty} p(y_i|x)=0$, unless $y_i$ is the mean closest to $x=\pm\infty$, in which case the posterior will be $1$.
This is also illustrated in Figure \ref{subfig:NB_Gaussian}.

\begin{figure}
  \centering
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{NB_Gaussian_LDA.pdf}

    \caption{Gaussian distribution.}
     \label{subfig:NB_Gaussian}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{NB_t_LDA.pdf}
    \caption{Student's $t$-distribution.}
    \label{subfig: NB_t}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width = 1 \textwidth]{LR_distance_pos_change_marco.pdf}
    \caption{Logistic regression.}
    \label{subfig: regression}
  \end{subfigure}
  \caption{LDA and logistic regression with exponential and polynomial assumptions. Posteriors $p(y_0|x)$ are compared. Subfigure \ref{subfig:NB_Gaussian} and \ref{subfig: NB_t} show the predicted posterior by LDA with class conditional Gaussian or $t$-distributions. Subfigure \ref{subfig: regression} compares the posterior of softmax and softRmax. LDA with Gaussian assumption and softmax with exponential functions show overconfident predictions in the distribution tails. Conservative prediction is achieved by substituting polynomial for exponential behavior.}
 \label{fig: conservativeness}
\end{figure}
\paragraph{Polynomial substitute.}
We propose to substitute the Gaussian distribution with the (noncentral) Student's $t$-distribution in the density estimation. Other distributions that fall of polynomially can be considered as long as the power of the leading terms are the same for all $k$ class conditional distributions. In this way, conservative posteriors with a behavior as in Equation \eqref{eq: conservative definition} are obtained.

The reason for this is that the limit of $x$ going to $\pm$infinity for Equation \eqref{eq:bayes} behaves rather different when the numerator and denominator contain polynomial instead of exponential terms.  For the former, convergence is controlled by the polynomial decay rate of the posteriors $p(x|y_k)$.  When equal, the limit posterior, assuming all priors equal, is $\tfrac{1}{k}$.

\paragraph{Example.}
We consider a binary classification task in 1D data. We assume a uniform distribution in the range $[-2, -1]$ for class $y_0$ and $[1, 2]$ for class $y_1$. With Gaussian distributions for the class conditional distributions $p(x|y_0)$ and $p(x|y_1)$---fitted using maximum likelihood, we get the change of posterior $p(x|y_0)$ w.r.t. input $\mathbf{x}$ as in Figure \ref{subfig:NB_Gaussian}.
When $x\to-\infty$, $p(y_0|x)=1$ and when $x\to \infty$ $p(y_1|x)=1$. Substituting the $t$-distribution for the Gaussian, as shown in Figure \ref{subfig: NB_t}, for samples that are in the bulk of the class conditional distribution, we still obtain a posterior $p(y_0|x)$ close to 1. But for samples in the tail, we find more conservative prediction where $p(y_0|x)$ and $p(y_1|x)$ are approximately $\tfrac{1}{2}$.

\subsubsection{Logistic Regression and Softmax}\label{section: method_logistic regression}
The softmax in neural network, as employed in the last layer to come to posterior estimates, works in the same way as multi-class logistic regression for classification tasks. Here, we consider a basic linear transformation $f(\mathbf{x}, \mathbf{w}) = \mathbf{w}^T\mathbf{x} + \mathbf{b} = \mathbf{z}$, though our analysis can be readily generalized to nonlinear neural networks.

With the standard softmax activation $\varsigma^S$, the embedding $
\mathbf{z}$ is mapped to a vector of posteriors with $p(y_i|\mathbf{x}) = \varsigma_i^S(\mathbf{z}) =e^{\mathbf{z}_i}/\sum_ke^{\mathbf{z}_k}$.  Equivalent to Equation \eqref{eq:other}, we have
\begin{equation}\label{eq: regression infinity}
\begin{split}
    &\varsigma_i^S(\mathbf{z})=\frac{\exp(\mathbf{w}_i^T\mathbf{x} + \mathbf{b}_i)}{\sum_k \exp(\mathbf{w}_k^T\mathbf{x} + \mathbf{b}_k)}= \\
    & \left( 1 + \sum_{k\neq i}\exp \left( (\mathbf{w}_k-\mathbf{w}_i)^T\mathbf{x} + (\mathbf{b}_k - \mathbf{b}_i) \right) \right)^{-1}.
\end{split}
\end{equation}
When $\|\mathbf{x}\|\to \infty$, if $(\mathbf{w}_k - \mathbf{w}_i)^T\mathbf{x}$ is negative for all $k, k \neq i$, then the posterior will be 1, otherwise it is 0.

We make the connection of softmax with LDA here. Let us position $k$ normal distributions with identity covariance, $N(\cdot|\mathbf{m},\mathbf{I})$, in $Z$.
Their means are the $k$ standard basis vector $\mathbf{e}_k$.
Based on these distributions---every single one of them representing one of the $k$ classes, we can map every $\mathbf{z} \in Z$ to a vector of posteriors $\varsigma^G(\mathbf{z})$, simply by setting
\begin{equation}\label{eq: fit_Gaussian_softmax_conditional}
    \varsigma_i^G(\mathbf{z}) := \frac{N(\mathbf{z}|\mathbf{e}_i,\mathbf{I})}{\sum_k N(\mathbf{z}|\mathbf{e}_k,\mathbf{I})}.
\end{equation}
This, in turn, can be directly related to the softmax $\varsigma^S$.  First, we realize that, for $\mathbf{z}$ fixed,
\begin{equation}
\begin{split}
&N(\mathbf{z}|\mathbf{e}_i,\mathbf{I}) \propto \exp(-\tfrac{1}{2} \|\mathbf{z}-\mathbf{e}_i\|^2)\label{eq: approximate_softmax_Gaussian_conditional}\\
\propto & \exp\left(-\frac{1}{2}\sum_{k} \mathbf{z}_k^2\right) \exp(\mathbf{z}_i) \exp(-\tfrac{1}{2}) \propto e^{\mathbf{z}_i}.
\end{split}
\end{equation}
From this, we immediately see that
\begin{equation}\label{eq: approximate_softmax}
    \varsigma_i^G(\mathbf{z}) = \frac{N(\mathbf{z}|\mathbf{e}_i,\mathbf{I})}{\sum_k N(\mathbf{z}|\mathbf{e}_k,\mathbf{I})} = \frac{e^{\mathbf{z}_i}}{\sum_k e^{\mathbf{z}_k}} = \varsigma_i^S(\mathbf{z}).
\end{equation}

\paragraph{Polynomial substitute.}
Inspired by the standard Cauchy distribution $p_C(x) = \frac{1}{\pi(1+x^2)}$---a specific $t$-distribution, we use a polynomial term with the power of $-2$ to substitute the Gaussian class conditional distribution $N(\mathbf{z}|\mathbf{e}_i,\mathbf{I})$ in Equation \eqref{eq: approximate_softmax}, which gives our \emph{softRmax} activation function $\varsigma^C$:
\begin{equation}\label{eq: our_softmax}
    \varsigma_i^C(\mathbf{z}) := \frac{\frac{1}{\|\mathbf{z}-\mathbf{e}_i\|^2}}{\sum_k \frac{1}{\|\mathbf{z}-\mathbf{e}_k\|^2}}.
\end{equation}
By adopting the polynomial function, the posterior becomes conservative, because
\begin{equation}\label{eq: regression posterior polynomial}
\begin{aligned}
p(y_i|\mathbf{x}) =  \varsigma_i^C(\mathbf{z})&= \frac{\|\mathbf{w}^T\mathbf{x} + \mathbf{b}-\mathbf{e}_i\|^{-2}}{\sum_k \|\mathbf{w}^T\mathbf{x} + \mathbf{b}-\mathbf{e}_k\|^{-2}}\\
&=\frac{1}{1 + \sum_{k\neq i} \left\|\frac{\mathbf{w}^T\mathbf{x} + \mathbf{b} - \mathbf{e}_i}{\mathbf{w}^T\mathbf{x }+\mathbf{b} -\mathbf{e}_k}\right\|^{2}}
\end{aligned}
\end{equation}
and the terms $\left\|\frac{\mathbf{w}^T\mathbf{x} + \mathbf{b} - \mathbf{e}_i}{\mathbf{w}^T\mathbf{x }+\mathbf{b} -\mathbf{e}_k}\right\|^{2}$ converge to 1 when $\|\mathbf{x}\|\to \infty$.
\paragraph{Example.}
We consider logistic regression for binary classification in 1D. Similar to the previous example, we assume a uniform distributions in the ranges $[-1,0]$ and $[1,2]$ for the two classes $y_0$ and $y_1$. The sigmoid/softmax function is substituted with the softRmax activation function from Equation \eqref{eq: our_softmax} to construct a conservative regressor. In Figure \ref{subfig: regression}, we see that the posterior $p(y_0|x)$ goes to $\tfrac{1}{2}$ on both ends.

\subsection{Robustness}\label{section: Robustness}

Next to softRmax being conservative, simply substituting the standard softmax with it in any probabilistic deep net also brings more adversarial robustness. We show that this comes from conservativeness in the tail and an inherent weight regularization that leads to an enlarged margin between samples and the decision boundary.
 \begin{figure*}[hbt]
  \centering
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_False_epoch_1.png}

    \caption{Softmax 1}
     \label{subfig: linear_softmax_epoch1}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_False_epoch_5.png}

    \caption{Softmax 5}
     \label{subfig: linear_softmax_epoch5}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_False_epoch_10.png}

    \caption{Softmax 10}
     \label{subfig: linear_softmax_epoch10}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_False_epoch_15.png}

    \caption{Softmax 15}
     \label{subfig: linear_softmax_epoch15}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_False_epoch_30.png}

    \caption{Softmax 30}
     \label{subfig: linear_softmax_epoch30}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_False_epoch_50.png}

    \caption{Softmax 50}
     \label{subfig: linear_softmax_epoch50}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_marco_epoch_1.png}

    \caption{Ours 1}
     \label{subfig: linear_marco_epoch1}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_marco_epoch_5.png}

    \caption{Ours 5}
     \label{subfig: linear_marco_epoch5}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_marco_epoch_10.png}

    \caption{Ours 10}
     \label{subfig: linear_marco_epoch10}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_marco_epoch_15.png}

    \caption{Ours 15}
     \label{subfig: linear_marco_epoch15}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_marco_epoch_30.png}

    \caption{Ours 30}
     \label{subfig: linear_marco_epoch30}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{linear_conservative_marco_epoch_50.png}

    \caption{Ours 50}
     \label{subfig: linear_marco_epoch50}
  \end{subfigure}
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_False_epoch_1.png}

    \caption{Softmax 1}
     \label{subfig: softmax_epoch1}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_False_epoch_10.png}

    \caption{Softmax 10}
     \label{subfig: softmax_epoch10}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_False_epoch_15.png}

    \caption{Softmax 15}
     \label{subfig: softmax_epoch15}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_False_epoch_20.png}

    \caption{Softmax 20}
     \label{subfig: softmax_epoch20}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_False_epoch_30_circle.png}

    \caption{Softmax 30}
     \label{subfig: softmax_epoch30}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_False_epoch_50.png}

    \caption{Softmax 50}
     \label{subfig: softmax_epoch50}
  \end{subfigure}

  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_marco_epoch_1.png}

    \caption{Ours 1}
     \label{subfig: marco_epoch1}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_marco_epoch_10.png}

    \caption{Ours 10}
     \label{subfig: marco_epoch10}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_marco_epoch_15.png}

    \caption{Ours 15}
     \label{subfig: marco_epoch15}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_marco_epoch_20.png}

    \caption{Ours 20}
     \label{subfig: marco_epoch20}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_marco_epoch_30_circle.png}

    \caption{Ours 30}
     \label{subfig: marco_epoch30}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.14\linewidth}
    \includegraphics[width = 1 \textwidth]{conservative_marco_epoch_50.png}

    \caption{Ours 50}
     \label{subfig: marco_epoch50}
  \end{subfigure}
  \caption{Margin change for linearly separable dataset and the moon dataset with softmax and our softRmax. Different colored points represent the two classes. The color bands show the posterior develops in the input space. The number in the title of each subfigure is the training epoch. With the standard softmax, the model makes the posterior change around the decision boundary sharp to minimize the loss. Due to the regularization of weights $w$ with softRmax, it is harder to minimize the loss by increasing the posterior fast at the decision boundary, which enables the model to find a larger margin.}
\label{fig: linear_softmax-epochs}
\end{figure*}

\subsubsection{Robustness from Conservativeness}\label{section: robustness from the tail}

Most gradient-based adversarial attacks try to maximize the overall loss \cite{goodfellow2015explaining} or minimize the loss of a target class \cite{papernot2016limitations}. For a properly converged network that employs the standard softmax, attacking a correctly classified sample pushes it away from the tail, as the overall loss would not increase moving towards it (and the target class loss would not decrease).
This is because the posterior of the correct class does not decrease towards the direction of the tail (see Figure \ref{subfig: regression}).
The loss landscape using softRmax is different due to the conservativeness in the tail, as also illustrated in Figure \ref{subfig: regression}.  For samples that are already positioned in the direction of the tail, an attack would actually push them even further into the tail.  This increases the overall loss or decrease the target class loss. A perturbation towards the tail does, however, not change the accuracy so the attack fails.
This leads a neural network using our softRmax to be more robust to gradient-based attacks. Note that this defense is different from gradient obfuscation because our loss landscape is not unnecessarily rough but simply has a different structure. This will be further elaborated in the corresponding experiment in Subsection \ref{section: exp_gradient_obfuscation}.

\subsubsection{Robustness from Enlarged Margin}
The other factor that contributes to the robustness of softRmax, is the enlarged margin. To illustrate, we again consider a simple linear mapping of the input: $\mathbf{z} = \mathbf{w}^T\mathbf{x} + \mathbf{b}$. The theory can be generalized to nonlinear mappings by substituting the weight $\mathbf{w}$ with the gradient $\nabla_{\mathbf{x}}\mathbf{z}$ in the following derivation. With the output of the activation function being the general $\varsigma$, the posterior gradient equals:
\begin{equation}\label{eq: chain}
    \frac{\partial \varsigma(\mathbf{z})}{\partial \mathbf{x}} = \frac{\partial \varsigma(\mathbf{z})}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \mathbf{x}} = \nabla_{\mathbf{z}}\varsigma \mathbf{w}.
\end{equation}

The network weights are optimized by minimizing a posterior based loss function, which means the posterior of the labeled class should be maximized. For a separable dataset, there are many possible decision boundaries that can be learned by the network. When a decision boundary is biased by some samples close to the decision boundary (like in Figures \ref{subfig: linear_softmax_epoch10} and \ref{subfig: softmax_epoch20}), the network generally has two options to further decrease the loss. It can either move the decision boundary to enlarge the classifier margin, or make the transient of posterior steeper at the decision boundary so posteriors of correctly classified samples saturate to 1 quicker. Both of the two approaches decrease the loss.

We observe that with softmax being the activation, the network tends to increase the posterior by making the posterior transient steep (as shown in Figures \ref{subfig: linear_softmax_epoch50} and \ref{subfig: softmax_epoch30}). We believe that this is because the magnitude $m$ of $\mathbf{w}$ is not regularized, so the network can simply increase $m$ during the optimization process to increase the posterior gradient in Equation \eqref{eq: chain}.  This leads to the fast transient of the posterior around the decision boundary. A problem in the optimization is that the posteriors of samples will quickly saturate to 1 and do not contribute to gradient updates anymore. If a decision boundary is biased like in Figure \ref{subfig: linear_softmax_epoch10}, it hardly changes in subsequent epochs. Such decision boundary correctly separates all data, but is more vulnerable to adversarial attacks because the classifier margin is not maximized.

Different from softmax, softRmax optimizes the loss in the other way: by enlarging the margin. It maps $\mathbf{x}$ to $\mathbf{z}$ around the $k$th row of the identity matrix $\mathbf{e}_k$ for class $k$, so $\|\mathbf{z}\| \approx 1$. Correspondingly, the magnitude of weight $\mathbf{w}$ is inherently regularized by $\|\mathbf{w}^T\mathbf{x}+\mathbf{b}\| \approx 1$.
This avoids increasing the weights to values that can lead to a sharp decision boundary and leaves just one of the two above-mentioned optimization options to decrease the loss, i.e., enlarging the margin (see Figs. \ref{subfig: linear_marco_epoch50} and \ref{subfig: marco_epoch30}), resulting in increased robustness.

\section{Background Material and Related Methods}\label{section: related work}

Adversarial attacks are used for robustness evaluation in our work. They are categorized into white-box and black-box attacks, depending on whether the network is available or not \cite{papernot2016towards}. Black-box attacks do not need the network architecture and usually involve the training of a substitute network that mimics the decision boundary of the target network \cite{papernot2017practical}. A gradient-based adversarial attack is a typical white box attack \cite{goodfellow2015explaining, kurakin2016physical}. It aims to find the perturbation direction that can lead to the fastest change in the prediction.

FGSM \cite{goodfellow2015explaining} is a simple yet effective approach where a small perturbation $\mathbf{\eta}$ is added to the input $\mathbf{x}$ to increase the overall loss. The perturbation $\mathbf{\eta}$ is $\epsilon$ multiplied by the sign of the loss gradient $\nabla_xJ(\mathbf{w}, \mathbf{x}, y)$.  The perturbed input becomes:
\begin{equation}\label{eq: FGSM}
    \mathbf{x}' = \mathbf{x} + \epsilon \,  \text{sign}(\nabla_xJ(\mathbf{w}, \mathbf{x}, y)).
\end{equation}
Similarly, a gradient-based target attack \cite{papernot2016limitations} aims to perturb the sample to a target class $y_t$ by decreasing the loss that corresponds to the target class:
\begin{equation}
    \mathbf{x}' = \mathbf{x} - \epsilon \, \text{sign}(\nabla_xJ(\mathbf{w}, \mathbf{x}, y_t)).
\end{equation}
BIM \cite{kurakin2016physical} performs the attack iteratively in $T$ steps. With the same attacking scale $\epsilon$, BIM applies the attack at the scale of $\alpha = \epsilon/T$ in each step to form an attacked input $\mathbf{x}'_t$ at step $t$:
\begin{equation}\label{eq: BIM}
    \mathbf{x}'_{t+1} = \mathbf{x}'_t + \alpha \, \text{sign}(\nabla_xJ(\mathbf{w}, \mathbf{x}, y)).
\end{equation}

We need the notion of a prediction margin $M_z$ \cite{tsuzuku2018lipschitz} to measure the robustness to adversaries, which has an indirect link to the classical (geometrical) margin in the input space \cite{ScholkopfS02}. Our work uses it to evaluate the margin and the robustness of our method. For this, we consider a mapping from the input $\mathbf{x}$ to the latent or representation space: $\mathbf{z} = f(\mathbf{x}, \mathbf{w})$, with $\mathbf{z}\in \mathbb{R}^{k}$ and $\mathbf{z}_i$ the output of the final layer corresponding to class $i\in \{0,1,\ldots,k\}$. Assuming a sample $\mathbf{x}$ is correctly classified to its class $y$, $\mathbf{z}_y$ takes on the maximum value in $\mathbf{z}$. The prediction margin is defined as the distance between $\mathbf{z}_y$ and the second largest value in $\mathbf{z}$:
\begin{equation}\label{eq: prediction margin}
    M_z := z_y - \max_{i\neq y}\{z_i\}.
\end{equation}

Adversarial defenses for deep learning have been achieved by adversarial training \cite{kurakin2016adversarial}, distillation \cite{ba2014deep,papernot2016distillation}, constructing a maximum margin in the latent space \cite{pang2018max, pang2019rethinking, wan2018rethinking, hess2020softmax} and gradient regularization \cite{drucker1992improving,ross2018improving,hess2020softmax,tsuzuku2018lipschitz,nayebi2017biologically,goroshin2013saturating}. Explanations for gradient regularization approaches are heuristic and their successes often hinge on gradient obfuscation \cite{finlay2021scaleable}. The latter refers to an unnecessarily rough loss landscape that hinders gradient-based adversarial attacks, which get readily stuck in the local minima of the roughened loss. It should be noted, however, that this approach does not solve the problem of adversarial attacks inherently \cite{athalye2018obfuscated}. Increasing the iteration number in BIM attacks \cite{kurakin2016physical} and using black-box attacks are standard to detect gradient obfuscation. We use both in our work to show that our approach does not rely on gradient obfuscation.

Covariate shift is a specific problem within domain adaptation. Domain adaptation refers to the scenario where the training data and the test data are not i.i.d. \cite{daume2009frustratingly, kouw2019review}. The training and test data are referred to as the source domain and the target domain, respectively. One standard solution of this problem is to approximate the target domain by assigning the source samples weights determined by the source and target distribution. In the original work \cite{sugiyama2007covariate}, these are estimated using Gaussian distributions.  With this approach, adding very few samples in the tail of the source distribution can lead to considerable overfitting to the added outliers. We illustrate that, if a polynomial $t$-distribution instead of Gaussian distribution is adopted in the procedure of density estimation, the influence of outliers is limited.

redpajama_set_name: RedPajamaArXiv


Mix and combine all ingredients.
Pour into non-stick skillet and when you shake the pan and the pancake moves, it's time to flip.
Brown on both sides, and enjoy!
(I also add about 1/4 cup of sugar-free maple syrup).


text:
Q: How to group data already group in MongoDB I have a MongoDB query based in Objects as "Ticket" structure, each ticket usually contains Tasks Object and each of them have an Owner, if the Ticket is Open, the associate Owner is "OpnPrps.CurAgtNme", by other hand if the ticket is Closed, the associate Owner is "Nms.CloAgt".
This is how my database (JSON) look like:
{
    "result" : [
        {
            "_id" : NumberLong(3131032306336),
            "TicId" : 1147552,
            "OrgId" : 729,
            "Sts" : "Closed",
            "CrtDat" : ISODate("2015-04-23T18:50:46.000Z"),
            "CloDat" : ISODate("2015-04-23T19:46:26.000Z"),
            "ShtDes" : "Copy of Employment agreement",
            "Des" : "EE wants a copy of employment agreement. address was verified.",
            "DesSum" : "EE wants a copy of employment agreement. address was verified.",
            "Sol" : "ISIGHT TICKET NUMBER:

US-04-15-01109
",
            "CrtAgtId" : 20444,
            "CloAgtId" : 20149,
            "CrtApp" : null,
            "IsInt" : false,
            "HasPrtTsk" : null,
            "PexBodId" : "",
            "PayGrp" : "",
            "RvwDat" : null,
            "RclDat" : null,
            "IsDynDueDatDef" : false,
            "OlaDat" : ISODate("2015-04-25T00:50:50.000Z"),
            "SlaDat" : ISODate("2015-04-25T00:50:50.000Z"),
            "DynDueDatElaDat" : null,
            "ReaTim" : "00:00:00",
            "LstUpd" : ISODate("2015-04-23T19:46:26.000Z"),
            "VrsOnl" : 2,
            "VrsArc" : 1,
            "OpnPrps" : null,
"Nms" : {
                "Cmp" : "Organization1 US",
                "Org" : "Organization1",
                "Srv" : "Policies",
                "SrvGrp" : "17. Workforce Administration (NGR)",
                "Wkg" : "WORKGROUP1",
                "Pri" : "",
                "Tir" : "T1",
                "Src" : "Call",
                "CrtAgt" : "Arun ",
                "CloAgt" : "Felicia"
            },
"Olas" : {
                "_id" : 2,
                "EntNam" : "ENTITY",
                "DueDat" : ISODate("2015-04-25T00:50:50.000Z"),
                "AmbDat" : ISODate("2015-04-24T20:50:49.000Z"),
                "DueDatElaDat" : null,
                "AmbDatElaDat" : null,
                "SlaDuration" : 18,
                "TotTim" : NumberLong(0),
                "TotTimPndEnt" : NumberLong(0),
                "TotTimPndEmp" : NumberLong(0),
                "RclInSla" : false
            },
"Tsks" : {
                "_id" : 1,
                "Typ" : "Planned",
                "CrtDat" : ISODate("2015-04-23T18:50:46.000Z"),
                "CloDat" : null,
                "LstUpd" : ISODate("2015-04-23T18:50:46.000Z"),
                "DueDat" : ISODate("2015-04-25T00:50:50.000Z"),
                "TimCplTsk" : 1080,
                "DueDatEla" : false,
                "AgtOwnId" : null,
                "WkgSklId" : 45387,
                "EntId" : 2,
                "Sts" : "Open",
                "PrdTskId" : 201,
                "Ttl" : "Provide Navigational Assistance",
                "Des" : "Provide Navigational Assistance",
                "SrvSklId" : 45792,
                "PriSklId" : null,
                "TotTim" : 0,
                "PtnId" : null,
                "PtnTic" : null,
                "DepTskId" : null,
                "IsAct" : true,
                "IsMndOnCloTic" : false,
                "Nms" : {
                    "AgtOwn" : null,
                    "Wkg" : "CM_T1",
                    "Ent" : "ENTITY",
                    "SrvSkl" : "Policies",
                    "PriSkl" : null,
                    "Ptn" : null,
                    "Frm" : ""
                },
                "AscTicItm" : null,
                "FrmId" : null,
                "Flds" : []
            },

And the query I'm using to group the data looks like:
db.tickets.aggregate([
    {$match:{
        'Nms.Org': 'Organization1',
        'Nms.Cmp':'Company',
        'Nms.Wkg':'Workgroup'

    }},
    {$project:{
        _id:0,
        'Tsks._id':1,
        'Tsks.Sts':1,
        'Tsks.DueDat':1,
        'Sts':1,
        'Nms.Org':1,
        'Nms.Cmp':1,
        'Nms.Wkg':1,
        'Nms.CloAgt':1,
        'OpnPrps.CurAgtNme':1,
        'OpnPrps.CurEntNme':1,
        'Olas.EntNam':1
        }},

    {$unwind : "$Olas" },
    {$unwind : "$Tsks" },
    {$match:{$and:[{
        'Tsks.DueDat': {$ne: null},
        'Olas.EntNam': 'Entity',
        'Tsks._id':{$gt:0}
        }]}},

        {$group:
    {_id:{
                Org:'$Nms.Org',
                Cmp:'$Nms.Cmp',
                Wkg:'$Nms.Wkg',
                CurEntNme:'$Olas.EntNam',
                CurTskId: '$Tsks._id',
                DueDate:'$Tsks.DueDat',
                Owner1:'$OpnPrps.CurAgtNme',
                Owner2:'$Nms.CloAgt'
                },
                All: {$sum: { $cond:
                             [
                                {   $eq: [ '$Tsks.DueDat' , null ] } ,0,1
                             ]}}

            },

        }
    ])

With this query I'm able to get every ticket for every owner (even if they're open or closed) and their associate tasks, my problem is that I want to group all the tickets without taking in consideration the status, something like a second group after the results I have now, see below:
The table I get with Jasper Studio with this query looks like:
Owner       Inventory
---------   --------------
Noemi          1                   Owner1:Noemi | Owner2:null
Carl           2                   Owner1:null | Owner2:Carl
Darla          2                   Owner1:Darla| Owner2:null
Carl           1                   Owner1:Carl| Owner2:null
Paola          2                   Owner1:null| Owner2:Paola
Noemi          2                   Owner1:null | Owner2:Noemi

As you can see, I'm getting repeated values due to the different field from each ticket. The table I'm looking for should be like this:
 Owner       Inventory
 ---------   --------------
 Noemi          3                   Owner1:Noemi | Owner2:Noemi
 Carl           3                   Owner1:Carl | Owner2:Carl
 Darla          2                   Owner1:Darla| Owner2:null
 Paola          2                   Owner1:null| Owner2:Paola

So, my problem is that I can't find the way to group these results again to obtain the second table.

redpajama_set_name: RedPajamaStackExchange


text:
Q: iOS memory usage variable? I am developing an iPhone game with cocos2d-iphone.
I am particularly interested in how much memory is CCSpriteFrameCache "holding" at the moment. I am wondering - is there a way to know that? Without using any Xcode tools?
Perhaps there is a variable that will already let me know an estimate memory consumption value on my app?

A: Generally speaking the problem you are posing is not easy to solve.
In the case of CCSpriteFrameCache, since this class contains a pointer to an NSMutableDictionary of sprite frames, which are textures, you could iterate the dictionary and accumulating the texture dimensions (multiplied by the size of each pixel).
Another approach would be converting the dictionary into NSData like this:
NSData * data = [NSPropertyListSerialization dataFromPropertyList:spriteFrameDictionary
    format:NSPropertyListBinaryFormat_v1_0 errorDescription:NULL];
NSLog(@"size: %d", [data length]);

but this would require you to implement the NSCoding protocol for the CCSpriteFrame class.
About accumulating the textures size, you can multiply width by height by pixel size; the pixel size depends on the pixel format: RGBA8888 is 32 bits, RGB565 is 16 bits; also you have to take into account that open gl textures have only sizes that are power of 2: 256x256, 512x512, 1024x512 etc.

A: Actually if you are concerned about memory consumption of your textures, they are stored by CCTextureCache.  There is a CCTextureCache (Debug) method called dumpCachedTextureInfo method in there. I have not tried it myself, but here :
@implementation CCTextureCache (Debug)

-(void) dumpCachedTextureInfo
{
NSUInteger count = 0;
NSUInteger totalBytes = 0;
for (NSString* texKey in textures_) {
    CCTexture2D* tex = [textures_ objectForKey:texKey];
    NSUInteger bpp = [tex bitsPerPixelForFormat];
    // Each texture takes up width * height * bytesPerPixel bytes.
    NSUInteger bytes = tex.pixelsWide * tex.pixelsHigh * bpp / 8;
    totalBytes += bytes;
    count++;
    CCLOG( @"cocos2d: \"%@\" rc=%lu id=%lu %lu x %lu @ %ld bpp => %lu KB",
          texKey,
          (long)[tex retainCount],
          (long)tex.name,
          (long)tex.pixelsWide,
          (long)tex.pixelsHigh,
          (long)bpp,
          (long)bytes / 1024 );
}
CCLOG( @"cocos2d: CCTextureCache dumpDebugInfo: %ld textures, for %lu KB (%.2f MB)", (long)count, (long)totalBytes / 1024, totalBytes / (1024.0f*1024.0f));
}

You want to calculate per texture the bit format, since it is possible to store different texture formats in the cache, depending on your current needs. It will give you (last line) the summary of contents, including total memory consumed.


text:
Teaching Evolution and the Nature of Science
The Gerald W. Lynch Theater at John Jay College
This symposium will address the nature of scientific inquiry and the teaching of evolution as fundamental to modern biology. Aims of this symposium are to broaden the audience's understanding of the basic tenets of the concept of evolution, to illustrate the central role evolution plays in modern science, to elaborate on the investigative nature of science, and to recognize approaches to the teaching of science that reflect non-scientific propositions.
The audience for this event is high school science teachers, college faculties who teach science classes or train future science teachers, and state and local officials responsible for education policy. The symposium will also be of interest to others with an interest in science, education and evolution.
The primary goals of this symposium are to (1) address the need to respond to the increasingly effective challenge of intelligent design (ID) advocates in science classes; (2) provide science educators with the tools, both rhetorical and scientific, that can help them deal with the current issues in the context of the day-to-day delivery of science education; (3) enable science teachers to develop skills of scientific inquiry among their students in order to train both scientifically literate citizens and future generations of scientists and engineers; and (4) help science education meet an obligation in ensuring students understand that evidence is a necessary, not optional, component of the scientific process.
Hessy L. Taft (St. John's University)
Claudia M. Toback (Science Council of New York City)
Sydel Silverman (Graduate Center, City University of New York)
Lucie W. Saunders (Lehman College)
Federica Raia (City College of New York)

redpajama_set_name: RedPajamaCommonCrawl


text:
Center for Childhood Cancer Research
Maris Laboratory
John M. Maris
Nikolaos G. Sgourakis
CAR T Cells
STAT Wunderkind "Blazing New Trails:" Q&A With Mark Yarmarkovich, PhD
Mark Yarmarkovich, PhD, is a 2021 STAT Wunderkind.
shafere1 [at] chop.edu (By Emily Shafer)
Editor's note: A hearty congratulations to Mark Yarmarkovich, PhD, for being named a 2021 STAT Wunderkind! Each year, STAT, a news site that reports on health, medicine, and the life sciences, names its STAT Wunderkinds, the next generation of scientific superstars, who are selected from hundreds of nominations across North America. The Wunderkinds are "on the cusp of launching their careers, but not yet fully independent," and "blazing new trails as they attempt to answer some of the biggest questions in science and medicine," according to STAT. Dr. Yarmarkovich is a senior scientist in the Maris Laboratory, and his research focuses on CAR T cells and their application in the treatment of rare and aggressive cancers. We asked Dr. Yarmarkovich to discuss his research journey and his future research goals.
How did you find your way to studying CAR T therapy? What interests you about the field?
The path that led me to CAR T cells starts with my early experiences as an undergraduate when I first learned about engineering antibodies for therapeutic purposes, particularly against cancer. I was on track to enter medical school to become an oncologist, but I was particularly drawn to this form of therapy as antibodies had already evolved to solve the issues of precision targeting that is essential to cancer therapeutics.
I decided to stray from my plans of becoming an oncologist, as I was fascinated by this field and thought this was an area where I could have impact. I was able to get an internship at Genentech, where antibody engineering was pioneered, and decided to join a rotational program focused on developing the next generation of antibody therapeutics.
When I saw the work being done at Penn/CHOP with CAR T cells, I was compelled by the idea of combining the specificity of antibodies with the potency of T cells. My interest in the field is rooted in a fascination with the immune system and how to deploy the principles of immunity safely and effectively against cancer. The work I do challenges me and takes advantage of skills I have built across various disciplines that I am interested in. Ultimately, what drives my work is my belief these therapies have great potential to help cancer patients.
What do you think is the most important research finding you've made in your career thus far, and why?
I am most excited about the methods that we are developing for targeting new classes of tumor targets (non-mutated intracellular cancer drivers) that I think can significantly expand the applications of immunotherapy to a much larger set of cancers. Also, the technology to target these cancer drivers across patients with different HLA types will open the ability to expand the groups of patients eligible for these therapies, including ethnic groups who are currently underserved.
What in your current research are you most excited about, and why?
I'm very excited about moving our current CARs into clinical trials, expanding this class of therapies into other lethal cancers, and developing new technologies that will result in better immunotherapies.
What are your long-term research goals?
My long-term goal is to help enable the treatment of any cancer patient with immunotherapy. I'm aiming to help develop a suite of technologies that will allow the identification of tumor-specific targets in any tumor and processes for engineering target-specific immunotherapies specifically suited for each patient's cancer.
How has CHOP supported your research/career path? What makes it such an ideal place to conduct such groundbreaking research?
It's been an absolute privilege to work with John Maris, MD, who has been such a supportive mentor through all stages of this project. Dr. Maris allowed me to get this project off the ground in his lab when it was just a high-risk idea and supported the expansion of this work in so many directions since then.
CHOP has developed an excellent certificate program in Cell and Gene Therapy that I completed last year and is investing in translating our work by funding the pathway of our CARs into clinical trials through the Cell and Gene Therapy Collaborative Acceleration Grant. CHOP also is investing in more upstream research with the recent recruitment of Nik Sgourakis, PhD, whose work is highly complementary to ours in characterizing the molecular basis of immune interactions that we hope will inform the development of future therapies. I think that the investments CHOP is making in these efforts will ultimately yield better therapies for more patients in the years to come.


text:
GH Collective – Quarantine Special
Posted by Ryan Perrow | May 21, 2020 | Editorials | 0 |
"GET TO THE SH*TTER!"
So, the nation is still within a semi-lockdown and we're right in the middle of our own individual quarantine as COVID-19 continues ravages the globe, but we gamers are built for situations like this. Stay inside for hours on end and avoiding physical contact with people? Oh, so, act normally? Before we continue to propagate our rather outdated stereotypes further, I'll explain what we're are doing here. We here at the Game Hype towers are knee deep in self isolation at the minute, and as we love our games, we all thought you (yes you) dear reader, might want to know what we are playing as part of our civic duty to defeat this virus (and avoid developing cabin fever).
Ryan Perrow (@NBFlying)
What I'm playing: Call of Duty Warzone and Final Fantasy Type – 0 HD
It's a mixed bag keeping me from going completely stir crazy, the fast frantic action of COD: Warzone balanced by the steady level griding with the characters of Class 0. Battle Royale games may not be as hot as they were this time last year but I must admit that I think Warzone is the best of them, and trust me, giving the bloated Call of Duty any credit since it the 360 days stings. I'm happy to report that my buddies and I managed to win a round, even if one of us was dead when it happened. The bounty contracts in the middle of the battle royale add a huge amount of drama to the already tense battle royale genre. As for Final Fantasy Type-0 HD, it's a Final Fantasy game, the story is great and has a much more mature slant than most games in the franchise, just take a look at the opening FMV. Originally released on PSP and released as an HD remaster a few years ago Type-0 tends to be overlooked but if you want a decent length experience and enjoyed Final Fantasy XV both games having been directed by Hajime Tabata.
Final Fantasy Type – 0 HD
Kyle Doherty (@Antigenetic92)
What I'm playing: GWENT (Android) and Fallout: New Vegas (PC)
So after binge watching Twin Peaks for the third time in the last 6 months (that opening music is the most calming melody on the planet, fight me) I was stuck for something to pass the time durng the nationwide quarantine that is currently afoot, and ended up landing on CD Projekt Red's lastest release for Android: GWENT. Behaving nothing like the original card game from within their stellar 2015 RPG: The Witcher 3: Wild Hunt, GWENT is silimar in playstyle to the likes of The Elder Scrolls: Legends and Hearthstone, in that it's foremost a tactical card game, with a strong emphasis on strategy and deck building. While the former is reserved primarily for those who really sink themselves into the game full time, it acts as a decent distration to pass the time of the current commute from the office (currently my bedroom) to my toilet, which i'll sit on for close to an hour (sometimes two) trying to perfect my Northern Realms deck (because the Nilfgaard deck is for absolute scrublords). Most of my time spent since Bojo's lockdown however has been spent on a fresh playthrough of the criminally underrated Fallout: New Vegas from Obsidian. While Bethesda have gone from being a top-tier studio to one that people wince upon hearing their name in recent years, it's always made me smile how Obsidian Entertainment (made up somewhat of former staff from the now defunct Black Isle Studios) were able to take what Bethesda did with Fallout 3 (which is still an amazing game) and create a Fallout title that was more akin to the first two in terms of thematic maturity and lore, and is downright better than Fallout 3, 4 and 76 in a myriad of different ways. With Obsidian's success with The Outer Worlds recently, I think Bethesda needs to quit while they're ahead and let the OG boys have another crack at making another Fallout game, because lets be real (intellectual property ownership aside) the guys and gals over at Obsidian do it better.
"Ah Paciencia, my sweet Paciencia..."
Aaron Moger (@aaronvaanmoger)
What I'm Playing: World of Warcraft and Atelier Sophie: Alchemist of the Mysterious Book
There's no better way to spend time in quarantine than taking advantage of experience boosts. That's exactly what Blizzard has done with World of Warcraft, giving everyone a 100% boost to EXP and giving me time to level some other classes which I've always wanted to play. On top of this, the gold earned on top of this is enough to keep my subscription going for a bit and I may even be able to get Shadowlands. I am very much enjoying my time back in WoW after a hiatus. When I'm not at my PC, my game of choice currently is Atelier Sophie Alchemist of the Mysterious Book on the PS Vita. This game is perfect for isolation, the game features quality JRPG battles blended with an addictive Item Synthesis system that is easy to get lost in. On top of this, it is one of the few Atelier games that doesn't require you to meet deadlines. The best part is I'm playing it on Vita so if I get bored with the Living Room I can go and play it in the Kitchen no problem.
Atelier Sophie: Alchemist of the Mysterious Book
Adam O'Connell (@mynameisfrog_gh)
What I'm Playing: Devil May Cry 5 and Yakuza Kiwami
Stuck in Isolation? Love smacking demons around? How about getting revenge on your former Oath Brother? Well, if you're like me then I have the perfect pair of games for you to keep your mind off this trying time. Firstly, Devil May Cry 5. A franchise I haven't given the time of day since 2008′ Devil May Cry 4 and refamiliarising myself with this legendary hack and slash/run series as well as diving back into the rich lore that is both intricate yet uncomplicated, and in my humble opinion the series has reached its apex with its savage 5th entry. In recent years, Capcom has made a habit of creating some beautifully rendered worlds complete with some absolutely magnificent looking character models (this is evident with the revival of the Resident Evil franchise) and none have looked better in the DMC series than DMC5. With amazing detail to its concrete jungle and visceral artistry along with lightning-fast combat from three playable characters all unique in their own way, each slash and bullet fired feels like your enemies are being hit by a freight train. That, alongside the fantastic plot that joins the rest of the timeline seamlessly, with the return of familiar characters and new faces alike. Also if you find yourself somehow NOT satisfied and with space to spare of your hardware, give Yakuza Kiwami your attention, a full remake of the franchise's first game released in 2005 on the PS2. Rebuilt to match the overall awesome look of more recent entries and changes the gameplay style to match the fantastic gameplay style of 2015's prequel Yakuza 0. The original Yakuza has been reworked profoundly making walking the streets of Kamuroucho as entertaining as the hard-hitting fights, substories, karaoke, and the antics of the Mad Dog Mjima! All with a gripping story. For newcomers, this along with Yakuza 0 is a great place to get acquainted with the trials and tribulations of Kazuma Kiryu.
Bettsy (@JohnBGameHype)
What I'm playing: Sonic Mania, Gears 5, NieR, WWE 2K20.
First off, I hope all of you are safe and healthy. These are trying times for us all but by working together, Covid 19 will be beaten. Much like how Covid 19 has united communities, gaming also provides this incredible escape from the harsh reality of the pandemic, whilst uniting players all over the world with the passion and joy that gaming evokes from us all. Which is why blasting locust into two halves, to f5'ing noobs on WWE2k20, or chasing after emeralds and reliving memories of yesteryear on Sonic Mania, have been immensly fun. Three very different games all boasting unique qualities. With Gears 5 you have the superb story of Kait and James Fenix, a tale of two cogs living in the shadow of their parents, whilst fighting their own personal demons and looking for answers. With beautiful visuals, high production values and some kick ass gun fights. Gears 5 is one hell of a joy ride. Sonic Mania is one massive trip down the lane of nostalgia and delivers the best 2D sonic game to date. With fantastic attention to detail, compelling boss fights, creative, original and fun stages to enjoy, plus a killer soundtrack. Not only is Sonic Mania raising the bar for future outings in the world of Dr Eggman and the choas emeralds, in my opinion the public have spoken and quite clearly the popularity of the game offers an obvious indication towards what direction SEGA should take the little blue creature and his friends. (We all know they'll likely continue down the bug ridden 3d outings sadly, THEY SUCK). I can't comment too much On NieR as i'm like 1 hour into the game, but the initial signs are promising that it's a title i'll enjoy. As for WWE 2k20 despite it's poor reviews across the board (I scored it poorly), a few patches later and we at last have a game in a playable state. Yes the graphics and production values leave much to be desired. Yet strangly enough, there's something immensely satisfying about taunting your opponent (DELETE, DELETE, DELETE) before hitting the twist of fate off the top rope, and then adding further insult to injury by prancing around the ring with a ridiculous dance or sexual taunt before pinning your defeated opponent. So there you have my two cents. Also, I just want to thank all our readers for your continued support, I hope you stay well and enjoy the collab article.
Kyle Byrne (@StealthFrag)
What I'm playing: XCOM: Chimera Squad, Hyperlight Drifter, Pokemon Sword.
The Covid-19 pandemic has very much put the world on hold currently. With the need to stay inside and work from home with only shore excursions to the shops for supplies like one of Metro's Stalkers coming out of the Metro to the not so dead outside world running from bathroom to bathroom to wash your hands as if they were one of Left4Deads many safehouses. Yes the "end times" are certainly far duller than anticipated plenty of time too look your bedroom walls and scroll through Netflix to find something you haven't watched ..already. While we all do are part and stay indoors I hope you are all keeping safe and well. Fortunately for us to stave of boredom (and Insanity) we have video games to keep us occupied with more time in the house and if you have tried to get hold of a Nintendo Switch recently you will know many other people of have had the same idea! Well of course why not? We all know as gamers how much a game can suck you in. "One more turn" in Civ can quite easily lead to it being 4am running on nothing more than your energy drink and the desire to Nuke Gandhi before he nukes you.
Talking of one more turn and the folks at Firaxis recently they released XCOM Chimera squad a smaller scale and scope XCOM game you take control of Chimera Squad a specialist police strike force made to promote unity and to deal with the more extreme elements post defeating advent, Alien and human haters like. Chimera are named appropriately as it is consists and is comprised of both humans, Aliens and former advent elements post defeating advent, Alien and human haters like. Chimera are named appropriately as it is consists and is comprised of both humans, Aliens and former advent being set after XCOM 2. Chimera Squad compared to the former title in the series are all named characters this time around with their own skill trees, personalities and appearances even talking and playing of each other. This does means they also dont suffer from permanent death. Of course it isnt all that easy you can still have them be wounded and suffer negative effects if they arent treated, after all at its core this is still XCOM and very much plays that way. Being more close quarter based though they've added some fun new mechanics like breaching but also is somewhat more story focused it more XCOM but with enough difference to keep it fresh.
I also recently managed to somehow pick up an Animal Crossing Nintedo Switch, and while something I have been dabbling in as well. Nostalgia has found me diving into Pokèmon a series I havent played since Diamond on the DS. More specifically Pokemon Sword as my friend picked up Shield allowing us to trade the exclusive pokemon for each game and while I haven't had much time to play it, its quickly took up much of my spare time bringing an old 7 year olds itch to catch them all!
I also picked up Hyperlight Drifter a top down RPG reminiscent of something like the original Zelda games trawling dungeons while you perform quests and collect better equipment, it is entirely its own beast taking place after some sort of Apocalypse or Catastrophe you play as the Drifter, trying to piece together or avert vesions or perhaps prophecy. It doesn't have anything in the way of dialogue alot of the story and tale of the world being told through pictures when you talk to NPCs or by your surroundings and musical score which so far is excellent. All this combined leaves you much in the same spot as our protagonist lost and confused as to the bigger picture trying to piece together what's going on as events unfold. I've currently just beaten the first boss and comparison can be drawn to the Souls series only being able to carry so many healing items at a time and needing a second to use and restore to full health, along with varied enemies with specific attack patterns this transfers over to bosses too having their own phases. Making sure you make full use of the sword, gun and dash you're equipped with. I'm definitely enjoying what I've delved into so far and am looking forward to getting to the bottom of the mystery I'm unravelling as I play!
Hyperlight Drifter has a bold colourful pixel art style.that only adds to it's hopeless and somber tone of its world and its mysteries more stark in contrast.
Joanna Hughes (@x_jamjar_x)
What I'm playing: Animal Crossing: New Horizons
I would be lying if I didn't say I was dubious, it didn't seem like my thing, and I hate jumping on the bandwagon – but Animal Crossing stopped being a game I resented and very quickly became my evening routine. Considering that I continue to work as usual and I refrained from buying it for at least a week after release, I was shocked to find that I have spent well over 75 hours perfecting my little island. Never have I become so engrossed in a game in all my life. There is never a good place to stop and you promise yourself you'll just plant a few more flowers and catch one more fish and then you'll go to bed, it is only as the sun rises in game that you realise that you never made it. I think it is fair to say that Animal Crossing becomes a part of life you couldn't possibly have anticipated. Not only is this true sandbox of a game fairly addictive just because it is peaceful and pretty, it has a fair amount of challenges that you really want to complete. Whether you want a 5 star island or a thriving turnip business, or even to complete your collections for the museum, Animal Crossing: New Horizons never ends with it's flurry of tasks. Combine this with the fact that there is a huge online community, it is constantly changing and evolving, I don't think I will ever get bored of going from island to island exploring.
That just about covers our list for what we've been developing Cabin Fever to. Been playing something that was on our list? Think we missed something? Let us know in the comments below!
PreviousMafia Trilogy Officially Revealed
NextBorderlands 3 Updates get first look
Ryan Perrow
Video Game Critic, Comic Book Geek, Spawn Fanboy, Halo Rage Monster, New Jersey Devils Fan. Nicknamed Sparkles and drinks an unbelievable amount of tea.
Retrogame – The Elder Scrolls IV: Oblivion
7 video game openings that got me hooked!
In Memoriam – Blood Omen: Legacy of Kain
GH Collective – Game of the Year Edition


Share your QI success stories. In 250 words or less, tell us about what you wanted to accomplish, what changes you made, and how you measured improvement. Client and staff quotes that illustrate the impact of the change are welcome too!
Send your stories to quality@e-qip.ca.
Our invitation is an open call. Contact us anytime.


The most exciting inning of baseball I've seen lasted forever. It was Game 5 of the Rangers versus the Blue Jays in the 2015 ALDS playoffs. It involved crazy rule interpretations, three errors and an amazing Jose Bautista go-ahead home run. In short, it was the perfect baseball inning. It lasted 53 minutes.
Rob Manfred, the Commissioner of Baseball for the MLB, is rightfully concerned with the tepid interest of the American public, specifically the younger generations. Baseball's viewers are the oldest demographic compared to other major American sports. His solution is shortening the game which has admittedly lengthened in the past few years.
However, the small rule changes, like removing intentional walks, only shave 11 seconds off per game. Bigger picture suggestions like adding base runners in extra innings have been rightfully met with disgust.
What Manfred fails to understand is that us youthful viewers do not blanch at the length of the game, but the tedium of it. Football games are longer and sodden with commercials, but the excitement of the sport draws viewers. While baseball can never match the barbaric nature of football, it can become more dynamic, and that starts with changing the unwritten rules.
Baseball players are encumbered with a code of ethics dating back to the start of the 20th Century. Players cannot show emotion on the field, and should always act like they've "been there before." Unlike football and basketball, anyone who celebrates or trash talks is seen as disrespectful to the other team and the institution of baseball.
What capped off that Blue Jays-Rangers inning was that the unwritten rules were broken. Bautista did not just hit a homerun, he flipped the bat afterwards. The bat flip was the GIF — the pimping of the hit after an emotional roller coaster is what we remember. The fact that it was followed next season by Rougned Odor punching him in the face just added to the excitement.
There are some indications that changes are coming, but it's slow. There is some phenomenal young talent in baseball, and players like Bryce Harper and Manny Machado add youthful energy and fiery quotes to the game. However, Mike Trout, the best player in baseball, has adopted the Derek Jeter strategy of being extraordinarily dynamic on the field, and having the personality of slightly curdled milk. He has been lauded for preserving the tradition of baseball, which is code for having no personality. Overseas teams can provide a blueprint — Hispanic players are known for being exuberant in their play, and the Korean league features cheerleaders and abundant bat flips.
I am not advocating a loss of decorum in baseball. A rookie should not be flipping off the pitcher after every hit. But if baseball is to stay relevant, there must be more "GIF-able" moments. Changing baseball's culture will not be easy. A big reason baseball has remained stagnant is that pitchers feel entitled to retaliate against hitters celebrating by beaning them next pitch. Hitters would rather not show excitement and risk getting hit by a 95-mph fastball. Therefore, MLB should strengthen rules allowing umpires to eject pitchers intentionally hitting batters. Displayed emotion must no longer be seen as disrespect but rather a part of baseball.
In short, Manfred should be less concerned about the seconds of gameplay and more bothered by the ambivalence exhibited by the players during it.


Our first box came wrapped in fancy tissue paper and everything! "Oooo!"
This box included samples of Rose Mist, Mim Special Summer Oolong, and Doomni Exotic Summer Oolong. I was a little surprised to be sent two oolong teas out of three.
My favorite was Rose Mist. It has luminous color with a sweet magical flavor. From the first whiff as you open the bag, you are transported to a fresh bed of roses interlaced with subtle strawberry and pomegranate notes. The brewed liquor is golden and luminous. It makes me think of a magical potion for a happily ever after. This tea flows softly over the tongue, filling the mouth with a fresh flavor. Its rosy sweet and velvety. This tea is excellent for easing you into a relaxing evening. Its low caffeine and high quality in every way.
Doomni i was not impressed with. This one was a bit flavorless for me. If anything, it mostly tasted bitter.
My second box included two flavors I had already recieved. This surprised me. I was also slightly disappointed. While I loved Rose Mist, Doomni was disappointing to see arrive again. After this box, I decided to sit down and post a few reviews. Hopefully, my next box will include a variety of teas of greens, whites, and blacks instead of being so oolong heavy. I like oolong tea very much, but I was under the impression the boxes were oing to allow me to sample a variety of teas from around the world. Hopefully, the next box will offer this.
I have purchased a few other flavors since my second box. Teabox was kind enough to send me three complementary samples with my order!!
A cup of ginger/almond/cinnamon deliciousness; versatile blend!
The first inhale I took from the opened bag instantly brought to mind a paper cup filled with deliciously foamy chai ready to tickle my lips with its frothy liquid. Steeped into a milk-chocolate brown liquor, the notes of ginger, almond, and cinnamon complement each other delightfully. With its hint of spice in the aroma as you bring the cup to your lips to its rich decadent body, this tea is a great morning and afternoon indulgence. With its ability to be drunk straight from the bag or with additions of honey and milk, I recommend this blend as a great versatile tea. Its is made with a base of black tea leaves, chai spices, and thick almond slices. Quite an inviting cup!
Nice low-caffeine tea. Brews very clear and is deep gold/orange in hue. I had hoped for more of a chamomile flavor in the taste profile, as hinted at on the packaging. The liquor is slightly too vegetal in its flavor profile for my tastes. I think its high in quality but not quite my favorite taste on a personal level. I think fans of earthy teas will enjoy this one. Thank you to Teabox for sending me this tea complementary with my order. I appreciate the sample very much!


The days are getting shorter. The weather is getting a little crisper. Before you know it, winter will be here for those of us living in cold climates. That means snow, ice and miserably cold temperatures, which can present a number of challenges and potential health risks for our four-legged companions. Don't stress out, though. If you follow the steps below, you can keep your dog healthy and safe during winter time.
As a rule of thumb, the lower the temperature outside, the shorter the time your pup should spend outdoors. Always watch the thermometer before sending a dog outside. When it gets extremely cold, make sure they spend only a limited time outdoors.
If for some reason your dog does spend extended periods of time outside, be sure to always have to have an option for shelter that is windproof and preferably heated.
Staying warm in cold weather takes a lot of energy, which in turn burns calories more quickly. If your dog plays outside in the cold, give them a little extra food to refuel, and always make sure their water bowl is full.
Winter is tough on your dog's paws. First, ice often accumulates between the pads and toes, leading to sore paws and even hypothermia or frostbite. When your dog comes back inside, check their feet for ice and remove gently with a towel.
Another issue is the salt used to melt ice on the road and sidewalks. This can lead to chemical burn on your poor pup's paws. Worse, your dog may lick their paws and ingest the chemicals. To avoid any problems, either try to limit the time your dog spends walking on the road or quickly wipe down their feet as soon as you get home.
Dog booties can help with both problems – if your pup will keep them on.
Another dangerous chemical in winter in antifreeze. The substance drips from a car's radiator and has a smell and taste that dogs are attracted to. The toxin ethylene glycol is what makes antifreeze so lethal, and only a small drop can have serious health impacts for your dog, including damage to the brain, liver and kidneys, and even death.
Signs of antifreeze poisoning include delirium, uncoordinated movement, vomiting, excess urination, rapid heart beat, weakness, diarrhea, fainting and seizures. If you think your dog ingested antifreeze, call your vet right away!
Of course you always want to make sure your dog is well trained and well behaved. But in the winter months, this will be put to the test. For example, it's annoying when your dog pulls on the leash in other seasons, but this becomes downright dangerous when there's ice on the ground that could cause injury to one or both of you. Or say your dog runs onto a lake that is only partially froze, or they are starting to drink antifreeze. You want your dog to be familiar with commands at these perilous times.
Your dog's fur provides warmth. Don't shave your dog down in winter months; they need that extra protection against the cold. However, if your pup has long hair, you'll want to trim it a bit to minimize ice and salt sticking to the fur, especially on the paw area.
For smaller dogs with short hair, you may want to get them a dog coat or sweater for extra warmth. You'll likely get some ribbing from friends for this, but wouldn't you rather have a warm, safe and healthy pup?
Washing your pup too often removes essential oils and can irritate the skin, especially during cold months. That's not to say you shouldn't bath your pup during the winter; it's just better to do it less frequently and only when needed. To keep your dog from getting too stinky, you can always use a wet washcloth to wipe down areas like the legs and paws that get dirty more frequently. And when you do give your dog a bath, make sure he is 100% dry before going outside. You don't want a dogsicle.
As we mentioned before, ice is a tricky bugger. One false move and you, your dog, or both of you are going down. Hard. Keep your dog away from extra icy areas to prevent any serious injuries. If you have to walk on ice, make sure your dog walks at a reasonable pace and doesn't pull.
Also, watch how much ice your dog eats. Not only could it be made up of dangerous chemicals, the cold ice may also chill your dog's body and cause damage. Our dog Toby loves to eat ice. One time in an extremely cold bout, he grabbed a giant chunk of ice and wolfed it down before we could stop him. Moments later, he started to shake and whine. The cold was just too much for his system! We held him tight and calmed him, trying to desperately to warm him up. Thankfully this worked, but had it not, we would have taken him to the vet right away. The experience was utterly terrifying!
Despite your best efforts, your dog may still do something silly in the winter time that has serious consequences. Always keep an eye out for strange, uncharacteristic behavior. If you observe anything out of the ordinary – excessive whining and shivering, slow movements, shallow breathing, etc. – call the vet right away.
Just a reminder: If you see a dog in the cold for an unreasonable amount of time, speak up. Talk to the owner if possible. If not, call authorities. Your actions could save the dog's life.


text:
Q: console.time doesn't support QLoggingCategory? I'm trying to reduce the amount of log output in my QtQuick application by creating many logging categories like documented here.

A logging category can be passed to console.log() and friends as the first argument. If supplied to to the logger the LoggingCategory's name will be used as Logging Category otherwise the default logging category will be used.

Now I'm trying to do that with console.time() and console.timeEnd() (documented here).
I get some errors in the log output:
qrc:/main.qml:16: Error: console.time(): Invalid arguments

How can I use logging categories for console.time()?
Example code:
import QtQuick 2.9
import QtQuick.Window 2.2

Window {
    visible: true
    width: 640
    height: 480
    title: qsTr("Hello World")

    LoggingCategory {
        id: category
        name: "some.not.so.important.stuff"
    }

    Component.onCompleted: {
        console.time(category, 'start');
        console.timeEnd(category, 'start');
    }
}

A: Where documentation says

A logging category can be passed to console.log() and friends as the
  first argument

it refers to console.log, console.debug, console.info, console.warn and console.error, which are all meant to just print messages.
The other console features (Assert, Timer, etc.) do not support logging categories.


The Eucharist is a Christian rite that is considered a sacrament in most churches. According to the New Testament, It was instituted by Jesus Christ during His Last Supper. Giving His disciples bread and wine during the Passover meal, Jesus commanded His followers to "do this in memory of Me" while referring to the bread as "My Body" and the wine as "My Blood". Through the Eucharistic celebration Christians remember Christ's sacrifice of himself on the cross.