\section{Introduction}
\label{sec:intro}

The majority of works in object detection mainly describe results on two datasets: Pascal VOC \cite{everingham2011pascal} and MS-COCO \cite{lin2014microsoft}. While this information might be helpful in forming the relative intuition of a model behavior for common scenes, it is still unobvious when it comes to a dataset whose distribution of objects deviates. Goldman et al. \cite{goldman2019precise} proposed a dataset with the following key features: 1) a high number of objects on an image, 2) objects are densely packed.

Although in the comparison table the authors declared the dataset to have 110,712 classes, it is important to note that in this context they used the term \emph{class} to describe the appearance variations among the objects. De facto the dataset represents a single-class object detection task. Considering this information, we might reference an example of a similar task from a different domain \cite{shao2018crowdhuman} where both RetinaNet \cite{lin2017focal} and Faster-RCNN \cite{ren2015faster} were taking as baseline solutions and showed good results. Hence, the first step of my plan was to revisit some of the baseline architectures.

A rapidly growing number of papers with different new methods in object detection makes it difficult to maintain the reliable and up-to-date benchmarking table especially when a custom implementation of the new method also comes with different parts of the training pipeline such as data pre-processing or results post-processing. In order to address this issue, many attempts have been made for the last several years to create a highly reusable codebase that covers most of the popular models in the field. For now there are plenty of such toolboxes based on different deep learning frameworks: Tensorflow Object Detection API \cite{huang2016speedaccuracy}, Detectron \cite{Detectron2018} and Detectron2 \cite{wu2019detectron2}, MMDetection \cite{chen2019mmdetection}, and so on. Each of these toolboxes has reusable parts of code, which makes it easier to compare the performance of models in vitro. For my experiments I chose MMDetection v1.

The rest of the report is structured as follows: in Section \ref{sec:dataset} I briefly review the parameters of the dataset and additionally highlight the features which I found important. Section  \ref{sec:exps}
contains all the experiment results where I consequentially inspect the results starting from baseline models \ref{sec:basleine} then testing a few approaches that are different from widely-used anchoring scheme \ref{sec:adv}. In the final subsection \ref{sec:scale} I describe two simple tricks that helped to achieve considerably good performance for Faster-RCNN, testing some commonly-used tricks along the way.

\section{Dataset description}
\label{sec:dataset}

The dataset for the challenge is almost the same version that was originally described in \cite{goldman2019precise} with minor differences: the train set contains 8,219 instead of 8,233 images and the test set contains 2,936 out of 2,941. The validation part contains exactly 588 images. During training a few more examples of broken images were found, so those might be easily excluded.
\begin{figure}[h!]
  \centering
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[width=\linewidth]{ratio_hist.png}
    \caption{Aspect ratio}
  \end{subfigure}
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[width=\linewidth]{size_hist.png}
    \caption{Area}
  \end{subfigure}
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[width=\linewidth]{width_hist.png}
    \caption{Width}
  \end{subfigure}
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[width=\linewidth]{height_hist.png}
    \caption{Height}
  \end{subfigure}
  \caption{Statistics of object geometry in histograms. Outliers are clipped by 0.01 and 0.99 quantiles.}
  \label{fig:fig1}
\end{figure}

Inspecting the train set statistics the most popular image sizes were found. It turned out that the majority of scenes were taken in portrait mode in the following resolutions: 2448x3264 - 3463 images, 1920x2560
- 1710 images, 2336x4160 and 2340x4160 might be combined to have 1133 samples in total. The distribution of image sizes definitely differs from Pascal VOC and MS COCO that is a premise to experimenting with related parameters. Secondly, object sizes and aspect ratios were reviewed (see Fig. \ref{fig:fig1}). While aspect ratio distribution was not found interesting, object size distribution seemed a more rewarding feature to work with. Although object sizes were distributed as 0.12\%/18.53\%/81.35\% among "small", "medium" and "large" according to MS COCO the object size classification, the picture is changing in the scenario of image resizing, which is a necessary trick for training with reasonable batch size. Decreasing the image size to 816x1088 \footnote{This size is used as a default parameter to "Resize" transformation for the rest of the report until otherwise is specified. Unlike default 1333x800 the proposed size takes into account the most popular image size within SKU110k.} which is 9 times smaller, we will see the following shift - 18.64\%/80.27\%/1.09\%. In this scale the number of "large" examples is substantially small. These findings drew much attention to the ideas described in \ref{sec:scale}.

\section{Experiments}
\label{sec:exps}

\subsection{Previously compared models}
\label{sec:basleine}
For the purpose of setting up a fair baseline two main models were chosen: Faster-RCNN and RetinaNet. Both models are used with default configurations utilizing ResNet-50 \cite{he2015deep} backbone and FPN \cite{lin2016feature}. In the initial experiments images are resized to a maximum scale of 1333x800, without changing the aspect ratio. The training schedule is “1x”, which means 12 epochs. "RandomFlip" image augmentation is used with probability 0.5 during training time. The model is trained on the training set, and the test set is used for evaluation. Evaluation is performed using official "pycocotools" package which is an implementation of detection evaluation metrics used by COCO with "maxDets" parameter changed to "[1 10 300]". \footnote{The results obtained by testing with pycocotools were slightly different from the results obtained with the evaluation code proposed by the authors. I did not set a goal of revealing the differences in code, so just be aware while comparing with any results taken from other technical reports.}

Given the results described in Table \ref{tab:table1} I decided to further experiment with stronger models.

\begin{table}
	\caption{Revisiting default models}
	\centering
	\begin{tabular}{llllll}
		\toprule
		Model               &  mAP  & AP@$^{0.5}$ & AP@$^{0.75}$ & AR$^{300}$   \\
		\midrule
		RetinaNet-r50-fpn    & 0.463 & 0.751  & 0.532 & 0.512 \\
		Faster-RCNN-r50-fpn  & 0.523 & 0.850  & 0.592 & 0.582 \\
		\bottomrule
	\end{tabular}
	\label{tab:table1}
\end{table}

\subsection{Different anchoring approaches}
\label{sec:adv}

In order to test anchoring strategies different from the dense scheme used for both previous models \emph{Guided Anchoring} \cite{wang2019region} was tried in a combination with RetinaNet on top of two different backbones. Surprisingly, GA-RetinaNet-r50-fpn showed results comparable with Faster-RCNN model. Another approach \emph{RepPoints} \cite{yang2019reppoints}, which is an example of anchor free detectors, did not show any significant improvement with default parameters.

\begin{table}[!htp]
	\caption{Non-dense anchoring}
	\centering
	\begin{tabular}{llllll}
		\toprule
		Model                        &  mAP  & AP@$^{0.5}$ & AP@$^{0.75}$ & AR$^{300}$   \\
		\midrule
		GA-RetinaNet-r50-fpn         & 0.523 & 0.870 & 0.579 & 0.583 \\
		GA-RetinaNet-x101-32x4d-fpn  & 0.537 & 0.882 & 0.602 & 0.598 \\
		RepPoints-moment-r50-fpn     & 0.505 & 0.815 & 0.578 & 0.562 \\
		\bottomrule
	\end{tabular}
	\label{tab:table2}
\end{table}

\subsection{Scale-related tricks}
\label{sec:scale}

\subsubsection{Anchor scale}
\label{sec:scale1}
MMDetection Faster-RCNN default implementation uses the following anchor settings: `anchor\_strides`=[4, 8, 16, 32, 64] that specifies the stride for an anchor grid and `anchor\_scale` = [8] that is used as a multiplication factor for strides to obtain 5 anchor scales with areas [32$^2$, 64$^2$, 128$^2$, 256$^2$, 512$^2$] respectively. In Section \ref{sec:dataset} I highlighted lack of "large" objects if resizing preprocessing is applied. Hence, in order to generate more dense proposals, it looked like a good idea to decrease `anchor\_scale` to [4] to form the following areas [16$^2$, 32$^2$, 64$^2$, 128$^2$, 256$^2$].
Table \ref{tab:table3} represent relative importance of this parameter.

\begin{table}[!htp]
	\caption{Comparison of different anchor scales}
	\centering
	\begin{tabular}{llllll}
		\toprule
		Model                &  Anchor scale &  mAP  & AP@$^{0.5}$ & AP@$^{0.75}$ & AR$^{300}$   \\
		\midrule
		Faster-RCNN-r50-fpn  & [8] & 0.522	&0.850	&0.591	&0.577 \\
		Faster-RCNN-r50-fpn  & [4] & 0.551	&0.912	&0.614	&0.613 \\
		\bottomrule
	\end{tabular}
	\label{tab:table3}
\end{table}

To pay respect to computer vision challenges widely-used tricks were tested on top of that such as: simple TTA with `RandomFlip`, additional data augmentation using albumentations \cite{info11020125} config\footnote{ \url{https://github.com/open-mmlab/mmdetection/blob/v1.0.0/configs/albu_example/mask_rcnn_r50_fpn_1x.py}}, multi-scale training. None of them gave any significant boost. See Table \ref{tab:table4}.

\begin{table}[!htp]
	\caption{Some of the common tricks}
	\centering
	\begin{tabular}{llllllllll}
		\toprule
		Model               & Lr schd & Extra augs & Train flip & Test flip & MS train &  mAP  & AP@$^{0.5}$ & AP@$^{0.75}$ & AR$^{300}$   \\
		\midrule
		Faster-RCNN-r50-fpn & 1x & \xmark & \xmark & \xmark & \cmark & 0.552	&0.912	&0.615	&0.616 \\
		Faster-RCNN-r50-fpn & 1x & \cmark & \xmark & \xmark & \xmark & 0.548	&0.911	&0.608	&0.612\\
		Faster-RCNN-r50-fpn & 2x & \cmark & \cmark & \xmark & \xmark & 0.540	&0.906	&0.596	&0.606 \\
		Faster-RCNN-r50-fpn & 2x & \cmark & \cmark & \cmark & \xmark & 0.510	&0.888	&0.543	&0.584 \\
		\bottomrule
	\end{tabular}
	\label{tab:table4}
\end{table}

In order to obtain even more precise detector Cascade-RCNN \cite{cai2017cascade} was also tested. Image size was reduced for models with ResNeXt-101 in order to keep batch size equal to 2 images. The results are available in Table \ref{tab:table5}. Soft-NMS \cite{bodla2017softnms} was applied as a post-processing, which usually gives better results.

\begin{table}
	\caption{Cascade-RCNN comparison}
	\centering
	\begin{tabular}{lllllllll}
		\toprule
		Model&Img size&Anchor scale& Soft-NMS&mAP& AP@$^{0.5}$ & AP@$^{0.75}$ & AR$^{300}$   \\
		\midrule
		Cascade-RCNN-r50-fpn        &(816, 1088)&[8]& \xmark & 0.525	&0.840 &0.604 &0.582 \\
		Cascade-RCNN-r50-fpn        &(816, 1088)&[4]& \xmark & 0.553	&0.902 &0.626 &0.615 \\
		Cascade-RCNN-r50-fpn        &(816, 1088)&[4]& \cmark & 0.556	&0.900 &0.632 &0.622 \\
		Cascade-RCNN-x101-32x4d-fpn &(768, 1024)&[4]& \xmark & 0.556	&0.903 &0.629 &0.617 \\
		Cascade-RCNN-x101-32x4d-fpn &(768, 1024)&[4]& \cmark & 0.560	&0.902 &0.635 &0.623 \\
		\bottomrule
	\end{tabular}
	\label{tab:table5}
\end{table}

\subsubsection{Tiling strategies}
\label{sec:scale2}
Image tiling as a trick for object detection for large images with small objects on them was previously explored in \cite{ozge2019power}. Following the proposed method 2x2 tiling grid was chosen with 20\% intersection between consecutive tiles. Keeping in mind dense object location bounding box annotations of objects with residual area less than 20\% of the original were removed.
In their work \cite{ozge2019power} proposed to also take a full frame for merging detectors. Although it might seem helpful for "large" objects, I'd like to emphasize that we almost never have those, so this option seems redundant.

Hence, I decided to test the two following approaches:
\begin{itemize}
	\item \textbf{Tile-combine}. In this scenario each image in train and test parts of the dataset is split into 4 tiles. Appropriate annotations are created. During test time inference is run on each tile and the results are saved into a JSON file. Then the results are manually merged. Duplicated objects on overlapping areas are resolved by NMS.
	\item \textbf{Train-time tiling}. Here the model on 4 tiles is trained. Test-time inference is run using a 4 times larger resolution. The last one is possible because we do not have to store gradients in the test time.
\end{itemize}

\emph{Tile-combine} seemed promising when evaluating on tiles (w/o merging), but the metrics dramatically decreased for full frames after combining the results (merged). I also tried to resolve multiple bounding boxes on tiles edges by NMS providing normalized areas instead of probability scores from the detector, which did not improve the score. Obviously when image tiling is applied `anchor\_scale` parameter becomes much less important. Finally, \emph{train-time tiling} and applying detection on a full frame with 1632x2176 resize showed the best results and I used this configuration for final training. See Table \ref{tab:table6}.

\begin{table}[!htp]
	\caption{Tiling strategies}
	\centering
	\begin{tabular}{lllllllll}
		\toprule
		Model                      & Comment & An. scale & Soft-NMS &  mAP  & AP@$^{0.5}$ & AP@$^{0.75}$ & AR$^{300}$   \\
		\midrule
		Faster-RCNN-r50-fpn        & w/o merging & [8] & \xmark & 0.561	&0.912 &0.632 &0.628 \\
		Faster-RCNN-r50-fpn        & w/o merging & [4] & \xmark & 0.566  &0.928 &0.636 &0.636 \\
		Faster-RCNN-r50-fpn        & merged & [4] & \xmark & 0.547	&0.894 &0.615 &0.611 \\
		Faster-RCNN-r50-fpn        & full frame & [4] & \cmark & \textbf{0.577} &0.928 &\textbf{0.659} &\textbf{0.654} \\
		\bottomrule
	\end{tabular}
	\label{tab:table6}
\end{table}

\subsubsection{Final submission approach}

For my final approach I used Faster-RCNN-r50-fpn with `anchor\_scale` = [4] trained on a combination of train, val and test sets tiled as described above. During inference on a provided leaderboard-test set without annotations I used resizing to 1632x2176 and Soft-NMS to resolve duplications. No TTA or ensembling was used.

\section{Conclusions}

In this report it was empirically shown that the simple two-stage detector is a strong baseline for the Product Detection in Densely Packed Scenes task. Also two scale-related approaches are described which improve the detector performance by a large margin. Hence, object scale is one of the aspects that should be considered for further investigation and I also hope to see new approaches being developed for the dataset taking this idea into account.

\bibliographystyle{plain}


\section*{Appendices}
\section{Algorithm}
\label{appendix:algorithm}
In this appendix, we provide a detailed algorithm for implementing \textit{PatchUp}{}. As with most regularization techniques, \textit{PatchUp}{} also has two modes (either inference or training). It also needs the combining type (either \textit{Soft PatchUp}{} or \textit{Hard PatchUp}{}), $\gamma$, and $block\_size$. This algorithm shows how \textit{PatchUp}{} generates a new hidden representation from $(g_{k}(x_{i}), y_{i})$ and $(g_{k}(x_{j}), y_{j})$. Lines 4 to 9 in the algorithm\ref{alg:patchup} are the binary mask creation process used in both \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{}.

\import{tables/}{algo.tex}

Figure~\ref{fig:block_selection} briefly illustrates and summarizes the binary mask creation process in \textit{PatchUp}{}. Lines 11 to 25 correspond to the interpolation and combination of hidden representations in the mini-batch in \textit{PatchUp}{}. Figure~\ref{fig:mask_cmp} compares the masks generated by \textit{PatchUp}{} and \text{CutMix}{}.

\begin{figure}[htbp]
  \centering
\begin{subfigure}{0.47\linewidth}
	\label{fig:mixer_mask}
	\includegraphics[width=1.\textwidth,height=55pt]{pics/blockmix_mask.png}
	\caption{Mask sampling in \textit{PatchUp}{}.}
\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\label{fig:cutmix_mask}
	\includegraphics[width=1.\textwidth]{pics/cutmix_mask.png}
	\caption{Mask sampling in \text{CutMix}{}.}
\end{subfigure}
\caption{Mask sampling in \textit{PatchUp}{} is applied in the hidden state, compared to \text{CutMix}{} which is applied in the input space. Red areas show the blocks that should be altered.}
\label{fig:mask_cmp}
\end{figure}

\section{Why random $k$?}
\label{appendix:random_k_selction}
\textit{PatchUp}{} applies block-level regularization at a randomly selected hidden representation layer $k$. The Information Bottleneck (IB) principle, introduced by \citet{DBLP:Tishby}, gives a formal intuition for selecting $k$ randomly. First, let us encapsulate the layers of the network into blocks where each block could contain more than one layer. Let $g_k$ be the $k$-th block of layers. In this case, sequential blocks share the information as a hidden representation to the next block of layers, sequentially. We can consider this case as a Markov chain of the block of layers as follows:
\begin{equation}
x \to g_{1}(x) \to  g_{2}(x) \to  g_{3}(x).
\end{equation}
In this scenario, the sequential communication between the intermediate hidden representations are considered to be an information bottleneck. Therefore,
\begin{equation}
I(g_{3}(x);\;g_{2}(x)) < I(g_{2}(x);\;g_{1}(x)) < I(g_{1}(x);\;x),
\end{equation}
where $I(g_{k}(x);\;g_{k-1}(x))$ is the mutual information between the $k$-th and $(k-1)$-th layer.

If $g_{k=3}(x)$ has enough information to represent $x$, then applying regularization techniques in $g_{k=3}(x)$ will provide a better generalization to unseen data. However, most of the current state-of-the-art CNN models contain residual connections which break the Markov chain described above (since information can skip the $g_{k=3}$ layer). One solution to this challenge is to randomly select a residual block and apply regularization techniques like ManifoldMixup or \textit{PatchUp}{}.

\begin{figure}[tbp]
\centering
\begin{minipage}{.51\textwidth}
  \centering
  \includegraphics[width=1.\linewidth, height=75pt]{pics/max_pool2d.png}
\caption {\textit{PatchUp}{} mask creation process ($block\_size = 5$). The left matrix shows the process of feature selection from feature maps. By using a $max\_pool2d$ function, we can create blocks around selected features. The $max\_pool2d$ function uses $stride = (1,\;1)$, $kernel\_size = (block\_size,\,block\_size)$, and $padding = (\frac{block\_size}{2}, \frac{block\_size}{2})$. Red and blue points are $1$ and $0$ in the generated binary mask, respectively.}
  \label{fig:block_selection}
\end{minipage}%
\hfill
\begin{minipage}{.47\textwidth}
  \centering
  \includegraphics[width=1.\textwidth]{pics/delta_theta.png}
\caption{The comparison of $\rho$ for flattened hidden representations of a mini-batch of samples at the second residual block (layer $k = 3$) of WideResNet-28-10 with corresponding regularization method.}
\label{fig:sim_comparison}
\end{minipage}
\end{figure}

\section{\textit{PatchUp}{} Interpolation Policy Effect}
\label{appendix:sim_comparison}
Assume that $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are flattened hidden representations of two examples produced at layer $k$. And, $\mathcal{H}$ is the flattened interpolated hidden representation of these two paired samples at layer $k$. First, we calculate the cosine distance of the pairs ($\mathcal{H}_{2}$, $\mathcal{H}_{1}$), ($\mathcal{H}_{1}$, $\mathcal{H}$), and ($\mathcal{H}_{2}$, $\mathcal{H}$). Reversing the cosine of these cosine similarities give the angular distance between each pair of vectors denoted as $\angle\theta$, $\angle\alpha_{1}$, and $\angle\alpha_{2}$, respectively. There is always a surface that contains $\mathcal{H}_{2}$ and $\mathcal{H}_{1}$ denoted as $\mathcal{S}$. Mathematically, we have:
\begin{equation}
\begin{array}{l}
\label{eq:lsim_study}
    \angle \alpha_{1} = \cos^{-1}(\frac{\mathbf{\mathcal{H}_{1}} \cdot \mathbf{\mathcal{H}}}{\|\mathbf{\mathcal{H}_{1}}\|\|\mathbf{\mathcal{H}}\|}) \And
    \angle \alpha_{2} = \cos^{-1}(\frac{\mathbf{\mathcal{H}_{2}} \cdot \mathbf{\mathcal{H}}}{\|\mathbf{\mathcal{H}_{2}}\|\|\mathbf{\mathcal{H}}\|}) \And
    \angle \theta =
    \cos^{-1}(\frac{\mathbf{\mathcal{H}_{2}} \cdot \mathbf{\mathcal{H}_{1}}}{\|\mathbf{\mathcal{H}_{2}}\|\|\mathbf{\mathcal{H}_{1}}\|}),
\end{array}
\end{equation}
Let us define \(\angle\rho=(\angle\alpha_{1}+ \angle\alpha_{2})-\angle\theta\) and \(\angle\rho'= |\angle\alpha_{1}-\angle\alpha_{2}|-\angle\theta\).
According to the triangle inequality principle, either $\angle\rho$ or $\angle\rho'$ will be zero if, and only if, $\mathcal{H}\in\mathcal{S}$.
Figure~\ref{fig:sim_vectors} illustrates three possible scenarios for two paired flattened hidden representations and their flattened interpolated hidden representations. $\angle\rho$ and $\angle\rho'$ are zero for the left and right figures, respectively. We try to empirically show that $\mathcal{H}_{2}$, $\mathcal{H}_{1}$, and $\mathcal{H}$ always lie in the same surface $\mathcal{S}$ and $\mathcal{H}$ lies between $\mathcal{H}_{2}$ and $\mathcal{H}_{1}$ in ManifoldMixup. This means that $\angle\rho = 0$ for ManifoldMixup because of its linear interpolation policy. The middle figure in~\ref{fig:sim_vectors} is the case that both $\angle\rho$ and $\angle\rho'$ are not equal to zero. This figure shows that one possible situation is that flattened interpolated hidden representation does not lie in the surface $\mathcal{S}$. Our goal is to produce the interpolated hidden representation that lies in all possible places towards all dimensions in the hidden space.
\begin{figure}[H]
\centering
  \includegraphics[width=.32\textwidth]{pics/vector_1.png} \hfill
  \includegraphics[width=.32\textwidth]{pics/vector_2.png} \hfill
  \includegraphics[width=.32\textwidth]{pics/vector_3.png}
    \caption{$\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are the flattened hidden representations. $\mathcal{H}$ is the flattened interpolated hidden representation that can be produced by either ManifoldMixup, \textit{Soft PatchUp}{} or \textit{Hard PatchUp}{}.}
    \label{fig:sim_vectors}
\end{figure}
As discussed in section~\ref{para:patchup_vs_manifold}, ManifoldMixup can provide interpolated hidden representation only in a limited space. However, \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{} can produce a wide variety of interpolated hidden representations towards different dimensions. To support that, in WideResNet-28-10, for a mini-batch of 100 samples, we calculated the $\angle\rho$ for the flattened interpolated hidden representation produced by ManifoldMixup, \textit{Hard PatchUp}{}, and \textit{Soft PatchUp}{} at the second residual block (layer $k=3$) with the same interpolation policy ($\lambda=.4$, $\gamma=.5$, and $block\_size=7$) for both \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{} for all samples in the mini-batch. The swarmplot~\ref{fig:sim_comparison} shows all $\angle\rho$ for the mini-batch are equal to zero in ManifoldMixup, which empirically supports our hypothesis. However, \textit{PatchUp}{} produces more diverse interpolated hidden representations towards all dimensions in the hidden space. It is worth mentioning that few $\angle\rho$ that are equal to zero in \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{} belong to the interpolated hidden representation that was constructed from the pairs with the same labels.

\section{\textit{PatchUp}{} Experiment Setup and Hyper-parameter Tuning}
\label{appendix:hyperparameters}
This section describes the hyper-parameters of each model in table~\ref{table:models_other_hp} following the hyper-parameter setup from ManifoldMixup \cite{manifold} experiments in order to create a fair comparison. First, we performed hyper-parameter tuning for the \textit{PatchUp}{} to achieve the best validation performance. Then we ran all the experiments five times, reporting the mean and standard deviation of errors and negative log likelihoods for the selected models. We let models train for defined epochs and checkpoint the best model in terms of validation performance during the training.
In our study, we used PreActResNet18, PreActResNet34, and WideResNet-28-10 models. Table~\ref{table:models_other_hp} shows the hyper-parameters used for training the models.

\begin{figure}[bt!]
\centering
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/manifold_alpha_ht.png}
	\caption{Impact of $\alpha$ in ManifoldMixup approach.}
	\label{fig:hyp_manifold}
	\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/cutmix_k_ht.png}
	\caption{Impact of $cutmix\_prob$ in \text{CutMix}{} approach.}
	\label{fig:hyp_cutmix}
	\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/softpatchup_gamma_ht.png}
	\caption{Impact of $\gamma$, $block\_size$ with $patchup\_prob$ as 1.0 for \textit{Soft PatchUp}{}.}
	\label{fig:hyp_soft}
	\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/harpatchup_gamma_ht.png}
	\caption{Impact of $\gamma$, $block\_size$ with $patchup\_prob$ as 0.7 for \textit{Hard PatchUp}{}.}
	\label{fig:hyp_hard}
	\end{subfigure}
\caption{Impact of hyper-parameters $\gamma$, $block\_size$ and $patchup\_prob$ on error rates in the CIFAR-10 validation set for PreActResNet18. We repeated each job three times to collect the mean and the standard deviation of errors. Marked points are the mean of the error rate in the validation set. And, the shadow shows the bootstrapping of results for each hyper-parameter setting.
The lower numbers on the y-axes correspond to better performance.}
\label{hyptune}
\end{figure}
{\renewcommand{\arraystretch}{1.1}
\begin{table*}[t]
\centering
    \captionsetup[subtable]{position = top}
    \captionsetup[table]{position=top}
    \caption{The hyper-parameters used for each model to compare the effect of each regularization technique. The learning rate is denoted as \textit{lr}. And, \textit{lr} is multiplied at each learning rate schedule step by the step factor.}
    \label{table:models_other_hp}
    \begin{subtable}{.7\linewidth}
        \resizebox{\linewidth}{!}{
	\begin{tabular}[h]{ ll rrr }
		\toprule
		{Model} & {\textit{lr}} & {\textit{lr} steps} & {step factor} & {Epochs}\\
        \midrule
		{PreactResnet18}  & $0.1$ & $500${-}$1000${-}$1500$ & $0.1$ & $2000$\\
		\cmidrule(lr){2-5}
		{PreactResnet34} & $0.1$ &$500${-}$1000${-}$1500$ & $0.1$ & $2000$\\
		\cmidrule(lr){2-5}
		{WideResnet-28-10} & $0.1$ & $200${-}$300$ & $0.1$ & $400$\\
        \bottomrule
	\end{tabular}}
\end{subtable}
\end{table*}}

\textit{PatchUp}{} adds $patchup\_prob$, $\gamma$ and $block\_size$ as hyper-parameters. $patchup\_prob$ is the probability that the \textit{PatchUp}{} operation is performed for a given mini-batch, i.e if there are $N$ mini-batches and $patchup\_prob$ is $p$, \textit{PatchUp}{} is performed in $p$ fraction of $N$ mini-batches. $\gamma$ and $block\_size$ are described in section~\ref{patchup}.
We tuned the \textit{PatchUp}{} hyper-parameter on CIFAR-10 with the PreActResNet18. To create a validation set, we split $10\%$ of training samples into a validation set. We set $\alpha$ to $2$ in \textit{PatchUp}{}. For \textit{Soft PatchUp}{}, we set $patchup\_prob$ to $1.0$ and applied \textit{PatchUp}{} to all mini-batches in training. Then, we did a grid search by varying $\gamma$ from $0.45$ to $0.9$ and $block\_size$ from $3$ to $9$. We found that $\gamma$ of $0.75$ and $block\_size$ of $7$ work best for \textit{Soft PatchUp}{} as shown in figure~\ref{fig:hyp_soft}. Similarly, for \textit{Hard PatchUp}{}, we set $patchup\_prob$ to $0.7$ and performed a grid search by varying $\gamma$ from $0.2$ to $0.6$ and $block\_size$ from $3$ to $9$. We found that $block\_size$ of $7$ and $\gamma$ of $0.5$ yield the best results for \textit{Hard PatchUp}{} as shown in figure~\ref{fig:hyp_hard}. Figure~\ref{fig:hyp_manifold} shows that ManifoldMixup with ($\alpha = 1.5$) achieves the best validation performance. For cutout, we used the same hyper-parameters proposed in \cite{cutout}, setting cutout to $16$ for CIFAR10, $8$ for CIFAR100, and $20$ for SVHN following~\cite{cutout}.
Figure~\ref{fig:hyp_cutmix} shows that \text{CutMix}{} achieves its best validation performance in PreActResNet18 in CIFAR-10 with $cutmix\_prob = 0.4$. Furthermore, DropBlock achieves its best validation performance on this task by setting the block size and $\gamma$ to $7$ and $0.9$, respectively~\cite{dropblock}.

\section{ Generalization on Deformed Images}
\label{appendix:affine_transformation}
\import{tables/}{affine_wrn.tex}
We created the deformed test sets from CIFAR100, as described in Section~\ref{exp:deformed_images_test}. Table~\ref{tab:affine_result} shows improved quality of representations learned by a WideResNet-28-10 model regularized by \textit{PatchUp}{} on CIFAR-100 deformed test sets. The significant improvements in generalization provided by \textit{PatchUp}{} in this experiment shows the high quality of representations learned with \textit{PatchUp}{}.

\section{Robustness to Adversarial Examples}
\label{appendix:adversarial-attack}
The adversarial attacks refer to small and unrecognizable perturbations on the input images that can mislead deep learning models \cite{goodfellow2014FGSM, goodfellow2016deep}.
One approach to creating adversarial examples is using the Fast Gradient Sign Method (FGSM), also known as a white-box attack~\cite{goodfellow2014FGSM}. FGSM creates examples by adding small perturbations to the original examples. Once a regularized model is trained then FSGM creates adversarial examples as follows \cite{goodfellow2014FGSM}:
\begin{equation}
\begin{array}{l}
\label{eq:fgsm_perturbation}
    x^{\prime}= x + \epsilon \times \operatorname{sign}\left(\nabla_{\boldsymbol{x}} J(\boldsymbol{\theta}, \boldsymbol{x}, y)\right).
\end{array}
\end{equation}
where $x^{\prime}$ is an adversarial example, $x$ is the original example, $y$ is the ground truth label for $x$, and $J(\boldsymbol{\theta}, \boldsymbol{x}, y)$ is the loss of the model with parameters of $\theta$. $\epsilon$ controls the perturbation.

Our experiments show the effectiveness of \textit{Soft PatchUp}{} against the attacks in most cases. However, \textit{Hard PatchUp}{} performed well against the FGSM attack only on PreActResNet34 for CIFAR-100. Figure~\ref{fig:appendix_adv_attack} shows the comparison of the state-of-the-art regularization techniques' effect on model robustness against the FGSM attack.

\begin{figure}[bt!]
\centering
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c10_p18.png}
	\caption{Comparison on PreActResNet18 for CIFAR-10.}
	\label{fig:appendix_attack_c10_p18}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c100_p18.png}
	\caption{Comparison on PreActResNet18 for CIFAR-100.}
	\label{fig:appendix_attack_c100_p18}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c10_p34.png}
	\caption{Comparison on PreActResNet34 for CIFAR-10.}
	\label{fig:appendix_attack_c10_p34}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c100_p34.png}
	\caption{Comparison on PreActResNet34 for CIFAR-100.}
	\label{fig:appendix_attack_c100_p34}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c10_wrn.png}
	\caption{Comparison on WideResNet28-10 for CIFAR-10.}
	\label{fig:appendix_attack_c10_wrn}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_c100_wrn.png}
	\caption{Comparison on WideResNet28-10 for CIFAR-100.}
	\label{fig:appendix_attack_c100_wrn}
	\end{subfigure}
\hfill
\begin{subfigure}{0.47\linewidth}
	\includegraphics[width=.9\linewidth]{pics/attack_svhn_wrn.png}
	\caption{Comparison on WideResNet28-10 for SVHN.}
	\label{fig:appendix_attack_svhn_wrn}
\end{subfigure}
\caption{Robustness to the FGSM attack, known as a white-box attack. We repeated each test for five trained models to report the mean and the standard deviation of the accuracy of each method against the FGSM attack. The higher values on the y-axes show the robustness of the model against the attack. And, $\epsilon$ is the magnitude that controls the perturbation. }
\label{fig:appendix_adv_attack}
\end{figure}

\section{Analysis of \textit{PatchUp}{}'s Effect on Activations}
\label{appendix:activation_comparison}
In our implementation WideResNet28-10 has a conv2d module followed by three residual blocks. Figure~\ref{fig:appendix_activation_cmp} illustrates the comparison of ManifoldMixup, cutout, CutMix, \textit{Soft PatchUp}{}, and \textit{Hard PatchUp}{}. Figure~\ref{fig:appendix_activation_conv1},~\ref{fig:appendix_activation_layer1}, and~\ref{fig:appendix_activation_layer2} show that \textit{PatchUp}{} produces more variety of features in layers that we apply \textit{PatchUp}{} on.
\begin{figure}[tb!]
\centering
\begin{subfigure}{0.47\textwidth}
	\includegraphics[width=1.\textwidth]{pics/act_cmp_conv1.png}
	\caption{Comparison on the first convolution module.}
	\label{fig:appendix_activation_conv1}
\end{subfigure}
\hfill
\begin{subfigure}{0.47\textwidth}
	\includegraphics[width=1.\textwidth]{pics/act_cmp_layer1.png} \caption{Comparison on 1st Residual Block.}
	\label{fig:appendix_activation_layer1}
\end{subfigure}
\hfill
\begin{subfigure}{0.47\textwidth}
	\includegraphics[width=1.\textwidth]{pics/act_cmp_layer2.png} \caption{Comparison on 2nd Residual Block.}
	\label{fig:appendix_activation_layer2}
\end{subfigure}
\hfill
\begin{subfigure}{0.47\textwidth}
	\includegraphics[width=1.\textwidth]{pics/act_cmp_layer3.png}
	\caption{Comparison on 3rd Residual Block.}
	\label{fig:appendix_activation_layer3}
\end{subfigure}
\hfill
\caption{The effect of the state-of-the-art regularization techniques on activations in WideResNet28-10 for CIFAR100 test set. Each curve is the magnitude of feature activations, sorted by descending value, and averaged over all test samples for each method. The higher magnitude shows a wider variety of the produced features by the model at each block.}
\label{fig:appendix_activation_cmp}
\end{figure}
\section{Conclusion}
\label{conclusion}
We presented \textit{PatchUp}{}, a simple and efficient regularizer scheme for CNNs that alleviates some of the drawbacks of the previous mixing-based regularizers. Our experimental results show that with the proposed approach, \textit{PatchUp}{}, we can achieve state-of-the-art results on image classification tasks across different architectures and datasets. Similar to previous mixing based approaches, our approach also has the advantage of avoiding any added computational overhead. The strong test accuracy achieved by \textit{PatchUp}{}, with no additional computational overhead, makes it particularly appealing for practical applications.

\section*{Acknowledgments}

We would like to acknowledge Compute Canada and Calcul Quebec for providing computing resources used in this work.
The authors would also like to thank Damien Scieur, Hannah Alsdurf, Alexia Jolicoeur-Martineau, and Yassine Yaakoubi for reviewing the manuscript.
SC is supported by a Canada CIFAR AI Chair and an NSERC Discovery Grant.
\section{Relation to Other Methods}
\label{discussion}
\begin{figure}[htbp]
\centering
\begin{minipage}{.49\textwidth}
  \centering
  \includegraphics[width=.47\linewidth]{pics/manifoldmixup.png} \ \
    \includegraphics[width=.47\linewidth]{pics/blockmix.png}
    \caption{Left: ManifoldMixup interpolated samples for any combination of the three blue hidden states selected only from along orange line. Right: \textit{PatchUp}{} can produce interpolated hidden representations for these three hidden states in almost all possible places in all dimensions except the samples which lie directly on the orange lines.}
    \label{fig:diagram_mix_3d}
\end{minipage}%
\hfill
\begin{minipage}{.49\textwidth}
    \centering
\includegraphics[width=.481\linewidth]{pics/cat_2.png} \hfill \ \
  \includegraphics[width=.481\linewidth]{pics/dog_2.png}
    \caption{The two possible block selections from \text{CutMix}{} for two samples (cat and dog) with a large background. Swapping a similar part of the background or an essential element correlated to the label in the selected images can have a negative effect on the \text{CutMix}{} learning objective.}
    \label{fig:cat_dog}
\end{minipage}%
\end{figure}
\paragraph{\textit{PatchUp}{} Vs. ManifoldMixup:}\label{para:patchup_vs_manifold}Both \textit{PatchUp}{} and ManifoldMixup try to improve the generalization of a model by combining the latent representations of a pair of examples. ManifoldMixup combines two hidden representations by using the mixing operation defined in Equation~\ref{equ:mix_lam} which produces a new latent representation in a linear way for a pair of two hidden representations. \textit{PatchUp}{} uses a more complex approach to find a combination of two hidden representations, ensuring that a more diverse subspace of the hidden space gets explored. To understand the behaviour and the limitation that exist in the ManifoldMixup, assume that we have a 3D hidden space representation as illustrated in figure~\ref{fig:diagram_mix_3d}. Figure~\ref{fig:diagram_mix_3d} presents the possible combinations of hidden representations explored via ManifoldMixup and \textit{PatchUp}{}. Blue dots represent real hidden representation samples. ManifoldMixup can produce new samples that lie directly on the orange lines which connect the blue point pairs due to its linear interpolation strategy. On the other hand, \textit{PatchUp}{} can select various points in all dimensions, and can also select points extremely close to the orange lines. The proximity to the orange lines depends on the selected pairs and $\lambda$ sampled from the beta distribution. Appendix-\ref{appendix:sim_comparison} provides the mathematical and experimental justifications to show the comparative advantage of \textit{PatchUp}{} over ManifoldMixup.
\paragraph{\textit{PatchUp}{} Vs. \text{CutMix}{}:}The \text{CutMix}{} strategy is to cut and fill some parts of the selected pairs instead of using interpolation for creating a new sample in the input space. Therefore, the \text{CutMix}{} method has less potential for a manifold intrusion problem, however, \text{CutMix}{} may still suffer from a manifold intrusion problem. Figure~\ref{fig:cat_dog} shows two samples with small portions that correspond to their labels. \text{CutMix}{} cuts and fills the rectangular parts of the selected image randomly. In this example, if only the parts within the yellow bounding boxes are swapped, then the label does not change. However, if the parts within the white bounding boxes are swapped, then the entire label is swapped. In both scenarios, \text{CutMix}{} only learns the interpolated target based on the fraction of the images that is swapped. In contrast, these scenarios cannot occur in \textit{PatchUp}{} since it works in the hidden representation space. Another difference between \text{CutMix}{} and \textit{PatchUp}{} is how the masks are created. \textit{PatchUp}{} can create arbitrarily shaped masks while \text{CutMix}{} masks can only be rectangular. Figure~\ref{fig:mask_cmp} shows an example of \text{CutMix}{} Mask and \textit{PatchUp}{} mask in input space and hidden representation space, respectively.

Feature-CutMix applies \text{CutMix}{} in the latent space. According to \citet{cutmix}, Feature-CutMix is not as effective as \text{CutMix}{}. Both the learning objective of \textit{PatchUp}{}, as well as the binary mask selection are different from Feature-CutMix.
\section{Experiments}
\label{experiments}
To evaluate the generalization improvements that \textit{PatchUp}{} can provide with either \textit{Hard} or \textit{Soft PatchUp}{}, we applied \textit{PatchUp}{} to image classification tasks on CIFAR-10, CIFAR-100~\cite{dataset:cifar}, and SVHN~\cite{netzer2011reading} datasets with PreActResNet18, PreActResNet34, and WideResNet-28-10 models\footnote{The code to reproduce all the results is available at \href{https://github.com/chandar-lab/PatchUp}{\textcolor{WildStrawberry}{https://github.com/chandar-lab/PatchUp}}.}. We used the same set of base hyper-parameters for all the models to be able to compare and evaluate the generalization improvements due to different regularization methods in a fair way. Appendix-\ref{appendix:hyperparameters} explains the experiment setup and the hyper-parameter tuning processes in detail. \textit{PatchUp}{} adds $patchup\_prob$, $\gamma$ and $block\_size$ to the set of hyper-parameters. $patchup\_prob$ is the probability that \textit{PatchUp}{} is performed for a given mini-batch. We set $\alpha$ to $2$ in \textit{PatchUp}{}. And, based on the hyper-parameter tuning described in Appendix-\ref{appendix:hyperparameters}, \textit{Hard PatchUp}{} yields the best performance with $patchup\_prob$, $\gamma$, and $block\_size$ as $0.7$, $0.5$, and $7$, respectively.
\textit{Soft PatchUp}{} achieves the best performance with $patchup\_prob$, $\gamma$, and $block\_size$ as $1.0$, $0.75$, and $7$, respectively.
\import{tables/}{cifar.tex}

\import{tables/}{svhn.tex}
\subsection{Generalization on Image Classification}
Table~\ref{tab:tbl-cifar} shows the comparison of the generalization performance of \textit{PatchUp}{} with five other state-of-the-art regularization techniques on the CIFAR-10 and CIFAR-100 datasets. Our experiments show that \textit{Hard PatchUp}{} leads to a lower test error for all the models on both CIFAR-10 and CIFAR-100. \textit{Soft PatchUp}{} has the second-best result for this task. From Table~\ref{tab:tbl-cifar}, we see that \textit{PatchUp}{} provides a significant improvement over previous state-of-the-art regularization techniques across different architectures. We observed that \textit{Soft PatchUp}{} has the second-best performance across all the architectures indicating the potential that \textit{Soft PatchUp}{} is beneficial in fine-grained classification. Table~\ref{tab:tbl-svhn} shows that \textit{Hard PatchUp}{} achieves the best top-1 error across the different models. Our experiments show that ManifoldMixup has the second-best performance for PreActResNet18 on SVHN. \textit{Soft PatchUp}{} also achieves the second-best performance for WideResNet-28-10 on SVHN. It is also worth noting that \textit{Soft PatchUp}{} performs reasonably well and is comparable to ManifoldMixup for PreActResNet34 on SVHN.
\import{tables/}{affine_p34.tex}

\subsection{Generalization on Deformed Images}
\label{exp:deformed_images_test}
Regularization methods aim to improve the generalization of a model to unseen data. Applying affine transformations on the test set can provide novel deformed data that can be used to evaluate and compare the minimality and sufficiency of the representations learned by models with state-of-the-art regularization techniques \cite{manifold}. We trained PreActResNet34 and WideResNet-28-10 on the CIFAR100 dataset. And then, we created deformed test sets from CIFAR100 by applying random rotations, random shearings, and different rescalings. Table~\ref{tab:affine_p34_result} shows that \textit{PatchUp}{} provides the best performance on affine transformed test sets and better generalization in PreActResNet34. Table~\ref{tab:affine_result} in Appendix-\ref{appendix:affine_transformation} illustrates that the quality of representations is improved by \textit{PatchUp}{} and it also shows better generalization in deformed test sets on WideResNet-28-10. Generalization is significantly improved by \textit{PatchUp}{}, as are the quality of representations learned by \textit{PatchUp}{} as demonstrated by this experiment.
\begin{figure}[tb]
\centering
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=.95\linewidth]{pics/attack_c100_p18.png}
	\caption{Comparison on PreActResNet18 for CIFAR-100.}
	\label{fig:attack_c100_p18}
	\end{subfigure}
\hfill
\begin{subfigure}{0.48\linewidth}
	\includegraphics[width=.95\linewidth]{pics/attack_svhn_wrn.png}
	\caption{Comparison on WideResNet28-10 for SVHN.}
	\label{fig:attack_svhn_wrn}
	\end{subfigure}
\caption{Robustness to FGSM attack. Plots are based on the mean and standard deviation of the accuracy of five trained models for each method against FGSM attack. The x-axis represents $\epsilon$ which is the magnitude that controls the perturbation.}
\label{fig:adv_attack}
\end{figure}
\begin{figure}[tb]
\centering
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/act_cmp_layer1.png}
	\caption{Comparison on 1st Residual Block.}
	\label{fig:activation_layer1}
	\end{subfigure}
\hfill
\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=1.\linewidth]{pics/act_cmp_layer2.png}
	\caption{Comparison on 2nd Residual Block.}
	\label{fig:activation_layer2}
	\end{subfigure}
\caption{The effect of the state-of-the-art regularization techniques on activations in WideResNet28-10 for CIFAR100 test set on the first and second Residual Blocks. Each curve is the magnitude of the feature activations, sorted in descending value, and averaged over all test samples for each method. The higher magnitude indicates a wider variety of the produced features by the model at each block.}
\label{fig:activation_cmp}
\end{figure}
\subsection{Robustness to Adversarial Examples}
Since neural networks are trained based on Empirical Risk Minimization (ERM), slight changes in the data distribution have a significant effect on the model performance~\cite{szegedy_Zaremba, mixup}. Such unseen data used to confuse the models are known as adversarial examples. Certain data-dependent regularization techniques can alleviate such fragility to adversarial examples by training the models with interpolated data. Therefore, the robustness of a regularized model to adversarial examples can be considered as a criterion for comparison~\cite{mixup, manifold, cutmix}. To evaluate the robustness of \textit{PatchUp}{} against adversarial attacks, we compared the performance of PreActResNet18, PreActResNet34, and WideResNet-28-10 on CIFAR-10 and CIFAR-100 with adversarial examples created by the FGSM attack described in~\cite{goodfellow2014FGSM}. We compared the performance of WideResNet28-10 for SVHN against the FGSM attack. Figure~\ref{fig:adv_attack} shows the robustness of \textit{PatchUp}{} against FGSM attack. Figure~\ref{fig:appendix_adv_attack} in the appendix shows further comparisons. Based on the results, we can see that \textit{Soft PatchUp}{} is more robust to adversarial attacks when compared to other regularization methods. \textit{Hard PatchUp}{} and ManifoldMixup achieve the second-best performance in most experiments. While \textit{Hard PatchUp}{} achieves better performance in terms of classification accuracy, \textit{Soft PatchUp}{} seems to trade-off a slight loss of accuracy in order to achieve more robustness.

\subsection{Effect on Activations}
To study the effect of the state-of-the-art regularization techniques on the activations in the residual blocks, we compared the mean magnitude of feature activations in the residual blocks following~\cite{cutout} in WideResNet28-10 for the test set in CIFAR-100. We first train the models with regularization techniques and then calculate the magnitudes of activations in the validation set. The higher mean magnitude of features shows that the models tried to produce a wider variety of features in the residual blocks~\cite{cutout}. Our WideResNet28-10 has a conv2d module followed by three residual blocks. For this ablation study, we selected $k$ randomly such that $k\in \{0, 1, 2, 3\}$. Therefore, we apply the ManifoldMixup and \textit{PatchUp}{} in either input space, first conv2d, first residual block, or second residual block. Figure~\ref{fig:activation_cmp} illustrates the comparison of ManifoldMixup, cutout, CutMix, \textit{Soft PatchUp}{}, and \textit{Hard PatchUp}{}.
Figures~\ref{fig:activation_layer1} and~\ref{fig:activation_layer2} illustrate that \textit{PatchUp}{} produces more diverse features in the layers where we apply \textit{PatchUp}{}. In Appendix-\ref{appendix:activation_comparison}, figure~\ref{fig:appendix_activation_cmp} shows this ablation study results in first conv2d, first residual block, second residual block and third residual block. Since we are not applying the \textit{PatchUp}{} in the third residual block, the mean magnitude of the feature activations are below, but very close to, cutout and \text{CutMix}{}. This experiment also shows that producing a wide variety of features can be an advantage for a model. However, according to our experiments, a larger magnitude of activations does not always mean better performance. Figure~\ref{fig:activation_cmp} shows that for ManifoldMixup, the mean magnitude of the feature activations is less than other approaches. But, it performs better than cutout and \text{CutMix}{} in image classification, affine transformations, and FGSM attacks.

\subsection{Significance of loss terms}
\label{appendix:patchup_loss}
\textit{PatchUp}{} uses the loss that is introduced in Equation~\ref{eq:swap_loss}. We can paraphrase the \textit{PatchUp}{} learning objective for this ablation study as follow:
\begin{equation}
\begin{aligned}
\label{eq:swap_losss}
    L(f) = &\expectation_{(x_i,\;y_i) \sim P}\,
    \expectation_{(x_j,\;y_j) \sim P}\,
    \expectation_{\lambda \sim \text{Beta}(\alpha,\;\alpha)}
    \expectation_{k \sim \mathcal{S}} ~~(L_1 + L_2) ,
\end{aligned}
\end{equation}
where
\begin{align}
L_{1} &= \text{Mix}_{p_{u}} [ \ell(f_{k}(\phi_k), y_i), \ell(f_{k}(\phi_k), Y)] ,\\
L_{2} &= \ell(f_{k}(\phi_k), W(y_i, y_j)) ,
\end{align}

This section is an ablation study to show the effect of $L_{1}$ and $L_{2}$ in \textit{PatchUp}{} loss. Table~\ref{tab:loss_study} shows the error rate on the validation set for WideResNet-28-10 on CIFAR-100. This study shows that the summation of the $L_{1}$ and $L_{2}$ reduces error rate by $.1\%$ in both \textit{Soft PatchUp}{} and \textit{Hard PatchUp}{}.

{\renewcommand{\arraystretch}{1.1}
\begin{table*}[hbt!]
\centering
    \captionsetup[subtable]{position = top}
    \captionsetup[table]{position=top}
    \caption{The error rate on the validation set on CIFAR-100 for WideResNet-28-10 with \textit{Hard PatchUp}{} and \textit{Soft PatchUp}{}. The result is the mean and standard deviation of the experiment for five runs. A smaller number indicates better performance.}
    \label{tab:loss_study}
    \begin{subtable}{.9\linewidth}
        \resizebox{\linewidth}{!}{
        \begin{tabular}{llll}
        \toprule
        Simple WideResNet-28-10& & & Error Rate:  \(23.256 \pm 0.586\) \\
        \toprule
        {}     & Error rates with $L1$ & Error rates with $L2$ & Error rates with $L (f)$ \\ \cmidrule(lr){2-4}
        \textit{Soft PatchUp} & \(16.856 \pm 0.666\)   & \(16.865 \pm 0.339\)   &  \(\mathbf{16.75 \pm 0.291}\)\\
        \cmidrule(lr){2-4}
        \textit{Hard PatchUp} & \(16.135 \pm 0.229\)   & \(16.79 \pm 0.457\)   &  \(\mathbf{16.02 \pm 0.358}\) \\

        \bottomrule
        \end{tabular}}
    \end{subtable}
\end{table*}
}

\section{Introduction}
\label{introduction}
Deep Learning (DL), particularly deep Convolutional Neural Networks (CNNs) have achieved exceptional performance in many machine learning tasks, including object recognition~\cite{krizhevsky}, image classification~\cite{krizhevsky,NIPS2015_Ren,Kaiming}, speech recognition~\cite{hinton} and natural language understanding \cite{seq2seq, transformers}. However, in a very deep and wide network, the network has a tendency to memorize the samples, which yields poor generalization for data outside of the training data distribution~\cite{arpit2017closer, goodfellow2016deep}. To address this issue, noisy computation is often employed during the training, making the model more robust against invariant samples and thus improving the generalization of the model~\cite{information_dropout}. This idea is exploited in several state-of-the-art regularization techniques.

Such noisy computation based regularization techniques can be categorized into data-dependent and data-independent techniques~\cite{DBLP:Hongyu}. Earlier work in this area has been more focused on the data-independent techniques such as Dropout~\cite{article:dropout}, SpatialDropout~\cite{DBLP:spatial_dropout}, and DropBlock~\cite{dropblock}. Dropout performs well on fully connected layers~\cite{JMLR:v15:srivastava14a}. However, it is less effective on convolutional layers~\cite{jonathan}. One of the reasons for the lack of success of dropout on CNN layers is perhaps that the activation units in the convolutional layers are correlated, thus despite dropping some of the activation units, information can still flow through these layers. SpatialDropout~\cite{jonathan} addresses this issue by dropping the entire feature map from a convolutional layer. DropBlock~\cite{dropblock} further improves SpatialDropout by dropping random continuous feature blocks from feature maps instead of dropping the entire feature map in the convolutional layers.

Recent works show that data-dependent regularizers can achieve better generalization for CNN models. Mixup~\cite{mixup}, one such data-dependent regularizer, synthesizes additional training examples by interpolating random pairs of inputs $x_i$, $x_j$ and their corresponding labels $y_i$, $y_j$ as:
\begin{equation}
\begin{array}{ll}{\tilde{x}=\lambda x_{i}+(1-\lambda) x_{j}} \text{\quad and \quad} {\tilde{y}=\lambda y_{i}+(1-\lambda) y_{j}} \end{array}
\end{equation}
where $\lambda \in [0,1]$ is sampled from a Beta distribution such that \(\lambda \sim Beta(\alpha, \alpha)\) and ($\tilde{x},\;\tilde{y}$) is the new example. By using these types of synthetic samples, Mixup encourages the model to behave linearly in-between the training samples.

The mixing coefficient $\lambda$ in Mixup is sampled from a prior distribution. This may lead to the \textit{manifold intrusion problem}~\cite{DBLP:Hongyu}: the mixed synthetic example may \textit{collide} (i.e. have the same value in the input space) with other examples in the training data, essentially leading to two training samples which have the same inputs but different targets. To overcome the manifold intrusion problem, \citet{mai2019metamixup} used a meta-learning approach to learn  \(\lambda\) with a lower possibility of causing such collisions. However, this meta-learning approach adds significant computation complexity. ManifoldMixup~\cite{manifold} attempts to avoid the manifold intrusion problem by interpolating the hidden states (instead of input states) of a randomly chosen layer at every training update.

Different from the interpolation based regularizers discussed above, cutout~\cite{cutout} drops the contiguous regions from the image in the input space. This kind of noise encourages the network to learn the full context of the images instead of overfitting to the small set of visual features.
CutMix~\cite{cutmix} is another data-dependent regularization technique that cuts and fills rectangular shape parts from two randomly selected pairs in a mini-batch instead of interpolating two selected pairs completely. Applying \text{CutMix}{} at the input space improves the generalization of the CNN model by spreading the focus of the model across all places in the input instead of just a small region or a small set of intermediate activations. \text{CutMix}{} also improves the generalization performance of a very deep and wide CNN model such as PyramidNet. According to the \text{CutMix}{} paper, applying \text{CutMix}{} at the latent space, Feature \text{CutMix}{}, is not as effective as applying \text{CutMix}{} in the input space~\cite{cutmix}.

In this work, we propose \textit{PatchUp}{} which is a regularization technique that operates in the hidden space by masking out contiguous blocks of the feature map of a random pair of samples, and then either mixes (\textit{Soft PatchUp}{}) or swaps (\textit{Hard PatchUp}{}) these selected contiguous blocks.
Our experiments verify that \textit{Hard PatchUp}{} achieves a better generalization performance compared to other state-of-the-art regularization techniques for CNNs such as Mixup, cutout, CutMix and ManifoldMixup on CIFAR-10, CIFAR-100, and SVHN datasets. \textit{Soft PatchUp}{} achieves the second-best performance on CIFAR-10, CIFAR-100 with PreactResnet18, PreactResnet34, and WideResnet-28-10 models while achieving comparable results to ManifoldMixup on SVHN with PreactResnet18 and PreactResnet34. Furthermore, \textit{PatchUp}{} provides significant improvements in the generalization on deformed images and better robustness against Fast Gradient Sign Method (FGSM) adversarial attack.

\section{PatchUp}
\label{patchup}
\textit{PatchUp}{} is a hidden state block-level regularization technique that can be used after any convolutional layer in CNN models. Given a deep neural network $f(x)$ where $x$ is the input, let $g_k$ be the $k$-th convolutional layer. The network $f(x)$ can be represented as $f(x) = f_k(g_k(x))$ where $g_k$ is the mapping from the input data to the hidden representation at layer $k$ and $f_k$ is the mapping from the hidden representation at layer $k$ to the output \cite{manifold}. In every training step, \textit{PatchUp}{} applies block-level regularization at a randomly selected convolutional layer $k$ from a set of intermediate convolutional layers. Appendix-\ref{appendix:random_k_selction} gives a formal intuition for selecting $k$ randomly.
\subsection{Binary Mask Creation}
\label{binary_mask}
\begin{figure}[tbp]
\centering
	\includegraphics[width=\textwidth]{pics/patchup_process.png}
\caption{\textit{PatchUp}{} process for two hidden representations associated with two samples randomly selected in the mini-batch ($a,\;b)$. $X_{1} = g_{k}^{(i)}(a)$ and $X_{2} = g_{k}^{(i)}(b)$ where $i$ is the feature map index. Right top shows \textit{Hard PatchUp}{} output and the right bottom shows the interpolated samples with \textit{Soft PatchUp}{}. The yellow continuous blocks represent the interpolated selected blocks.}
\label{fig:patchup_process}
\end{figure}
Once a convolutional layer $k$ is chosen, the next step is to create a binary mask $M$ (of the same size as the feature map in layer $k$) that will be used to \textit{PatchUp}{} a pair of examples in the space of $g_k(x)$. The mask creation process is similar to that of DropBlock~\cite{dropblock}. The idea is to select contiguous blocks of features from the feature map that will be either mixed or swapped with the same features in another example. To do so, we first select a set of features that can be altered (mixed or swapped). This is done by using the hyper-parameter $\gamma$ which decides the probability of altering a feature. When we alter a feature, we also alter a square block of features centered around that feature which is controlled by the side length of this square block, $block\_size$. Hence, the altering probabilities are readjusted using the following formula \cite{dropblock}:
\begin{equation}
    \label{equ:gamma}
    \gamma_{adj} = \frac{\gamma \times \textit{(feature map's area)}}{\textit{(block's area)} \times \textit{(valid region to build block)}}.
\end{equation}
where the area of the feature map and block are the $feat\_size^2$ and $block\_size^2$, respectively, and the valid region to build the block is $(feat\_size - block\_size + 1)^2$.

For each feature in the feature map, we sample from $Bernoulli(\gamma_{adj})$. If the result of this sampling for feature $f_{ij}$ is 0, then $M_{ij} = 1$. If the result of this sampling for $f_{ij}$ is 1, then the entire square region in the mask with the center $M_{ij}$ and the width and height of the square of $block\_size$ is set to 0. Note that these feature blocks to be altered can overlap which will result in more complex block structures than just squares. The block structures created are called patches. Figure-\ref{fig:patchup_process} illustrates an example mask used by \textit{PatchUp}{}. The mask $M$ has 1 for features outside the patches (which are not altered) and 0 for features inside the patches (which are altered).
\subsection{\textit{PatchUp}{} Operation}

Once the mask is created, we can use the mask to select patches from the feature maps and either swap these patches (\textit{Hard PatchUp}{}) or mix them (\textit{Soft PatchUp}{}).

Consider two samples $x_i$ and $x_j$. The \textit{Hard PatchUp}{} operation at layer $k$ is defined as follows:
\begin{equation}
       \mathbf{\phi_{hard}}(g_k(x_i), g_k(x_j)) = \mathbf{M} \odot g_k(x_i) + (\mathbf{1}- \mathbf{M}) \odot g_k(x_j),
\end{equation}
where $\odot$ is known as the element-wise multiplication operation and $\mathbf{M}$ is the binary mask described in section ~\ref{binary_mask}.

To define \textit{Soft PatchUp}{} operation, we first define the mixing operation for any two vectors $a$ and $b$ as follows:
\begin{equation}
\label{equ:mix_lam}
\text{Mix}_\lambda(a,\;b) = \lambda \cdot a + (1 - \lambda) \cdot b,
\end{equation}
where $\lambda \in[0,1]$ is the mixing coefficient. Thus, the \textit{Soft PatchUp}{} operation at layer $k$ is defined as follows:
\begin{equation}
    \mathbf{\phi_{soft}}(g_k(x_i), g_k(x_j)) = \mathbf{M} \odot g_k(x_i) + \text{Mix}_\lambda[((\mathbf{1}- \mathbf{M}) \odot g_k(x_i)),\;((\mathbf{1}- \mathbf{M}) \odot g_k(x_j))].
\end{equation}
where \(\lambda\) in the range of $[0,\;1]$ is sampled from a Beta distribution such that \(\lambda \sim Beta(\alpha, \alpha)\). $\alpha$ controls the shape of the Beta distribution. Consequently, it controls the strength of interpolation \cite{mixup}. Both \textit{PatchUp}{} operations are illustrated in Figure~\ref{fig:patchup_process}.
\subsection{Learning Objective}

After applying the \textit{PatchUp}{} operation, the CNN model continues the forward pass from layer $k$ to the last layer in the model. The output of the model is used for the learning objective, including the loss minimization process and updating the model parameters accordingly.

Again, consider the example pairs $(x_i, y_i)$ and $(x_j, y_j)$. Let $\phi_k = \phi(g_k(x_i), g_k(x_j))$ be the output of \textit{PatchUp}{} after the $k$-th layer. Mathematically, the CNN with \textit{PatchUp}{} minimizes the following loss function:
\begin{equation}
\begin{aligned}
\label{eq:swap_loss}
    L(f) = &\expectation_{(x_i,\;y_i) \sim P}\,
    \expectation_{(x_j,\;y_j) \sim P}\,
    \expectation_{\lambda \sim \text{Beta}(\alpha,\;\alpha)}
    \expectation_{k \sim \mathcal{S}}\\
    &~~~~\text{Mix}_{p_{u}} [ \ell(f_{k}(\phi_k),\;y_i) \; , \ell(f_{k}(\phi_k),\;Y)]\; +
    \ell(f_{k}(\phi_k),\; W(y_i,\;y_j)).
\end{aligned}
\end{equation}
where $p_{u}$ is the fraction of the unchanged features from feature maps in $g_{k}(x_i)$ and $\mathcal{S}$ is the set of layers where \textit{PatchUp}{} is applied randomly. $\phi$ is $\phi_{hard}$ for \textit{Hard PatchUp}{} and $\phi_{soft}$ for \textit{Soft PatchUp}{}.

$Y$ is the target corresponding to the changed features. In the case of \textit{Hard PatchUp}{}, $Y = y_j$ and in the case of \textit{Soft PatchUp}{}, $Y = \textit{Mix}_{\lambda}(y_i, y_j)$. $W(y_i, y_j)$ calculates the re-weighted target according to the interpolation policy for $y_i$ and $y_j$. $W$ for \textit{Hard PatchUp}{} and \textit{Soft PatchUp}{} is defined as follows:
\begin{align}
W_{hard}(y_i, y_j) &= \text{Mix}_{p_{u}}(y_i, y_j).    \\
W_{soft}(y_i, y_j) &= \text{Mix}_{p_{u}}(y_i,\;\text{Mix}_{\lambda}(y_i, y_j)).
\label{eq:target_reweighted}
\end{align}

The \textit{PatchUp}{} loss function has two terms where the first term is inspired from the CutMix loss function and the second term is inspired from the MixUp loss function.
\subsection{\textit{PatchUp}{} in Input Space}
When $k = 0$, \textit{PatchUp}{} only gets applied to the input space. To apply \textit{PatchUp}{} to the input space, only the \textit{Hard PatchUp}{} operation is used since swapping in the input space provides better generalization compared to mixing~\cite{cutmix}. Only one random rectangular patch is selected in the input space (similar to CutMix) because the \textit{PatchUp}{} binary mask is too strong for the input space, which has only three channels, compared to hidden layers in which each layer has numerous channels.


\section{Introduction}
The complex Ginzburg-Landau equation is a canonical model
for weakly nonlinear, dissipative systems and one of the
most-studied nonlinear equations in the physics community. It can
be used to describe a vast variety of nonlinear phenomena such as
superconductivity \cite{A.M.}, Bose-Einstein condensation
\cite{E.Kengne}, superfluidity \cite{A. Berti}, strings in field
theory \cite{R.J. Rivers}, liquid crystals \cite{M. C.} and lasers
\cite{M.S. Osman,Osman2}. Exact solitary pulse solutions of complex Ginzburg-Landau equation with the cubic nonlinearity are available, but these
pulses are stable only in a region where the background is
unstable \cite{akh}. However, it is possible to create stable
solitary pulses, in a region where background is also stable, with the
inclusion of an extra term, represents delayed Raman
scattering, in the model \cite{Facao3,facao4}. The existence and stability of solitary wave solutions for cubic complex Ginzburg-Landau equation is also studied in the presence of driven term \cite{malomed,raju,amit}. Recently, lot of attention has been paid to obtain exact analytical solutions for complex systems modeled by nonautonomous partial differential equations \cite{osman01,osman02,osman03,osman04,osman05}.

\par Soliton propagation in fibers with linear and nonlinear gain and spectral filtering \cite{4M. Mats} or pulse generation in fiber lasers with additive pulse mode-locking or nonlinear polarization rotation \cite{5H.A.,6J.D.} have been studied by considering complex cubic-quintic Ginzburg-Landau equation (CQGLE) as model equation. Also Kengne and Vaillancourt have used modified Ginzburg–Landau equation that describes the pulse propagation in a lossy electrical transmission
line \cite{kengne}. The CQGLE supports a class of localized solutions such as stationary solitons, sources, sinks, moving solitons and fronts with fixed velocity \cite{saarlos,akh2,crespo}. Apart from these solutions, the CQGLE also possesses the solutions with special propagation properties: pulsating, creeping, and erupting solitons \cite{J.Soto2,2001PRE,2001PLA}. The erupting solitons are those that periodically exhibits explosive instability. These solitons were found numerically \cite{J.Soto2} and also experimentally in passively mode-locked lasers \cite{2002PRL}. The effect of higher-order terms, namely, third-order dispersion, self-steepening and intrapulse Raman scattering has been investigated on erupting solitons and it is found that the explosions of an erupting soliton can be controlled or even canceled due to the inclusion of one or more higher-order terms \cite{tian,latas,M.Facao1,M.Facao2,carvalho,latas2}. Recently, work has been done to study the transitions of stationary to pulsating solutions \cite{2018} and on the selection mechanism of soliton explosions in CQGLE under the influence of higher-order terms \cite{2019}. Fac\~{a}o et al. have studied the effect of intrapulse Raman scattering (IRS) on erupting solitons of CQGLE and numerically shown the propagation of stable traveling solitons for a specific range of IRS parameter \cite{M.Facao1,M.Facao2}. Although, CQGLE is a well studied dynamical system, the exact solutions of CQGLE with IRS term have rarely appeared in the literature. However, in Ref. \cite{anki}, the authors have presented the exact stationary front solutions for CQGLE with Raman term and generalized these solutions into moving fronts using energy and momentum balance equations for particular cases by assuming either the quintic or Raman term to be zero.

\par In this work, we consider the CQGLE in the presence of IRS and report the existence of exact localized  solutions in the form of dark and front solitons. The dark solitons are presented by a new kind of kink solution in terms of Lambert W function which we shall refer to as Lambert W-kink solitons. The Lambert W function is an implicitly elementary function, also
known as the product logarithm, has rich variety of applications in
number of areas of physics, computer science, pure and applied
mathematics and ecology, and is defined as the inverse of $f(W)=W e^W$ \cite{R.M.,S.R.}. Several
well-known problems in electrostatics and in quantum mechanics can
be solved with greater ease using the notation of Lambert W function. Biswas et al. used the notation of Lambert W function to obtain soliton solutions of modified nonlinear Schr\"odinger equation using variational principle \cite{Biswas2}. This function is also used as a step potential for which the one-dimensional stationary Schr\"odinger equation is exactly solved in terms of the confluent hypergeometric functions \cite{potential}. Recently, soliton solution in the form of Lambert W function was obtained for analytically solvable parity-breaking $\phi^6$ model and the results so obtained were compared with kink of $\phi^4$ theory \cite{Azadeh}. Apart from Lambert W-kink solitons, we have also explored moving front solitons for this model. The evolution of optical solitons can be controlled by judicious choice of model parameters. The frequency chirp is found to be directly proportional to the
intensity of the wave and saturates at some
finite value as $t\rightarrow\pm\infty$. Frequency chirp is a well-known result of the
interaction of the group velocity dispersion and the nonlinear self phase modulation.
Chirp is very useful in the process of optical pulse compression and found potential applications in optical communication systems \cite{chirp1,chirp2,chirp3}. A significant work has been done on the existence of chirped solitons in the context of nonlinear optics \cite{chirp4,chirp5,Alka,goyal}.

 \section{Model Equation}
We begin our analysis by considering the complex cubic-quintic
Ginzburg-Landau equation (CQGLE) with intrapulse Raman scattering (IRS) term
    \begin{equation}\label{eq1} i U_z+\frac{1}{2} U_{tt}+\gamma\vert U \vert^2 U=i\delta U+i \beta U_{tt}+i \epsilon\vert U\vert^2U-\nu\vert U \vert^4 U+i \mu \vert U \vert^4 U+T_r{(\vert U \vert^2)}_t U,\end{equation}
where $U$ is the normalized envelope of
the pulse, $z$ and $t$ are the normalized propagation distance
and retarded time, respectively. For laser system \cite{6J.D.}, the physical meaning of various coefficients is the following:
$\delta$ is a constant gain (or loss if negative), $\beta$ describes
spectral filtering or gain dispersion, $\epsilon$ represents
nonlinear gain (or two-photon absorption if negative), $\mu$ represents a
higher order correction to the nonlinear amplification or absorption, $\nu$ is a higher order correction term to the nonlinear refractive index, $T_r$ represents the IRS coefficient and $\gamma$ represents positive Kerr effect (or negative Kerr effect if negative) .

\section{Chirped soliton-like solutions}
In order to find the exact solution of Eq. (\ref{eq1}), we choose the following ansatz
 \begin{equation}\label{eq2} U(z,t)=\rho(\xi)e^{i(\phi(\xi) -k z)}, \end{equation}
where $\xi=t-u z$ is the traveling coordinate, $\rho$ and
$\phi$ are real functions of $\xi$. Here $u = \frac{1}{v}$, where $v$ indicates the group
velocity of the pulse envelope. The corresponding intensity of the propagating pulse is given by $|U(z,t)|^2=|\rho(\xi)|^2$. The spectral changes introduced across the pulse at any distance $z$ are a direct consequence of time dependence of nonlinear phase
shift. The frequency change across the pulse is the time
derivative of phase and is given by $\delta\omega(z,t)=-\frac{\partial}{\partial t}[\phi(\xi)-k
z]=-\phi'(\xi)$. This time dependence of $\delta\omega$ is referred to as frequency
chirping. Now, substituting Eq. (\ref{eq2}) into
Eq. (\ref{eq1}), and separating out the real and imaginary parts of the equation, we obtain
the following coupled equations in $\rho$ and $\phi$,
    \begin{equation}\label{eq3}u\rho\phi'+k\rho+\frac{1}{2}\rho''-\frac{1}{2}\rho\phi'^2+\gamma\rho^3=-2\beta\rho'\phi'-\beta\rho\phi''-\nu\rho^5+2T_r\rho^2\rho',\end{equation}
    \begin{equation}\label{eq4}-u\rho'+\rho'\phi'+\frac{1}{2}\rho\phi''=\delta\rho+\beta\rho''-\beta\rho\phi'^2+\epsilon\rho^3+\mu\rho^5.\end{equation}
Assuming that the qualitative features of frequency chirp depend considerably on the exact pulse shape through the relation $\delta\omega(z,t)=-\phi'(\xi)=-(A\rho^2+B)$, where A and B are the nonlinear and constant chirp parameters, respectively,
the coupled equations given by Eq. (3) and Eq. (4) reduce to

    \begin{equation}\label{eq6} \rho''+4(2\beta A-T_r)\rho^2 \rho'+4\beta B\rho'+(2uB-B^2+2k)\rho+2(u A+\gamma-A B)\rho^3+(2\nu-A^2)\rho^5=0,\end{equation}
    \begin{equation}\label{eq7} \rho''-\frac{2A}{\beta}\rho^2 \rho'+\frac{(u-B)}{\beta} \rho'+\(\frac{\delta}{\beta}-B^2\)\rho+\(\frac{\epsilon}{\beta}-2AB\)\rho^3+\(\frac{\mu}{\beta}-A^2\)\rho^5=0,\end{equation}
for $\beta \neq 0$. By assuming the following identifications:

\begin{equation}\label{eq8} M\equiv\(8\beta A-4T_r=-\frac{2A}{\beta}\),\end{equation}

\begin{equation}\label{eq9} N\equiv \(4\beta B=\frac{u-B}{\beta}\),\end{equation}

\begin{equation}\label{eq10} Q\equiv \(2uB+2k-B^2=\frac{\delta}{\beta}-B^2\),\end{equation}

\begin{equation}\label{eq11} R\equiv \(2Au-2AB+2\gamma=\frac{\epsilon}{\beta}-2AB\),\end{equation}

\begin{equation}\label{eq12} S\equiv \(2\nu-A^2=\frac{\mu}{\beta}-A^2\),\end{equation}

Eqs. (\ref{eq6}) and (\ref{eq7}) can be mapped into a single equation
    \begin{equation}\label{eq13}\rho''+M\rho^2\rho'+N\rho'+Q\rho+R\rho^3+S\rho^5=0.\end{equation}
Solving Eqs. (\ref{eq8})-(\ref{eq12}), we obtain the constraint conditions as
     \begin{equation}\label{eq14}\nonumber A=\frac{2\beta  T_r}{1+4\beta^2},~~u=\frac{1}{A}\left(\frac{\epsilon }{2\beta }-\gamma \right),~~B=\frac{u}{1+4\beta^2}\end{equation}
     \begin{equation}\label{eq15}k=\frac{\delta }{2\beta }-u B,~~\mu =2\beta \nu.\end{equation}
Eq. (\ref{eq13}) can be solved to obtain exact localized solution for compatible form of first-order differential equation for the function $\rho(\xi)$. In this work, we have explored the Lambert W-kink and moving front soliton solutions for this equation.
\subsection{Lambert W-kink solitons}
In order to explore exact analytical solution of Eq. (12), use shall be made of the differential equation
    \begin{equation}\label{eq17}\rho'=(a^2-\rho^2)~(a-\rho),\end{equation}
 ($a$ is a real parameter here), that admits Lambert W-kink solution of the form \cite{R.M.,Azadeh}
    \begin{equation}\label{eq21}\rho(\xi)=a\(1-\frac{2}{1+W(e^{4a^2\xi+1})}\),\end{equation}
where $W$ represents Lambert W function. The corresponding second-order differential equation for $\rho(\xi)$ reads
     \begin{equation}\label{eq18}\rho''-3\rho^5+5a\rho^4+2a^2\rho^3-6a^3\rho^2+a^4\rho+a^5=0. \end{equation}
For $\rho'$ given by Eq. (\ref{eq17}), Eq. (\ref{eq13}) is consistent with the Eq. (\ref{eq18}) by the identification of various unknown parameters as $M=-5,N=a^2,Q=2a^4,R=-4a^2,S=2$. Solving these conditions along with constraints given by Eq. (\ref{eq15}), the model coefficients and solution parameter $`a$' fixed as
    \begin{equation}\label{eq19}\nonumber T_r=\frac{5}{4}\left(1+4\beta ^2\right),~~\epsilon =\frac{11\beta \gamma} {2\left(4+5\beta^2\right)},~~\nu=1+\frac{25}{8}\beta^2,\end{equation}
    \begin{equation}\label{eq20} \delta=\frac{\gamma^2\left(1+32\beta^2\right)}{4\beta\left(4+5\beta^2\right)^2},~~a=\sqrt{\frac{-2\gamma}{4+5\beta ^2}}.\end{equation}

\begin{figure}[ht!]
    \begin{center}
         \includegraphics[angle=0,scale=.3]{parameters3}
    \caption{\label{fig01} Curves of model coefficients versus spectral filtering term $`\beta$' for $\gamma=-1$.}
    \end{center}
\end{figure}
It should be noted that $\gamma$ will take only negative values, as solution parameter $`a$' should be real, and $\beta$ can be chosen arbitrarily $(\beta \neq 0)$ while the other model coefficients depend on $\beta$ and $\gamma$. In Fig. \ref{fig01}, we have presented the allowed values of the model coefficients with $\beta$ lying in the interval $[0,1]$ for $\gamma=-1$. In Fig. \ref{fig1}, we have shown the amplitude profile of Lambert W-kink solution, given by Eq. (\ref{eq21}), for different values of the spectral filtering term $`\beta$' and $\gamma=-1$. From this plot, one can observe that kink wave has large amplitude and becomes more steep for small values of $\beta$ as solution parameter $`a$' is inversely proportional to $\beta$. It should be noted that, from Fig. \ref{fig01}, these Lambert W-kink solutions are possible only for $\delta>0, ~\epsilon<0$ and $\gamma<0$.
\begin{figure}[ht!]
\begin{center}
\includegraphics[angle=0,scale=.60]{ampW}
\caption{\label{fig1} Amplitude profiles of Lambert
W-kink solution for different values of $\beta$, $\beta=0.1$ (thick line), $\beta=0.5$
(dashed line) and $\beta=0.9$ (dotted line).}
\end{center}
\end{figure}

\begin{figure}[ht!]
    \begin{center}
        \subfigure[]{
    \includegraphics[angle=0,scale=.5]{intensity1}
    }
     \subfigure[]{
    \includegraphics[angle=0,scale=.5]{intensity2}
     }
     \subfigure[]{
    \includegraphics[angle=0,scale=.7]{chirpW}
    }
    \caption{\label{fig2}(a,b) Intensity profile of Lambert W-kink soliton for $\gamma=-1$ and different values of coefficient $\beta$, $0.1$ and $0.5$, respectively. (c) The corresponding chirp profiles
     for $\beta=0.1$ (thick line) and $\beta=0.5$ (dashed line).}
    \end{center}
\end{figure}

Using Eq. (\ref{eq21}) into the Eq. (\ref{eq2}), the intensity expression of Lambert W-kink solitons, for the model equation Eq. (\ref{eq1}), reads
    \begin{equation}\label{i1}I_W(z,t)=a^2\(1-\frac{2}{1+W(e^{4a^2\xi+1})}\)^2,\end{equation}
with chirping given by
    \begin{equation}\label{i2}\delta\omega(z,t)=-\[\frac{5a^2\beta}{2}\(1-\frac{2}{1+W(e^{4a^2\xi+1})}\)^2-\frac{\gamma }{2\beta \left(4+5 \beta ^2\right)}\].\end{equation}
The intensity profile of Lambert W-kink soliton is depicted in the Fig. \ref{fig2}(a,b) for $\gamma=-1$ and different values of coefficient $\beta$, $0.1$ and $0.5$, respectively. These intensity profiles are similar to dark solitons (albeit asymmetric in nature) and shows relative compression of pulses with the modulation of spectral filtering term $`\beta$'. Fig. \ref{fig2}(c) shows the profiles of corresponding frequency chirp $\delta\omega$ across the pulse of Lambert W-kink soliton at $z=0$. One can observe that frequency chirp saturates to negative value as the retarded time approaches its asymptotic limit and the amplitude of chirp can be controlled for judicious choice of parameter $`\beta$'.

\subsection{Moving front solitons}
To exemplify the existence of moving front solitons as exact solutions of this model,
let us consider the differential equation
in $\rho$
    \begin{equation}\label{eq22}\rho'=c\rho\(1-\frac{\rho^2}{2b}\),\end{equation}
where $b,c$ are real parameters. The explicit moving front soliton is given by \cite{S.N.,rpal}
    \begin{equation}\label{eq25}\rho(\xi)=\sqrt{b\(1+\mbox{tanh}(c\xi)\)},\end{equation}
for $b>0$.
\begin{figure}[ht!]
\begin{center}
\includegraphics[angle=0,scale=.60]{ampk}
\caption{\label{fig4} Amplitude profiles of kink solution for different values of $\beta$, $\beta = 0.3$ (thick line), $\beta = 0.4$
(dashed line) and $\beta = 0.5$ (dotted line). The other parameters used in the plots are $\epsilon =0.8,~\gamma =1,~\delta =-1$ and $T_r=0.2$.}
\end{center}
\end{figure}
The corresponding second-order differential equation for $\rho(\xi)$ reads
     \begin{equation}\label{eq23}\rho''-\frac{3c^2}{4b^2}\rho^5+\frac{2c^2}{b}\rho^3-c^2\rho=0. \end{equation}
Substituting Eq. (\ref{eq22}) into Eq. (\ref{eq13}) and comparing the resultant equation with Eq. (\ref{eq23}), the solution parameters found to be
    \begin{equation}\label{eq24} c=\frac{-N\pm\sqrt{N^2-4Q}}{2},~~b=\frac{(N+4c)c}{2(R+Mc)},\end{equation}
along with constraint on model coefficient $\nu=\frac{A^2}{2}+\frac{c M}{4b}-\frac{3c^2}{8b^2}$. Here, the parameters $M,N,Q$ and $R$ can be obtained from Eqs. (\ref{eq8})-(\ref{eq11}), using Eq. (\ref{eq15}), for different values of the model coefficients $\beta,~\epsilon,~\gamma,~\delta$ and $T_r$. For illustrative purpose, we choose the model coefficients $\epsilon>0$, $\gamma>0$ and $\delta<0$, just opposite to the case of Lambert W-kink solution, to depict the evolution of kink solutions. The amplitude profile of kink solution is shown in Fig. \ref{fig4} for $\epsilon =0.8,~\gamma =1,~\delta =-1,~T_r=0.2$ and different values of $\beta$.
\begin{figure}[ht!]
    \begin{center}
     \subfigure[]{
    \includegraphics[angle=0,scale=.5]{intensityk1}
    }
     \subfigure[]{
    \includegraphics[angle=0,scale=.5]{intensityk2}
    }
     \subfigure[]{
    \includegraphics[angle=0,scale=.7]{chirpK}
    }
    \caption{\label{fig5} (a,b) Intensity profile of front soliton for different values of $\beta$, $0.3$ and $0.5$, respectively. (c) The corresponding chirp profiles for $\beta = 0.3$ (thick line) and $\beta = 0.5$ (dashed line). The other parameters are same as in Fig. \ref{fig4}.}
    \end{center}
\end{figure}
The expression of the intensity for these kink solution can be written as
    \begin{equation}\label{eq28}I_K(z,t)=b\(1+\mbox{tanh}(c\xi)\).\end{equation}
The corresponding chirping is given by
    \begin{equation}\label{eq29}\delta\omega(z,t)=-\(\frac{u+2 \beta  T_r b\(1+\mbox{tanh}(c\xi)\)}{1+4 \beta^2}\). \end{equation}
The intensity and chirp profiles for front solitons are shown in Fig. \ref{fig5} for
different values of $\beta$. For $\beta = 0.3$, the frequency chirp approaches negative value as $t\rightarrow\pm\infty$ while it approaches negative and positive value as $t\rightarrow~+\infty$ and $t\rightarrow~-\infty$, respectively, for $\beta = 0.5$.

\newpage

\section{Conclusion}
In conclusion, we have shown the existence  of exact explicit solution for the complex cubic-quintic Ginzburg-Landau equation in terms of Lambert W function or omega function, under the influence of  of intrapulse Raman scattering. Parameter domains are delineated in which these optical solitons exit in the ensuing model. It is observed that these optical solitons are possible for negative values of nonlinear gain and Kerr effect and positive value of constant gain. Whereas, no such restrictions are imposed on the moving fronts or optical shock-type solitons that are obtained as a byproduct of this model. We have observed that the intensity of these fronts has been doubled with a slight change in the value of the spectral filtering or gain parameter. The frequency chirp associated with these nonlinear waves has been identified. Furthermore, we have explicated the pivotal role played by this nonlinear chirp on the intensity of these waves. These results may be useful for experimental realization of undistorted transmission of optical waves in optical fibers and further understanding of their optical transmission properties. Finally, we hope that the exact nature of these nonlinear waves presented here may be profitably exploited in designing the optimal Raman fiber laser experiments.

\section{Acknowledgment}  A.G. gratefully acknowledges Science and Engineering
Research Board (SERB), Department of Science and Technology,
Government of India for the award of SERB Start-Up Research Grant
(Young Scientists) (Sanction No: YSS/2015/001803) during the course of this work. Nisha is thankful to SERB-DST,
India for the award of fellowship during the work tenure. We sincerely thank the referees for their useful comments.