text:
\section{Introduction}
\input{introduction.tex}

\section{Preliminary}
\input{definition.tex}
\label{sec:definition}

\section{Representation Learning Module}
\label{sec:representation}
\input{representation.tex}

\section{Clustering Module}
\label{sec:clustering}
\input{clustering.tex}

\section{Taxonomy of Deep Clustering}
\label{sec:work}
In this section, we summarize existing deep clustering methods into four branches based on the interaction between the representation learning module and the clustering module:
\begin{enumerate}
    \item Multi-stage deep clustering: the representation learning module is sequential connected with the clustering module.
    \item Iterative deep clustering: the representation learning module and the clustering module are iteratively updated.
    \item Generative deep clustering: the clustering module is modeled as a prior representation module.
    \item Simultaneous deep clustering: the representation learning module and the clustering module are simultaneously updated.
\end{enumerate}

\subsection{Multistage Deep Clustering}
\label{multistage}
\input{multistage.tex}

\subsection{Iterative Deep Clustering}
\input{iterative.tex}

\subsection{Generative Deep Clustering}
\input{generative.tex}

\subsection{Simultaneous Deep Clustering}
\input{simultaneous.tex}

\input{Advantages-and-disadvantages.tex}

\input{application.tex}

\section{Datasets and Evaluation Metrics}
\label{sec:datasets}
\input{dataset-evaluation.tex}

\section{Applications}
\label{sec:application}
\input{real-application.tex}

\section{Future Directions}
\label{sec:future}
\input{future.tex}

\section{Conclusion}
\input{conclusion}

\bibliographystyle{ACM-Reference-Format}

\subsection{Overview}

\begin{sidewaystable}
\centering
\caption{The proposed taxonomy of existing methods and their properities}
\label{Label}
\scalebox{0.95}{
\begin{tabularx}{\textwidth}{|p{0.095\textwidth}<{\centering}|p{0.20\textwidth}<{\centering}|p{0.245\textwidth}<{\centering}|p{0.16\textwidth}<{\centering}|p{0.05\textwidth}<{\centering}|X<{\centering}|}
 \hline
 Taxonomy&Advantages & Disadvantages&Clustering Module&Type&Citation  \\ \hline
    \multirow{7}{*}{Multi Stage} & \multirowcell{7}[0pt][l]{ 1. Easy to implement  \\
2. Better interpretability} &\multirowcell{7}[0pt][l]{1. Sub-optimal representations \\
2. Limit on clustering performance} &   \multirow{7}{*}{Multi Stage}  &  \multirow{2}{*}{Image} &\cite{tian2014learning}\cite{ji2017deep}\cite{zhang2021learning}\cite{zhou2018deep}\newline \cite{peng2016deep}\cite{tao2021clustering}\cite{dang2021nearest}    \\ \cline{5-6}
                          &     &  &                                    &  Graph  &  \cite{huang2014deep}\cite{chiang2019cluster}\cite{nasrazadani2022sign}\cite{wu2016cccf}  \\ \cline{5-6}
                          &     &  &                                   &  \multirow{2}{*}{Textual}  &  \cite{chang2005using}\cite{tseng2010generic}\cite{cerquitelli2017data}\cite{shen2018improving}\newline \cite{janani2019text} \cite{li2020text}\cite{kong2020learning}\cite{alami2021unsupervised}   \\ \cline{5-6}
                          &     &  &                                   &  Video  &  \cite{peng2021deep}\cite{wang2020cluster}   \\ \cline{5-6}
                          &     &  &                                   &  Medical  &  \cite{hassan2021novel}\cite{dao2010inferring}   \\ \hline
    \multirow{5}{*}{Generative}& \multirowcell{5}[0pt][l]{ 1. Latent cluster structure \\ discovery \\ 2. Able to model complex \\ data distribution} & \multirowcell{5}[0pt][l]{1. Unstable training \\ 2. High computational complexity \\ 3. Sensitive to prior distribution \\ assumptions}& \multirow{2}{*}{VAE} &\multirow{2}{*}{Image}&\cite{dilokthanakul2016deep}\cite{yang2019deep}\cite{jiang2016variational}\cite{prasad2020variational}\newline \cite{ji2021decoder}  \cite{li2020unsupervised}\cite{wang2022neural}   \\ \cline{4-6}
     &  &  & \multirow{3}{*}{GAN} &\multirow{2}{*}{Image}&\cite{mukherjee2019clustergan}\cite{ben2018gaussian}\cite{yu2018mixture}\cite{ntelemis2021image}\newline \cite{gan2021learning} \cite{ghasedi2019balanced}  \\ \cline{5-6}
      &  &  & & Graph & \cite{jia2019communitygan}   \\ \hline
    \multirow{5}{*}{Iterative}  & \multirowcell{5}[0pt][l]{ 1. Clustering oriented \\ representation learning \\ 2. Discriminative space for \\ clusteing} & \multirowcell{5}[0pt][l]{ 1. Error propagation \\ 2. Limit on pseudo-label quality} & \multirow{4}{*}{Individual supervision} & \multirow{2}{*}{Image} & \cite{yang2016joint}\cite{yang2017towards}\cite{song2013auto}\cite{zhang2019self}\newline  \cite{lv2021pseudo}\cite{van2020scan}\cite{niu2021spice}  \\ \cline{5-6}
                          &  &   &                                &Video&\cite{alwassel2020self} \\ \cline{5-6}
                          &  &   &                                &Medical&\cite{kart2021deepmcat}\cite{mittal2021new}   \\ \cline{4-6}
                          &  & &\multirow{1}{*}{Relational supervision}&Image&\cite{chang2017deep}\cite{caron2018deep}\cite{zhang2019neural}\cite{niu2020gatcluster}   \\  \hline
    \multirow{12}{*}{Simultaneous} & \multirowcell{12}[0pt][l]{ 1. Clustering oriented \\ representation learning  \\ 2.Discriminative space for \\ clusteing\\ 3. Unified optimization \\ framework} & \multirowcell{12}[0pt][l]{1. Undesired prejudice of \\ optimization  focus \\ 2. Sensitive to degenerate solutions    }  &\multirow{4}{*}{Auto-encoder based}&\multirow{2}{*}{Image}& \cite{xie2016unsupervised}\cite{guo2017improved}\cite{ghasedi2017deep}\cite{guo2018deep}\cite{hsu2017cnn}\newline
    \cite{li2018discriminatively}\cite{peng2017cascade}\cite{rezaei2021learning} \\ \cline{5-6}
    &  & & &Graph&\cite{bo2020structural}   \\ \cline{5-6}
    &  & & &Medical&\cite{tian2019clustering}\cite{hu2020iterative}   \\ \cline{4-6}
    &  & & Mutual Information based clustering &\multirow{2}{*}{Image}&\cite{wu2019deep}\cite{ji2021variational}\cite{ji2019invariant}\cite{ntelemis2021information}   \\ \cline{4-6}
    &  & &\multirow{5}{*}{Contrastive}&\multirow{3}{*}{Image}&\cite{li2021contrastive}\cite{dang2021doubly}\cite{huang2020deep}\cite{wu2019deep}\newline \cite{zhong2020deep}  \cite{do2021clustering}\cite{huang2021deep}\cite{tsai2020mice}\newline \cite{shen2021you}\cite{huang2021exploring}  \cite{dang2021nearest}   \\ \cline{5-6}
    &  & & &Textual&\cite{zhang2021supporting}   \\ \cline{5-6}
    &  && &Graph&\cite{zhong2021graph}    \\ \cline{4-6}
    &  & &\multirow{2}{*}{Hybrid methods}& Image &\cite{sundareswaran2021cluster}\cite{chang2019deep}\cite{shah2017robust}   \\ \cline{5-6}
    &  &  & &Textual &\cite{zhang2021supporting}\cite{meng2020hierarchical} \cite{thirumoorthy2021hybrid}  \\ \hline
    \end{tabularx}%
    }
\end{sidewaystable}

\subsection{Relation Matching Deep Clustering}
In deep clustering, each data instance can be represented in two spaces, namely the $d$-dimensional \textit{embedding space} and $K$-dimensional \textit{label space}.
The relationships among instances are expected to be consistent during dimension reduction, which has been utilized to bridge the semantic gap between representation learning and clustering.
Borrowing the idea from the domain adaptation \cite{long2015learning}, relation matching can be realized in bidirectional ways:
\begin{equation}
    \mathcal{L}_{RM} = \sum^{N}_{i}\sum^{N}_{j}\ell(R^{s}_{ij},R^{t}_{ij})
\end{equation}
where $\ell$ is a measure of relation matching, e.g.,  cosine similarity or Euclidean distance, $R^{s}$ and $R^{t}$ are the relations in source and target space, which can be either embedding space or label space.
It is worth noting that the relations defined here narrowly refers to the continuous ones and we introduce the discrete relations as pseudo labeling in subsection \ref{subsec:pseudo}.

\textbf{Analysis.}
The relation matching deep clustering explicitly connect the representation learning and clustering, which is straightforward and easy to implement.
However, calculating $N^{2}$ pairs of instances is computational inefficient.
To tackle this challenge, some methods only preserve the k-nearest-neighbor relations \cite{van2020scan,dang2021nearest} for each instance or the relations with high confidence \cite{van2020scan}.
Although this can somehow improve the training efficiency, the extra hyper-parameter is hard to set in the unsupervised manner.
Furthermore, among all pairs of relations, many of them are noisy especially in the early training phases with limited capability. How to filter out the clean relations to boost the performance while discard the noisy relations is still an open research question.

\subsection{Pseudo Labeling Deep Clustering}
\label{subsec:pseudo}
The pseudo labeling can be viewed as another type of relation matching where the relations are discrete based on the consistency of labels.
It has been widely studied in semi-supervised learning \cite{lee2013pseudo}, and recently being introduced to deep clustering.
According to the way of utilizing pseudo labels, existing methods can be largely divided into two groups: instance-wise pseudo labeling \cite{van2020scan,niu2021spice,caron2018deep} and relation-wise pseudo labeling \cite{chang2017deep,niu2020gatcluster}.

The instance-wise pseudo labeling filters out a subset of instances with high confidence and trains the network in a supervised manner with cross-entropy loss:
\begin{equation}
    \mathcal{L}_{IPL} = -\frac{1}{|\mathcal{X}^{c}|} \sum^{\mathcal{X}^{c}}_{i} \sum_{k=1}^{K} \tilde{y}_{i k} \log \left(z_{i k}\right)
\end{equation}
where $\mathcal{L}_{IPL}$ denotes the loss of instance-wise pseudo labeling, $\mathcal{X}^{c}$ denotes the subset of instances with high confidence, $\tilde{y}_{i k}$ and $z_{i k}$ are the predicted hard label and soft cluster assignment.
The confidence is usually estimated by entropy or maximum of the assignment probabilistic distribution.

The general idea of relation-wise pseudo labeling is to enforce instances with same pseudo labels closer while instances with different pseudo labels away from each other in the embedding space.
Given the filtered instances, relation-wise pseudo labeling construct the discrete relations among instances to guide the representation learning: the \textit{must-link} for pairs of instances with same pseudo labels and \textit{cannot-link} for pairs of instances with different pseudo labels:
\begin{equation}
    \mathcal{L}_{RPL} = \frac{1}{|\mathcal{C}|}\sum_{\{i,k\}\in \mathcal{C}}  R_{ik} - \frac{1}{|\mathcal{M}|}\sum_{\{i,j\}\in \mathcal{M}}  R_{ij}
\end{equation}
where $\mathcal{M}$ is the set of must-link relations and $\mathcal{C}$ is the set of cannot-link relations,
$R_{ij}$ is the similarity of instance $x_{i}$ and $x_{j}$ in the low-dimensional embedding space.

\textbf{Analysis.}
The pseudo labeling has brought the power of semi-supervised learning into the unsupervised clustering task.
However, the performance highly relies on the quality of filtered pseudo labels which is susceptible to model capability and hyper-parameter tuning, especially in the unsupervised manner. The current methods \cite{van2020scan,niu2021spice} have taken the pre-training as the early stage before pseudo labeling to tackle this challenge, but more attention deserved to be paid on it.

\subsection{Self-training Deep Clustering}
\label{subsec:self-training}
The self-training strategy is introduced to the deep clustering task \cite{xie2016unsupervised} and open up an active branch of methods named self-training deep clustering.
More specifically, the cluster assignment distribution is optimized by minimizing the KL-divergence with an auxiliary distribution:
\begin{equation}
    \mathcal{L}=D_{K L}(\mathbb{P} \| \mathbb{Q})=\sum^{N}_{i} \sum^{K}_{k} p_{i k} \log \frac{p_{i k}}{q_{i k}}
\end{equation}
where $\mathbb{Q}$ is the cluster assignment distribution and $\mathbb{P}$ is the auxiliary distribution.
$q_{ik}$ and $p_{ik}$ denotes the probability of instance $x_{i}$ belong to cluster $k$.
The assignment distribution $\mathbb{Q}$ follows the assumption of K-means and is produced by the embedding distance between instance and cluster centroids:
\begin{equation}
    q_{i k}=\frac{\left(1+\left\|h_{i}-c_{k}\right\|_{2}^{2} / \alpha\right)^{-\frac{\alpha+1}{2}}}{\sum^{K}_{j}\left(1+\left\|h_{i}-c_{j}\right\|_{2}^{2} / \alpha\right)^{-\frac{\alpha+1}{2}}}
\end{equation}
where $h_{i}$ is the representation of data instance $x_{i}$ and $c_{k}$ is the representation of cluster $k$, $\alpha$ is the freedom degree of the Student's t-distribution \cite{van2008visualizing}.
The auxiliary distribution $\mathbb{P}$ is a variant of assignment distribution $\mathbb{Q}$ with both instance-wise and cluster-wise normalization:
\begin{equation}
    p_{i k}=\frac{q^{2}_{i k} / f_{k}}{\sum^{K}_{j} q_{i j}^{2} / f_{j}}
\end{equation}
where $f_{k}=\sum^{N}_{i}q_{i k}$ are soft cluster frequencies.

\textbf{Analysis.}
The self-training deep clustering has been widely adopted in several existing works \cite{xie2016unsupervised,guo2017improved,guo2018deep,ghasedi2017deep,li2018discriminatively,rezaei2021learning,peng2017cascade}.
The success relies on the following properties:
First, the square of cluster assignment probability with cluster-wise normalization will guide the model put more attention (gradient) on the instances with higher confidence, which in turn reduce the impact of low confidence ones.
As a result, the cluster assignment vector tends to be one-hot.
Second, the soft cluster frequencies $f_{k}$ can be viewed as the sum of the probability that instance belongs to the $k$-th cluster.
This can prevent the degenerate solution that all instances belong to the same cluster.
The above two properties have a profound effect on the later works and many deep clustering methods can be viewed as variants of it.

\subsection{Mutual Information Maximization based Clustering}
\label{subsec:mi_clustering}
As introduced in subsection \ref{subsec:mi_rl}, mutual information maximization has achieved tremendous success in representation learning.
Benefiting from its unrestriction to the feature dimension and semantic meaning, it has also been introduced to the clustering module for measuring the dependence between instance and cluster assignment.
The mutual information maximization based clustering can be formulated as:
\begin{equation}
        \mathcal{I}(X, \tilde{Y}) =D_{K L}(p(x, \tilde{y}) \| p(x) p(\tilde{y})) =\sum_{x \in \mathcal{X}} \sum_{\tilde{y} \in \mathcal{Y}} p(x, \tilde{y}) \log \frac{p(x, \tilde{y})}{p(x) p(\tilde{y})}.
\end{equation}
where $X$ is the original data, $\tilde{Y}$ is the predicted labels, $p(x,\tilde{y})$ is the joint distribution and $p(x), p(\tilde{y})$ are the marginal distribution.
The estimation of mutual information is same as that in representation learning.

\textbf{Analysis.} The mutual information maximization based clustering and representation learning are quite similar despite the dimension of optimization object.
The major advantage is that it overcomes the gap between the representation learning and clustering.
As a consequence, the rapid development of deep representation learning techniques can be naturally introduced to the clustering task and optimized in a unified framework.

\subsection{Contrastive Deep Clustering}
Inspired by the success of contrastive representation learning and mutual information maximization based deep clustering, contrastive learning has also been introduced to deep clustering.
Similar to contrastive representation learning, the target of contrastive deep clustering is to pull the positive pairs close while pushing the negative pairs far away.
The major difference lies in the definition of positive pairs and negative pairs, which can be further divided into three groups:

\subsubsection{Instance-Instance contrast}
The instance-instance contrast treats the cluster assignment of each instance as the representation and directly reuses the contrastive representation learning loss:
\begin{equation}
    \mathcal{L}_{IIC}=-\log \sum^{N}_{i=1}\frac{\exp \left(f\left(z_{i}, z^{\mathcal{T}}_{i}\right) / \tau\right)}{\sum_{j=1}^{N} \exp \left(f\left(z_{i}, z_{j}^{\mathcal{T}}\right) / \tau\right)}
\end{equation}
where $z_{i}$ is the cluster assignment of instance $x_{i}$ predicted by the clustering module.

\subsubsection{Cluster-Cluster contrast}
The cluster-cluster contrast treats each cluster as an instance in the embedding space, the target is pulling the cluster and its augmented version close while pushing different clusters far away, which can be formulated as:
\begin{equation}
    \mathcal{L}_{CCC}=-\log \sum^{K}_{k=1}\frac{\exp \left(f\left(c_{k}, c^{\mathcal{T}}_{k}\right) / \tau\right)}{\sum_{j=1}^{K} \exp \left(f\left(c_{k}, c_{j}^{\mathcal{T}}\right) / \tau\right)}
\end{equation}
where $c_{k}$ is the representation of cluster $k$.
It is worth noting that cluster-cluster contrast satisfy the basic requirement of clustering that each cluster should be dissimilar, which agrees with the clustering friendly representation learning described in subsection \ref{subsec:cluster_friend}.

\subsubsection{Instance-Cluster contrast}
The instance-cluster contrast is similar to the K-means which utilizes the cluster centroid as an explicit guidance.
Given the representation of each instance and cluster centroid in the same low-dimensional space, each instance is expected to be close to the corresponding cluster centroid while far from the other cluster centroids.
Such similarity and dissimilarity can be naturally modeled by the contrastive learning:
\begin{equation}
    \mathcal{L}_{ICC}=-\log \sum^{N}_{i=1}\frac{\exp \left(f\left(h_{i}, c'_{i}\right) / \tau\right)}{\sum_{j=1}^{K} \exp \left(f\left(h_{i}, c_{j}\right) / \tau\right)}
\end{equation}
where $c'_i$ is the corresponding cluster centroid of $x_i$ which is usually estimated by an alternative clustering method.
This can be also understood as maximizing the mutual information between the representation and cluster assignment with data augmentations.

\textbf{Analysis.}
Besides the advantages inherited from the mutual information maximization clustering,
the primary advantages of contrastive deep clustering
is that the data augmentation helps improve the robustness of clustering, which has been ignored by most existing methods.
For detailed advantages of contrastive learning, please refer to the former contrastive learning surveys \cite{jing2020self,jaiswal2020survey}.

\subsection{Datasets}

\subsubsection{Image Datasets}

Image is the most commonly used data type in real-world deep clustering. The early attempts of deep clustering are applied to image datasets including COIL-20\footnote{http://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php}, CMU PIE\footnote{http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html}, Yale-B\footnote{http://vision.ucsd.edu/\textasciitilde leekc/ExtYaleDatabase/Yale\%20Face\%20Database.htm}, MNIST\footnote{http://yann.lecun.com/exdb/mnist/index.html}, CIFAR\footnote{http://www.cs.toronto.edu/\textasciitilde kriz/cifar.html} and  STL-10\footnote{https://cs.stanford.edu/\textasciitilde acoates/stl10/}.
Recently, efforts has been paid to perform clustering on large volume vision datasets (e.g. ImageNet). Although existing methods have achieved promising performance on ImageNet-10 and ImageNet-Dogs dataset, clustering on Tiny-ImageNet (200 clusters) or full-size ImageNet is still challenging.

\subsubsection{Textual Datasets}
The widely used textual datasets in early applications of texutal data clustering include
Reuters-21578\footnote{https://archive.ics.uci.edu/ml/datasets/Reuters-21578+Text+Categorization+Collection} and 20 Newsgroups\footnote{http://qwone.com/\textasciitilde jason/20Newsgroups/ } datasets, which have already been vectorized and little feature engineering is needed. Currently, raw textual datasets including IMDB\footnote{http://ai.stanford.edu/\textasciitilde amaas/data/sentiment/}, stackOverflow\footnote{https://github.com/jacoxu/StackOverflow}, and more in nlp-datasets github repository\footnote{https://github.com/niderhoff/nlp-datasets} is still challenging for deep textual clustering.

\subsubsection{Video Datasets}
The ultimate task of video clustering varies from action classification \cite{peng2021deep} to video anomaly detection \cite{wang2020cluster}. Kinetics-400 and Kinetics-600\footnote{https://www.deepmind.com/open-source/kinetics} are two of the most famous video datasets.
The others include UCF-101 dataset\footnote{https://www.crcv.ucf.edu/research/data-sets/ucf101/} and HMDB-51 dataset\footnote{https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/\#dataset}.

\subsubsection{Graph Datasets}
Commonly used graph datasets for node clustering can be referred from the following papers \cite{shun2016parallel,chiang2019cluster,nasrazadani2022sign} and Stanford Network Analysis Project\footnote{http://snap.stanford.edu/}. And there are also graph-level classification datasets like PROTINS\cite{borgwardt2005protein} and MUTAG\cite{debnath1991structure}, which can be used to perform and evaluate graph-level clustering.

\subsection{Evaluation Metrics}
Evaluation metrics aim to evaluate the validity of methods. In the field of Deep Clustering, three standard clustering performance metrics are widely used: Accuracy(ACC), Normalized Mutual Information(NMI) and Adjusted Rand Index(ARI).

\subsubsection{Accuracy}
ACC indicates the average correct classification rate of clustering samples. Given the ground truth labels $Y=\left\{ y_i|1\leq i\leq N\right\}$ and the predicted hard assignments $\tilde{Y}=\left\{ \tilde{y}_i|1\leq i\leq N \right\}$, ACC can be computed as follows:
\begin{equation}
    ACC(\tilde{Y}, Y)=\mathop{\max}_{g} \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\{ y_i = g(\tilde{y}_i)\}
\end{equation}
where $g$ is the set of all possible one-to-one mappings between the predicted labels and ground truth labels. The optimal mapping can be efficiently obtained by the Hungarian algorithm \cite{kuhn1955hungarian}.

\subsubsection{Normalized Mutual Information}
NMI quantifies the mutual information between the predicted labels and ground truth labels into $[0,1]$:
\begin{equation}
    NMI(\tilde{Y}, Y)=\frac{\mathcal{I}(\tilde{Y}; Y)}{\frac{1}{2} \left[H(\tilde{Y})+H(Y)\right]}
\end{equation}
where $H(Y)$ is the entropy of $Y$ and $\mathcal{I}(\tilde{Y}; Y)$ is the mutual information between $\tilde{Y}$ and $Y$.

\subsubsection{Adjusted Rand Index}
ARI comes from Rand Index(RI), which regards the clustering result as a series of pair-wise decisions and measures it according to the rate of correct decisions:
\begin{equation}
    RI=\frac{TP + TN}{C_N^2}
\end{equation}
where $TP$ and $TN$ denote the number of true positive pairs and true negative pairs, $C_N^2$ is the number of possible sample pairs. However, the RI value of two random partitions is not a constant approaching 0, thus ARI was introduced:
\begin{equation}
    ARI=\frac{RI-\mathbb{E}(RI)}{\max(RI)-\mathbb{E}(RI)}
\end{equation}

Both ACC and NMI $\in [0, 1]$ while ARI $\in [-1, 1]$, in which higher values indicate better performance.

\subsection{Initialization of Deep Clustering Module}
The initialization of deep neural networks usually play an important role in training efficiency and stability \cite{glorot2010understanding}.
This is more critical in deep clustering where both the representation learning module and clustering module are modeled by deep neural networks.
Recently, the model pre-training \cite{qiu2020pre} has been a popular network initialization technique which has also been introduced to the deep clustering \cite{van2020scan}.
However, the pre-training based initialization is appropriate for representation learning module but has not been well studied for the clustering module.
Although there has been some initialization schemes on the shallow clustering \cite{celebi2013comparative}, the initialization for the cluster module with deep neural networks is still under investigation.

\subsection{Overlapping Deep Clustering}
The deep clustering methods discussed in this paper largely focus on the partitioning clustering where each instance belong to only one cluster.
Meanwhile, in the real-world scenario, each instance may belong to multiple clusters, e.g.,  users in a social network \cite{xie2013overlapping} may belong to several communities, and the video/audio in the social media may have several tags \cite{wu2020multi}.
Among the deep clustering methods discussed in this paper, if the clustering constraints are conducted on the cluster assignment probabilistic matrix, they can be directly adapted to the overlapping clustering setting.
However, if the training relies on the pseudo hard label of data instances, they may fail in the overlapping clustering setting.
Although the multi-label classification has been widely studied in the literature \cite{liu2021emerging,zhang2013review}, how to adapt to unsupervised clustering learning is still an open research question.

\subsection{Degenerate Solution VS Unbalanced Data}
The degenerate solution \cite{caron2018deep} has been one of the most noteworthy concerns in deep clustering, where all instances may be assigned to a single cluster.
Many deep clustering methods have added extra constraints to overcome this problem \cite{huang2014deep, caron2018deep,ji2019invariant,huang2020deep,van2020scan,li2021contrastive,park2021improving}, among which the entropy of cluster size distribution is the most widely used one.
By maximizing the entropy, the instances are expected to be evenly assigned to each cluster and avoid degenerating to single cluster.
It is worth noting that the success relies on the uniform distribution of ground truth labels which is coincidentally satisfied by the benchmark dataset such as CIFAR10 and CIFAR100.
However, in real-world scenarios, such assumption is too strict and most datasets are unbalanced or long tailed \cite{wang2016training}.
The divergent target of uniform distribution and unbalanced dataset will seriously weaken the deep clustering.
Recently, clustering unbalanced data \cite{fang2021unbalanced,liang2020lr} has attracted increasing attention and this will boost the performance of clustering in real-world applications.

\subsection{Boosting Representation with Deep Clustering}
Throughout this paper, we can find that a good representation is essential for clustering.
Although the clustering friendly representation learning has been studied in the literature, they are designed for the specific shallow clustering method.
On the contra, the clustering structure denotes the high-order pattern of the dataset, which should be preserved in the comprehensive representation \cite{li2020prototypical}.
The deep clustering methods discussed in this paper focus on how to incorporate the representation learning to boost the clustering, meanwhile, how to in turn boost the representation learning by the clustering is still to be studied.

\subsection{Deep Clustering Explanation}
As an unsupervised task, the clustering process usually lacks human priors such as label semantics, number of clusters\cite{moser2007joint}, which makes the clustering results hard to explain or understand.
Some methods \cite{zhang2021deep} have already combined the tags provided by the users to boost the explanation of the clustering results.
However, it relies on the accurate human tagging which may not be realized in practical.
With the development of causal inference in deep learning, the explanation of clustering among instances is hopefully to be improved.
Both the research and industry community are expecting a generalized explanation framework for clustering, especially on the high-dimensional data.
To conclude, how to utilize the casual inference techniques in the clustering is of great importance and deserve more attention.

\subsection{Transfer Learning with Deep Clustering}
Transfer learning \cite{pan2009survey} aims at bridging the gap between the training and test datasets with distribution shift.
The general idea is to transfer the knowledge from the known data to the unknown test data.
Recently, deep clustering is playing an increasing important role in unsupervised transfer learning \cite{xue2016multi,deng2019cluster,tang2020unsupervised,zhou2021cluster,Zhou_2021_ICCV}, where the target domain is in the unsupervised manner.
For example, ItClust \cite{hu2020iterative} and MARS \cite{brbic2020mars} has achieved success in scRNA-seq clustering (section \ref{medical}), AD-Cluster \cite{zhai2020ad} that use clustering has improved domain adaptive person re-identification.
On the other way around, unsupervised transfer learning methods can also benefit deep clustering.
Take UCDS \cite{menapace2020learning} as an example, the unsupervised domain adaptation is used to perform clustering among variant domains. Distribution shift is one of the key factor affecting the performance of machine learning models, including deep clustering. Perform clustering analysis can give further understanding on unsupervised target domain, but how to transfer clustering result to knowledge and effectively minimize the distribution shift based on clustering result can be further explored.

\subsection{Clustering with Anomalies}
In subsection \ref{app:anomaly}, we have discussed the applications of deep clustering in anomaly detection where the instances are well clustered.
Concerning the existence of anomaly instances in the dataset, the clustering may also be influenced and mutually restrict, since most existing deep clustering methods have no specific response to the outliers.
The classic K-means method is known to be sensitive to outliers, although there has been a few works on overcoming this problem \cite{liu2019clustering}, they are designed for shallow clustering methods.
To this end, how to improve the deep clustering robustness to the anomaly instances and gradually improve the clustering performance by reducing the detected anomaly instances is still an open research question.

\subsection{Efficient Training VS Global Modeling}
To improve the training efficiency and scalability, most existing deep clustering methods have utilized the mini-batch training strategy, where the instances are separated into batches and the model is updated with each batch.
This is suitable for the task where the instances are independent on each other, such as classification and regression.
However, as the deep clustering heavily relies on the complicated relationship among instances, such mini-batch training may lose the ability of global modeling.
Although some existing methods have utilized the cluster representations or prototypes \cite{li2020prototypical} to store the global information, how to balance the training efficiency and model capability is still worth studying.

\subsubsection{Deep Generative clustering based on Variational Auto-Encoder}

For clustering of high dimensional and complex data, one promising solution is to directly stack GMM with a deep neural network --- GMM generates a latent vector $z$, and the deep neural network further transforms the latent vector $z$ into the complex data instance $x$. In this way, the stacked model can enjoy the merit of the latent cluster structure and meanwhile has sufficient capacity to model the complex data. For example, the representative models, VaDE \cite{jiang2016variational} and GMVAE \cite{dilokthanakul2016deep}, assume the following generative process for each instance:
\begin{enumerate}
    \item Choose a cluster: $c \sim \operatorname{Mult}(\pi)$
 \item Draw a latent vector: $\mathbf{z}|c \sim \mathcal{N}\left(\mathbf{\mu_z}(c;\beta), \mathbf{\sigma}_z^{2}((c;\beta)) \mathbf{I}\right)$
 \item Draw a sample: $\bf{x}|\bf{z} \sim \mathcal{N}\left(\mathbf{\mu_x}(\mathbf{z};\theta), \mathbf{\sigma}_x^{2}(\mathbf{z};\theta) \mathbf{I}\right)$
\end{enumerate}
where $\mathbf{\mu_z}(.;\beta)$, $\bf{\sigma_z^2}(.;\beta)$, $\mathbf{\mu_x}(.;\theta)$ and $\mathbf{\sigma_x^2}(.;\theta)$ are given by neural networks with parameters $\beta$ and $\theta$, which determinize the mean and variance of Gaussian distributions, respectively. Given the above generative process, the optimal parameters and cluster assignment can be obtained by maximizing the likelihood of the given data points as:
\begin{align}
    \log p({\bf{x}}) = \log \int_{\bf{z}} {\sum\limits_c {p ({\bf{x}}|{\bf{z}})p(\mathbf{z}|c)p(c)\bf{d}{\bf{z}}}}
    \end{align}

However, directly optimizing the above likelihood is intractable as it involves integration and complex neural network. Variational Auto-Encoder (VAE)\cite{kingma2013auto} sheds a light to tackle this problem such that the parameters and the posterior can be efficiently estimated via backpropagation. Specifically, the generative model is trained with the following variational inference objective, a.k.a.  the evidence lower bound (ELBO):
\begin{align}
{{\mathcal L}_{{\mathop{\rm ELBO}\nolimits} }}({\bf{x}}) = {\mathbb E_{q({\bf{z}},c| {\bf{x}})}}\left[ {\log \frac{{p({\bf{x}},{\bf{z}},c)}}{{q({\bf{z}},c|{\bf{x}})}}} \right]
\end{align}
where $q(\bf{z}, c|\bf{x};\varphi)$ is the variational posterior to approximate the true posterior, which can be modelled with the recognition networks $\varphi$. Monte Carlo \cite{kalos2009monte} and the reparameterization trick \cite{kingma2013auto} can be employed to learn the parameters.

More recently, upon VaDE and GMVAE, some improved variants have been proposed. For example, Prasad et al.~ \cite{prasad2020variational} introduced a data augmentation technique that constrains an input instance (e.g.,  image) to share a similar clustering distribution with its augmented one; Li et al.~ \cite{li2020unsupervised} employed Monte Carlo objective and the Std Annealing track for optimizing mixture models, which would generate better-separated embeddings than the basic VAE-based methods; Ji et al.~ \cite{ji2021decoder} proposed to replace decoder in VAE with an improved mutual-information-based objective; Wang et al.~ \cite{wang2022neural} proposed to separate the latent embeddings into two parts which capture the particularity and commonality of the clusters, respectively.

\subsubsection{Deep Generative clustering based on Generative Adversarial Network}

Recent years have witnessed the great success of Generative Adversarial Network (GAN) in estimating complex data distribution \cite{goodfellow2014generative,jabbar2021survey,arjovsky2017wasserstein}. A standard GAN contains two components: a generator $G$ that targets synthesizing "real" samples to fool the discriminator, and a discriminator $D$ tries to discriminate the real data from the generated samples. With the adversary between the two components, the generator could generate samples that have a similar distribution to the data. Inspired by such outstanding ability, it would be promising to integrate GAN into generative clustering models. Specifically, Ben-Yosef et al.~ \cite{ben2018gaussian} proposed to stack a GMM with a GAN, where GMM serves as a prior distribution for generating data instances. Formally, they optimized the following objective function:
\begin{align}
\min _{G} \max _{D} V(D, G)=\underset{\mathbf{x} \sim p_{\mathcal{X}}(\mathbf{x})}{\mathbb{E}}[\log D(\mathbf{x})]+\underset{\mathbf{z} \sim p_{\mathcal{Z}}(\mathbf{z})}{\mathbb{E}}[\log (1-D(G(\mathbf{z})))]
\end{align}
where $p_{\mathcal{X}}(\mathbf{x})$ denotes the training data distribution; $p_{\mathcal{Z}}(\mathbf{z})$ is a prior distribution of $G$ and defined as mixture of Gaussians:
\begin{align}
    {p_{\mathcal Z}}({\bf{z}}) = \sum\limits_{k = 1}^K {{\pi _k}} * {\mathcal N}\left( {{\bf{\mu }}_c,{{\bf{\sigma }}^2_c}{\bf{I}}} \right)
\end{align}
By equipping GAN with such multi-modal probability distribution, the model could provide a better fit to the complex data distribution especially when the data includes many different clusters.

There are also some improved variants. For example, Yu et al.~\cite{yu2018mixture} proposed to directly replace the Gaussian distribution of GMM with a GAN and developed a $\epsilon$-expectation- maximization learning algorithm to forbid early convergence issues; Ntelemis et al.~ \cite{ntelemis2021image} proposed to employ Sobel operations prior to the discriminator of the GAN; Mukherjee et al.~ \cite{mukherjee2019clustergan} proposed to sample the latent vector $z$ from a mixture of one-hot encoded variables and continuous latent variables. An inverse network with a clustering-specific loss is introduced to make the model more friendly to the clustering task. Analogously, an inverse network is introduced in \cite{ghasedi2019balanced,gan2021learning} for the feature-level (i.e.,  latent vector) adversary.

\textbf{Summary.} Although deep generative clustering models can generate samples while completing clustering, they also have some weaknesses: 1) Training a generative model usually involves Monte Carlo sampling, which may incur training unstable and high computational complexity; 2) The mainstream generative models are based on VAE and GAN, and inevitably inherit the same disadvantages of them. VAE-based models usually require prior assumptions on the data distributions, which may not be held in real cases; although GAN-based algorithms are more flexible and diverse, they usually suffer from mode collapse and slow convergence, especially for the data with multiply clusters.

\subsubsection{Iterative deep clustering with individual supervision}
The individual supervision in iterative deep clustering depends on the pseudo labels generated by the clustering module, which can be utilized to train the representation learning module in a supervised manner.

In early works \cite{yang2017towards,song2013auto}, the cluster centroids and assignments are updated in a K-means way.
$\text{S}^2\text{ConvSCN}$ \cite{zhang2019self} and PSSC \cite{lv2021pseudo} combine subspace clustering and pseudo labeling, which obtain pseudo labels by spectral clustering or partitioning the pseudo similarity graph.
Later, a lot of works \cite{caron2018deep,van2020scan,niu2021spice} tend to utilize neural networks for both representation learning and clustering, where these two parts are combined together as one neural network. The clustering module is usually a multilayer perceptron(MLP) that produces soft clustering assignments. In this way, the hard pseudo labels can guide both clustering and representation learning through gradient backpropagation with proper constraints.
The representative method is DeepCluster \cite{caron2018deep},
which alternates between K-means clustering
and updating the backbone along with the classifier by minimizing the gap between predicted clustering assignments and pseudo labels.
In fact,  DeepCluster has already been applied as a mature clustering algorithm in video clustering \cite{alwassel2020self}.

Recently, SCAN \cite{van2020scan} follows a pretraining-with-finetuning framework. The clustering results are fine-tuned with self-labeling, which selects the highly confident instances by thresholding the soft assignment probability, and updates the whole network by minimizing the cross-entropy loss on the selected instances.
SPICE \cite{niu2021spice} is another representative iterative deep clustering method, where the classification model is first trained with the guidance of pseudo labels and then retrained by the semi-supervised training on the set of reliably labeled instances.

\subsubsection{Iterative deep clustering with relational supervision}
The relational supervision in iterative deep clustering refers to the relationship based on the pesudo labels, which provides pairwise guidance to the representation learning module.
More specifically, the relationship is usually modeled by whether two instances have same discrete pseudo labels \cite{yang2016joint,chang2017deep} and the model is trained as a binary classification task.
Another popular branch of methods \cite{zhang2019neural, niu2020gatcluster} model the relationship by the similarity of cluster assignment probabilities, which train the representation learning as a regression task.

\textbf{Summary.}
Iterative deep clustering methods benefit from the mutual promotion between representation learning and clustering.
However, they also suffer from the error propagation in the iterative process. More specifically, the inaccurate clustering results can lead to chaotic representations where the performance are limited by the self-labeling effectiveness.
Furthermore, this will in turn affect the clustering results especially in the early stage of training.
Therefore, existing iterative clustering methods heavily rely on the pretraining of representation module.
\subsection{Deep Generative Clustering}

\subsection{Not grouped yet}

\subsubsection{Multi-stage}
\begin{enumerate}

    \item (KBS2022) Deep image clustering by fusing contrastive learning and neighbor relation mining\cite{xu2022deep}
\end{enumerate}

\subsubsection{Multi-view Clustering}
\begin{enumerate}
    \item (CVPR2021) COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction\cite{lin2021completer}
    \item (TPAMI2021) Efficient and effective regularized incomplete multi-view clustering\cite{liu2020efficient}
    \item (ICCV2021) One-pass Multi-view Clustering for Large-scale Data\cite{liu2021one}
    \item (AAAI2020) Unified graph and low-rank tensor learning for multi-view clustering\cite{wu2020unified}
\end{enumerate}

\subsubsection{Others}
\begin{enumerate}
    \item The single-noun prior for image clustering
    \item Agglomerative clustering using the concept of mutual nearest neighbourhood.
    \item Self-tuning spectral clustering
    \item Progressive self-supervised clustering with novel category discovery
    \item Associative deep clustering: Training a classification network with no labels
    \item $\color{green}\checkmark$ (ICASS)Deep clustering: Discriminative embeddings for segmentation and separatio
\end{enumerate}

\subsection{Representation Learning}
\begin{enumerate}
    \item $\color{green}\checkmark$ (CVPR 2018) Unsupervised Feature Learning via Non-Parametric Instance Discrimination\cite{wu2018unsupervised}

    \item $\color{green}\checkmark$ (Arxiv 2021) Learning Representations by Contrasting Clusters While Bootstrapping Instances\cite{lee2020learning}

    \item $\color{green}\checkmark$ Consensus Clustering With Unsupervised Representation Learning\cite{regatti2021consensus}
\end{enumerate}

\subsection{Application}
\begin{enumerate}
    \item $\color{green}\checkmark$(ICCV2021) End-to-end robust joint unsupervised image alignment and clustering\cite{zeng2021end}
\end{enumerate}

\subsubsection{Subspace based Multistage Clustering}
deep subspace clustering was proposed to learn an affinity matrix and instance representations first, and then performs clustering by spectral clustering on the affinity matrix \cite{ji2017deep,zhang2021learning,zhou2018deep} or the K-means on the instance representations \cite{peng2016deep}.
Thanks to the contribution of scikit-learn \cite{scikit-learn} and many other open-source machine learning libraries, clustering algorithms has been applied to many fields with a limited cost of programming. For example, in the scenario of textual/video/graph data clustering, relation (similarity) matching was used in \cite{chang2005using, tseng2010generic,kong2020learning}, K-Means in \cite{cerquitelli2017data}, Spectral Clustering in \cite{janani2019text,peng2021deep,alami2021unsupervised} and Hierarchical Agglomerative Clustering in \cite{shen2018improving}, so as many other clustering algorithms being directly applied. In special, graph cut based node clustering like Metis \cite{karypis1998fast}, Graclus \cite{dhillon2007weighted} and Balance Normalized Cut (BNC) \cite{chiang2012scalable} were used in graph clustering applications \cite{chiang2019cluster,nasrazadani2022sign,wu2016cccf}.

The most recent multistage methods have explicitly incorporated the clustering prior into the representation learning, then conduct clustering on the target friendly representations.
For example,
IDFD \cite{tao2021clustering} learn representations with two aims: learning similarities among instances and reducing correlations within features.
With the above explicit purposes, a naive K-means on the learned representations can also achieve competitive clustering results over many existing deep clustering methods.

\textbf{Summary.}
Multistage methods enjoy the property of fast deployment, programming friendly, and intuitive understanding.
However, such a simple combination of deep representation learning and shallow (traditional) clustering has the following weakness:

    1) Most representation learning methods are not intently designed for clustering tasks, which can not provide sufficient discriminative power for clustering. Specifically, clustering reflects global patterns among data instances, while existing representation learning methods have largely focused on the individual patterns.
    2) The clustering results can not be further utilized to guide the representation learning, which is of great importance for comprehensive representation. In particular, the cluster structure implies the inherent relationships among data instances, and can in turn serve as critical guidance for improving representation learning.
To conclude, such straightforward cascade connection will cut off the information flow and interactions between representation learning and clustering, thus the limitations of both side will influence the final performance together.

\subsection{Transfer Learning}
Despite the success of deep clustering in mining global pattern among instances, it has also benefit various downstream tasks.
In this section, we discuss some typical applications of deep clustering.

\subsection{Community Detection}
Community Detection \cite{fortunato2010community,liu2020deep,jin2021survey} aims at partitioning the graph network into several sub-graphs mainly according to connection density, which can be treated as node-level graph clustering task.
Early works are mainly based on modularity measurement \cite{girvan2002community,blondel2008fast}, Maximum flows \cite{flake2002self}, graph cut \cite{shi2000normalized} and its extension, spectral methods \cite{von2007tutorial}.
With the development of Graph Nerual Networks (GNNs)\cite{ma2019disentangled,zhou2019hahe}, nodes are represented as individual instances in the low-dimensional space.
As a result, the border between modern community detection \cite{su2022comprehensive} and graph clustering\cite{li2017semi,li2020schain,zhou2020cross} is gradually getting blurry, and GNN based graph clustering \cite{tsitsulin2020graph,bianchi2020spectral} have already been applied to many applications.
However, different from early community detection that focus on the network topology, the graph clustering usually incorporates the node attributes and other side information.
How to release the power of GNNs while reserve the topology characteristic is still under study.

\subsection{Anomaly Detection}
\label{app:anomaly}
Anomaly Detection (a.k.a.  Outlier Detection, Novelty Detection) is a technique for identifying abnormal instances or patterns among data.
Early before deep clustering, density based clustering methods \cite{ester1996density,schubert2017dbscan,comaniciu2002mean} have specifically mentioned and addressed the problem of noise during clustering, which has enlighted a group of density based anomaly detection methods \cite{ccelik2011anomaly,chen2011anomaly}.
The later anomaly detection methods \cite{markovitz2020graph,wang2020cluster,ma2021comprehensive} have utilized the clustering results and identify the anomaly instances as those far from the cluster centroids or border of each cluster.
Currently, the deep clustering has shown great potential in forming a better clustering space for anomaly detection.
Instead of performing anomaly detection after the deep clustering, the recent efforts have been put on conducting them in a unified framework: identify and remove the anomaly instances to reduce the impact on clustering \cite{liu2019clustering}, and anomaly detection can be further improved with better clustering results.

\subsection{Segmentation and Object Detection}
Image Segmentation is one of the most important approaches to simulating human understanding of images which aims at dividing the pixels into disjoint regions.
Generally speaking, image segmentation is doing pixel classification in a supervised manner and pixel clustering in an unsupervised manner \cite{coleman1979image,arifin2006image}.
Currently, the deep clustering has been successfully applied in segmentation by
utilizing the clustered regions to generate Scene Graph \cite{yang2018graph}.
Yi et al.~ \cite{yi2012image} surveyed graph cut based image segmentation, where graph cut is one of the most fundamental solutions to perform clustering (Spectral Clustering).
3D clustering can be a solution to 3D Object Detection, like in \cite{chen2022self}, 3D points were clustered to represent an object with geometric consistency.
But such clustering based segmentation and object detection has no guarantee for small region and object, where the expected clustering result is highly unbalanced. And the global positional information of a pixel may be ignored when peoforming clustering.

\subsection{Medical Applications}
\label{medical}
The convolutional neural network has successfully promoted the development of medical image processing in a supervised manner.
However, the manual dataset labeling process is often labor-intensive and requiring expert medical knowledge \cite{kart2021deepmcat}, which is hard to realize in the real-world scenarios.
Recently, the deep clustering has been introduced to automatically categorize large-scale medical images \cite{kart2021deepmcat}.
Mit al et al.~ \cite{mittal2021new} introduce medical image clustering analysis for faster COVID-19 diagnostic.
In the biological science field, single cell RNA sequencing (scRNA-seq) \cite{eraslan2019single} gives an output cell-gene matrix for cell population and behaviour analysis and even new cell discovery. For this purpose, ScDeepCluster \cite{tian2019clustering} and ItClust \cite{hu2020iterative} develop their models based on DEC \cite{xie2016unsupervised} to cluster scRNA-seq data, and MARS \cite{brbic2020mars} combining transfer learning and clustering to discover novel cell types. More application can be found in gene data clustering area\cite{gao2006discovering}.

\subsection{Summary}
In addition to the above successful deep clustering applications, the clustering also has great potential in many other domains, such as
financial analysis \cite{jaiswal2020green,govindasamy2018cluster,close2020combining},
trajectory analysis \cite{bandaragoda2019trajectory,olive2019trajectory},
and social media understanding \cite{yue2019survey,orkphol2019sentiment,zhu2014tripartite}.
Although most of the aforementioned methods has not incorporate deep learning techniques, with the increasing data volume and dimension, we believe that deep clustering will pave the way for these scenarios.
\subsection{Auto-Encoder based Representation Learning.}
Auto-Encoder \cite{rumelhart1985learning} is one of the most widely adopted unsupervised deep representation learning methods for its simplicity and effectiveness.
The auto-encoder is a linear stack of two deep neural networks named encoder and decoder.
The encoder $\mathbf{f}_{e}$ encodes the input data $x$ into low-dimensional representation $h = \mathbf{f}_{e}(x)$, and
the decoder $\mathbf{f}_{d}$ decodes the low-dimensional representation $h$ to the input data space $\hat{x}=\mathbf{f}_{d}(h)$.
A good auto-encoder is expected to reconstruct the input data without dropping any information, thus the optimization target of auto-encoder is usually formulated as:
\begin{equation}
    \mathcal{L}_{AE}=\sum_{i}^{N}\left\|x_{i}-\hat{x}_{i}\right\|_{2}^{2} = \sum_{i}^{N}\left\|x_{i}-\mathbf{f}_{d}\left(\mathbf{f}_{e}\left(x\right)\right)\right\|_{2}^{2}
\end{equation}
where $\left\|\cdot\right\|_{2}$ is the L2-norm, $\hat{x}_{i}$ is the reconstructed data.
The auto-encoder is a general structure and can be customized for different data types.
For example, deep neural networks for vectorized features \cite{zhai2018autoencoder}, convolutional networks for images and graph neural networks for graphs \cite{pan2018adversarially}, 3D convolutional networks and LSTM auto-encoder for videos \cite{srivastava2015unsupervised} etc..

\textbf{Analysis.}
The auto-encoder based representation learning framework enjoys the property of easy implementation and efficient training, and has been widely adopted in the early works of deep clustering.
However, the representations are learned in an instance-wise manner while the relationships among different instances are largely ignored.
As a result, the instances in the embedding space may not be well discriminated from each other, resulting in poor clustering performance.

\subsection{Deep Generative Representation Learning.}
Another line of deep unsupervised representation learning lies in the generative model.
Generative methods assume that the data $x$ are generated from latent representation $h$ and then reversely deduce the posterior of the representation $p(h|x)$ from the data.
Among these, the most typical method is variational auto-encoder (VAE) \cite{kingma2013auto}. VAE resorts to the variance inference technique and maximizes the evidence lower bound (ELBO) on the data likelihood:
\begin{equation}
\log p(x) \geq\mathbb{E}_{q(h|x)}[\log p(x|h)]-D_{K L}(q(h|x) \| p(h)) \label{eq:vae}
\end{equation}
where $D_{KL}{(\cdot\|\cdot)}$ denotes the KL-divergence between two distributions, $p(h)$ is the prior distribution of the latent representation, $q(h|x;\varphi)$ is the variational posterior of the representation to approximate the true posterior (\textit{i.e.,} $q(h|x;\varphi)\approx p(h|x)$), which can be modelled with the recognition networks $\varphi$. By utilizing the reparameterization trick \cite{kingma2013auto} and Monte Carlo approximation \cite{kalos2009monte}, the posterior can be efficiently learned from the equation (\ref{eq:vae}) via backpropagation.

\textbf{Analysis.} Deep generative models enjoy some merits such as being flexible, explainable and capable to recreate data points. It would be promising to transform the generative representation models for deep clustering task so that the clustering models could inherit these advantages.

\subsection{Mutual Information Maximization Representation Learning.}
\label{subsec:mi_rl}
Mutual information (MI) \cite{kinney2014equitability} is a fundamental quantity for measuring the dependence between random variables $X$ and $Y$, which is formulated as:
\begin{equation}
\mathcal{I}(X ; Y)=\int \log \frac{d \mathbb{P}_{X Y}}{d \mathbb{P}_{X} \otimes \mathbb{P}_{Y}} d \mathbb{P}_{X Y}
\end{equation}
where $\mathbb{P}_{X Y}$ is the joint distribution, $\mathbb{P}_{X} = \int_{Y} d \mathbb{P}_{X Y}$ and $\mathbb{P}_{Y} = \int_{X} d \mathbb{P}_{X Y}$ are the marginal distribution, $\mathbb{P}_{X} \otimes \mathbb{P}_{Y}$ is the product of the marginal distributions.
Traditional mutual information estimations \cite{kraskov2004estimating} are only tractable for discrete
variables or known probability distributions.
Recently, MINE \cite{belghazi2018mutual} is proposed to estimate the mutual information with deep neural networks.
The widely used mutual information estimation is the Jensen-Shannon divergence (JSD) \cite{nowozin2016f} formulated as:
\begin{equation}
    \mathcal{I}_{JSD}(X;H)=\mathbb{E}_{\mathbb{\mathbb{P}}_{X H}}[-\operatorname{sp}(-D(x, h))]-\mathbb{E}_{\mathbb{P}_{X}\times \mathbb{P}_{H}}[\operatorname{sp}(D(x, h))]
\end{equation}
where $\operatorname{sp}(x)=\log \left(1+e^{x}\right)$ is the softplus function.
$D$ is a discrimintor function modeled by a neural network.
Another popular mutual information estimation is InfoNCE \cite{oord2018representation} which will be introduced in subsection \ref{subsec:contrastive}.
Benefiting from the neural estimation, the mutual information has been widely adopted in unsupervised representation learning \cite{hjelm2018learning,bachman2019learning}.
More specifically, the representation is learned by maximizing the mutual information between different layers \cite{bachman2019learning} or different parts of the data instances \cite{hjelm2018learning}, so that the consistency of representation can be guaranteed.
This can be viewed as an early attempt at self-supervised learning which has an extensive impact on the later works.

\textbf{Analysis.}
As a fundamental measure of correlation and dependence, the mutual information has several advantages.
The major advantage taken by the deep clustering task is that variables measured by mutual information are not restricted to same dimension and semantic space, such as instances and clusters.
The detailed applications will be introduced in subsection \ref{subsec:mi_clustering} and \ref{subsec:mi_model}.
Similar to auto-encoder based and deep generative representation learning, the objective of mutual information maximization methods is also instance-wise, which may also have the aforementioned problems in capturing the relationships among instances.
However, the marginal distribution in mutual information estimation depends on all the observed samples.
In other words, the relationships among instances are captured in an implicit way, which has also boosted the performance in deep clustering.

\subsection{Contrastive Representation Learning}
\label{subsec:contrastive}
Contrastive learning is one of the most popular unsupervised representation learning techniques in the past few years.
The basic idea is to pull positive pair of instances close while push negative pair of instances far away, which is also known as instance discrimination.
The representative target of contrastive learning is the InfoNCE loss \cite{oord2018representation} formulated as:
\begin{equation}
    \mathcal{L}_{InfoNCE}=-\log \sum^{N}_{i=1}\frac{\exp \left(f\left(h_{i}, h^{\mathcal{T}}_{i}\right) / \tau\right)}{\sum_{j=1}^{N} \exp \left(f\left(h_{i}, h_{j}^{\mathcal{T}}\right) / \tau\right)}
\end{equation}
where $h_{i}$ is the representation of the anchor sample, $h^{\mathcal{T}}_{i}$ is the positive sample representation and $h_{j}$, $h_{j}^{\mathcal{T}}$ are the negative sample representations,  $f$ is a similarity function, $\tau$ is the temperature parameter \cite{hinton2015distilling}.
The positive samples are usually conducted by data augmentation which may vary from different data types.
For example, the flip, rotate and crop augmentation for image data \cite{chen2020simple}, the node dropping, edge perturbation, attribute masking and subgraph sampling for graph data \cite{you2020graph,li2021disentangled}.
The negative samples are selected from augmented view of other instances in the dataset \cite{chen2020simple} or a momentum updated memory bank of old negative representations \cite{he2020momentum}, which can be viewed as approximation of noise.

\textbf{Analysis.}
There has been several theoretical analyses on contrastive learning, and substantial evidences have shown that the representation learned by contrastive learning can benefit the clustering task.
In \cite{wang2020understanding}, contrastive learning is explained with two properties: alignment of features from positive pairs and uniformity of the feature distribution on the hypersphere.
The alignment property encourages the samples with similar features or semantic categories to be close in the low-dimensional space, which is essential for clustering.
Such discriminative power has also been proved in the supervised manner \cite{khosla2020supervised}.

The former work \cite{oord2018representation} has proved that minimizing the InfoNCE loss is equivalent to maximizing the lower bound of mutual information, while the data augmentation is not considered in the mutual information maximization.
Recent study \cite{wang2021chaos} has shown that the augmentation has played the role of `ladder' in connecting instances within same category. As a result, instances from same cluster may be pulled closer which can benefit the clustering in the discriminative space.

\subsection{Clustering Friendly Representation Learning.}
\label{subsec:cluster_friend}
Although the aforementioned representation learning methods have somehow implicitly boosted the performance of clustering, they are not explicitly designed for the clustering task.
In this subsection, we will introduce some representation learning methods that explicitly support the clustering task.

K-means \cite{lloyd1982least} friendly representation is first defined in DCN \cite{yang2017towards} where samples in the low-dimensional space are expected to scatter around their corresponding cluster centroids.
This can well support the assumption of K-means that each sample is assigned to the cluster with the minimum distance to the centroid.
The objective $\mathcal{L}_{KF}$ can be formulated as:
\begin{equation}
    \mathcal{L}_{KF} = \sum^{N}_{i=1}\left\|\mathbf{f}\left(x_{i}\right)-\tilde{y}_{i}\mathbf{C} \right\|_{2}^{2}
\end{equation}
where $\mathbf{f}(\cdot)$ is the deep neural network for representation learning,
$\tilde{y}_{i}$ is the hard assignment vector of data instance $x_{i}$ which has only one non-zero elements,
$\mathbf{C}$ is the cluster representation matrix where $k$-th column of $\mathbf{C}$, i.e., $c_{k}$ denotes the centroid of the $k$-th cluster.

Spectral clustering friendly representation learning is inspired by the eigen decomposition in spectral clustering \cite{ng2001spectral} that projects instances into the space with orthogonal bases.
In deep clustering, the orthogonal basis is modeled explicitly by reducing correlations within features \cite{tao2021clustering}, which can be formulated as:
\begin{equation}
    \mathcal{L}_{SF}=\sum_{m=1}^{d}\left(-h_{m}^{T} h_{m} / \tau+\log \sum_{n}^{d} \exp \left(h_{n}^{T} h_{m} / \tau\right)\right)
\end{equation}
where $d$ is the number of feature dimensions, $h_{m}$ is the $m$-th dimension feature vector, $\tau$ is the temperature parameter.
The objective is to learn the independent features so that redundant information is reduced.

\textbf{Analysis.}
The clustering friendly representation learning benefits from the direct optimization for clustering, which may significantly boost the corresponding clustering performance.
However, such simplicity also limits the generalization to other clustering methods.
Currently, the research community has put more efforts on the inspirations of clustering methods and express them in a deep learning perspective, rather than learning specific representations for each clustering method.

\subsection{Subspace Representation Learning}
\label{subsec:subspace}
Subspace representation learning is the early stage of subspace clustering \cite{vidal2009sparse}, which aims at mapping the data instances into a low-dimensional subspace where instances can be separated.
Basically, current subspace representation learning methods \cite{zhang2019self,zhang2019neural,ji2017deep,zhang2021learning,peng2016deep,zhou2018deep} have relied on the self-expressiveness assumption where a data instance can be expressed as a linear combination of other data instances from the same subspace,
i.e., $\mathbf{X=X \Theta_c}$, where $\mathbf{X}$ is the data matrix and $\mathbf{\Theta_c}$ is the self-expression coefficient matrix. For representation learning, the self-expression property leads to the following objective:
\begin{equation}
    \mathop{\text{min}}_\mathbf{\Theta_c}\ \|\mathbf{\Theta_c}\|_p + \frac{\lambda}{2} \|\mathbf{H-H \Theta_c} \|_F^2 \quad \text{s.t.} \quad \text{diag}(\mathbf{\Theta_c})=\textbf{0}
\end{equation}
where $\|\cdot\|_p$ is a matrix norm, $\lambda$ controls the weight balance. $\textbf{H}$ denotes the sample representations learned by a network. Further, $\mathbf{\Theta_c}$ can be implemented as the parameters of an additional network layer \cite{ji2017deep}.

\textbf{Analysis.}
The success of subspace representation learning relies on the theoretical assurance, which provides explicit modeling of relationships among data instances.
However, it suffers from the limitation of solving the coefficient matrix of size $N \times N$, which is computationally difficult for large-scale data.

\subsection{Data type specific representation learning}
The former subsections have summarized the universal architectures of representation learning module.
In real-world scenarios, representation learning for different data types can be viewed as the variants of the aforementioned architectures.
In this subsection, we summarize four widely studied data types and their corresponding representation learning methods in deep clustering.

\subsubsection{Image Representation Learning}
Learning representations of images using CNN \cite{lecun1998gradient} and ResNet \cite{he2016deep} as backbone has achieved great success in the past decades.
In the image deep clustering, they still play active roles as feature extractors or backbones in representation learning module.
Beyond the above two methods, recent advances have been made by introducing modern representation learning techniques such as vision transformer \cite{dosovitskiy2020image} to deep clustering.
As one of most popular directions, the unsupervised representation learning for image data will play a central role in deep clustering and affect the other data types.

\subsubsection{Text Representation Learning}
The early attempts of text representation learning have utilized the statistical based methods like TF-IDF \cite{steinbach2000comparison}, Word2Vec\cite{le2014distributed} and Skip-Gram \cite{mikolov2013efficient}.
Later, some works focus on the topic modeling \cite{hofmann2013probabilistic} and semantic distances \cite{dzisevivc2019text,prasad2019hybrid} for text representation learning, and more \cite{chung2019unsupervised} on unsupervised scenarios.
Recently, the pre-trained language models like BERT \cite{devlin2018bert} and GPT-3 \cite{brown2020language} are gradually dominating the area of text representation learning.
However, the fine tuning \cite{zhang2021fast} of these methods in deep clustering task is still an open question.

\subsubsection{Video Representation Learning}
The video representation learning is a challenging task which combines the spatial-temporal learning, multi-model learning (with audio) \cite{alwassel2020self}, and natural language processing (with video abstract and subtitles) into one place.
The early methods utilize LSTM Autoencoder \cite{srivastava2015unsupervised}, 3D-ResNets \cite{qian2021spatiotemporal} and 3D-U-Net \cite{peng2021deep} as feature extractor.
The recent methods have focused on spatial-temporal modeling \cite{feichtenhofer2021large,srivastava2015unsupervised,zhang2016unsupervised,zhang2018salient} and Qian et al.~ \cite{qian2021spatiotemporal} in particular incorporates contrastive learning for self-supervision.

\subsubsection{Graph Representation Learning}
\label{community}
The classic graph representation learning aims at learning low dimensional representation for nodes so that the proximity among nodes can be preserved in the embedding space.
Graph Neural Networks (GNNs) \cite{zhang2020deep,wang2016structural} are widely used including
GCN \cite{kipf2016semi}, GraphSAGE \cite{hamilton2017inductive} and GAT \cite{velivckovic2017graph}, brings infinite possibility of graph node representation learning combining node features and graph topology\cite{wang2021explainable,zhang2020learning,zhu2020deep,zhou2020dge}.
Furthermore, the graph-level information also has great potential in tasks like proteins classification \cite{borgwardt2005protein}, which has drown increasing attention in graph-level representation learning \cite{zhang2019hierarchical,zhang2018anrl,wu2014bag,guo2017combining}.

\textbf{Analysis.}
The data type specific representation learning mentioned above can be naive backbone for feature extraction or end-to-end unsupervised representation learning, which are most active research directions in deep learning.
With more types of data being collected and fast development of deep learning, we believe that the deep clustering with grow along with the data type specific representation learning techniques.
\subsubsection{Auto-encoder with self-training}
Auto-encoder is a powerful tool to learn data representations in an unsupervised way, and has been utilized since the first attempts of simultaneously deep clustering \cite{huang2014deep}.

The representative method is DEC \cite{xie2016unsupervised} which combines the auto-encoder with the self-training strategy.
Such simple but effective strategy has deeply influenced the follow-up works.
The auto-encoder is pre-trained and only the encoder is utilized as the initialization of the representation learning module.
The self-training strategy mentioned in subsection \ref{subsec:self-training} is then introduced to optimize the clustering and representation learning simultaneously.

Based on the vanilla DEC method, many variants and improvements are proposed.
To preserve the local structure of each instance,
IDEC \cite{guo2017improved} further integrates the reconstruction loss to the auto-encoder.
The general formulation of auto-encoder with self-training can be summarized as:
\begin{equation}
    \mathcal{L}_{AEST}=\mathcal{L}_{AE}+\mathcal{L}_{ST}
\end{equation}
where $\mathcal{L}_{AE}$ is the loss of auto-encoder and $\mathcal{L}_{ST}$ is the loss of clustering oriented self-training, e.g. the neighborhood constraint in DEC.
To improve the capability of auto-encoder, some efforts are made to adapt different data types.
In \cite{ghasedi2017deep,li2018discriminatively,guo2018deep}, the linear layer of auto-encoder is replaced with fully convolutional layers so that the image feature can be well captured.
In CCNN \cite{hsu2017cnn}, the clustering convolutional neural network is proposed as a new backbone to extract the representations which are friendly to the clustering task.
In DEPICT \cite{ghasedi2017deep}, an additional noisy encoder is introduced, and the robustness of auto-encoder is improved by minimizing the reconstruction error of every layer between the noisy decoder and the clean encoder.

Although the self-training strategy has achieved success, later works also make attempts for specific problem.
To boost the robustness of clustering, self-training is applied between two branches: clean and augmented \cite{guo2018deep} (noisy \cite{ghasedi2017deep}). Specifically, the target distribution $\mathbb{P}$ is computed via the clean branch to guide the soft assignments $\mathbb{Q}$ of the augmented or noisy branch.
Self-training strategy can be combined with subspace clustering. CSC \cite{peng2017cascade} introduces the assumption named invariance of distribution, i.e. the target distribution $\mathbb{P}$ should be invariant to different distance metrics in subspace space. Therefore, two metrics (Euclidean and Cosine distance) are used to compute the target distributions $\mathbb{P}_E$ and $\mathbb{P}_C$, with the KL divergence between them minimized.

According to aforementioned analysis, self-training is similar to the K-means clustering and suffers from the unbalance problem \cite{buda2018systematic} between different clusters.
To solve the problem of the unbalanced data and out-of-distribution samples, StatDEC \cite{rezaei2021learning} improves target distribution by adding normalized instance frequency of clusters.
In this way, the model can preserve discrimination of small groups and form a local clustering boundary which is insensitive to unbalanced clusters.

\subsubsection{Mutual Information Maximization based Clustering}
\label{subsec:mi_model}
As illustrated in subsection \ref{subsec:mi_rl} and \ref{subsec:mi_clustering}, the mutual information maximization has been successfully applied in both representation and clustering.
The unified form of mutual information in both modules has brought convenience for understanding and implementation.

The representative mutual information maximization based clustering method is DCCM \cite{wu2019deep}.
For each data instance, the mutual information between deep and shallow layer representation is maximized so that the consistency of representation can be guaranteed.
Such consistency is further extended to the cluster assignment space by encouraging the instances with the same pseudo labels to share similar representations.
Many later works can be regarded as variants of this method.
In VCAMI \cite{ji2021variational} and IIC \cite{ji2019invariant}, the augmented mutual information (AMI) is introduced to improve the robustness.
Such augmentation invariant has inspired the later contrastive deep clustering methods, which will be introduced in the next subsection.
In ImC-SWAV \cite{ntelemis2021information}, the mutual information between the integrated discrete representation and a discrete probability distribution is maximized, which improves the vanilla SWAV \cite{caron2020unsupervised} method.

\subsubsection{Contrastive Deep Clustering}
Similar to mutual information maximization based deep clustering, contrastive learning has also been successfully applied in both representation learning module and clustering module.
The main idea of contrastive learning is pulling similar instances closer while pushing different instances away, which is in the spirit of clustering that instances from the same cluster should be close while instances from different clusters should be apart.

The representative contrastive deep clustering method is CC \cite{li2021contrastive}.
The basic idea is to treat each cluster as the data instance in the low-dimensional space.
The instance discrimination task in contrastive representation learning can be migrated to the clustering task by discriminating different clusters, which is the fundamental requirement of clustering.
Furthermore, the advantage of augmentation invariant and local robustness can be preserved in the clustering task.

Take CC as the fundamental architecture, many contrastive deep clustering can be viewed as variants of it.
PICA \cite{huang2020deep} can be viewed as the degeneration of CC without augmentation, it directly separates different clusters by minimizing the cosine similarity between the cluster-wise assignment statistic vectors.
In DCCM \cite{wu2019deep}, the augmentation is introduced to guarantee the local robustness of the learned representation.
DRC \cite{zhong2020deep} has the same contrastive learning as CC where the cluster representation is called assignment probability.
The difference lies in the cluster regularization which is inspired by group lasso \cite{meier2008group}.
In CRLC \cite{do2021clustering}, contrastive learning is performed between the cluster assignments of two augmented versions of the same instance, rather than the cluster representations.
Also, the dot product in contrastive learning is replaced by the log-dot product, which is more suitable for the probabilistic contrastive learning.
SCCL \cite{zhang2021supporting} extends this approach with textual data augmentation, which proves that this contrastive learning based self-training framework is universally applicable.

The later works further adopt the contrastive clustering in the semantic space.
In SCL \cite{huang2021deep}, the negative samples are limited by different pseudo labels,
so that instances from different clusters can be further distinguished.
MiCE \cite{tsai2020mice} follows a divide-and-conquer strategy where the gating function divides the whole dataset into clusters and the experts in each cluster aim at discriminating the instances in the cluster.
However, compared with InfoNCE that models alignment and uniformity implicitly\cite{wang2020understanding}, MiCE model these two properties in a more explicit way.
Recently, TCC \cite{shen2021you} further improve the efficiency with the reparametrization trick and explicitly improve the cluster discriminability.

Some other methods make attempts on overcoming the problems of vanilla contrastive learning.
In GCC \cite{zhong2021graph}, the positive pair and negative pair are selected by the KNN graph constructed on the instance representation.
This may be related to overcoming the `false negative' problem in contrastive learning.
In NCC \cite{huang2021exploring}, the contrastive learning module is replaced from SimCLR to BYOL \cite{grill2020bootstrap}, so that the over-reliance on the negative samples can be solved.

\subsubsection{Hybrid simultaneous deep clustering}
The aforementioned simultaneous deep clustering methods have remarkable characteristics and advantages, some other works are hybrids of the above techniques.
SCCL \cite{zhang2021supporting} and Sundareswaran et al.~ \cite{sundareswaran2021cluster} combine contrastive representation learning and self-training.
DDC \cite{chang2019deep} and RCC \cite{shah2017robust} combine relation matching clustering and pseudo labeling to boost the clustering performance. DCC \cite{shah2018deep} combines the auto-encoder based representation and the relation matching clustering.
The auto-encoder based representation learning and spectral clustering are combined in \cite{yang2019deep} by incorporating the augmentations from the contrastive learning.

\textbf{Summary.} The simultaneous deep clustering has attracted the most attention for its unified optimization.
Intuitively, the learned representation is clustering oriented and the clustering is conducted on the discriminative space.
However, it may incur from undesired prejudice of optimization focus between representation learning module and clustering module, which can only be mitigated by the manually setting of the balanced parameter for now.
Also, the model is easy to sink into degenerate solutions where all instances are assigned into one single cluster.

redpajama_set_name: RedPajamaArXiv


text:
Q: Laravel Response::json not returning JSON properly? Routes registered in app/routes.php
Route::resource('users', 'UsersController',
    array('except' => array('new', 'update')));

Route::post('users/authenticate', array('as' => 'authenticate', 'uses' => 'UsersController@authenticate'));

Route::get('users/is_authenticated', array('as' => 'authenticated', 'uses' => 'UsersController@is_authenticated'));

The method is_authenticated is not returning JSON, but when I put the Response::json() in the index method it returns the JSON schema.
Here is my is_authenticated method:
public function is_authenticated()
{
    return Response::json(['authenticated' => Auth::check()]);
}

What's going wrong here?  I ran php artisan routes and it returns this for the route:
GET|HEAD api/users/is_authenticated | authenticated     | UsersController@is_authenticated
I don't get a 404 Not Found when visiting the page, but there is no content.  What's the problem?
Edit: routes are prefixed with api

A: The problem is that your first route is activated when calling users/is_authenticated. The order of the routes is important in Laravel, as the first matching route is executed. You can just change the order of your routes to make the route users/is_authenticated available, like so:
Route::post('users/authenticate', array('as' => 'authenticate', 'uses' => 'UsersController@authenticate'));

Route::get('users/is_authenticated', array('as' => 'authenticated', 'uses' => 'UsersController@is_authenticated'));

Route::resource('users', 'UsersController', array('except' => array('new', 'update')));

redpajama_set_name: RedPajamaStackExchange


text:
Die montenegrinische Fußballnationalmannschaft () ist die Fußball-Auswahl des Fußballverbandes des südosteuropäischen Staates Montenegro.

Formierung
Bis 2006 war der Staat Montenegro Teil der Föderation Serbien und Montenegro und somit bestand auch die serbisch-montenegrinische Fußballnationalmannschaft, die zuletzt an der WM 2006 teilnahm und dort als schlechteste Mannschaft nach der Vorrunde ausschied. Im Kader standen aber keine Spieler aus Montenegro. Danach wurde die Mannschaft aufgelöst und die serbische Auswahl trat an Stelle der serbisch-montenegrinischen Auswahl, da Serbien als politischer Nachfolgestaat Serbien-Montenegros gilt. Der montenegrinische Fußballverband musste sich neu gründen.

Weil zum Zeitpunkt der Gruppenauslosung zur Qualifikation zur EM 2008 noch die serbisch-montenegrinische Mannschaft gelistet war, trat die serbische Auswahl in der zugelosten Gruppe an.

Geschichte
Montenegro wurde am 5. Oktober 2006 als provisorisches und am 26. Januar 2007 als vollwertiges UEFA-Mitglied aufgenommen und spielte das erste Länderspiel am 24. März 2007 gegen Ungarn im Pod Goricom in Podgorica. Montenegro gewann das Spiel durch zwei verwandelte Elfmeter von Mirko Vučinić und Igor Burzanović mit 2:1. Jedoch wird dieses Spiel von der FIFA nicht anerkannt, da Montenegro kein Mitglied war.

Am 31. Mai 2007 wurde Montenegro FIFA-Mitglied. Die Niederlage gegen Japan (0:2) am 1. Juni 2007 beim Kirin Cup wird von der FIFA als erstes offizielles Länderspiel gewertet.

Die Mannschaft konnte sich bisher weder für eine Fußball-Weltmeisterschaft noch für eine Fußball-Europameisterschaft qualifizieren. Montenegro nahm während der Qualifikation zur WM 2010 erstmals an einem offiziellen Wettbewerb teil und traf in Gruppe 8 auf Italien, Bulgarien, Irland, Zypern und Georgien. Der Mannschaft gelang ein Sieg gegen Georgien, sie spielte sechsmal unentschieden und belegte in der Abschlusstabelle den fünften Rang.

In der Qualifikation zur Fußball-Europameisterschaft 2012 spielte die Mannschaft gegen die Schweiz, England, Bulgarien und Wales. Am ersten Spieltag der Qualifikation siegte man mit einem 1:0 gegen die Waliser, wodurch der zweite Platz erreicht wurde. Auch im Laufe der Qualifikation konnte man den zweiten Platz halten. Hierbei gelangen auch sowohl ein 1:0-Sieg gegen die Schweiz sowie ein 0:0-Unentschieden gegen England. Zeitweise kämpfte man gar um den Gruppensieg. Dieses Vorhaben wurde nach einer 1:2-Niederlage gegen Wales wieder verworfen. Nun konzentrierte man sich auf den zweiten Platz – in einem Fernduell mit den Schweizern. Am vorletzten Spieltag der Qualifikation konnte man sich im Fernduell, gegen die Schweiz, durchsetzen. Im Rückspiel, gegen England, konnte man ein 2:2 erringen. Nachdem man bereits Mitte der ersten Halbzeit mit 0:2 zurücklag, markierte Elsad Zverotić kurz vor dem Pausenpfiff das 1:2. In der Nachspielzeit markierte Andrija Delibasic das 2:2. Zuvor unterlagen die Schweizer bei den Walisern mit 0:2. Das 2:2 der Montenegriner war auch gleichbedeutend mit der sicheren Teilnahme an den Play-offs. Gegner in den Play-offs war Tschechien. Das Hinspiel am 11. November in Prag ging 2:0 verloren, ebenso das Rückspiel mit 1:0. Damit verpasste man die Qualifikation zur Europameisterschaft.

Auch bei der Qualifikation für die Weltmeisterschaft 2014 reichte es in Gruppe H nur zu Platz 3 hinter England und der Ukraine. In der Qualifikation für die Fußball-Europameisterschaft 2016 traf Montenegro in Gruppe G auf Russland, Schweden, Österreich, Moldawien und Liechtenstein.

Auch die WM 2018 wurde verpasst, da Montenegro in der Qualifikation hinter Polen und Dänemark nur den dritten Platz belegte. Die Relegation der Gruppenzweiten wurde durch eine 2:4-Niederlage am letzten Spieltag gegen Polen bei gleichzeitigem Remis der Dänen gegen Rumänien verpasst, wodurch den Montenegrinern vier Punkte fehlten.

In der erstmals stattgefundenen UEFA Nations League 2018/19 spielte Montenegro in Liga C und traf dabei zwischen September und November 2018 auf Litauen, Rumänien und Serbien. Mit zwei Siegen, einem Remis und drei Niederlagen wurde der dritte Platz belegt.

In der Qualifikation für die Europameisterschaft 2021 traf Montenegro in Gruppe A auf England, Tschechien, Kosovo sowie Bulgarien, wurde Gruppenletzter und kassierte mit 0:7 gegen England seine höchste Länderspielniederlage.

In der UEFA Nations League 2020/21 spielt Montenegro wieder in Liga C und trifft dabei auf Aserbaidschan, Luxemburg und Zypern. Mit vier Siegen bei einer Heimniederlage gegen Luxemburg kam Montenegro letztendlich auf den ersten Tabellenplatz und stieg damit in die Liga B auf.

Zur Qualifikation für die Weltmeisterschaft 2022 traf Montenegro in Gruppe G auf Niederlande, Türkei, Norwegen, Lettland und Gibraltar. Nach vier Niederlagen fand man sich zuletzt mit 12 Punkten auf Platz 4 wieder.

In der UEFA Nations League 2022/23 dann in Liga B traf Montenegro im Sommer 2022 auf die Gruppengegner Rumänien, Bosnien-Herzegowina und Finnland.

Rekordspieler

Stand: 20. November 2022

Rekordtorschützen

Trainerstab
 Trainer:
 Co-Trainer: Miodrag Džudović, Nenad Stanković, Srđan Kljajević
 Torwarttrainer: Dragoje Leković

Trainer
  Zoran Filipović (2007–2009)
  Zlatko Kranjčar (2010–2011)
  Branko Brnović (2011–2015)
  Ljubiša Tumbaković (2016–2019)
  Faruk Hadžibegić (2019–2020)
  Miodrag Radulovic (seit 2020)

Internationale Wettbewerbe
Zur Zeit vor der Gründung des Montenegrinischen Fußballverbandes siehe jugoslawische Fußballnationalmannschaft und serbisch-montenegrinische Nationalmannschaft

Teilnahme Montenegros an der Fußball-Weltmeisterschaft

Teilnahme Montenegros an der Fußball-Europameisterschaft

UEFA Nations League
 2018/19: Liga C, 3. Platz mit 2 Siegen, 1 Remis und 3 Niederlagen
 2020/21: Liga C, 1. Platz mit 4 Siegen, 1 Remis und 1 Niederlage
 2022/23: Liga B, 3. Platz mit 2 Siegen, 1 Remis und 3 Niederlagen
 2024/25: Liga B

Kleinere Turniere

Länderspiele

Siehe auch
 Montenegrinische Fußballnationalmannschaft der Frauen
 Liste der montenegrinischen Fußballnationalspieler

Weblinks

 Offizielle Website
 Fifa (Montenegro)
 Uefa (Montenegro)
 Profil bei Weltfussball.de

Einzelnachweise

Fußballnationalmannschaft (UEFA)

Fussball

redpajama_set_name: RedPajamaWikipedia


Watch highlights from the 2018 Oita Japan Festival and Austin-Oita Sister City Committee Chairperson Kristie Bryant talk more about the different activities at the festival!
Austin-Oita Sister City Committee Board Member and 2018 Oita Japan Festival Chair Elise Krentzel and Kinokuniya USA Regional Manager Shigekazu Watanabe talk to Lauren Petrowski and Amanda Salinas of Fox 7 Austin on Good Day Austin.
Austin-Oita Sister City Committee (AOSCC) Chairperson Kristie Bryant and AOSCC Board Member and 2018 Oita Japan Festival Chair Elise Krentzel talk with 2018 Oita Japan Festival emcee and host of The Community Pilot Show Greg Ciotti.
Austin-Oita Sister City Committee (AOSCC) Chairperson Kristie Bryant, AOSCC Board Member and 2018 Oita Japan Festival Chair Elise Krentzel, and Friend of AOSCC Miwa Gardner-Page talk about the Oita Japan Festival and living in Japan on KUTX 98.9 Eklektikos with John Aielli.
Austin-Oita Sister City Committee Board Member Susan Hammons talks about the 2018 Oita Japan Festival and Rhonda Rolf of Central Texas Chado demonstrates tea ceremony on We Are Austin.


If you have the opportunity to get between $2,510 and $50,000 in as little as one business day, would you believe it?¹ ³ ⁵ Well, depending on your ability to make on-time monthly payments and the equity of your vehicle, it may be a reality for those who apply with LoanMart for auto title loans in Lake Forest.
How can we do this? The process is done by using your vehicle's title as collateral. You don't lose the use of your vehicle, however, as long as you're making regular monthly payments on time. The terms of the loan may be between 12 and 48 months.
Since 2002, LoanMart has assisted customers suffering from various financial struggles. We may be able to help you out as well, but you may never know if you don't apply. Call 888-578-5449 to learn more.
Online Information Request Form: Go onto our secure website and located on the right side of each page is our online information request form. By smart device, computer or tablet, you may be able to complete and submit this form.
Phone: Dial 888-578-5449 and you may be able to turn in an information request form by talking to our auto title loan agents.
The third step is to pick up your money if you're a qualified applicant in Lake Forest. If your money situation is an issue, please reach out to LoanMart for an auto title loan!
Click here if you want to apply in person for a title loan in Lake Forest.


Lash growth oil! longer, thicker and stronger lashes!
Throw away your 'falsies' and start your journey to stronger, longer and fuller looking lashes and brows in just a few weeks!
This eyelash growth oil will noticeably increase the length and thickness of your lashes within few weeks! Our all natural eyelash oil is made to deliver dramatically full and thick lashes! Its designed to help you get naturally lush lashes without the use of harmful chemicals. It activates the hair follicle and tissue by using nutrient packed ingredients that are known to strengthen and promote growth of the eyelashes.
Who doesn't dream of having long, thick and dark eyelashes? Castor oil is effective in growing and thickening eyelashes, brows and hair. It is also capable of preventing them from breaking and can also trigger the growth of eyelashes at places where they are not present in abundance. These days people are ready to spend any amount of money on cosmetic products for achieving beautiful thick eyelashes. If you want to achieve attractive eyelashes choose an effective natural alternative rather than spending a lot of money on artificial products. Use eyeliner applicator to apply oil to the roots for growth. Draw a line with the brush at the base of your eyelashes. For best results, apply every day before bed. To avoid discomfort use tiny amount of product!
An intensive overnight lash conditioning treatment for longer fuller and stronger lashes.
Potent blend of nourishing oils and vitamins.
Replenishes and hydrates each lash from the root.
Can be worn on both your eyelashes and eyebrows.
INGREDIENTS: VITAMIN E, CASTOR OIL, AVOCADO OIL, HEMP SEED OIL, ALMOND OIL, MACADAMIA OIL, JOJOBA OIL, CAMELLIA OIL, ARGAN OIL. ROSEMARY, LEMON AND PEPPERMINT ESSENTIAL OILS.
Free of barabens, sulphates, mineral oil, pthaltes, PEG free, Gluten free, non GMO, cruelty free, no animal testing, suitable for vegans.