\section{Introduction}

Due to the rapid development of deep convolutional neural networks (CNN)~\cite{simonyan2014very,he2016deep}, substantial progress is achieved in image recognition.
However, they usually assume that the training and testing data is balanced.
In practice, training or testing data appears to be long-tailed, e.g., there exist few samples for rare diseases in medical diagnosis~\cite{zhang2020exploring,zhang2021cross,zhao2021deep,zhao2021contralaterally,pan2022computer,zhang2021automatic} or endangered animals in species classification~\cite{zhang2022generalized,zhang2022onfocus,cheng2022compound}.
As mentioned by~\cite{zhang2021weakly}, the case becomes even worse in weakly and semi-supervised learning scenarios~\cite{zhang2018spftn,zhang2019leveraging,zhang2020weakly,zhao2021weakly,huang2021scribble,pan2022learning,zhao2022cross,wang2022double,zhang2022generalized}.
The conventional training process of CNNs is dominated by frequent classes while rare classes are neglected.
A large number of methods focus on rebalancing the training data through biasing the training process towards rare classes~\cite{cao2019learning,hong2021disentangling}.
However, they usually assume the testing data is balanced, while the distribution of real testing data is unknown and arbitrary. The performance of existing algorithms for learning with long-tailed distribution (LLTD) remains to be validated under such circumstance.
This paper revisits existing evaluation strategies for long-tailed image classification and provides a new strategy to evaluate LLTD algorithms on testing data with unknown distribution.

\label{sec:intro}
\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\linewidth]{pics/adapt.pdf
    \caption{In real-world tasks for learning with long-tailed distribution, the distribution of testing data is unknown and may be different from that of training data.}
    \label{fig:motivation}
\end{figure}

\begin{table*}[t]
    \fontsize{7.5}{9} \selectfont
    \centering
    \caption{Evaluation strategies for long-tailed image classification, including: 1) testing algorithms on balanced subset~\cite{zhou2020bbn,park2021influence}; 2) testing algorithms on subsets having uniform distribution and forward/backward-trend distribution~\cite{zhang2021test}; 3) testing algorithms on a series of subsets with dynamic evolving distributions.
    \setlength{\tabcolsep}{1mm}{
    \begin{tabular}{r|l|l|r}
    \toprule
    \toprule
    \multicolumn{1}{l|}{\textbf{Test Data}} & \multicolumn{1}{l}{\textbf{Metrics}} & \textbf{Descriptions} & \multicolumn{1}{l}{\textbf{Evaluation Properties}} \\
    \midrule
    \midrule
          & \cellcolor[rgb]{ .949,  .949,  .949}ACC$_{all}$ & \cellcolor[rgb]{ .949,  .949,  .949}Accuracy of all classes & \multicolumn{1}{l}{\cellcolor[rgb]{ .949,  .949,  .949}1. Only reflect the classification performance under a} \\
    \multicolumn{1}{l|}{Balanced} & ACC$_{many}$ & Accuracy of many-shot classes & \multicolumn{1}{l}{specific testing distribution.} \\
    \multicolumn{1}{l|}{Distribution} & \cellcolor[rgb]{ .949,  .949,  .949}ACC$_{mid}$ & \cellcolor[rgb]{ .949,  .949,  .949}Accuracy of medium-shot classes & \multicolumn{1}{l}{\cellcolor[rgb]{ .949,  .949,  .949}2. Not able to reflect the stability and upper/lower} \\
          & ACC$_{few}$ & Accuracy of few-shot classes & \multicolumn{1}{l}{bound of algorithms on real-world testing data.} \\
    \midrule
    \midrule
    \multicolumn{1}{l|}{Multiple} & \cellcolor[rgb]{ .949,  .949,  .949}ACC$_{forw}$ & \cellcolor[rgb]{ .949,  .949,  .949}Accuracy under forward-trend test distribution & \multicolumn{1}{l}{\cellcolor[rgb]{ .949,  .949,  .949}1. Reflect the classification performance under several} \\
    \multicolumn{1}{l|}{Independent} & ACC$_{uni}$ & Accuracy under uniform test distribution & \multicolumn{1}{l}{dependent imbalanced testing distributions.} \\
    \multicolumn{1}{l|}{Distributions} & \cellcolor[rgb]{ .949,  .949,  .949}ACC$_{back}$ & \cellcolor[rgb]{ .949,  .949,  .949}Accuracy under backward-trend test distribution & \multicolumn{1}{l}{\cellcolor[rgb]{ .949,  .949,  .949}2. Coarsely reflect stability and upper/lower bound.} \\
    \midrule
    \midrule
          & AUC   & Area under accuracy curve &  \\
    \multicolumn{1}{l|}{Dynamic} & \cellcolor[rgb]{ .949,  .949,  .949}ACC$_{avg}$ & \cellcolor[rgb]{ .949,  .949,  .949}Average accuracy & \multicolumn{1}{l}{\cellcolor[rgb]{ .949,  .949,  .949}1. Reflect the classification performance under real} \\
    \multicolumn{1}{l|}{Evolving} & ACC$_{std}$ & Standard deviation of accuracy & \multicolumn{1}{l}{testing data more comprehensively.} \\
    \multicolumn{1}{l|}{Distributions} & \cellcolor[rgb]{ .949,  .949,  .949}ACC$_{max}$ & \cellcolor[rgb]{ .949,  .949,  .949}Maximum accuracy & \multicolumn{1}{l}{\cellcolor[rgb]{ .949,  .949,  .949}2. Evaluate stability and upper/lower bound more finely.} \\
          & ACC$_{min}$ & Minimum accuracy &  \\
          & DR    & Drop ratio of accuracy &  \\
    \bottomrule
    \bottomrule
    \end{tabular}%
    }

    \label{tab:comp-eval}
\end{table*}

Recently, learning models with long-tailed training data attracts lots of research interest.
Existing methods concentrate on different procedures including data preparation, feature representation learning, objective function design, and class prediction to tackle this task.
According to the focused procedures, they can be categorized into four types, i.e., data balancing~\cite{chawla2002smote,drummond2003c4}, feature balancing~\cite{cui2021parametric}, loss balancing~\cite{lin2017focal,park2021influence,ren2020balanced}, and prediction balancing~\cite{wang2020long,zhang2021test,li2022trustworthy}.
Most of them train models with imbalanced subsets of existing datasets, such as CIFAR-10/100~\cite{krizhevsky2009learning}, ImageNetLT~\cite{liu2019large}, and iNaturalist~\cite{van2018inaturalist}. Then, they evaluate the classification performance on a balanced testing set with accuracy values of all classes and partial classes (e.g., many/middle/few-shot classes).
This evaluation process differs from the real-world situation where the testing data distribution is unknown.
Hence, it may not reflect the actual classification performance objectively.
Moreover, those metrics can not indicate the stability and performance bounds of algorithms on testing data which may have arbitrary distributions.
\cite{zhang2021test} attempts to estimate LLTD algorithms on testing data with multiple distributions: 1) Testing data has an imbalanced distribution sharing the forward trend as the training data; 2) Testing data has uniform sample sizes across classes, namely the distribution is balanced; 3) Testing data has imbalanced distributions with the backward trend of the training data. However, this manner is still limited in comprehensively evaluating LLTD algorithms on unknown testing data. It can only coarsely reflect algorithmic stability and bounds.

To address the above issues, we devise new evaluation metrics based on testing data with dynamic evolving distributions. For covering possible test distributions as fully as possible, we simulate a series of testing sets by shifting the frequent classes according to the class index. Then, each case of testing data is used to calculate the accuracy of LLTD algorithms. A corpus of evaluation metrics are estimated from the accuracy on all testing sets: 1) The area under the curve formed by those accuracy values and the average accuracy can be used for evaluating the performance on universal testing data; 2) The standard deviation and the accuracy drop ratio reflect the stability against the variation of test data distributions; 3) The maximum and minimum accuracy values can approximate the upper and lower bound, respectively. The comparison between our evaluation strategy and existing strategies is provided in Table~\ref{tab:comp-eval}.

Based on the new evaluation metrics, we provide an elaborated analysis about the classification accuracy, robustness, and bounds for existing LLTD algorithms on two benchmarks in which the testing data has same or different imbalance ratio with the training data.
This can guide the selection of data rebalancing techniques.
We observe that a few methods such as the contrastive representation learning algorithm in~\cite{cui2021parametric},  ensembling models learned with different distributions~\cite{zhang2021test}, logit rebalancing strategies~\cite{ren2020balanced,li2022long}, and data distribution disentangling~\cite{hong2021disentangling} have distinguished performance on addressing the long-tailed learning problem.
However, there still exists large space for improving the classification of tail classes under large distribution shift or large-scale number of classes. Meanwhile, middle classes also need attentions during training.

Main contributions of this paper are as follows.

\begin{itemize}
\item[1)] We provide a simple survey for existing long-tailed learning algorithms and classify them into four types according to the key procedures during the pipeline for tackling the long-tailed learning, i.e., data rebalancing, loss design, feature representation learning, and category prediction.
\item[2)] We design a corpus of new evaluation metrics for analyzing the classification accuracy, stability, and bounds of LLTD algorithms on testing data with dynamic evolving distributions more comprehensively.
\item[3)] Based on the new evaluation metrics, two benchmarks where training data and testing data have same or different imbalance ratios are set up to evaluate existing LLTD algorithms.
\end{itemize}

\begin{figure*}[thbp]
    \centering
    \includegraphics[width=1\linewidth]{pics/long-tail-categorization-v4.pdf}
    \put(-482,278){\cite{chawla2002smote}}
    \put(-491,256){\cite{drummond2003c4}}
    \put(-479,233){\cite{zhang2017mixup}}
    \put(-310,278){\cite{wang2020long}}
    \put(-232,256){\cite{tang2022invariant}}
    \put(-307,233){\cite{zhang2021test}}
    \put(-207,278){\cite{cai2021ace}}
    \put(-200,233){\cite{li2022trustworthy}}
    \put(-473,156){\cite{kang2019decoupling}}
    \put(-452,133){\cite{zhou2020bbn}}
    \put(-492,111){\cite{tang2020long}}
    \put(-457,78){\cite{cui2021parametric}}
    \put(-480,55){\cite{li2022targeted}}
    \put(-475,32){\cite{alshammari2022long}}
    \put(-51,172){\cite{lin2017focal}}
    \put(-34,146){\cite{park2021influence}}
    \put(-30,119){\cite{cao2019learning}}
    \put(-68,97){\cite{ren2020balanced}}
    \put(-34,77){\cite{hong2021disentangling}}
    \put(-72,55){\cite{menon2020long}}
    \put(-38,34){\cite{li2022long}}
    \put(-38,11){\cite{zhao2022adaptive}}
    \vspace{-5pt}
    \caption{Categorization of existing long-tailed image classification works. Based on four primary procedures, existing works can be categorized into four groups: balanced data, balanced feature representation, balanced loss, and balanced prediction.}
    \label{figCate}
\end{figure*}

\section{Survey of Long-tailed Learning}
\label{sec:related}
\subsection{Problem Definition}
\label{sec:prob_def}
Real-world data often has a long-tailed distribution.
Suppose the training data be $\mathbb D^{trn}=\{(x_n, y_n)\}_{n=1}^{N^{trn}}$. $x_n$ and $y_n$ denote the $n$-th training image and its class label, respectively; $N^{trn}$ represents the number of training samples.
Let the number of training samples belonging the $c$-th class be $N^{trn}_c$. The imbalance ratio of the training data is denoted as $\rho^{trn}=\max_c N^{trn}_c / \min_c N^{trn}_c$.
Suppose there exist $C$ target classes, i.e., $y_n\in[1,C]$.
We define the testing dataset as  $\mathbb D^{tst}=\{(x_n, y_n)\}_{n=1}^{N^{tst}}$.
$N^{tst}$ denotes the number of testing samples. The number of testing samples belonging the $c$-th class is $N^{tst}_c$. The imbalance ratio of the testing data is denoted as $\rho^{tst}=\max_c N^{tst}_c/\min_c N^{tst}_c$.

We define the distribution shift between training and testing data with the JS divergence,
\begin{equation}
\delta(\mathbb D^{trn}, \mathbb D^{tst})=-\sum_{c=1}^C [q_c^{trn} \ln(\frac{q_c^{trn}}{q_c^{tst}})+q_c^{tst} \ln(\frac{q_c^{tst}}{q_c^{trn}})],
\end{equation}
where $q_c^{trn}=\frac{N_c^{trn}}{N^{trn}}$ and $q_c^{tst}=\frac{N_c^{tst}}{N^{tst}}$.

Given the training data $\mathbb D^{trn}$, the goal is to learning a CNN model which can well adapt to the testing data $\mathbb D^{tst}$. We define the inference process of the CNN model as $f(\cdot)$. Namely, given an image $x$, the CNN model can generate a classification probability vector $\mathbf p\in \mathbb R^C$. The inferred label is denoted as $\hat{y}=\arg\max_c p[c]$. Here, $p[c]$ indicates the $c$-th class's probability value.

\subsection{A Survey of Prior Works}
Under long-tailed distribution, head classes are prone to dominate the learning process, thus impairing the accuracy of tail classes. The core factor to address this problem is balancing the learning process. As shown in Figure~\ref{figCate}, we categorize existing works into four groups: balanced data, balanced feature representation, balanced loss, and balanced prediction.

\subsubsection{Balanced Data}
A straightforward method to address the data imbalance problem is to construct balanced data distribution, i.e., re-sampling the training data. However, excessively sampling tail classes~\cite{chawla2002smote} induces the over-fitting issue, while under-sampling head classes~\cite{drummond2003c4} hampers the representation learning and weakens the accuracy of head classes. Based on the effectiveness of mixup-based methods \cite{zhang2017mixup, yun2019cutmix}, combining samples from different classes can alleviate the long-tailed challenge. However, a naive implementation is prone to generate more head-head pairs. To this end, Xu \etal \cite{Xu2021TowardsCM} propose a balance-oriented mixup algorithm by biasing the mixing factor towards tail classes and increasing the occurrence of head-tail pairs.

\subsubsection{Balanced Feature Representation}
Designing algorithms to learn balanced feature representations is the other promising direction for addressing the long-tailed learning problem. cRT~\cite{kang2019decoupling} finds that data imbalancement does not impair the representation ability. Hence, cRT~\cite{kang2019decoupling} learns the feature extraction backbone with the conventional training strategy and then employ the data re-balancing algorithms to train the classifier. Similarly, BBN~\cite{zhou2020bbn} unifies the traditional and re-balanced data sampling strategies, and gradually shifts focus from the former to the latter. Aiming to jointly acquire representative features and discriminative classification scores, Tang \etal \cite{tang2020long} use the moving average of momentum to measure the misleading effect of head classes during training, and build a causal inference algorithm to remove the misleading effect during inference.

Another feature balancing approach is to enhance the representation ability for each class, \eg by contrastive learning.
Cui \etal \cite{cui2021parametric} propose to explicitly learn a feature center for each class, which is used to increase the inter-class separability.
Li \etal \cite{li2022targeted} reveal that the imbalanced sample distribution leads to close feature centers for tail classes. Thus, they propose to constrain feature centers to be uniformly distributed.
Alshammari \etal \cite{alshammari2022long} tackle the long-tailed challenge from the perspective of weight balancing, and apply the weight decay strategy to penalize large weights.

\subsubsection{Balanced Loss}
Another reasonable approach to the imbalance challenge is assigning relatively higher attention to tail classes during the network optimization process. For example, Focal loss~\cite{lin2017focal} assigns larger weights to samples with lower prediction confidences, \ie, the so-called hard samples. However, if the dataset exhibits severe imbalance, excessively emphasizing tail classes would lead to an over-fitting dilemma. To alleviate this challenge, Park \etal \cite{park2021influence} propose to re-weight samples according to the reverse of their influences on decision boundaries. LDAM~\cite{cao2019learning} tackles the data imbalance challenge by increasing margins of tail classes' decision boundaries, considering decision boundaries of head classes are more reliable than those of tail classes.

The other line of works focus on balancing gradients for head and tail classes by adjusting the \textit{Softmax} function. Ren \etal \cite{ren2020balanced} make an early attempt to balance the \textit{Softmax} function and develop the meta-sampling strategy to dynamically adjust the data distribution in the training process. LADE~\cite{hong2021disentangling} proposes the post-compensated \textit{Softmax} strategy to disentangle the source data distribution from network predictions. In addition, Menon \etal~\cite{menon2020long} introduce the logit adjustment strategy. If testing samples obey the independent and identical distribution of training samples, the logit adjustment strategy can generate accurate predictions. However, it is impractical to guarantee the independent and identical distribution between training and testing samples. Moreover, the underlying distribution of testing samples is usually unknown. To cope with this problem, GCL~\cite{li2022long} introduces Gaussian clouds into the logit adjustment process, and adaptively sets the cloud size according to the sample size of each class. Recently, Zhao \etal~\cite{zhao2022adaptive} reveal that previous logit adjustment techniques primarily focus on the sample quantity of each class, while ignoring the difficulty of samples. Thus, Zhao \etal \cite{zhao2022adaptive} propose to prevent the over-optimization on easy  samples of tail classes, while highlighting the training on difficult samples of head classes.

\subsubsection{Balanced Prediction}
Another type of methods try to tackle the long-tailed challenge by improving the inference process.
Aiming to balance the accuracy of head and tail classes, Wang \etal~\cite{wang2020long} learn multiple experts simultaneously and ensemble their predictions to reduce the bias of single models. A dynamic routing module is developed to control the computational costs.
ACE~\cite{cai2021ace} attempts to learn multiple complementary expert models. Specifically, each expert model is responsible for distinguishing a specific set of classes while its responses to non-assigned classes are suppressed.
Considering the real-world testing data may exhibit a distinct distribution compared to the training data, Zhang \etal \cite{zhang2021test} learn multiple models under different distributions and combines them with weights generated via testing-time adaptation.
For decreasing the computational cost, Li \etal~\cite{li2022trustworthy} propose to measure the uncertainty of each expert, and assign experts to each sample dynamically.
Tang \etal~\cite{tang2022invariant} utilize uniform intra-class data sampling and confidence-aware data sampling strategies to construct different training environments for learning features invariant to diversified attributes.

\section{The New Evaluation Metrics}
\label{sec:data}

\input{tables/cmp-cifar-same-01}

In real-world applications, limited training data can not reflect the actual data distribution. To evaluate LLTD algorithms more comprehensively, we set up testing datasets which have dynamic evolving distributions.
The percent of the $c$-th class's samples namely $q_c^{tst}$ is determined according to the following formulation,
\begin{equation}
q_c^{tst}(\alpha) = \frac{({\rho^{tst}})^{-\frac{|c-\alpha|}{C-1}}}{\sum_{c=1}^C ({\rho^{tst}})^{-\frac{|c-\alpha|}{C-1}}},
\end{equation}
where $\alpha$ is a variable controlling the peak of the testing data distribution. Varying $\alpha$ can derive testing data with diversified distributions.

Suppose the maximum class sample size be $N_{max}^{tst}$. We choose the the total number of samples namely $N^{tst}$ as below,
\begin{equation}
N^{tst} = \sum_{c=1}^C N_{max}^{tst} ({\rho^{tst}})^{-\frac{|c-1|}{C-1}}.
\end{equation}

Then, the sample size of the $c$-th class in testing data can be obtained as, $N_c^{tst}=N^{tst} q_c^{tst}(\alpha)$. Finally, new testing samples can be randomly drawn out from the original dataset.

\input{tables/cmp-cifar-same-05}

Accuracy is the basic metric for evaluating performance of image classification models.
Intrinsically, it records the percent of correctly predicted testing samples as in the following equation,
\begin{equation}
V^{acc} = \frac{\sum_{n=1}^{N^{tst}} (\hat y_n = y_n) }{N^{tst}}.
\end{equation}

We synthesize a series of testing sets by varying $\alpha$ in $\{\frac{(t-1)C}{T}+1\}_{t=1}^T$, where $T$ denotes the times of data synthesization.
For each times of synthesization, we implement five replaceable samplings.
The overall performance on this testing set synthesization can be evaluated by averaging the five samplings' accuracy, which is defined as $V^{acc}_{t}$.
The distribution shift between the training set and the $t$-th synthesized testing set is estimated as,
\begin{equation}
    \delta_t = -\sum_{c=1}^C [q_c^{trn} \ln(\frac{q_c^{trn}}{q_c^{tst}(\alpha_t) })+q_c^{tst}(\alpha_t) \ln(\frac{q_c^{tst}(\alpha_t)}{q_c^{trn}})],
\end{equation}
where $\alpha_t=\frac{(t-1)C}{T}+1$.

One simple manner to unify the results of $T$ synthesizations is directly averaging them:
\begin{equation}
V^{avg} = \frac{1}{T}\sum_{t=1}^T V^{acc}_{t},
\end{equation}
where $V^{avg}$ denotes the averaged accuracy. We can also evaluate the sensitivity to distribution variation with the accuracy drop ratio $V^{dr}$ and the standard deviation $V^{std}$:
\begin{align}
V^{dr}  & = \frac{\max_{t} V^{acc}_{t} - \min_{t} V^{acc}_{t}}{\max_{t} V^{acc}_{t}}, \\
V^{std} & = \sqrt{\sum_{t=1}^T (V^{acc}_{t} - V^{avg})^2/T}.
\end{align}

\input{tables/cmp-cifar-same-1}

The other manner for combining the accuracy values is calculating the area under the curve of $T$ synthesizations' results. All testing sets are ranked with respect to the distribution shifts between training and testing sets in the ascending order. Denote the ranked set indices as $\{t_k\}_{k=1}^T$. This metric is calculated as below:
\begin{equation}
V^{auc} = \frac{\sum_{k=1}^{T-1} (V^{acc}_{t_k} + V^{acc}_{t_{k+1}})(\delta_{t_{k+1}} - \delta_{t_{k}})/2}{ \delta_{t_{T}} - \delta_{t_{1}} }.
\end{equation}

\section{Benchmarks}
\label{sec:exper}

\subsection{Experimental settings.}
We use two datasets for training and testing, including CIFAR10 and CIFAR100~\cite{krizhevsky2009learning}.
Both CIFAR10 and CIFAR100 contain 60,000 images with size of 32$\times$32.
The number of classes of CIFAR10 and CIFAR100 is 10 and 100, respectively.
The original datasets are split into 50,000 images for training and 10,000 images for testing.
We follow~\cite{kang2019decoupling,zhou2020bbn} to synthesize imbalanced training set with the imbalance ratio $\rho^{trn}\in\{0.01,0.05,0.1\}$.
$N_c^{trn}$ is set as 5,000 and 500 for CIFAR10 and CIFAR100, respectively.
During training, 10\% images of each class are used for validation.
When generating testing data, we choose $\rho^{tst}$ from $\{ 0.01, 0.05, 0.1\}$ and set $N_{max}^{tst}$ as 1,000 and 100 for CIFAR10 and CIFA100, respectively.

\subsection{Benchmark 1: Testing Data having Same Imbalance Ratio as Training Data} \label{sec:exper-same}
In this subsection, we re-evaluate the performance of existing methods, including Focal~\cite{lin2017focal}, LDAM~\cite{cao2019learning}, cRT~\cite{kang2019decoupling}, BBN~\cite{zhou2020bbn}, MetaS~\cite{ren2020balanced}, DecTDE~\cite{tang2020long}, RIDE~\cite{wang2020long}, IBLoss~\cite{park2021influence}, TADE~\cite{zhang2021test}, LADE~\cite{hong2021disentangling}, Prior-LT~\cite{Xu2021TowardsCM}, PCL~\cite{cui2021parametric}, and GCLLoss~\cite{li2022long}.
The baseline method (CE) is implemented using the conventional uniform data sampling and standard cross-entropy loss function.
For all methods, we use ResNet32~\cite{he2016deep} as the classification backbone model.
Here, we use the same imbalance ratio for training and testing data.

\begin{figure*}[t]
    \centering
    \begin{subfigure}{0.328\linewidth}
        \centering
        \includegraphics[width=1\linewidth, trim=10 10 5 5, clip]{pics/acc_curves/CIFAR10_0.01_0.01.pdf}
        \caption{$\rho^{trn}=\rho^{tst}=0.01$}

    \end{subfigure}
  \begin{subfigure}{0.328\linewidth}
        \centering
        \includegraphics[width=1\linewidth,trim=10 10 5 5, clip]{pics/acc_curves/CIFAR10_0.05_0.05.pdf}
    \caption{$\rho^{trn}=\rho^{tst}=0.05$}

    \end{subfigure}
  \begin{subfigure}{0.328\linewidth}
        \centering
        \includegraphics[width=1\linewidth, trim=10 10 5 5, clip]{pics/acc_curves/CIFAR10_0.1_0.1.pdf}
        \caption{$\rho^{trn}=\rho^{tst}=0.1$}

    \end{subfigure}

\caption{Accuracy curves of existing methods on CIFAR10 against the JS divergence. (a) The decay coefficients $\rho^{trn}$ and $\rho^{tst}$ are set to 0.01; (b) $\rho^{trn}$ and $\rho^{tst}$ are set to 0.05; (c) $\rho^{trn}$ and $\rho^{tst}$ are set to 0.1.
\label{fig:acc-curve-cifar10}
\end{figure*}

\begin{figure*}[t]
    \centering
    \begin{subfigure}{0.328\linewidth}
        \centering
        \includegraphics[width=1\linewidth, trim=10 10 5 5, clip]{pics/acc_curves/CIFAR100_0.01_0.01.pdf}
        \caption{$\rho^{trn}=\rho^{tst}=0.01$}

    \end{subfigure}
  \begin{subfigure}{0.328\linewidth}
        \centering
        \includegraphics[width=1\linewidth,trim=10 10 5 5, clip]{pics/acc_curves/CIFAR100_0.05_0.05.pdf}
        \caption{$\rho^{trn}=\rho^{tst}=0.05$}

    \end{subfigure}
  \begin{subfigure}{0.328\linewidth}
        \centering
        \includegraphics[width=1\linewidth, trim=10 10 5 5, clip]{pics/acc_curves/CIFAR100_0.1_0.1.pdf}
        \caption{$\rho^{trn}=\rho^{tst}=0.1$}

    \end{subfigure}

\caption{Accuracy curves of existing methods on CIFAR100 against the JS divergence.
\label{fig:acc-curve-cifar100}
\end{figure*}

\input{tables/cmp-cifar-smaller}
We report the experimental results on re-distributed versions of CIFAR10 and CIFAR100 which have the same imbalance ratios for training and testing data in Table~\ref{tab:CIFAR-01} ($\rho^{trn}=\rho^{tst}=0.01$), Table~\ref{tab:CIFAR-05} ($\rho^{trn}=\rho^{tst}=0.05$), and Table~\ref{tab:CIFAR-1} ($\rho^{trn}=\rho^{tst}=0.1$). The accuracy curves with respect to the JS divergence are illustrated in Fig.~\ref{fig:acc-curve-cifar10} and Fig.~\ref{fig:acc-curve-cifar100}.

\input{tables/cmp-cifar-larger}

We can observe that, the parametric contrastive learning~\cite{cui2021parametric} is very good at coping with the data imbalance problem.
It achieves the highest classification accuracy overall, and showcases high stability across different testing distributions.
This means that learning good representations is a very effective strategy for relieving the influence of data imbalance, due to the improvement in feature generalization ability and the prevention of overfitting with tail classes.
TADE~\cite{zhang2021test} also has very promising ability in tackling the data imbalance problem, due to the combination of multiple expert models learned with different simulated training distributions.
Another effective direction is adjusting the classification logits according to classes' sample sizes like GCLoss~\cite{li2022long} and MetaS~\cite{ren2020balanced}, which can help to enlarge the embedding space of tail classes.
LADE~\cite{zhang2021test} is targeted at disentangling the label distribution of the training data from the model prediction. It also has fine effect in adapting to arbitrary testing distributions. For example, it produces the second best AUC and average ACC metrics on CIFAR10 with $\rho^{trn}=\rho^{tst}=0.01$.

From the metric values on standard deviation (STD) and drop ratio (DR), most methods with relatively better classification performance such as PCL, TADE, and GCLoss are robust against the variance of testing data distribution. Prior-LT~\cite{Xu2021TowardsCM} which uses class-balanced mixup to construct training samples and prior-compensated Softmax for probability prediction have relatively more stable performance than other methods on CIFA100.

The upper bound of LLTD algorithms on testing data with unknown distribution can be approximated by the maximum accuracy (ACC-MAX) metric. Most of algorithms are unable to improve this metric. On CIFA100, a few algorithms such as cRT, BBN, IBLoss and Prior-LT  even achieve severely decreased ACC-MAX value compared to the baseline. The reason may be that those methods excessively bias the training process towards tail classes. Several methods are capable of improving the ACC-MAX metric, such as Meta-Softmax, PCL, LADE, focal loss, etc.

The minimum accuracy (ACC-MIN) can approximate the lower bound of LLTD algorithms. This metric is usually obtained when there exists large or moderate shift between training and testing distributions.  We can see that most methods are capable of improving this metric, since they have specific designs for enhancing the significance of tail classes during training. The most three effective methods in boosting ACC-MIN are PCL, TADE, and LADE.

\subsection{Benchmark 2: Testing Data with Imbalance Ratios Different to Training Data} \label{sec:larger}
We report the experimental results on re-distributed versions of CIFAR10 and CIFAR100 where the imbalance ratios of the training data are different with that of the testing data in Table~\ref{tab:CIFAR-05-01} ($\rho^{trn}=0.05$, $\rho^{tst}=0.01$) and Table~\ref{tab:CIFAR-05-1} ($\rho^{trn}=0.05$, $\rho^{tst}=0.1$).
We can see that existing methods have similar performance rankings with the situation where the imbalance ratios of training and testing distributions are same.

\section{Discussions and Conclusions}
\label{sec:conclu}

\textbf{Discussions}. Based on the experimental results, we recommend the following directions to improve algorithms for learning with long-tailed distribution.
\begin{itemize}
  \item The minimum accuracy of most methods is not high under severe data imbalance or large number of classes (e.g., CIFAR100). This indicates that there still exists huge space for improving the performance on tail classes.
  \item From Fig.~\ref{fig:acc-curve-cifar10}, on CIFAR10, a few methods, such as  PCL, GCLoss, and LADE, are capable of achieving high accuracy on small and large JS divergence. However, the accuracy on middle JS divergence is relatively low. This means that they are able to achieve good performance on head and tail classes but the middle classes lack attention during training.
\end{itemize}

\textbf{Conclusions}. In this paper, we set up new Benchmarks to analyze the performance of methods for learning with long-tailed distribution. Based on a series of testing sets with evolving data distribution, we devise new metrics to analyze the accuracy, stability, and upper/lower bound of existing methods comprehensively. Extensive experiments on CIFAR10 and CIFAR100 are conducted to evaluate existing methods. We also summarize existing methods into data, feature, loss, and prediction balancing types according to the focused stage in the working pipeline.

{\small
\input{main.bbl}
}

\end{document}


\section*{Acknowledgement}
\vspace{-0.2cm}
The work is partially supported in by the Young Scientists Fund of the National Natural Science Foundation of China under grant No. 62106154, by the Natural Science Foundation of Guangdong Province, China (General Program) under grant No.2022A1515011524, and by Shenzhen Science and Technology Program ZDSYS20211021111415025.
\section{Experiment Setup}
\label{sec:app_exp}
\textbf{Dataset.}
Our experiments are mainly conducted on the popular COCO2017 Keypoint Detection benchmark~\citep{lin2014microsoft}, which contains about $250K$ person instances with $17$ keypoints. We compare with other state-of-the-art methods on both the \texttt{val2017} set and \texttt{test-dev} set.
To verify the superiority of explicit detection, we also evaluate our approach on the CrowdedPose dataset~\citep{li2019crowdpose} which is more challenging and includes many crowded and occlusion scenes. It consists of $20K$ images containing about $80K$ persons with $14$ keypoints.
For ablation studies, we report results on the COCO \texttt{val2017} set.
The OKS-based Average Precision (AP) is employed as the main evaluation metric on both datasets.

\textbf{Implementation details.} In the training stage, we augment input images by random crop, random flip, and random resize with the shorter sides in $[480,800]$ and the longer sides less or equal to $1333$ following DETR~\citep{carion2020end} and PETR~\citep{shi2022end}. To accelerate the early explicit human detection, we use a human query denoising training strategy from DN-DETR~\citep{li2022dn}.
We use the AdamW~\citep{kingma2014adam,loshchilov2017decoupled} optimizer with weight decay of $1 \times 10^{-4}$ and train our model on Nvidia A100 GPUs with batch size 16 for $60$ epochs and $80$ epochs on COCO and CrowdPose, respectively. The initial learning rate is $1 \times 10^{-4}$ and is decayed at the $55$th epoch and $75$th epoch by a factor of 0.1 on COCO and CrowdPose, respectively. The channel dimension $D$ is set to 256. \Revise{The number of layers in Human Detection Decoder and Human-to-Keypoint Detection Decoder are $2$ and $4$ respectively.}
In the test stage, the input images are resized to have their shorter sides being $800$ and their longer sides less or equal to $1333$.

\Revise{\textbf{Loss Function.}
the overall loss function of ED-Pose can be formulated as:
\begin{gather}
        L=L_h+L_c+L_k \\
        L_h=\mu \left|H-H^*\right|+\beta(1-\mathrm{GIOU}) \\
        L_c=-\lambda \alpha(1-p_t)^{\gamma}log(p_t), \mathrm{where}~p_t=p~\mathrm{if}~y=1,~p_t=1-p~\mathrm{if}~y\neq1  \\
        L_k= \omega \left|P-P^*\right| + \theta \frac{\sum^{K}_i\mathrm{exp}(-\left|P_i-P_i^*\right|/2s^2k_i^2)\delta(v_i>0)}{\sum^{K}_i\delta(v_i>0)}
\end{gather}}
\Revise{where $L_h$ is for human box regression that contains L1 loss and GIOU~\citep{rezatofighi2019generalized} loss, $L_c$ is for human classification that is focal loss~\citep{lin2017focal} with $\alpha$ = 0.25, $\gamma$ = 2, and $L_k$ is for keypoint regression that includes L1 loss and the constrained L1 loss-OKS loss~\citep{shi2022end}. $\left|H-H^*\right|$ is the L1 distance between the predicted human boxes and the ground-truth ones. $y\in{\pm1}$ specifies the ground-truth class, and $p\in[0, 1]$ is the ED-Pose's estimated probability for the class with label $y=1$. $\left|P-P^*\right|$ is the L1 distance between predicted keypoints inside a human and the ground-truth ones. $\left|P_i-P_i^*\right|$ is the L1 distance between the $i$-th predicted keypoint and ground-truth one, $v_i$ is the visibility flag of the ground truth, $s$ is the object scale, and $k_i$ is a per-keypoint constant that controls falloff.}

\Revise{The loss coefficients $\mu, \beta, \lambda, \omega, \theta$ are $5$, $2$, $2$, $10$, $4$.}

\textbf{The Detailed Interactive Learning between Human Detection and Keypoint Detection.}
The relations from internal human-keypoint, internal keypoint-keypoint, and external human-human are learned from the self-attention mechanism.
As shown in Figure.~\ref{fig:decoder}-(c), Human-to-Keypoint Interactive Attention can be computed as follow:
\begin{gather}
    \mathbf{Q}_{H,K}^{c'}=\mathrm{softmax}(\frac{f(\mathbf{Q}_{H,K}^c+\mathrm{PE}(\mathbf{Q}_{H,K}^p))\cdot f(\mathbf{Q}_{H,K}^c+\mathrm{PE}(\mathbf{Q}_{H,K}^p))^\mathrm{T}}{\sqrt{D}}+\mathcal{M})\cdot f(\mathbf{Q}_{H,K}^c)
\end{gather}
where $\rm{PE}$ denotes positional encoding, $f(\cdot)$ is linear projection, $\mathbf{Q}_{H,K}^c \in \mathbb{R}^{(M+M*K)\times D}$ and $\mathbf{Q}_{H,K}^p \in \mathbb{R}^{(M+M*K)\times4}$ are human-keypoint content queries and human-keypoint position queries for $M$ human candidates and the corresponding $K$ keypoints. $D$ is 256 by default. In practice, to implement the three interactive learning processes in a simple way, we use an attention mask $\mathcal{M} \in \mathbb{R}^{(M+M*K)\times(M+M*K)}$ to block the interactiveness between external keypoint-keypoint. Thus, $\mathcal{M}$ can be formulated as:
\begin{equation}
\mathcal{M}(i,j)=
\begin{cases}
0& \mathcal{M}(i,j)=\mathrm{True} \\
-\infty& \mathcal{M}(i,j)=\mathrm{False}
\end{cases}
\end{equation}
where $\mathcal{M}(i,j)$ is the location of the attention mask. We use the $\mathcal{M}$ to keep internal human-keypoint attention (True), internal keypoint-keypoint attention (True), and external human-human attention (True) while avoiding external keypoint-keypoint attention (False).

\Revise{\textbf{Inference time.} 1) Comparison Methods: All of the methods in Table.~\ref{tab:sota} are tested on our A100 machine for a fair comparison and the detector of top-down methods is tested by MMdetection. In tabel.~\ref{tab:one-stage}, the inference time with \emph{\textbf{emphasized bold number}} is tested by our A100 machine, otherwise, it is from PETR's paper using a V100 GPU due to no available source code.  2) Testing Rules: We omit the time for data pre-processing and only measure the time for model forwarding and data post-processing (\textit{i.e.}, grouping operation in bottom-up methods). For bottom-up methods and one-stage methods, we set the batch size to $1$. For top-down methods, we set the batch size to $5$ for simulating the multi-person situation.}

\Revise{\textbf{A Unique Merit of ED-Pose for Human Detection Pre-training.}
Thanks to the explicit introduction of human detection, ED-Pose has a unique merit compared to other existing two-stage methods and one-stage methods, which can pre-train human detection to improve the detection performance to assist the subsequent pose estimation in a fully end-to-end manner.
Objects365 \citep{shao2019objects365} is a large-scale detection dataset with over 1.7M annotated images for training and 80, 000 annotated images for validation.
We pre-train the human detection of ED-Pose (\textit{i.e.}, backbone, encoder, human detection decoder) on Objects365 for 26 epochs using 64 Nvidia A100 GPUs.}

\section{Comparison of COCO Test-dev.}
\label{sec:app_cocotest}
From Table.~\ref{tab:one-stage}, our method without dense heatmap loss can significantly outperform all existing one-stage methods, such as DirecetPose~\citep{tian2019directpose}, FCPose~\citep{WeianMao2021FCPoseFC}, Inspose~\citep{DahuShi2021InsPoseIN}, CenterNet~\citep{zhou2019objects}, and PETR~\citep{shi2022end}. To be specific, our proposed ED-Pose has $2.2$ AP gains compared with the fully end-to-end method PETR~\citep{shi2022end} with a ResNet-50 backbone or Swin-L backbone with about $50\%$ inference time reduction.

\label{sec:app_coco}
\begin{table*}[h]
\centering
\vspace{-0.4cm}
\caption{Comparisons with state-of-the-art one-stage methods on COCO \texttt{test-dev} dataset. \underline{underlined} highlights the compared results. \Revise{\emph{Emphasized bold number} in inference time is tested by our A100 machine}.}
\resizebox{0.9\linewidth}{!}{
    \begin{threeparttable}
  \begin{tabular}{p{0.4cm}|l|l|l|ccc|cc|c}
    \toprule
       \multicolumn{2}{c|}{\textbf{Method}} & Backbone & Loss & ${\rm AP}$ & ${\rm AP}_{50}$ & ${\rm AP}_{75}$ & ${\rm AP}_{M}$ & ${\rm AP}_{L}$ & Time [ms]\\

    \midrule
    \multirow{7}*{{\small{\rotatebox{90}{Non E-2-E}}}}
    &DirectPose & ResNet-50 & R.& 62.2 & 86.4  & 68.2  & 56.7 & 69.8 & 74\ \\
    &DirectPose & ResNet-101 & R.& 63.3 & 86.7 &  69.4 & 57.8 & 71.2 & -\ \\
    &FCPose  & ResNet-50 & R+HM.   & 64.3 & 87.3 & 71.0 & 61.6 & 70.5 & 68 \\
    & FCPose & ResNet-101 & R+HM.  & 65.6 & 87.9 & 72.6 & 62.1 & 72.3 & 93\\
    & InsPose & ResNet-50 &  R+HM.  & 65.4 & 88.9 & 71.7 & 60.2 & 72.7 & \Revise{\emph{\textbf{78}}}\\
    &InsPose & ResNet-101 &R+HM.  &  66.3 & 89.2 & 73.0 & 61.2 & 73.9 & 100\\
    &CenterNet & Hourglass & R.& 63.0 & 86.8 & 69.6 & 58.9 & 70.4 & 160\\
        \midrule
    \multirow{4}*{{\small{\rotatebox{90}{Fully E-2-E}}}}
    &PETR & ResNet-50 &R+HM.  &  \underline{67.6} & 89.8 & 75.3 & 61.6 & 76.0 & \emph{\textbf{105}}\\
    &PETR & Swin-L & R+HM.  & \underline{70.5} & 91.5 & 78.7 & 65.2 & 78.0 & \emph{\textbf{206}}\\
    \cmidrule{2-10}
    &ED-Pose & ResNet-50 & R.  &  69.8{\color{Red}$\uparrow_{2.2}$} & 90.2 & 77.2 & 64.3 & 77.4& \emph{\textbf{{51}}}{\color{Red}$\downarrow_{51.4\%}$}\\
    &ED-Pose & Swin-L & R.  &  \textbf{72.7}{\color{Red}$\uparrow_{2.2}$} & \textbf{92.3} & \textbf{80.9} & \textbf{67.6} & \textbf{80.0}& \emph{\textbf{88}}{\color{Red}$\downarrow_{57.3\%}$} \\

  \bottomrule
\end{tabular}
\begin{tablenotes}
\item[1] \Revise{The inference time without \emph{Emphasized bold number} is from PETR's paper because there is no public code for replication.}
\end{tablenotes}
  \end{threeparttable}}
    \label{tab:one-stage}
\end{table*}

\section{Results of Human Detection}
\label{sec:app_det}
Due to the explicit person detection in our methods, we also report the human detection performance on the COCO \texttt{val2017} set and CrowdPose \texttt{test} set in Figure.~\ref{tab:box2}.
For the COCO dataset, ED-Pose with Swin-L* backbone gains $7.7$ ${\rm AP}_{M}$ and $6.7$ ${\rm AP}_{L}$ improvement compared with Faster-RCNN~\citep{ShaoqingRen2015FasterRT}. For the CrowdPose dataset,
most top-down methods directly use YOLOv3~\citep{redmon2018yolov3} pre-trained on COCO \texttt{trainval} to generate the detected human bounding boxes,
making their performance even worse than recent bottom-up methods.
ED-Pose can provide human detection results with nearly doubling performance on ${\rm AP}_{M}$ and ${\rm AP}_{L}$ to serve future work.
Moreover, in Table~\ref{tab:layer}, we study the impact of different decoders (\textit{i.e.}, human detection decoder and human-keypoint detection decoder) for human detection and keypoint detection.
First, we discover that the keypoint initialization provided by the human detection decoder has already achieved good results on COCO and CrowdPose datasets, which proves the effectiveness of explicit human detection.
Second, the keypoint detection introduced via the human-keypoint detection decoder is also helpful to improve large-scale human detection and obtain similar performance on medium-scale human detection.
Different from previous observations of severe  optimization conflicts between human detection and keypoint detection, ED-Pose unifies the contextual learning between human-level and keypoint-level information and gains benefits from each other.

\begin{table*}[h]
\vspace{-0.4cm}
\caption{The Average Precision comparisons of the commonly used human detection methods (e.g., Faster-RCNN and YOLOv3) with ours on COCO (the left) and CrowdPose (the right). Notably, We only compare ${\rm AP}_{M}$ (medium object) and ${\rm AP}_{L}$ (large object) here as small objects in the two datasets are not labeled with keypoints. ED-Pose can provide better human detection to serve future work, especially for top-down and one-stage methods}
    \begin{center}
        \begin{minipage}[t]{0.35\textwidth}
        \centering
        \resizebox{\linewidth}{!}{
            \makeatletter\def\@captype{table}\makeatother
                \begin{tabular}{l|cc}
                    \toprule
                       {\textbf{Methods}}  &${\rm AP}_{M}$ & ${\rm AP}_{L}$  \\
                    \midrule
                        Faster-RCNN &  63.3 & 74.5  \\
                      ED-Pose (ResNet-50) &  65.9 & 77.6  \\
                         ED-Pose (Swin-L)  &69.7 & 80.2 \\
                         ED-Pose (Swin-L*)  &\textbf{71.0} & \textbf{81.2} \\
                  \bottomrule
                \end{tabular}
            }

        \end{minipage}
        \hspace{-0.5cm}
        \quad
        \begin{minipage}[t]{0.35\textwidth}
        \centering
        \resizebox{\linewidth}{!}{
            \makeatletter\def\@captype{table}\makeatother
                    \begin{tabular}{c|cccccc}
                        \toprule

                                                   {\textbf{Methods}} & ${\rm AP}_{M}$ & ${\rm AP}_{L}$  \\
                        \midrule
                         YOLOv3  & 37.4 & 46.5   \\
                          ED-Pose (ResNet-50)  & 58.0 & 75.3   \\
                         ED-Pose (Swin-L)   & 61.8 & 78.4 \\
                         ED-Pose (Swin-L*)  & \textbf{66.8} & \textbf{82.2} \\
                      \bottomrule
                    \end{tabular}
            }

        \end{minipage}
    \end{center}
     \label{tab:box2}
\vspace{-0.4cm}
\end{table*}

\begin{table*}[h]
\vspace{-0.4cm}
\centering
\caption{The performance comparisons of results from different decoders (\textit{i.e.}, human detection decoder and human-keypoint detection decoder) on both COCO and CrowdPose datasets with the Swin-L as the backbone. }
\resizebox{1\linewidth}{!}{
  \begin{tabular}{l|l|l|cc|ccc}
    \toprule
       \textbf{Method} & Dataset & Decoder &  ${\rm AP}_{M}$ (Human) & ${\rm AP}_{L}$ (Human) & ${\rm AP}$ (Keypoint)  \\
       \midrule
           ED-Pose & COCO & Human Det. &  \textbf{70.4}  & 79.6 & 70.4  \\
          ED-Pose & COCO & Human Det.+Human-Keypoint Det. & 69.8  & \textbf{80.1} & \textbf{74.3}   \\       \midrule
                     ED-Pose & CrowdPose & Human Det. &  \textbf{61.7}  & 76.9 & 68.0    \\
          ED-Pose & CrowdPose & Human Det.+Human-Keypoint Det. & 61.6  & \textbf{78.4} & \textbf{73.1}    \\
       \midrule
\end{tabular}}
\label{tab:layer}
\vspace{-0.4cm}
\end{table*}

\section{Qualitative Results}
\label{sec:app_viz}
\textbf{Qualitative ablations for explicit human detection}:
In the main article, we verify the effectiveness of explicit human detection for convergence speed and precision of the keypoint detection. Here, we present the corresponding qualitative results on the CrowdPose \texttt{test} set as shown in Fig.~\ref{fig:viz2}. In general, explicit human detection has several advantages: 1). It can provide global human-level information, enabling the model to be aware of flipping easily like the case in the first row. 2). It can give a clear human identity to the crowded scene to avoid the keypoint mismatching across persons, such as the case in the second row. 3) It is friendly for small human pose estimation as it can provide a precise prior of human box position, e.g., the case in the third row.

\textbf{Qualitative comparisons between PETR and ED-Pose}:
We present the visualization comparisons of ED-Pose and PETR~\citep{shi2022end} on the COCO dataset, as shown in Fig~\ref{fig:viz3}. From the first row, ED-Pose can pay attention to the flipping issue due to the introduction of explicit detection and the realization of human-keypoint feature propagation. The second row further reflects that explicit detection in ED-Pose can make the model aware of the human position when conducting pose estimation, rather than the chaotic and unconscious query on a full image like PETR. All possible detected keypoints from PETR have low scores, making it hard to distinguish the right and wrong results. The third row shows that the enhancement of the global-local interactivity could relieve the estimation problem of the hard pose with heavy occlusions.

\section{More Discussion}
\label{sec:app_dis}

\Revise{\textbf{Discussion for Additional Related Works.} (1) N. xue \citep{xue2022learning} focuses on improving the performance of the heatmap-based bottom-up method by learning the local-global contextual adaptation without any explicit box detection. (2) Different from the one-stage method, D. Wang \citep{wang2022contextual} decouples persons into multiple instance-aware feature maps to obtain the keypoints without any explicit box detection, and they also use heatmap-based supervision.}

\Revise{Different from their works, ED-Pose aims to utilize two explicit box detection processes with a unified box representation and light L1 losses. It abandons all post-processings in a fully end-to-end manner. In terms of performance, ED-Pose surpasses them by a margin and even outperforms heatmap-based Top-down methods under the same backbone.}

\Revise{\textbf{The More Detailed Advantage of ED-Pose.}
From the results, ED-Pose is efficient and effective even with the same backbone compared with previous works, indicating it really helps with pose estimation.
From the method with two box detection processes, we have admitted it is inspired by recent DETR-based methods. However, the motivation and usage have obvious differences due to the task differences between object detection and multi-person pose estimation. Multi-person pose estimation focuses on either human-level (global) or keypoint-level (local) information. ED-Pose re-considers this task as two explicit box detection processes with a unified representation and regression supervision.
Moreover, we also compare with other DETR-based pose estimators, and ED-Pose shows its superiorities in both efficiency and performance. The first human box detection can extract global features and provide a good initialization for the latter keypoint detection, making the training process converge fast. The second keypoint box detection can bring in local contextual information near keypoints.
However, previous DETR-based pose estimation methods only directly regressed the keypoints' 2D coordinates, we argue that the keypoint representation as a point lacks local contextual information. They directly regress local keypoints without global feature extraction, making the convergence slow and inaccurate estimation. In Figure.~\ref{fig:viz2} and Figure.~\ref{fig:viz3}, we demonstrate the qualitative comparisons of whether or not we needed to use explicit detection and PETR with ours. Lastly, we conduct comprehensive experiments on how explicit detection works in Table.~\ref{tab:box} and Table.~\ref{tab:hw}.}
\begin{figure}[h]
\vspace{-0.4cm}
\centering
 	{
 		\begin{minipage}[t]{1\linewidth}
 			\centering
 			\includegraphics[width=1\linewidth]{fig/ed_viz.png}
 		\end{minipage}
 	}
\vspace{-0.6cm}
\caption{Visualization results of the ablation study for explicit human detection on the CrowdPose dataset. Notably, we ignore the keypoint box to compare the keypoint localization of the two settings clearly. The white circle is the failure part with a description of the type of failure.}
\label{fig:viz2}
\vspace{-0.3cm}
\end{figure}

\begin{figure}[h]
\vspace{-0.2cm}
\centering
 	{
 		\begin{minipage}[t]{1\linewidth}
 			\centering
 			\includegraphics[width=1\linewidth]{fig/comp_viz.png}
 		\end{minipage}
 	}
\vspace{-0.6cm}
\caption{Visualization results of PETR and ED-Pose on COCO dataset. Both methods are based on ResNet-50 as the backbone. Notably, there are several candidates in the PETR's result at the second line as their classification scores are close and relatively low.}
\label{fig:viz3}
\end{figure}

\section{Conclusion}
\vspace{-0.1cm}
In this work, we re-consider the multi-person pose estimation task as two explicit box detection processes. We unify the global person and local keypoint into the same box representation, and they can be optimized by the consistent regression loss in a fully end-to-end manner. Based on the novel methods, the proposed ED-Pose surpasses existing end-to-end methods by a large margin and shows superiority over the long-standing two-stage methods on both COCO and CrowdPose.
This is a simple attempt to make the whole pipeline succinct and effective. We hope this work could inspire further one-stage designs.

\section{Experiments}
\vspace{-0.2cm}
Due to the page limit, we leave the detailed experiment setup
in \ref{sec:app_exp}, comparison results on COCO test-dev in \ref{sec:app_coco}, comparisons on human detection in \ref{sec:app_det}, more qualitative results and analyses in \ref{sec:app_viz}, and more discussion for ED-Pose (in Sec.~\ref{sec:app_dis}).

\vspace{-0.1cm}
\subsection{Results on CrowdPose}
\vspace{-0.2cm}
We first verify the effectiveness of ED-Pose with other state-of-the-art methods on the CrowdPose \texttt{test} set in Table ~\ref{tab:crowdpose}.
Compared with the top-down methods, we surpass the SimpleBaseline~\citep{xiao2018simple} under the same backbone by $9.1$ AP.  we also show superiority on all previous methods combined with Swin-L~\citep{liu2021Swin} backbone and outperform PETR by $1.5$ AP under the same backbone. When we enlarge the scales of backbones, ED-Pose will achieve the state-of-the-art 76.6 AP without multi-scale and flip tests.

\begin{table*}[h]
\vspace{-0.2cm}
  \centering
  \caption{Comparisons with state-of-the-art methods on CrowdPose \texttt{test} dataset. \textbf{TD, BU, OS} mean top-down, bottom-up, and one-stage methods, respectively. We use ``HM.'' and ``R.'' for heatmap-based losses and regression losses. $\dag$ denotes the flipping test. The model with * is pre-trained on Objects365~\citep{ShuaiShao2019Objects365AL} with $5$ feature scales. The \underline{underlined} highlights the compared results. The \textbf{best results} are highlighted in {\textbf{bold}}.}
  \resizebox{0.8\linewidth}{!}{
  \begin{tabular}{ll|l|ccc|ccc}
    \toprule
       &{\textbf{Method}}  & Loss & ${\rm AP}$ & ${\rm AP}_{50}$ & ${\rm AP}_{75}$ & ${\rm AP}_{E}$ & ${\rm AP}_{M}$ & ${\rm AP}_{H}$\\
       \midrule

	\parbox{2.5mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{\textbf{TD}}}}

    &Sim.Base. (ResNet-50)  & HM. & \underline{60.8} & 81.4 & 65.7 & 71.4 &61.2 & 51.2  \\
    &HRNet (HRNet-w48) $\dag$ & HM. & 71.3 & 91.1 & 77.5 & 80.5 & 71.4 & 62.5\\
    &TransPose-H & HM. & 71.8 & 91.5 & 77.8 & 79.5 & 72.9 & 62.2 \\
    &HRFormer-B & HM. & 72.4 & 91.5 & 77.9 & 80.0 & 73.5& 62.4\\
    \midrule
    \parbox{2.5mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{\textbf{BU}}}}

    &HrHRNet-w32$\dag$  & HM.  &  65.9  & 86.4 & 70.6 & 73.3 & 66.5 & 57.9\\
    &DEKR (HrHRNet-w32)$\dag$& HM.  & 65.7 & 85.7 & 70.4 & 73.0 & 66.4 & 57.5 \\

    &SWAHR (HrHRNet-w32)$\dag$& HM. & 71.6 & 88.5 & 77.6 & 78.9 & 72.4 & 63.0\\
    \midrule
    \parbox{2.5mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{\textbf{OS}}}}

    &PETR (Swin-L)  & R.+HM.  & \underline{71.6} & 90.4 & 78.3 & 77.3 & 72.0 & 65.8 \\
       \cmidrule{2-9}
    &ED-Pose (ResNet-50)  & R.  &  69.9{\color{Red}$\uparrow_{9.1}$} & 88.6& 75.8& 77.7& 70.6&60.9 \\
    &ED-Pose (Swin-L)  & R.  &  73.1{\color{Red}$\uparrow_{1.5}$} & 90.5& 79.8& 80.5& 73.8&63.8\ \\
    &ED-Pose (Swin-L*) & R.  &  \textbf{76.6}{\color{Red}$\uparrow_{5.0}$}& \textbf{92.4}& \textbf{83.3}& \textbf{83.0}& \textbf{77.3}& \textbf{68.3} \ \\
  \bottomrule
\end{tabular}}
\label{tab:crowdpose}
\vspace{-0.4cm}
\end{table*}

\begin{table*}[h]
\centering
\caption{Comparisons with state-of-the-art methods on COCO \texttt{val2017} dataset. $\dag$ denotes the flipping test. $\ddag$ removes the prediction uncertainty estimation in Poseur as a fair regression comparison. The \underline{underlined} highlights the compared results.
\Revise{The inference time of all methods is tested on an A100, except that the detector of top-down methods is tested by the MMdetection (\textit{i.e.}, $45$ms).}
}
\resizebox{0.9\linewidth}{!}{
  \begin{tabular}{p{0.4cm}l|l|l|l|ccc|cc|c}
    \toprule
       &\multicolumn{2}{c|}{\textbf{Method}} & Backbone & Loss & ${\rm AP}$ & ${\rm AP}_{50}$ & ${\rm AP}_{75}$ & ${\rm AP}_{M}$ & ${\rm AP}_{L}$ & Time [ms]\\
    \midrule
    \parbox{2.5mm}{\multirow{8}{*}{\rotatebox[origin=c]{90}{\textbf{Two-stage}}}}

    &\multirow{6}*{{\rotatebox{90}{\textbf{TD}}}}
    & Sim.Base.$^{\dag}$ & ResNet-50 & HM. &70.4 &88.6 & 78.3 & 67.1 & 77.2 & 45$+${{86}}\\
    && HRNet$^{\dag}$ &  HRNet-w32 & HM. & 74.4 & 90.5& 81.9& 70.8& 81.0& 45$+$\Revise{{112}} \\
    &&PRTR$^{\dag}$ & ResNet-50 & R. & 68.2 & 88.2 & 75.2 & 63.2 & 76.2 & 45$+$\Revise{{85}}\\
    &&Poseur &  ResNet-50 & RLE$^{\ddag}$ & 70.0 & - & -&- &- &  45$+${82} \\
    &&Poseur &  ResNet-50 & RLE & 74.2 & 89.8 & 81.3&\textbf{71.1} &80.1 &  45$+${{82}} \\
    \cmidrule{2-11}
    &\multirow{3}*{{\rotatebox{90}{\textbf{BU}}}}
    & HrHRNet$^{\dag}$ &  HRNet-w32 & HM. & 67.1 & 86.2 &73.0 &61.5 &76.1 &  {{322}}\\
    && DEKR$^{\dag}$  &  HRNet-w32 & HM. & 68.0 & 86.7 & 74.5 & 62.1 & 77.7  & -\\
    && SWAHR$^{\dag}$ &  HRNet-w32 & HM. & 68.9 & 87.8 & 74.9 & 63.0 & 77.4 & - \\
        \midrule
    \parbox{2.5mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{\textbf{E2E TD}}}}

    && Mask R-CNN &  ResNet-101 & HM. &66.0 & 86.9 & 71.5 &- &- & -\\
    &&PRTR$^{\dag}$ & ResNet-101 & HM. & 64.8 &85.1 &70.2& 60.4 &73.8& -\\
    &&Poseur$^{\dag}$ & ResNet-101 & RLE  & 68.6 & 87.5 & 74.8 & - & - & - \\
    &&Poseur$^{\dag}$ & HRNet-w48 & RLE  & 70.1&88.0 & 76.5 & - &- & -\\
    \midrule
    \parbox{2.5mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{\textbf{OS}}}}

    &&InsPose  & ResNet-50 &  R.+HM.  & 65.2& 87.2 & 71.3 & 60.6 & 72.2 & \Revise{78}\\
    &&PETR & ResNet-50 & R.+HM. & \underline{68.8}	& 87.5	& 76.3	& 62.7	& 77.7 & {105}  \\

    &&PETR & Swin-L & R.+HM.  & \underline{73.1}	& 90.7 & 80.9 & 67.2 & 81.7& {206} \\
     \cmidrule{3-11}
    &&ED-Pose & ResNet-50 & R.  &  71.6{\color{Red}$\uparrow_{2.8}$}&  89.6 & 78.1 & 65.9 & 79.8 & {\textbf{51}}{\color{Red}$\downarrow_{51.4\%}$}\\

    &&ED-Pose & Swin-L & R.  &  74.3{\color{Red}$\uparrow_{1.2}$} & 91.5 & 81.6 & 68.6 & 82.6 & {{88}}{\color{Red}$\downarrow_{57.3\%}$}  \\
    &&ED-Pose & Swin-L* & R.  &  \textbf{75.8}{\color{Red}$\uparrow_{2.7}$} & \textbf{92.3} & {82.9} & 70.4& \textbf{83.5}& {{142}}{\color{Red}$\downarrow_{31.1\%}$} \\
  \bottomrule
\end{tabular}}
\label{tab:sota}
\vspace{-0.4cm}
\end{table*}

\subsection{Results on COCO}
\vspace{-0.2cm}
We further make comparisons with the state-of-the-art
methods on COCO \texttt{val2017} and \texttt{test-dev} in Table.~\ref{tab:sota} and Table.~\ref{tab:one-stage} respectively.
In general, our ED-Pose outperforms all existing bottom-up methods and one-stage methods under the same backbone without any tricks, even the dense heatmap-based top-down methods. The proposed method achieves $71.6$ AP (by $2.8$ AP improvement) with a $51.4\%$ inference time reduction via the ResNet-50 backbone.
Additionally, our best model with the Swin-L* backbone achieves a $75.8$ AP, showing a consistent performance improvement. The improvements in the \texttt{test-dev} are similar.

\vspace{-0.2cm}
\subsubsection{Comparison of Effectiveness}
\vspace{-0.2cm}
\textbf{Comparison with one-stage methods}: Our method significantly outperforms all existing one-stage methods, especially in Table.~\ref{tab:one-stage}, such as DirecetPose~\citep{tian2019directpose}, FCPose~\citep{WeianMao2021FCPoseFC}, Inspose~\citep{DahuShi2021InsPoseIN}, CenterNet~\citep{zhou2019objects}, and PETR~\citep{shi2022end}, showing that ED-Pose with explicit box detection is an effective solution for the end-to-end framework.

\textbf{Comparison with two-stage methods}: For bottom-up methods, ED-Pose outperforms state-of-the-art methods by a large margin, such as HigherHRNet~\citep{cheng2020higherhrnet}, DEKR~\citep{geng2021bottom}, and SWAHR~\citep{luo2021rethinking}. Specifically, ED-Pose significantly surpasses the recently proposed SWAHR by $\textbf{3.7}$ AP ($71.6$ AP \textit{vs}. $67.9$ AP) even with a much smaller backbone (ResNet-50 \textit{vs}. HRNet-w32).
For top-down methods, ED-Pose is $1.2$ AP and $3.4$ AP higher than SimpleBaseline~\citep{xiao2018simple} (Sim.Base.) and PRTR~\citep{li2021pose}, respectively. To the best of our knowledge, this is the first time that a fully end-to-end framework can surpass top-down methods.

\textbf{Comparison with end-to-end top-down methods}:
Poseur~\citep{mao2022poseur} extends its framework to end-to-end human pose estimation with Mask-RCNN. Similarly, PRTR presents its end-to-end variant. However, compared to their two-stage paradigms, their end-to-end frameworks produce substantially inferior performance. Our fully end-to-end framework achieves much higher AP scores than theirs, indicating the effectiveness of our proposed methods.

\textbf{Qualitative results:} With explicit box detection, ED-Pose can perform well on a wide range of poses, containing viewpoint change, occlusion, motion blur, and crowded scenes. Fig.~\ref{fig:viz1} demonstrates the results of the detected person and the corresponding keypoints. As can be observed, the human boxes are precise under many severe cases, thus they can provide effective global human information for further keypoint detection. The local regions of the keypoint boxes are also reasonable for bringing abundant contextual information near the keypoint.

\begin{figure}[h]
\vspace{-0.2cm}
\centering
 	{
 		\begin{minipage}[t]{1\linewidth}
 			\centering
 			\includegraphics[width=1\linewidth]{fig/viz.png}
 		\end{minipage}
 	}
\vspace{-0.6cm}
\caption{Qualitative results of ED-Pose on COCO (the first row) and CrowdPose (the second row). We present both explicitly detected person boxes and keypoint boxes to understand how they work.}
\label{fig:viz1}
\end{figure}

\subsubsection{Comparison of Efficiency}
\begin{figure}[h]
\vspace{-0.4cm}
\centering
 	{
 		\begin{minipage}[t]{0.9\linewidth}
 			\centering
 			\includegraphics[width=1\linewidth]{fig/figure1.jpg}
 		\end{minipage}
 	}
\vspace{-0.4cm}
\caption{Comparisons of convergence speeds in the training stage (the left) and trade-offs between inference time and performance (the right) of existing mainstream methods. Our proposed one-stage method ED-Pose shows the superiority of efficiency compared with the Bottom-Up (BU) model HigherHRNet~\citep{cheng2020higherhrnet}, Top-Down (TD) models Sim.Base.~\citep{xiao2018simple} and DETR-based Poseur~\citep{mao2022poseur}, the one-stage method PETR~\citep{shi2022end}.
}
\label{fig:convergence}
\end{figure}

For the inference time, ED-Pose surpasses all the bottom-up and one-stage methods in both speed and accuracy fields from Table.~\ref{tab:sota} and ~\ref{tab:one-stage}.
Notably, from the right of Fig.~\ref{fig:convergence}, we compare the inference time under different AP scores. ED-Pose achieves the best efficiency and effectiveness trade-offs.
Under the competitive performance (about $74.2$) with the DETR-Based top-down method (Poseur with RLE loss~\citep{li2021human}), ED-Pose can boost inference speed by $30.7\%$.
Moreover, the left of Fig.~\ref{fig:convergence} shows the convergence speeds in the training stage that are rarely discussed before. EP-Pose introducing explicit detection is faster than all methods and obtains better performance under the early epochs. Interestingly, all DETR-based methods are slower than even bottom-up methods.

\subsection{Ablation Study}

\textbf{Explicit human detection}: We first analyze the
effectiveness of explicit human detection on both the COCO \texttt{val2017} and the CrowdPose \texttt{test}, using the ResNet-50 backbone. We remove all human detection losses in  decoder layers to verify its effectiveness. Results in Table.~\ref{tab:box} clearly verify that explicit human detection supervision significantly improves the convergence speed of keypoint detection and precision, yielding +$4.5$ AP on COCO and +$9.9$ AP on CrowdPose.

\begin{table*}[h]
\vspace{-0.7cm}
\centering
\caption{Impact on explicit human detection (Human Det.) for convergence speed and precision on the COCO dataset (the left) and CrowdPose dataset (the right).}
    \begin{center}
        \begin{minipage}[t]{0.46\textwidth}
        \centering
        \resizebox{\linewidth}{!}{
            \makeatletter\def\@captype{table}\makeatother
                \begin{tabular}{c|cccccc}
                    \toprule
                       {\textbf{Human Det.}} & 12e & 24e & 36e & 48e & 60e \\
                    \midrule
                     & 41.1 & 56.6& 61.0 & 64.7 & 67.1    \\
                     $\checkmark $ & \textbf{60.5}&  \textbf{67.5} &  \textbf{69.7}  & \textbf{70.8} & \textbf{71.6} \\
                  \bottomrule
                \end{tabular}
            }
            \label{tab:hw}
        \end{minipage}
        \hspace{-0.5cm}
        \quad
        \begin{minipage}[t]{0.51\textwidth}
        \centering
        \resizebox{\linewidth}{!}{
            \makeatletter\def\@captype{table}\makeatother
                    \begin{tabular}{c|cccccc}
                        \toprule
                           {\textbf{Human Det.}} & 12e & 24e & 36e & 48e & 60e & 80e\\
                        \midrule
                         & 13.1 & 20.5& 31.1 & 42.6 & 51.3 & 60.0   \\
                         $\checkmark $ & \textbf{37.1} & \textbf{54.8} &  \textbf{62.4}  & \textbf{66.4} & \textbf{68.2} & \textbf{69.9} \\
                      \bottomrule
                    \end{tabular}
            }
            \label{tab:mask}
        \end{minipage}
    \end{center}
     \label{tab:box}
\vspace{-0.3cm}
\end{table*}

\textbf{Keypoint detection representation}:
ED-Pose reformulates pose estimation as the keypoint box detection, where the 2D center-point coordinate $(x,y)$ of keypoint is extended to the 4D representation $(x,y,w,h)$. As the region of feature aggregation for keypoint depends on the width and height of the keypoint box, we explore the effect of $(w,h)$  under five initialization ways: 1). \textbf{None} discards width and height and keeps the original 2D coordinate representation
; 2). \textbf{Min.} denotes that the keypoint is initialized by $1\%$ width and height of the corresponding human box; 3). \textbf{Max.} means that a keypoint is directly initialized by the width and height of its corresponding human box; 4). \textbf{FFN.} is that we apply an FFN network upon the human content query to regress a 2D weight of each keypoint within the human box
; 5). \textbf{Ours} means that we initialize learnable embeddings for each keypoint across the whole dataset and utilize it to weigh the width and height of the human box.
As shown in Table.~\ref{tab:hw}, the result without width and height representation is much lower than those of the other four settings, leading to a $13.3$ AP drop compared with the best initialization way (Ours) since it will lose contextual information to query the local and global relations. Learnable width and height obtain better performance than the fixed sizes.

\textbf{The interactive learning between human and keypoint detection}:
Our human-to-keypoint detection decoder enhances the interactivity among external human-to-human, internal human-to-keypoint, and internal keypoint-to-keypoint. Thus, we explore the impact of different interactive learning strategies: 1) \textbf{Full} makes all human queries and keypoint queries interact one by one; 2) \textbf{w/o H-K} removes the internal human-to-keypoint interaction; 3) \textbf{w/o H-H} discards the external human-to-human interaction; 4) \textbf{Ours} is our full interactive learning strategy. As shown in Table.~\ref{tab:mask}, direct full connection for all queries could lead to $1.2$ AP degradation. The reason is that external keypoint-keypoint interaction in a full connection strategy may cause a large disturbance, especially in crowded situations. Besides, other interactive strategies slightly affect performance as well.

\textbf{Hyperparameter tuning}: To explore the effect on the number $M$ of selected queries in Sec.~\ref{sec:N-to-M}, we conduct three numbers with $50$, $100$, and $200$, respectively. As shown in Table.~\ref{tab:query}, increasing selected human queries can improve performance, but it meets a bottleneck for further improvement.

\begin{table*}[h]
\vspace{-0.6cm}
    \begin{center}
        \begin{minipage}[t]{0.37\textwidth}
        \centering
        \caption{Impact on the width and weight initialization of the keypoint box.}
        \resizebox{\linewidth}{!}{
            \makeatletter\def\@captype{table}\makeatother
              \begin{tabular}{l|ccccc}
                \toprule
                   {{$\mathbf{(w,h)}$}} & None & Min. & Max. & FFN. & Ours \\
                \midrule
                  ${\rm AP}$ & 58.3  & 70.8 &  70.8 & 71.2 & \textbf{71.6} \\
                  ${\rm AP}_{M}$ & 57.5 & 65.1 &  64.9 & 65.7 & \textbf{65.9} \\
                  ${\rm AP}_{L}$ & 60.0 &  79.2 & 79.3 & 79.2 & \textbf{79.8}\\
              \bottomrule
              \end{tabular}
            }
            \label{tab:hw}
        \end{minipage}
        \hspace{-0.25cm}
        \quad
        \begin{minipage}[t]{0.37\textwidth}
        \centering
        \caption{Impact on interactive learning between human and keypoint detection.}
        \resizebox{\linewidth}{!}{
            \makeatletter\def\@captype{table}\makeatother
              \begin{tabular}{l|ccccc}
                \toprule
                   {\textbf{Strategy}} & Full & w/o H-K & w/o H-H & Ours \\
                \midrule
                ${\rm AP}$ & 70.4  & 71.2 &  71.3 & \textbf{71.6} \\
                  ${\rm AP}_{M}$ & 64.6 & 65.6& \textbf{66.0} & 65.9\\
                  ${\rm AP}_{L}$ & 78.8 & 79.3 & 79.0 & \textbf{79.8} \\
              \bottomrule
            \end{tabular}
            }
            \label{tab:mask}
        \end{minipage}
        \hspace{-0.25cm}
        \quad
        \begin{minipage}[t]{0.22\textwidth}
        \centering
        \caption{Impact on the number of $M$ selected queries.}
        \resizebox{\linewidth}{!}{
            \makeatletter\def\@captype{table}\makeatother
            \begin{tabular}{l|ccc}
            \toprule
               {\textbf{$M$}} & $50$ & $100$  & $200$\\
            \midrule
                ${\rm AP}$ & 70.9 & \textbf{71.6} & \textbf{71.6}  \\
                  ${\rm AP}_{M}$ & 65.2 & 65.9 & \textbf{66.0} \\
                  ${\rm AP}_{L}$ & 79.2 & \textbf{79.8} & 79.6 \\
            \bottomrule
            \end{tabular}
            }
            \label{tab:query}
        \end{minipage}
    \end{center}
\vspace{-0.4cm}
\end{table*}

\section{Introduction}

\begin{figure}[h]
\vspace{-0.4cm}
\centering
 	{
 		\begin{minipage}[t]{1\linewidth}
 			\centering
 			\includegraphics[width=1\linewidth]{fig/intro.pdf}
 		\end{minipage}
 	}
\vspace{-0.7cm}
\caption{Illustration of (a) the perception of the pose estimation task that usually captures global and local contexts concurrently; (b) a taxonomy of existing estimators. ED-Pose (Ours) is a novel one-stage method of learning both global and local relations in an end-to-end manner.}
\label{fig:intro}
\vspace{-0.2cm}
\end{figure}

Multi-person human pose estimation has attracted much attention in the computer vision community for decades for its wide applications in areas of augmented reality (AR), virtual reality (VR), and human-computer interaction (HCI). Given an image, it targets to localize the 2D keypoint positions for every person in the image.
Although many methods have been developed~\citep{xiao2018simple,sun2019deep,cheng2020higherhrnet,mao2022poseur,shi2022end}, it remains challenging and intractable for situations with heavy occlusions, hard poses, and diverse body part scales.

Intuitively, as shown in Figure~\ref{fig:intro}, this task needs to focus on both global (human-level) and local (keypoint-level) dependencies, which  concentrate on different levels of semantic granularity.
Mainstream solutions are normally two-stage methods, which divide the problem into two separate subproblems (e.g., the global person detection and local keypoint regression).
Such solutions include Top-Down (TD) methods~\citep{xiao2018simple,sun2019deep,li2021pose,mao2022poseur} which are of high performance yet with a high inference cost and Bottom-Up (BU) solutions~\citep{cao2017realtime,newell2017associative,cheng2020higherhrnet} which are fast in inference yet with relatively lower precision.
However, all of these methods are non-differentiable between their global and local stages due to hand-crafted operations, like Non-Maximum Suppression (NMS), Region of Interest (RoI) cropping, and keypoint grouping post-processing.
Lately, Poseur~\citep{mao2022poseur} tries to directly apply top-down methods to an end-to-end framework and finds that there will be a significant performance drop (about \textbf{8.7} AP on COCO), indicating the optimization conflicts between the learning of global and local relations.

Exploring a fully end-to-end trainable method to unify the two disassembled subproblems is attractive and important.
Inspired by the success of recent end-to-end object detection methods, like DETR~\citep{carion2020end}, there is a surge of related approaches that regard human pose estimation as a direct set prediction problem. They utilize a bipartite matching for one-to-one prediction with Transformers to avoid cumbersome post-processings~\citep{li2021pose,mao2021tfpose,mao2022poseur,stoffl2021end,shi2022end}.
Recently, PETR~\citep{shi2022end} proposes a fully end-to-end framework to predict instance-aware poses without any post-processings and shows a favorable potential.
Nevertheless, it directly uses a pose decoder with randomly initialized pose queries to query local features from images. The only local dependency makes keypoint matching across persons ambiguous and thus leading to inferior performance, especially for occlusions, complex poses, and diverse human scales in crowded scenes.
Moreover, either two-stage methods or DETR-based estimators suffer from slow training convergence and need more epochs (e.g., train a model above a week) to achieve high precision. Additionally, the convergence speed of DETR-based methods is even slower than bottom-up methods~\citep{cheng2020higherhrnet}. We address the details in Sec.\ref{sec:rethink}.

Based on the above observations, this work re-considers multi-person pose estimation as two Explicit box Detection processes named ED-Pose.
We realize each box detection by using a decoder and cascade them to form an end-to-end framework, which makes the model fast in convergence, precise, and scalable.
Specifically, to obtain global dependencies, the first process detects boxes for all persons via human box queries selected from the encoded image tokens. This simple step can provide a good initialization for the latter keypoint detection to accelerate the training convergence.
Then, to capture local contextual relations and reduce ambiguities in the feature alignment, we regard the following pose estimation task as a keypoint box detection problem, where it learns both box positions and local contents for each keypoint.
Such an approach can leverage contextual information near a keypoint by directly regressing the keypoint box position and learning local keypoint content queries without dense supervision (e.g., heatmap).
To further enhance the global-local interactivity among external human-human, internal human-keypoint, and internal keypoint-keypoint, we design interactive learning between human detection and keypoint detection.

Following the two Explicit box Detection processes, we can unify the global and local feature learning using the consistent regression loss and the same box representation in an end-to-end framework.
We summarize the related methods from the supervisions and representations. Compared with previous works, ED-Pose is more conceptually simple.
Notably, we find that explicit global box detection will gain \textbf{4.5} AP on COCO and \textbf{9.9} AP on CrowdPose compared with a solution without such a scheme.
In comparison to top-down methods, ED-Pose makes the human and keypoint detection share the same encoders to avoid redundant costs from human detection and further boost performance by \textbf{1.2} AP on COCO and \textbf{9.1} AP on CrowdPose under the same ResNet-50 backbone.
Moreover, ED-Pose surpasses the previous end-to-end model PETR significantly by \textbf{2.8} AP on COCO and \textbf{5.0} AP on CrowdPose.
In crowded scenes, ED-Pose achieves the state-of-the-art with \textbf{76.6} AP (by 4.2 AP improvement over the previous SOTA~\citep{yuan2021hrformer})  without any bells and whistles (e.g., without multi-scale test and flipping).
We hope this simple attempt at explicit box detection, simplification of losses, and no post-processing to unify the whole pipeline could bring in new perspectives to further one-stage framework designs.

\begin{table*}[h]
\caption{Comparisons of existing estimators from the losses and representations of the Human and Keypoints. Our proposed ED-Pose unifies both Human and Keypoint detection under the consistent L1 regression loss and the box representation, making the end-to-end training simple yet effective.}
\resizebox{\linewidth}{!}{
\begin{tabular}{c|c|cc|cc}
    \toprule
\multicolumn{2}{c|}{Methodology} &Human Loss& Keypoint Loss& Human Representation & Keypoint Representation \\\midrule
     \multirow{2}*{Two-Stage Methods} & Top-Down & Regression & Heatmap & $(x,y,h,w)$ & $(x,y)$\\
      & Bottom-Up & -& Heatmap &-&  $(x,y)$ \\
          \midrule
     \multirow{2}*{One-Stage Methods} & Previous & - & Regression + Heatmap & -& $(x,y)$\\
  & Ours  & Regression &Regression&  $(x,y,h,w)$ &$(x,y,h,w)$ \\
  \bottomrule
  \end{tabular}}
\label{tab:comp_loss}
\vspace{-0.3cm}
\end{table*}

\section{Methodology}
\vspace{-0.3cm}
\subsection{Overview}
\begin{figure}[h]
\vspace{-0.4cm}
\centering
 	{
 		\begin{minipage}[t]{1.0\linewidth}
 			\centering
 			\includegraphics[width=1.0\linewidth]{fig/method.pdf}
 		\end{minipage}
 	}
\vspace{-0.8cm}
\caption{ The overview architecture of our ED-Pose, which contains a Human Detection Decoder and a Human-to-Keypoint Detection Decoder to detect human and keypoint boxes explicitly.
}
\label{fig:query}
\vspace{-0.3cm}
\end{figure}

As illustrated in Figure~\ref{fig:query}, the proposed ED-Pose\xspace is a fully end-to-end multi-person pose estimation framework without any post-processing.
Given an input image, we first utilize a backbone
to extract multi-scale features and obtain tokenized representations $\mathbf{F}$,
and then feed them into the Transformer encoder (e.g., deformable attention modules~\citep{zhu2020deformable}) with Positional Embeddings ($\mathbf{PE}$)
to calculate the refined tokens $\mathbf{F}'$.
To improve the efficiency of the following decoding process, we conduct coarse human query selection from $\mathbf{F}'$ to obtain the sparse human content queries $\mathbf{Q}_{H}^c$.
Then we utilize $\mathbf{Q}_{H}^c$ to generate the corresponding position queries $\mathbf{Q}_{H}^p$ via a Feed-Forward Network (FFN) and feed both $\mathbf{Q}_{H}^c$ and  $\mathbf{Q}_{H}^p$ into the \textit{Human Detection Decoder} to update into the corresponding queries $\mathbf{Q}_{H}^{c'}$ and $\mathbf{Q}_{H}^{p'}$, respectively (see Sec.\ref{sec:box-decoder}).
For each output human content query, we attach a human box regression and class entropy supervision ($L_h^l$ and $L_c^l$) at the $l$-th decoder layer.
Next, we further perform fine human query selection to discard the redundant human queries, obtaining content queries $\mathbf{Q}_{H_s}^{c}$ and position queries $\mathbf{Q}_{H_s}^{p}$.
We initialize keypoint queries based on these retained high-quality human queries and concatenate the human and keypoint queries together to form human-keypoint queries.
We name such a scheme as human-to-keypoint query expansion, and the obtained human-keypoint content queries $\mathbf{Q}_{H,K}^{c}$ and position queries $\mathbf{Q}_{H,K}^{p}$ are fed into \textit{Human-to-Keypoint Detection Decoder}.
In this way, we realize the keypoint detection at an instance level and update these queries to $\mathbf{Q}_{H,K}^{c'}$ and $\mathbf{Q}_{H,K}^{p'}$ layer by layer (see Sec.\ref{sec:box-kpt-decoder}).
Also, the keypoint and human box regression loss and class loss ($L_k^l$, $L_h^l$ and $L_c^l$) are added to the $l$-th decoder layer.
Finally, we use several FFN heads to regress the keypoint positions in each human box.

\textbf{Loss:}
Following the DETR~\citep{carion2020end}, we employ a set-based Hungarian loss that forces a unique prediction for each ground-truth box and keypoint. Our overall loss functions contain classification $L_c$, human box regression $L_h$, and keypoint regression loss $L_k$. Notably, $L_k$ simply consists of the normal L1 loss and the constrained L1 loss named Object Keypoint Similarity (OKS) loss~\citep{shi2022end} without any dense supervision (e.g., heatmap).

\begin{figure}[ht]
\centering
\vspace{-0.3cm}
    \centering
    \includegraphics[width=1\textwidth]{fig/decoder.pdf}
    \vspace{-0.7cm}
    \caption{The detailed illustration of (a) Human Detection Decoder, (b) Human-to-Keypoint Detection Decoder and (c) the detailed Interactive Learning in Human-to-Keypoint Detection Decoder. }
    \label{fig:decoder}
    \vspace{-0.3cm}
\end{figure}

\subsection{Human Detection Decoder}
\label{sec:box-decoder}
\vspace{-0.2cm}
In Figure~\ref{fig:decoder} (a), the proposed Human Detection Decoder aims to predict the candidate bounding box positions for each person in the image, as well as the corresponding content representation by leveraging image context from all encoded tokens.
Given $N$ input human content queries $\mathbf{Q}_{H}^c\in \mathbb{R}^{N\times D}$ and human position queries $\mathbf{Q}_{H}^p \in \mathbb{R}^{N\times 4}$, where $D$ is the channel dimension, the Human Detection Decoder outputs $N$ refined human box positions $\mathbf{Q}_{H}^{p'}$ and content representations $\mathbf{Q}_{H}^{c'}$.

\textbf{Coarse-to-Fine Human Query Selection}:
\label{sec:N-to-M}
Unlike previous work~\citep{shi2022end} to randomly initialize the input queries,
we adopt the query selection (QS) strategy for better initialization~\citep{zhang2022dino}.
Specifically, we propose a coarse-to-fine scheme to progressively select high-quality human queries.
In practice, the \textit{coarse human QS} selects first from a large number of refined tokens $\mathbf{F}' \in \mathbb{R}^{T \times D}$,
where $T$ is the number of the tokens (e.g., about $15$K).
Thus, the human content queries $\mathbf{Q}_{H}^c\in \mathbb{R}^{N\times D}$ are initialized by the top-$N$ refined tokens' features, ranked by their instance classification scores.
Here $N$ indicates the number of candidate human boxes (e.g., $900$). %
Then we use these selected human content queries to calculate their position queries $\mathbf{Q}_{H}^p\in \mathbb{R}^{N\times 4}$ by employing a simple detection head.
After passing through the Human Detection Decoder, we further conduct the \textit{fine human QS} by only retaining $M$ (e.g., $100$) refined human content queries and their position queries according to the classification scores, denoted as $\mathbf{Q}_{H_s}^{c}\in \mathbb{R}^{M\times D}$ and $\mathbf{Q}_{H_s}^{p}\in \mathbb{R}^{M\times 4}$.

\textbf{Human Box Detection}:
As illustrated in Figure~\ref{fig:decoder} (a),
we combine $\mathbf{Q}_{H}^c$ with $\mathbf{Q}_{H}^p$ and feed them into Human-to-Human Attention (\textit{i.e.} a self-attention layer) to calculate the relations among human queries, and output contextual enhanced ones.
Motivated by the deformable DETR~\cite{zhu2020deformable}, these enhanced human content queries, together with their position queries are further fed into Deformable Token-to-Human Attention (\textit{i.e.} a cross-attention layer) to update human queries by conducting interaction with multi-scale tokens.
Next, to adjust the human box positions, we leverage updated human queries to calculate the 4D offsets following the DAB-DETR~\citep{liu2022dab}, and add them back to previous human position queries.
In this way, the Human Detection Decoder refine the human content and position queries progressively by stacking multiple aforementioned layers and output $\mathbf{Q}_{H}^{c'}\in \mathbb{R}^{N\times D}$ and $\mathbf{Q}_{H}^{p'}\in \mathbb{R}^{N\times 4}$.

\subsection{Human-to-Keypoint Detection Decoder}
\label{sec:box-kpt-decoder}
\vspace{-0.2cm}
To unify both human and keypoint representations as the box and facilitate follow-up interactive learning, we regard multi-person pose estimation as the multiple set keypoint box detection problems.
Based on the selected high quality human queries $\mathbf{Q}_{H_s}^{p}$ and  $\mathbf{Q}_{H_s}^{c}$,
we initialize multiple sets of keypoint queries (\textit{i.e.} including both content and position queries) through the Human-to-Keypoint Query Expansion process,
and concatenate the human and keypoint queries together as inputs of the Human-to-Keypoint Detection Decoder (Figure.~\ref{fig:decoder} (b)) to calculate the precise keypoint positions for each person.

\textbf{Human-to-Keypoint Query Expansion}:
After obtaining $\mathbf{Q}_{H_s}^{p}$ and  $\mathbf{Q}_{H_s}^{c}$, the keypoint content query and positional query are initialized by the relevant human query instead of random initialization.
In practice, we first initialize a set of learnable keypoint embeddings $\mathbf{V_e} \in \mathbb{R}^{1 \times K\times D}$ where $K$ is the total number of keypoints (e.g., 17).
To concretize these embeddings to multiple specific persons' keypoint queries, we broadcast the first dimension of $\mathbf{V_e}$ to the number of human queries $M$ and then add it with $\mathbf{Q}_{H_s}^{c} \in  \mathbb{R}^{M\times 1 \times D}$ to obtain $M$ set of keypoint content queries.
For a specific set of keypoint position queries,
we separate the initialization process into center coordinate initialization (e.g., \{x, y\}) and box size initialization (e.g., \{w, h\}).
For the former, we adopt an FFN to regress all of the $K$ positions' coordinates by employing the corresponding human content query.
For the latter, the sizes of $K$ boxes are conditioned by the width and height of the corresponding human box via dot-multiplying dynamic weights $\textbf{W} \in \mathbb{R}^{K\times 2}$.
After that, we consider each human box and the corresponding set of keypoint boxes as a whole and generate the human-keypoint content queries $\mathbf{Q}_{H,K}^{c} \in  \mathbb{R}^{(M+M*K)\times D}$ and position queries $\mathbf{Q}_{H,K}^{p} \in  \mathbb{R}^{(M+M*K)\times 4}$ for further predictions.

\textbf{The Interactive Learning between Human and Keypoint Detection}:
As illustrated in Figure~\ref{fig:decoder} (b) and (c),
after generating human-keypoint queries, we feed them into Human-to-Keypoint Interactive Attention to learn relations from internal human-keypoint, internal keypoint-keypoint, and external human-human.
Such an interactive learning method between global and local feature aggregations has been ingeniously introduced to ensure the effectiveness of keypoint feature extraction.
In practice, we find that the fully connected interactive learning with external keypoint-keypoint would cause a large disturbance, especially in crowded situations, similar to problems encountered in bottom-up methods.
In contrast, our external human-human interactions effectively propagate global context across different candidates and distinguish their relations clearly.
Then we take the enhanced human-to-keypoint content queries to conduct the interaction with multi-scale encoded tokens to obtain updated human-to-keypoint queries.
By stacking multiple layers illustated in Figure~\ref{fig:decoder} (b), we finally obtain the refined human-to-keypoint queries, denoted as $\mathbf{Q}_{H, K}^{c'}$ and  $\mathbf{Q}_{H, K}^{p'}$.

\section{Related work}
\vspace{-0.2cm}
\textbf{One-stage Multi-Person Pose Estimation:} With the development of anchor-free object detectors~\citep{ZhiTian2019FCOSFC,LichaoHuang2015DenseBoxUL}, DirectPose~\citep{tian2019directpose} directly predicts instance-aware keypoints for all persons from an image. The direct end-to-end framework provides a new perspective to avoid the above cumbersome issues met in two-stage methods.
Generally speaking, these methods densely locate a set of pose candidates, which consist of joint positions from the same person.
FCPose~\citep{WeianMao2021FCPoseFC} builds upon dynamic filters~\citep{XuJia2016DynamicFN} in compact keypoint heads to boost both accuracy and speed.
Meanwhile, Inspose~\citep{DahuShi2021InsPoseIN} designs instance-aware dynamic networks to adaptively adjust part of the network parameters for each instance.
Nevertheless, these one-stage methods still need NMS to remove duplicates in the post-processing stage.
To further remove such hand-crafted components, PETR~\citep{shi2022end} views pose estimation as a hierarchical set prediction problem and proposes the first \emph{fully} end-to-end pose estimation framework with the advent of DETR~\citep{carion2020end}.

\textbf{Detection Transformers:} For the first time, DETR~\citep{carion2020end} performs object detection in an end-to-end manner by using a set-based global loss that forces directly unique predictions via bipartite matching and a Transformer encoder-decoder architecture.
It simplifies object detection as a direct set prediction problem, dropping multiple hand-designed components and prior knowledge.
Due to the effectiveness of DETR and its varieties (\textit{e.g.}, Deformable DETR~\citep{zhu2020deformable}), their frameworks have been widely transferred in many complex tasks, such as Mask DINO~\citep{li2022mask} for segmentation and PETR~\citep{shi2022end} for pose estimation.
Following the top-down methods, PRTR~\citep{li2021pose} and TFPose~\citep{mao2021tfpose} adopt Detection Transformers to estimate the cropped single-person images as a query-based regression task.
To capture the underlying output distribution and further improve performance in the regression paradigm, Poseur~\citep{mao2022poseur} brings Residual Log-likelihood Estimation (RLE)~\citep{li2021human} into the DETR-based top-down framework, achieving the state-of-the-art performance of regression-based methods.
For the one-stage manner, POET~\citep{stoffl2021end} utilizes the property of DETR to directly regress the poses (instead of bounding boxes) of all person instances in an image.
Recently, PETR~\citep{shi2022end} designs a \textit{fully} end-to-end paradigm with hierarchical attention decoders to capture both relations between poses and kinematic joints.

All the aforementioned methods take advantage of Detection Transformers and densely regress a set of poses (only local relations). However, they ignore the importance of introducing explicit box detection in pose estimation to model both global and local dependencies well.

\section{Rethinking One-stage Multi-Person pose estimation}
\label{sec:rethink}
\vspace{-0.2cm}
\textbf{The Necessities of One-stage Methods:}
For a long time, the two-stage paradigm dominates the mainstream methods for multi-person pose estimators. It can be generally divided into top-down methods and bottom-up methods.
Top-Down (TD) methods~\citep{xiao2018simple,sun2019deep,mao2022poseur} disentangle the task into detecting and cropping each person with an object detector (\textit{e.g.}, Mask RCNN~\citep{he2017maskrcnn}) in an image from the global (person-level) dependency and then conduct the single-person pose estimation via another model.
They focus on local (keypoint-level) relation modeling and improving the accuracy of single-person pose estimation. However, these methods still suffer from 1). heavy reliance on the performance of the human detector, 2). redundantly computational costs for additional human detection and RoI operations, and 3). the separate training for the human detector and the corresponding pose estimator.
Instead, Bottom-Up (BU) methods~\citep{cao2017realtime,newell2017associative,cheng2020higherhrnet} first detect all the keypoints in an instance-agnostic fashion. Next, they employ a heuristical grouping algorithm to associate the detected keypoints that belong to the same person, which improves efficiency.
Even so, the complicated grouping scheme makes bottom-up methods hard to handle heavy occlusions and multiple-person scales, resulting in inferior performance.
More importantly, both of them suffer from non-differentiable optimization between the global and local features, which is not perceptive.
Intuitively, one-stage methods could alleviate the above issues since all modules can be optimized in an end-to-end manner, and balance effectiveness and efficiency.
Interestingly, the recent DETR-based top-down method Poseur~\citep{mao2022poseur} tries to directly apply it to a one-stage framework and finds that it suffers from a significant performance drop. It might be the optimization conflicts between the global and local dependency learning using the shared encoder. Thus, how to design a one-stage framework effectively is still challenging and questionable.

\textbf{The Bottlenecks of Existing One-stage Methods:}
In terms of existing DETR-based methods, most of them still adopt the top-down framework, and improve the second single-person pose estimation via regarding it as a sequence prediction problem~\citep{WeianMao2021FCPoseFC,DahuShi2021InsPoseIN}. PETR is the first work to make the whole pipeline end-to-end without any post-processing.
However, existing methods still have some limitations.
First, all of them only utilize local dependencies to regress keypoints. Directly regressing keypoints for each person via pose queries is semantically ambiguous as the bottom-up strategies to find all keypoints from raw images instead of from cropped images.
Second, the pose or keypoint queries proposed in the above methods are randomly initialized without utilizing previously extracted features, making the training phase slow and ineffective.
Third, the keypoint representation as a point lacks contextual information when it queries from the encoded features, leading to feature misalignment.
Last, the interactions among global-to-global, global-to-local, and local-to-local are complex, especially in crowd scenes. The current models do not pay attention to handling these complicated relations.
In this article, we try to tackle the above issues by using unified box representations and regression losses in a one-stage process.