\subsection{Datasets}

We evaluate our model on the VG200 dataset and two large-scale long-tail VRR datasets, named GQA-LT \cite{ltvrd} and VG8K-LT \cite{ltvrd}.

\noindent \textbf{GQA-LT.} This dataset contains $72,580$ training, $2,573$ validation and $7,722$ testing images. Overall, it contains $1,703$ objects and $310$ relationships. The GQA-LT has a heavy ``long-tail" distribution where the example numbers per each class range from merely $1$ to $1,692,068$.

\noindent \textbf{VG8K-LT.} It is collected from Visual Genome (v1.4) \cite{visualGenome} dataset, containing $97,623$ training, $1,999$ validation and $4,860$ testing images. It covers $5,330$ objects and $2,000$ different relation types in total, in which least frequent objects/relationships only have $14$ examples and the most frequent ones have $618,687$ examples.

\noindent \textbf{VG200.} This dataset has been widely studied in the literature \cite{messagePassing, lsvru, zhang2019graphical}. It contains 50 relationships, and the category frequency in this dataset is considerably more balanced than that in GQA-LT and VG8K-LT. We follow the same data split as in \cite{lsvru} in our experiment.

\subsection{Experimental Settings}

\noindent\textbf{GQA-LT \& VG8K-LT Baselines.} We compare RelTransformer with several state-of-the-art models. The most popular models in this benchmark are implemented based on LSVRU \cite{lsvru} framework. To improve over the long-tail performance, baseline models usually are combined with the following strategies: 1) class-imbalance loss functions such as weighted cross entropy (WCE), equalization loss (EQL) \cite{eql}, focal loss \cite{focalLoss}, and ViLHub loss \cite{ltvrd}. 2) relation augmentation strategies such as RelMix \cite{ltvrd} to augment more examples in long-tail relations. 3) Decoupling \cite{decoupling} which decouples the learning procedure into representation learning and classification. 4) Counterfactual \cite{tang2020unbiased}, which alleviates the biased scene graph generation via the counterfactual learning. 5) OLTR \cite{oltr}, which has memory module with augmented attention.

\noindent\textbf{VG200 Baselines.} We compare with several strong baselines including Visual Relationship Detection \cite{vrd}, Message Passing \cite{messagePassing}, Associative Embedding \cite{associative}, MotifNet \cite{motifNet}, Permutation Invariant Predication \cite{permutationInvariant}, LSVRU \cite{lsvru}, relationship detection with graph contrastive loss (RelDN) \cite{zhang2019graphical},  GPS-Net \cite{gpsNet}, Visual Relationship Detection with Visual-Linguistic Knowledge (RVL-BERT) \cite{chiou2021visual} and Relational Transformer Network (RTN) \cite{rtn}.

\begin{table}[t]
    \centering
    \scalebox{0.9}{
    \begin{tabular}{l ccc }

    \multirow{2}{*}{Models} & \multicolumn{3}{c}{PRDCLS}\\
    & R@20 & R@50 & R@100\\

    \toprule
     VRD \cite{vrd}& - & 27.9 & 35.0  \\
     Message Passing \cite{messagePassing}& 52.7 & 59.3 & 61.3  \\
     Associative Embedding \cite{associative} & 47.9 & 54.1 & 55.4\\
     MotifNet (Left to Right) \cite{motifNet} & 58.5 & 65.2 & 67.1  \\
     Permutation Invariant \cite{permutationInvariant} &  - & 65.1 & 66.9 \\
     LSVRU \cite{lsvru}  & 66.8 & 68.4 & 68.4 \\
     RelDN \cite{zhang2019graphical}  & 66.9 & 68.4 & 68.4\\
     Graph-RCNN \cite{graph-rcnn}  & - & 54.2 & 59.1 \\
     VCTREE-SL  & 59.8 & 66.2 & 67.9 \\
     GPS-Net \cite{gpsNet} & 60.7 & 66.9 & 68.8  \\
     RVL-BERT \cite{chiou2021visual} &-  &62.9& 66.6\\
     RTN \cite{rtn} & 68.3& 68.7 &68.7\\
     \midrule
     RelTransformer (ours) & \textbf{68.5} & \textbf{69.7} & \textbf{69.7} \\

    \end{tabular}
    }
    \caption{Relation prediction on VG200 dataset.
    }
    \label{vg200_result}
\end{table}

\noindent\textbf{Evaluation Metrics.} For the GQA-LT and VG8K-LT datasets, we report the average per-class accuracy, which is commonly used for long-tail evaluation \cite{eql, decoupling, ltvrd}. Following the same evaluation setting as \cite{ltvrd}, we split the relationship classes into the many, medium, and few based on the relation frequency in the training dataset as shown in Table \ref{main_results}. For the VG200 dataset, following the previous evaluation setting in \cite{lsvru,Suhail_2021_CVPR}, we measure the Recall@k and mean Recall@K for predicate classification (PRDCLS), which is to predict the relation labels given the ground truth boxes and labels of the subject and object.

\subsection{Quantitative Results}

\noindent\textbf{GQA-LT and VG8K-LT Evaluation.} We present our results for GQA-LT and VG8K-LT datasets in Table \ref{main_results}. There is a clear performance improvement over all the baselines with the addition of RelTransformer, especially on the med and few classes. The combination of RelTransformer with WCE improves the med and few category in GQA-LT by a considerable margin of $\approx$20\% compared to all the baselines. This huge gain can be attributed to the weighted assignment of different classes when WCE loss is applied. This further refines the attention weights assigned to different classes in the global context and hence helps the overall performance. While previous works \cite{tang2020unbiased} improved the tail performance at the cost of head class accuracy, RelTransformer consistently improves on the tail as well as the head as seen from the table, underlying the effectiveness of our model.

For VG8K-LT, we also see performance gains with the addition of RelTransformer on all baselines across ``many", ``medium" and ``few" categories. A considerable improvement of $\approx$5\% can be seen when RelTransformer is combined with DCPL \cite{decoupling}, performing the best for the VG8K-LT dataset. While we see consistent improvements with the addition of RelTransformer for the VG8K-LT dataset, the improvement margins are certainly lower as compared to GQA-LT as seen in Table \ref{main_results}. This is due to the more challenging nature of VG8K-LT, containing 2000 relation classes compared to 300 classes present in GQA-LT. Some qualitative examples with the addition of RelTransformer can be seen in Fig. \ref{demonstrations}.

\noindent\textbf{VG200 Evaluation.} We also evaluate our model on VG200 dataset in Table \ref{vg200_result}. We compare with many different message-passing approaches in our baselines including RNN-based \cite{vctree,motifNet}, GCN-based \cite{graph-rcnn,gpsNet} and Transformer-based \cite{rtn,chiou2021visual}. These two Transformer-based approaches either only focus on the relational triplet context \cite{chiou2021visual} or neglect it completely \cite{rtn}. Our model differs from them with a different message-passing strategy, different context construction and a novel memory attention. The experimental results show that our method can better exploit the relational features, and we improve over the best-performing baselines by $0.2\%$ on R@20, $1.0\%$ on R@50, and $0.9\%$ on R@100.

\noindent \textbf{Mean Recall@K on VG200.} We evaluate the mean recall@k performance on the VG200 dataset for elation predication, and compared RelTransformer with several strong baselines such as VCTREE and Motif with both cross-entropy and EBM losses \cite{Suhail_2021_CVPR}. The results are summarized in Table~\ref{vg200_mreall}. We observe that RelTransformer can outperform all the baselines on mR@(20, 50, 100) while only being combined with cross-entropy loss, which shows its robustness on other data-imbalanced datasets.

\begin{table}[t]
    \centering

    \scalebox{0.9}{
    \begin{tabular}{l cccc }

    \multirow{2}{*}{Models} &\multirow{2}{*}{Method} &  \multicolumn{3}{c}{PRDCLS}\\
    & & mR@20 & mR@50 & mR@100\\

    \toprule
     IMP  & CE & 8.85 & 10.97 & 11.77 \\
     IMP  & EBM \cite{Suhail_2021_CVPR}  & 9.43 & 11.83 & 12.77 \\
     Motif & CE &12.45 & 15.71 & 16.8 \\
     Motif & EBM &14.2 & 18.2 & 19.7 \\
     VCTREE  & CE & 13.07 & 16.53 & 17.77 \\
     VCTREE  & EBM & 14.17 & 18.02 & 19.53 \\
     \midrule
    Ours & CE & \textbf{18.51} & \textbf{19.58} & \textbf{20.19}\\
    \end{tabular}}
    \caption{Mean Recall@K Performance on VG200 Dataset.}
    \label{vg200_mreall}
\end{table}

\begin{table*}[t]
\centering
\scalebox{0.9}{
\begin{tabular}{ cc| ccccccccc}

 \multicolumn{2}{c}{} & \multicolumn{3}{c}{many} & \multicolumn{3}{c}{medium} & \multicolumn{3}{c}{few} \\
Architecture & Learning Methods & SO &  SR &  OR &   SO &  SR & OR   &  SO & SR & OR \\
\toprule
LSVRU & VilHub & 40.5 & 32.8 & 33.7  & 25.7 & 14.2 & 13.9  & 10.2 & 5.3 & 5.2 \\
\midrule
LSVRU & CE &38.6 & 30.3 & 31.5  & 21.8 & 11.3 & 10.8  & 7.5 & 4.3 & 4.2 \\
RelTransformer & CE & \underline{\textbf{$\text{54.2}$}} & \underline{\textbf{$\text{46.6}$}} &  \underline{\textbf{$\text{47.2}$}} & \underline{\textbf{$\text{37.4}$}} &  \underline{\textbf{$\text{20.8}$}} & \underline{\textbf{$\text{21.8}$}} & \underline{\textbf{$\text{16.1}$}} & \textbf{8.6}  & \textbf{7.7}\\
\midrule
LSVRU & Focal Loss & 39.2 & 31.1 & 32.3 & 23.2 & 11.9 & 11.5 & 8.2 & 4.3  & 4.2\\
RelTransformer & Focal Loss & \textbf{49.9} & \textbf{41.7}& \textbf{42.5} & \textbf{32.2} & \textbf{17.7} & \textbf{8.0} & \textbf{13.1} & \textbf{7.0}  & \textbf{6.4}\\
\midrule
LSVRU  &  WCE &  18.3 & 17.3 & 17.2  & 13.7 & 9.4 & 9.4  & 7.1 & 4.2 & 3.6  \\
RelTransformer & WCE  & \textbf{19.2} & \textbf{20.0} & \textbf{19.5} & \textbf{15.7} & \textbf{13.6} &  \textbf{13.5}& \textbf{10.3} & \underline{\textbf{$\text{8.7}$}} & \underline{\textbf{$\text{8.1}$}} \\
\end{tabular}
}
\caption{Relationship triplet performance on GQA-LT dataset. SO = (subject, object), SR = (subject, relation) and OR = (object, relation). }
\label{compositional_result}
\end{table*}

\subsection{Further Analysis}
To analyze our results in more depth, we quantify our model's improvement per each class and visualize them in Fig \ref{gqalt_improvement}. We provide the contrast between RelTransformer and LSVRU with cross entropy loss. From the figure, we can observe that RelTransformer can improve the the majority of classes on both datasets. In particular, RelTransformer improves 173 relations while only worsening 32 ones on VG8K-LT dataset with most performance gain from the medium and few classes.

\begin{figure}
\centering
 \includegraphics[width=1.0\linewidth]
    {Figure/gqa_vg8k_improvement_crop.pdf}
\caption{Per-class relationship accuracy comparisons between our RelTransformer and LSVRU \cite{lsvru} baseline on VG8K-LT(top) and GQA-LT(bottom) dataset. The green bars indicate the improvement of RelTransformer over LSVRU, red bars indicate worsening and no bars mean no change.  The left-side y-axis represents the number of examples per class. The right-side y-axis shows the absolute accuracy improvement. The x-axis represents the relation classes which are sorted by their frequency.}
\label{gqalt_improvement}
\end{figure}

\noindent\textbf{GQA-LT and VG8K-LT Compositional  Prediction.} The compositional prediction is the correct prediction of the subject, relation, and object together.
This could trigger a more skewed long-tail distribution due to its combinatorial nature. To evaluate our model's compositional behaviors, we follow the work \cite{ltvrd} to group the classification results by the pairs of (subject, object), (subject, relation), and (object, relation). The results are provided in Table \ref{compositional_result}, and we can see noticeable performance improvement on all the classes in contrast to the baselines. But we also observe that both RelTransformer(CE) and LSVRU (CE) perform better than the ones combining with focal loss and WCE in many categories, which differs from the results for only predicting the relations. The main reason is that those class imbalance losses hurt more ``head" performance on subject/object compared to CE (see supplementary) and they are reflected in the compositional prediction.

\begin{figure}
\centering
 \includegraphics[width=1.0\linewidth]
    {Figure/without_global_memory_crop.pdf}
\caption{Per-class accuracy comparisons between the RelTransformer and its version without global-context encoder and memory attention using CE loss from Table \ref{ablation}.
}
\label{gqalt_improvement}
\end{figure}

\begin{figure*}[t]
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {gqa_figure/figb1_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {gqa_figure/figb2_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {gqa_figure/figa1_crop.pdf}
    \title{ \hspace{20pt}  LSVRU  \hspace{70pt}  RelTransformer }
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {gqa_figure/figa2_crop.pdf}
    \title{  LSVRU (WCE)  \hspace{50pt}      RelTransformer (WCE) }
\end{minipage} \hfill
\caption{Long-tail relationship recognition visualization. We compare our model with LSVRU in the combination of CE and WCE loss functions. Yellow color denotes the relation, blue denotes the subject, and pink denotes the object.}
\label{demonstrations}
\end{figure*}

\begin{table}[t]
\centering
\small

\scalebox{0.9}{
\begin{tabular}{lcccccccc}
\multirow{2}{*}{Models} &\multirow{2}{*}{Losses} & many &  med &  few &  all \\
& &100 & 300 & 1,600 & 2,000\\
\toprule
\rowcolor{Gray}
Full model & CE & \textbf{26.8} & \textbf{18.6} & \textbf{15.0} & \textbf{16.1} \\
\xmark global & CE& 24.8 & 16.9 &13.9  & 14.8  \\
\xmark mem &CE&  26.0& 16.4& 14.0& 15.0 \\
\midrule
\rowcolor{Gray}
Full model & Focal Loss & \textbf{30.5} & \textbf{22.8}& \textbf{14.8} & \textbf{16.8} \\
\xmark global & Focal Loss&  28.9 & 19.8 & 13.3 & 15.1 \\
\xmark mem & Focal Loss& 30.1 &  20.8 & 14.2 & 15.9 \\
\midrule
\rowcolor{Gray}
Full model & DCPL & \textbf{37.3} & \textbf{27.6} & \textbf{16.5} & \textbf{19.2} \\
\xmark global & DCPL&  36.1 & 25.4 & 15.7& 18.2 \\
\xmark mem & DCPL& 37.0 &  26.8 & 16.1 & 18.8 \\
\midrule
\rowcolor{Gray}
Full model & WCE & \textbf{36.6} & \textbf{27.4} & \textbf{16.3} & \textbf{19.0} \\
\xmark global & WCE& 35.6 & 24.5& 15.2 & 17.6 \\
\xmark mem & WCE& 36.5 & 27.2 & 16.0& 18.7
\end{tabular}}
\caption{Ablation study of RelTransformer on VG8K-LT dataset. global and mem represent the global-context encoder and memory attention module, respectively. \xmark represents the removal operation. Our default setting is marked in \colorbox{Gray}{gray}.}
\label{ablation}
\end{table}

\begin{figure*}[h!]
\centering
\scalebox{0.7}{
\begin{minipage}{0.33\textwidth}
 \includegraphics[width=1.0\linewidth,height =1.0\linewidth]
    {vg200/image1.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height =1.0\linewidth]
   {vg200/image2.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height =1.0\linewidth]
    {vg200/image3.pdf}
\end{minipage} \hfill}
\caption{The visualization of relation recognition with our model on VG200 dataset}
\label{VG200_examples}
\end{figure*}

\subsection{Ablation Studies}
To quantify the contributions of each component to the whole model performance, we ablate and evaluate our RelTrasnformer in different versions on VG8K-LT dataset as shown in Table \ref{ablation}. We choose VG8K-LT instead of GQA-lT since it is more challenging and covers more classes.

\noindent\textbf{The role of global-context encoder.} To investigate the effect of the global-context encoder, we ablate our RelTransformer with the version without learning global context. The results in Table \ref{ablation} indicate that the performance will drop on all the categories if we exclude it. It brings the performance down by 1.45\% accuracy (acc) on many, 2.35\% acc on medium, and 1.13\% acc on few when we average all the combined loss functions' results. This analytic shows that incorporating global context can benefit all categories with gaining the most performance on medium classes. We also demonstrate its per-class analysis in Fig. \ref{gqalt_improvement}, from which we can observe that the performance drops in most classes.

\noindent\textbf{The role of memory attention.} The persistent memory vectors are aimed to augment the relation representation with more useful ``out-of-context" information, especially for the low-frequent relations. In Table \ref{ablation}, we evaluate the version without the memory attention, and we can observe that it brings a performance reduction for all the categories. It drops the performance by 0.4\% acc on many, 1.2\% acc on medium, and 0.58\% acc on few relations when we average the results from all the loss functions. The medium and few are more influenced than many relations, indicating that memory attention can benefit the infrequent relation classes more. We also show the per-class analysis with removing the memory in Fig. \ref{gqalt_improvement}, where we can find the performances for medium and few relations drop most.

\subsection{Qualitative Results}
We randomly sample $2$ images which include long-tail relationships from the testing dataset. We evaluate RelTransformer and LSVRU \cite{lsvru} in terms of CE and WCE loss functions and contrast them in Fig. \ref{demonstrations}. It can be observed that LSVRU predicts very trivial relationships such as ``behind", ``under," or ``to the left of". Those trivial relationships convey a vague and inaccurate understanding of the image content. Our RelTransformer instead can describe the relations more accurately. e.g., it predicts ``man sniffing wine" for the bottom image. ``sniffing" is a long-tail relation type and this prediction exactly matches the ground truth. We also randomly sample $3$ images from VG200 dataset and visualize our prediction results in Fig. \ref{VG200_examples}.

\section{Introduction}
\label{sec_intro}
\input{introduction}

\section{Background}
\input{relatedWork}

\section{Approach}

\input{method}

\section{Experiments}

\input{Experiment}

\section{Conclusion}
\input{conclusion}

{\small
\bibliographystyle{ieee_fullname}

\subsection{Problem Definition}
An image can be decomposed into a scene graph $G = (N, E)$, where each node $(n_i \in N)$ represents an object and each edge $(e_i  \in E)$ represents the spatial or semantic relationships between two interacting objects. We denote the visual relationship between a subject $n_s$ and an object $n_o$ as $r$. In the visual relationship recognition (VRR) task, the goal is to predict $r$ between the given $n_s$ and $n_o$.

\begin{equation}
y^{r} = f(b^s, b^o, b^r, I)
\label{problem_define}
\end{equation}
where $b^s$, $b^o$, and $b^r$ are the subject, object and relationship bounding boxes. $b^r$ is obtained by the minimum enclosing region of $b^s$ and $b^o$. $y^{r}$ is the relationship label. $I$ denotes the arbitrary information such as image raw RGB pixel features. $f$ is the inference model.

\subsection{RelTransformer Architecture}

RelTransformer mainly comprises two components, an $L$-layer global-context encoder and a $P$-layer relational encoder. The overall architecture can be seen in Fig. \ref{fig:main_model}. Given an image, we first extract the object and relation features via a faster R-CNN detection network \cite{ren2015faster}. Those features are grouped together as a tuple of triplets
$(\langle s_1 , r_1, o_1\rangle,\dots,\langle s_N, r_N, o_N\rangle)$
according to their spatial or semantic relationships provided in the dataset, where each $s_i$, $r_i$ and $o_i$ $\in$ $\mathbb{R}^{1\times D}$ and  $N$ is the number of triplets. We first feed all the triplets into a multi-layer global-context encoder to learn a scene-contextualized representation.
Then we concentrate on learning the target relation $r_i$ representation from the context of $\langle s_i , r_i, o_i \rangle$ in the relational encoder. An external memory module is also introduced here to augment the long-tail relation representations. Finally, we employ a meshed connection (see Fig. \ref{fig:main_model}) to integrate global-context features into the relation representation learning.

\noindent \textbf{Global-Context Encoder.} We employ an $L$-layer Transformer \cite{vaswani2017attention} to model the global-context information and learn their pairwise relationships.
We first concatenate each triplet from $(\langle s_1 , r_1, o_1\rangle,\dots,\langle s_N, r_N, o_N\rangle)$ together into a compact representation as $X^\text{sro} = (x^\text{sro}_1 \dots x^\text{sro}_{N}$) where  $X^\text{sro} \in \mathbb{R}^{N \times 3D}$. We then feed $X^\text{sro}$ into the global-context encoder in a permutation-invariant order.

The Transformer \cite{vaswani2017attention} is a stack of multi-headed self-attention (MSA) and MLP layers. Its core component is the self-attention as defined in Eq. \ref{eq:self-att}. In each Transformer encoding layer, the multi-head self-attention repeats the self-attention multiple times and concatenate the results together; the result is then projected back to the same dimensionality. After that, the result is fed into an MLP network and produces each layer's output.

\begin{equation}
f_\text{sa}(Q, K, V) =\text{softmax}\left( \frac{(W_{q}Q) (W_{k}K)^{\top}}{\sqrt{d}}\right) W_{v}V
\label{eq:self-att}
\end{equation}

\noindent where $Q$, $K$ and $V \in \mathbb{R}^{t \times h}$ are query, key and value vectors. $t$ is the number of input tokens and $h$ is the hidden size. $d$ is a scaling factor. $W_q$, $W_k$ and $W_v$ are the learnable weight parameters.

We feed $X^\text{sro}$ into the global-context encoder, each layer $l$ outputs a contextualized representation $z_l$. We gather them together as $Z$ = ($z_{1}, \dots, z_{L}$) where $z_{l} \in \mathbb{R}^{N \times h}$.

\noindent \textbf{Relational Encoder.} It has multiple functions: 1) it specifically aggregates the relation representation from its referred subject and object via self-attention 2) it incorporates a persistent memory to augment the relations with ``out-of-context" information, which could especially benefit the long-tail relations. 3) it also has a ``bridging" role to aggregate each global-context encoding layer's output into the relation representation through a meshed attention module. The detailed operating procedure is described as follows.

We first add a learnable positional embedding \cite{bert} to each position of $s_i$, $r_i$, and $o_i$ in order to distinguish their semantic differences in the input sequence, and we denote this sequence as $X_{p-1}$ in the layer $p$. We apply the self-attention defined in Eq. \ref{eq:self-att} on $X_{p-1}$ in each layer $p$ and model their pairwise relationship. Their result will be $X^{att}_p$ as shown in Eq. \ref{sf1}.
\begin{equation}
        X^{\text{att}}_p = f_\text{sa}(X_{p-1},  X_{p-1}, X_{p-1})
        \label{sf1}
\end{equation}

\noindent \textbf{Memory Augmentation.} The trained model can easily forget long-tail relations because the model training is dominated by the instance-rich (or head) relations, and hence it tends to underperform on lower-frequent relations. Also, the self-attention is limited to attending to the features only from the tokens in an input sequence; hence, each relation only learns a representation based on the immediate context. To alleviate this issue, we propose a novel memory attention module motivated by several successful persistent memory ideas \cite{persistentMemory,zhao2020speech} in the literature. We denote a group of persistent and differentiable memory vectors as $M$. Each time the relation passes its features to $M$ and retrieves the information from $M$ via attention. The memory here captures the information not dependent on the immediate context; instead, it is shared across the whole dataset \cite{persistentMemory}. Through this way, long-tail relations are able to access the information (e.g., from other relations or itself in different training steps) with relevance to itself, and they can augment the target relation with the useful ``out-of-context" information in a well-trained model.

To compute this memory, we first randomly initialize $m$ memory vectors in each relational encoding layer $p$ as $M_p \in \mathbb{R}^{m \times h}$. We then compute the memory attention between the input feature with $M_p$, we treat $X_{p-1}$ as the query, and $M$ here is the Key and Value. Same self-attention operation is applied here in Eq. \ref{sf2} and we can obtain $X^{\text{mem}}_p$. The memory is directly updated via SGD.
\begin{equation}
        X^\text{mem}_p = f_\text{sa}( X_{p-1},M_{p-1} ,M_{p-1})
        \label{sf2}
\end{equation}

To aggregate $X^{\text{mem}}_p$ into $X_p^\text{att}$, we design a fusion function as $g(x,y)$ in Eq. \ref{fusion}. $g(x,y)$ is a attention gate and determines how to effectively combine two input features. It computes the complementary attention weights to each input and weighted combine them as the output. Through this fusion function in combination with a skip connection, we can get the fused feature as $\bar{X}_p$ = $g(X_p^\text{att}, X_p^{\text{mem}})$ + $X_{p-1}$.
\begin{equation}
\begin{split}
   g(x,y) &=  \alpha \odot x  + (J-\alpha) \odot y\\
   \alpha &= \sigma( W[x ; y] + b)
\end{split}
\label{fusion}
\end{equation}
 where $W$ is a 2D $\times$ D matrix. $b$ is a bias term. $[;]$ denotes the concatenation. $\odot$ is the Hadamard product. $J$ is an all-one matrix with the same dimensions as $\alpha$.

\noindent \textbf{Meshed Attention Fusion.} The features from different global-context encoding layers capture different vision granularity, and leveraging features from all of them has shown to be better than the one only from the last encoding layer~\cite{m2transformer,visualgpt}. Therefore, we adopt a meshed connection in our model and contribute each layer's output $z_l$ to the relation representation. To compute the meshed attention, we first compute the cross-attention between $\bar{X}_p$ and each global-context encoding output in ($z_1,\dots,z_L$); Its attention output is fused with $\bar{X}_p$ through Eq. \ref{fusion}. Their results are averaged up for each layer. We then project the averaged output in an MLP network and incorporate a skip connection to compute the final fused relation representation $X_p$ in Eq. \ref{crossfusion} as the layer output.

\begin{equation}
\begin{split}
     z_p^l  =& f_\text{sa}(\bar{X}_p, z_l, z_l)\\
   {X}_p =&  \text{MLP}(\frac{1}{L} \sum_{l=1}^L g(\bar X_p, z_p^l))+\bar{X}_p
\end{split}
\label{crossfusion}
\end{equation}

\begin{table*}[t!]
\centering
\scalebox{0.86}{
\begin{tabular}{ ll| cccc|cccc}
 \multicolumn{2}{l}{} & \multicolumn{4}{c}{VG8K-LT} & \multicolumn{4}{c}{GQA-LT} \\

 \multirow{2}{*}{Architecture} & \multirow{2}{*}{Learning Methods} & many &  medium &  few &  all &  many &  medium &  few &  all \\
  &  & 100 &  300 &  1,600 & 2,000&  16 &  46 &  248 &  310 \\
\toprule

LSVRU & VilHub \cite{ltvrd} & 27.5 & 17.4 & 14.6 & 15.7 & \underline{$\text{63.6}$} & 17.6 & 7.2 & 11.7\\
LSVRU & VilHub + RelMix \cite{ltvrd}  & 24.5 &16.5 &14.4& 15.4 & 63.4 &14.9& 8.0 &11.9 \\

LSVRU & OLTR \cite{oltr} & 22.5 & 15.6 & 12.6 & 13.6 & 63.5 & 15.0 & 8.2 & 12.1 \\
LSVRU & EQL \cite{eql} & 22.6 & 15.6 & 12.6 & 13.6 & 62.3 & 15.8 & 6.6 & 10.8 \\

LSVRU & $\text{Counterfactual }^{\sharp}$ \cite{tang2020unbiased} & 12.1 & 25.6 & 14.9 & 17.1 & 38.6 & 38.0 & 9.4 & 15.2\\

\midrule
 LSVRU& CE &22.2 & 15.5 & 12.6 & 13.5 & 62.6 & 15.5 & 6.8 & 11.0\\
\rowcolor{Gray2}
 RelTransformer (ours)& CE & \textbf{$\text{26.8}$} &  \textbf{$\text{18.6}$}   & \textbf{15.0} & \textbf{16.1} & \textbf{63.4} & \textbf{16.6} & \textbf{7.0} & \textbf{11.2} \\
\midrule

LSVRU & Focal Loss \cite{focalLoss} & 24.5 & 16.2 & 13.7 & 14.7 & 60.4 & 15.7 & 7.7 & 11.6 \\
\rowcolor{Gray2}
RelTransformer (ours) & Focal Loss & \textbf{30.5} & \textbf{22.8} & \textbf{14.8} & \textbf{16.8} & \textbf{61.9} & \textbf{16.8} & \textbf{8.3} & \textbf{12.2}\\
\midrule
LSVRU & DCPL \cite{decoupling}& 34.3 & 15.4 & 12.9 & 14.4& \textbf{61.4} & 23.6 & 7.6 &12.7 \\
\rowcolor{Gray2}
RelTransformer (ours) & DCPL  & \textbf{\underline{37.3}} & \underline{\textbf{27.6}}& \underline{\textbf{16.5}} & \underline{\textbf{19.2}}& 58.4& \textbf{38.6}& \textbf{13.2}&  \textbf{19.3} \\
\midrule
LSVRU  &  WCE &  35.5 & 24.7 & 15.2 & 17.2 & 53.4 & 35.1 & 15.7 & 20.5 \\
\rowcolor{Gray2}
RelTransformer (ours) & WCE &  \textbf{36.6}  & \textbf{$\text{27.4}$}& \textbf{$\text{16.3}$} & \textbf{$\text{19.0}$}&    \underline{$\textbf{63.6}$} & \underline{\textbf{$\text{59.1}$}} & \underline{\textbf{$\text{43.1}$}} &\underline{\textbf{$\text{46.5}$}}

\end{tabular}
}
\newline
\caption{Average per-class accuracy in relation prediction on VG8K-LT and GQA-LT datasets. We evaluate the average per-class accuracy for many, medium, few, and all classes. The best performance for each column is \underline{underlined}. $\sharp$ denotes our reproduction. Learning methods include various class-imbalance loss functions, augmentation and counterfactual approaches. Our model is marked in \colorbox{Gray2}{gray}.}
\label{main_results}
\end{table*}

\noindent \textbf{Final Triplet Prediction.} In the last layer $P$ of the relational encoder, we extract subject $x_s$, relation $x_r$ and object $x_o$ from $X_P$ accordingly. In the prediction stage, we leverage the language prior knowledge following previous work \cite{lsvru}, and represent each ground-truth label in their Word2Vec \cite{word2vec} embeddings, we project them into hidden representations with a 2-layer MLP in the classifier. Finally, we maximize their cosine similarity with $x_s$, $x_r$, and $x_o$ individually during the training.

\section{Training and Implementation Details}
To have fair comparisons with all the baseline models, we follow the same experimental setup as \cite{ltvrd} for GQA-LT and VG8K-LT evaluation and also the same setup as \cite{lsvru} for VG200 evaluation. We train our model and all its ablations with 8 V100 GPUs. The batch size is 8. We train our models with 12, 8, and 7 epochs on GQA-LT, VG8K-LT, and VG200 datasets, respectively. The hidden size of $h$ is 768. The number of Transformer heads is 12. Relational and global-context encoders both have 2 layers. The memory size is 100 $\times$ 768. We use the Faster R-CNN \cite{ren2015faster} with VGG-16 \cite{vgg} backbone to extract the object proposal features. We also apply the pretrained Word2Vec \cite{word2vec} embeddings to represent the relation and object labels.
\begin{figure}[h!]
\centering
 \includegraphics[width=0.8\linewidth]{Figure/memory_attention_analysis_crop.pdf}
 \caption{Average memory attention scores per each relation class on evaluation of GQA-LT. The relation classes are ranked by its frequency (from high to low)}
\label{mem_attention}
\end{figure}

\section{Subject and Object Per-Class Accuracy}
We also provide the per-class accuracy for subjects and objects in Table \ref{sub_obj_results}. We can observe that our RelTransformer can also easily outperform many baselines. Combining RelTransformer with CE loss can outperform all the baseline models on each category and it significantly improves over the ``many" category. Combining it with WCE can furthermore improve over ``medium" and ``tail" categories on both datasets. These results demonstrate the effectiveness of our model on the subject and object prediction. We also notice that RelTransformer (WCE or DCPL) drops the performance over ``many" categories compared to CE loss function; We hypothesis that it is due to the assigned lower weights on the high-frequent subjects/objects, and lower weights will lead to the lower confidence values during the classification. This phenomenon can explain why RelTansformer (WCE) underperforms on the ``many" classes and why it underperforms on the compositional prediction for ``many" and ``medium" classes.

\begin{table*}[h]
\centering
\begin{tabular}{ ll| cccc|cccc}
 \multicolumn{2}{l}{} & \multicolumn{4}{c}{VG8K-LT} & \multicolumn{4}{c}{GQA-LT} \\
 \multirow{2}{*}{Architecture} & \multirow{2}{*}{Learning Methods} & many &  medium &  few &  all &  many &  medium &  few &  all \\
  &   &  267 & 799 &  4,264 &  5,330 &  86 &  255 &  1,362 & 1,703 \\

\toprule
LSVRU & VilHub \cite{ltvrd} & 61.6 & 20.3 & 10.1 & 14.2 & 68.6 & 44.0 & 10.3 & 18.3 \\
LSVRU & VilHub + RelMix \cite{ltvrd}  & 59.5 &15.1 & 10.4& 13.6 & 68.8 &42.1& 10.1 & 18.1 \\
LSVRU & OLTR \cite{oltr} & 56.8 & 12.0 & 9.6 & 12.3 & 68.2 & 37.2 & 7.0 & 14.6 \\
LSVRU & EQL \cite{eql} & 56.9 & 12.1 & 10.0 & 12.7 & 68.9 & 43.7 & 10.0 & 18.0\\
LSVRU & $\text{Counterfactual} ^{\sharp}$\cite{tang2020unbiased} & 57.3 & 11.1 & 8.5 & 11.4 & 68.3 & 37.0 & 6.9 & 14.5\\
\midrule
LSVRU & CE & 57.3 & 11.1 & 8.5 & 11.4 & 68.3 & 37.0 & 6.9 & 14.5\\
\rowcolor{Gray2}
RelTransformer & CE & \textbf{\underline{67.1}} & \textbf{25.9} & \textbf{11.5} & \textbf{16.5} &\underline{\textbf{78.0}} & \underline{\textbf{56.6}} & \textbf{14.2} & \textbf{23.8} \\
\midrule
LSVRU & Focal Loss \cite{focalLoss} & 58.1 & 13.9 & 8.9 & 12.1 & 68.2 & 39.2 & 7.5 & 15.3 \\ \rowcolor{Gray2}
RelTransformer & Focal Loss & \textbf{65.6} & \textbf{21.7} & \textbf{10.8} & \textbf{15.2} & \textbf{75.0} & \textbf{51.4} & \textbf{11.9} & \textbf{21.0}\\
\midrule
LSVRU & DCPL\cite{decoupling} & \textbf{53.8} & 5.9 & 7.9 & 9.9 & \textbf{64.0} & 35.3 & 6.4 & 13.7\\
\rowcolor{Gray2}
RelTransformer & DCPL & 50.3 & \textbf{30.9} & \textbf{13.4}& \textbf{17.8} & 51.8 & \textbf{44.6} & \textbf{19.2} & \textbf{24.7}\\
\midrule
LSVRU & WCE& \textbf{52.8} & 27.2 & 10.8 & 14.5 & \textbf{53.4} & 42.0 & 14.0 & 20.2 \\
\rowcolor{Gray2}
RelTransformer& WCE & 50.1& \underline{\textbf{31.3}}& \underline{\textbf{13.7}} &\underline{\textbf{18.0}} & 50.3& \textbf{46.2}&\underline{\textbf{28.7}}& \underline{\textbf{32.4}} \\
\end{tabular}
\caption{Average per-class accuracy for subject/object. We separately evaluate the average  per-class  accuracy for many, medium, few, and all categories. The best performance from each category is underlined. $\sharp$ denotes our reproduction. Our model is denoted in \colorbox{Gray2}{gray}}
\label{sub_obj_results}
\end{table*}

\begin{table*}[h]
\centering
\begin{tabular}{ ll| cccc|cccc}

 \multicolumn{2}{l}{} & \multicolumn{2}{c}{Per-example Accuracy} \\
 Architecture & Learning Methods & Subject/Object & Relation \\
\toprule
LSVRU & CE & 51.9 & 94.8\\
LSVRU & VilHub \cite{ltvrd} &53.9 & 91.2 \\
LSVRU & VilHub + RelMix \cite{ltvrd}  & 53.5 &91.0  \\
LSVRU & EQL \cite{eql} & 51.1 & 93.9\\
LSVRU & WCE& 37.6 & 72.6  \\
\rowcolor{Gray2}
RelTransformer & CE & \textbf{\underline{62.5}} & \textbf{\underline{95.1}} \\
\rowcolor{Gray2}
RelTransformer & Focal Loss \cite{focalLoss} & 59.4 & 95.0\\
\rowcolor{Gray2}
RelTransformer & DCPL \cite{decoupling} &34.3 & 80.5 \\
\rowcolor{Gray2}
RelTransformer& WCE & 34.2& 74.2 \\
\end{tabular}
\caption{Per-example Accuracy on GQA-LT. The best performance from each category is underlined. Our model is denoted in \colorbox{Gray2}{gray}}
\label{per_example_acc}
\end{table*}

\section{Per-Example Accuracy}
We provide the per-example accuracy for the subjects and objects on GQA-LT dataset. It shows that RelTransformer (CE) achieves the best performance on predicting subjects/objects and relations among all the baselines. Per-example accuracy is mainly dominated by the ``head" classes since they are very large in example numbers. This indicates that RelTransformer (CE) improves both head and tail classes. Compared to the best baseline LSVRU (CE), RelTransformer (CE) improves it by 10.6 acc on subject/object and 0.3 acc on relation predictions. However, we could also notice that RelTransformer (WCE or DCPL) brings the performance down on per-example accuracy, and this phenomenon is also observed in LSVRU baselines. But the results in Table \ref{sub_obj_results}(Supplementary) and Table 1 (main paper) shows that RelTransformer (WCE or DCPL) has a good-performing result on ``medium" and ``tail" categories. This indicates that combining with class-imbalance loss functions can benefit low-frequent class predictions with the cost of the performance from a few number of top-frequent classes; RelTransformer improves the results more general.

\section{Memory Attention Further Analysis}

To further investigate the role of memory attention, we compute the memory attention scores, $J-\alpha$ in our fusion function Eq. \ref{fusion}, for each testing example from well-trained RelTransformer (WCE) on the GQA-LT dataset. We average the attention scores per each relation-class and demonstrate them in Fig. \ref{mem_attention}. The relation classes are ranked according to their frequency in the training data. We can observe a clear increasing trend for the attention scores from high-frequent relations to low-frequent ones, meaning that the features from memory can contribute to the long-tail relations more than the head-relations, which can reflect why memory can gain more improvement on the ``medium" and ``tail" classes.
\begin{equation}
\begin{split}
   g(x,y) &=  \alpha \odot x  + (J-\alpha) \odot y\\
   \alpha &= \sigma( W[x ; y] + b)
\end{split}
\label{fusion}
\end{equation}
 where $W$ is a 2D $\times$ D matrix. $b$ is a bias term. $[;]$ denotes the concatenation. $\odot$ is the Hadamard product. $J$ is an all-one matrix with the same dimensions as $\alpha$.

\section{Additional Qualitative Examples}
We show the relation prediction results on VG200 dataset in Fig. \ref{VG200_examples} and provide more long-tail relation prediction results on GQA-LT and VG8K-LT dataset in Fig. \ref{gqalt_examples} and \ref{vg8k_examples}.
\begin{figure*}[h!]

\centering
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
    {vg200/visualization1.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
   {vg200/visualization2.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
    {vg200/visualization3.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
   {vg200/visualization4.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
    {vg200/visualization5.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
   {vg200/visualization6.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
    {vg200/visualization7.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
   {vg200/visualization8.pdf}
\end{minipage} \hfill
\begin{minipage}{0.33\textwidth}
\centering
 \includegraphics[width=1.0\linewidth,height=1.0\columnwidth]
    {vg200/visualization9.pdf}
\end{minipage} \hfill
\caption{
Qualitative results on VG200 dataset}
\label{VG200_examples}

\end{figure*}

\begin{figure*}[h!]

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_1a_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_1b_crop.pdf}
\end{minipage} \hfill

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_2a_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_2b_crop.pdf}
\end{minipage} \hfill

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_3a_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_3b_crop.pdf}
\end{minipage} \hfill

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_4a_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_4b_crop.pdf}
\end{minipage} \hfill

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_5a_crop.pdf}
    \title{ \hspace{20pt}  LSVRU  \hspace{70pt}  RelTransformer }
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_5b_crop.pdf}
    \title{  LSVRU (WCE)  \hspace{50pt}      RelTransformer (WCE) }
\end{minipage} \hfill

\caption{
More long-tail relation prediction examples on GQA-LT dataset}
\label{gqalt_examples}

\end{figure*}

\begin{figure*}

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_6a_crop.pdf}

\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_6b_crop.pdf}

\end{minipage} \hfill

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_7a_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_7b_crop.pdf}
\end{minipage} \hfill

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_8a_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_8b_crop.pdf}
\end{minipage} \hfill

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_9a_crop.pdf}
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_9b_crop.pdf}
\end{minipage} \hfill

\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_10a_crop.pdf}
    \title{ \hspace{20pt}  LSVRU  \hspace{70pt}  RelTransformer }
\end{minipage} \hfill
\begin{minipage}{0.5\textwidth}
\centering
 \includegraphics[width=1.0\linewidth]
    {vg8k_figure/fig_10b_crop.pdf}
    \title{  LSVRU (WCE)  \hspace{50pt}      RelTransformer (WCE) }
\end{minipage} \hfill

\caption{
More long-tail relation prediction examples on VG8K-LT dataset}
\label{vg8k_examples}

\end{figure*}

\newpage
\clearpage
\clearpage

{\small
\bibliographystyle{ieee_fullname}


\subsection{Multi-agent trajectory modelling}

Our problem description closely follows \citet{alcorn2021baller2vec}, whom we quote here:

\begin{quote}
Let $A = \{1, 2, \dots, B\}$ be a set indexing $B$ agents and $P = \{p_{1}, p_{2}, \dots, p_{K}\} \subset A$ be the $K$ agents involved in a particular sequence.
[Let] $C_{t} = \{(x_{t,1}, y_{t,1}), (x_{t,2}, y_{t,2}), \dots, (x_{t,K}, y_{t,K})\}$ [be] an unordered set of $K$ coordinate pairs such that $(x_{t,k}, y_{t,k})$ are the coordinates for agent $p_{k}$ at time step $t$.
The ordered sequence of sets of coordinates $\mathcal{C} = (C_{1}, C_{2}, \dots, C_{T})$, together with $P$, thus defines the trajectories for the $K$ agents over $T$ time steps.
\end{quote}

In multi-agent trajectory modeling, the goal is to model a joint probability of the form:

\[
     p(\Delta x_{t,1}, \Delta x_{t, 2}, \dots, \Delta x_{t, K} | x_{1:t,1}, x_{1:t,2}, \dots, x_{1:t,K})
\]

\noindent
i.e., the joint probability of the $K$ agents' trajectories $\Delta x_{t,k}$ at time step $t$ given the agents' location histories $x_{1:t,k}$.
We note here that the common practice of \textit{simultaneously} predicting the trajectories for all of the agents at a specific time step is not required by theory.
Using the chain rule of probability, the joint probability of the agents' trajectories can be factorized as, e.g.:

\[
    p(\Delta x_{t,1}, \Delta x_{t, 2}, \dots, \Delta x_{t, K}) = p(\Delta x_{t,1}) p(\Delta x_{t, 2} | \Delta x_{t,1}) \dots p(\Delta x_{t, K} | \Delta x_{t,1}, \Delta x_{t, 2}, \dots, \Delta x_{t, K - 1})
\]

where we omit the conditional historical trajectories for brevity.
As a result, it is perfectly acceptable to generate trajectories agent-wise, using the previously generated trajectories as additional conditioning information when generating the trajectories for later agents (see Figure \ref{fig:legal_dep}).

\begin{figure}[ht]
\centering
\subfigure{\includegraphics[width=\columnwidth]{legal}}
\caption{
At inference time, a model is not \textit{required} to \textit{simultaneously} generate the trajectories for all of the agents at a specific time step.
An alternative strategy is to allow the model to generate the agents' trajectories one at a time, and let the model use the previously generated trajectories to inform the trajectories it generates for the remaining agents.
}
\label{fig:legal_dep}
\end{figure}

\subsection{\texttt{baller2vec}{} is a (conditional) generative model.}\label{sec:baller2vec_generative}

\texttt{baller2vec}{} is a recently described \textit{multi-entity} Transformer that can model sequences of \textit{sets} (the underlying data structure for multi-agent spatiotemporal systems), as opposed to sequences of individual inputs (like words in a sentence).
When used to model the game of basketball, the input at each time step for \texttt{baller2vec}{} is a set of feature vectors where each feature vector contains information about the identity and location of a player on the court.
\texttt{baller2vec}{} maps each input feature vector to an output feature vector, which is then used to ``classify'' the binned trajectory for that specific player at that specific time step.

Here, we provide a probabilistic interpretation of \texttt{baller2vec}{}, which establishes the theoretical grounds for using the chain rule to generate trajectories agent-wise at each time step in \texttt{baller2vec++}{}.
Without loss of generality, we only consider one-dimensional trajectories for a single agent here.
To briefly summarize, the outputs of the softmax function over the $n$ binned trajectories in \texttt{baller2vec}{} can be interpreted as mixture proportions for a mixture of uniform distributions with predetermined bounds that partition the Euclidean trajectory space.
Further, because $x_{t + 1} = x_{t} + \Delta x_{t}$, \texttt{baller2vec}{} is in fact a conditional generative model that assigns a probability to a sequence of trajectories given the initial position of the agent, i.e., $p(\Delta x_{1}, \Delta x_{2}, \dots, \Delta x_{T} | x_{1})$.
Using the chain rule, we decompose the joint probability of the trajectories as:

\[
    p(\Delta x_{1}, \Delta x_{2}, \dots, \Delta x_{T}) = p(\Delta x_{1}) p(\Delta x_{2} | \Delta x_{1}) \dots p(\Delta x_{T} | \Delta x_{1}, \Delta x_{2}, \dots, \Delta x_{T - 1})
\]

\noindent
to reflect their temporal structure (we omit the conditional initial position term for brevity).
Therefore, new trajectories can be generated from \texttt{baller2vec}{} with the following procedure (see Figure \ref{fig:graphical_model}):

\begin{enumerate}
    \item First, sample one of the $n$ different mixture components using the mixture proportions output from the classifier $f$ (i.e., \texttt{baller2vec}{}) conditioned on the agent's current position, i.e., $i \sim \text{Categorical}(\pi_{1}, \pi_{2}, \dots, \pi_{n})$ where $[\pi_{1}, \pi_{2}, \dots, \pi_{n}] = f(x_{t})$.
    \item Next, sample a trajectory from the uniform distribution associated with the sampled component, i.e., $\Delta x_{t} \sim \mathcal{U}(a_{i}, b_{i})$.
    \item Finally, add the sampled trajectory to the agent's input position to generate the agent's position at the start of the next time step, i.e., $x_{t+1} = x_{t} + \Delta x_{t}$.
\end{enumerate}

\begin{wrapfigure}{r}{0.5\linewidth}
\vskip -0.2in
\includegraphics[width=\linewidth]{graphical_model_tight}
\caption{
\texttt{baller2vec}{} can be viewed as a conditional generative model that assigns a probability to a sequence of trajectories given the initial positions of the agents.
Here, we show a graphical model depiction of a \texttt{baller2vec}{} model that generates a sequence of one-dimensional trajectories for a single agent.
Given the initial position of the agent (the circle containing $x_{1}$), one of $n$ different uniform distributions (the square containing $i_{1}$) is sampled using the mixture proportions ($\pi_{i}$) output by \texttt{baller2vec}{} ($f$).
The agent's trajectory (the diamond containing $\Delta x_{1}$) is then sampled from the selected uniform distribution, which has bounds $-\infty < a_{i} < b_{i} < \infty$.
At the start of the next time step, the agent's position is $x_{2} = x_{1} + \Delta x_{1}$.
Maximizing the likelihood of \texttt{baller2vec}{} as a classifier over the binned trajectories is thus equivalent to maximizing its likelihood when assuming the trajectories are generated from a mixture of uniform distributions that partition the Euclidean trajectory space (see Section \ref{sec:baller2vec_generative} for details).
}
\label{fig:graphical_model}
\end{wrapfigure}
Let $[\Delta x_{min}, \Delta x_{max})$ be an interval on the real line such that any trajectory $\Delta x < \Delta x_{min}$ or $\Delta x \geq \Delta x_{max}$ has zero density (i.e., such trajectories are humanly impossible).
Let $\{[a_{i}, b_{i})\}_{i=1}^{n}$ be a set of $n$ intervals that partition the interval $[\Delta x_{min}, \Delta x_{max})$ into $n$ bins, i.e., $\cup_{i=1}^{n} [a_{i}, b_{i}) = [\Delta x_{min}, \Delta x_{max})$ and $i \neq j \implies [a_{i}, b_{i}) \cap [a_{j}, b_{j}) = \emptyset$.
Recall that the probability density function (PDF) for a uniform distribution with bounds $-\infty < a < b < \infty$ is:

\[
    p(\Delta x) =
    \begin{cases}
        \frac{1}{b - a} & \text{for } \Delta x \in [a,b)  \\
        0               & \text{otherwise}
    \end{cases}
\]

\noindent
Letting $c_{i} = \frac{1}{b_{i} - a_{i}}$, the PDF for a mixture of uniforms with these bounds is thus:

\begin{equation}\label{eq:mix_uniforms}
    p(\Delta x) = \sum_{i=1}^{n} \pi_{i} \mathcal{U}(\Delta x; a_{i}, b_{i}) = \sum_{i=1}^{n} \pi_{i} c_{i}
\end{equation}

\noindent
where $p(\Delta x)$ is the density assigned to $\Delta x$ by the mixture, $\pi_{i}$ is the mixture proportion for the mixture component indexed by $i$ (i.e., $0 \leq \pi_{i} \leq 1$ and $\sum \pi_{i} = 1$), and $\mathcal{U}(\Delta x; a_{i}, b_{i})$ is the density assigned to $\Delta x$ by the uniform distribution with bounds $-\infty < a_{i} < b_{i} < \infty$.
Because the bounds of the uniform distributions partition $[\Delta x_{min}, \Delta x_{max})$, Equation \eqref{eq:mix_uniforms} reduces to:

\[
    p(\Delta x) = \pi_{i'} c_{i'}
\]

\noindent
where $\Delta x \in [a_{i'}, b_{i'})$ (because the other uniform distributions will assign a density of zero to $\Delta x$).
The likelihood for data $D$ (with $|D| = N$) is then:

\[
    \mathcal{L}(D) = \prod_{j=1}^{N} p(\Delta x_{j}) = \prod_{j=1}^{N} \pi_{j,i'} c_{j,i'}
\]

\noindent
where $\pi_{j,i'}$ is the mixture proportion assigned to the component with  $\Delta x_{j} \in [a_{i'}, b_{i'})$ and $c_{j,i'}$ is the associated density.
Taking the negative logarithm of the likelihood gives:

\begin{equation}\label{eq:mixture_nll}
    -\ln(\mathcal{L}(D)) = -\sum_{j=1}^{N} \ln(\pi_{j,i'}) -\sum_{j=1}^{N} \ln(c_{j,i'})
\end{equation}

\noindent
Because the bounds are fixed, the second summation is a constant, and Equation \eqref{eq:mixture_nll} becomes:

\[
    -\ln(\mathcal{L}(D)) = -\sum_{j=1}^{N} \ln(\pi_{j,i'}) + C
\]

\noindent
where $C = -\sum_{j=1}^{N} \ln(c_{j,i'})$.
Therefore, minimizing the loss of \texttt{baller2vec}{} as a classifier of binned trajectories is equivalent to minimizing the loss of the model when assuming the trajectories are generated from a mixture of uniform distributions as specified in Equation \eqref{eq:mix_uniforms}.

\subsection{Toy dataset}
Each training sample was initialized with the agents starting at $(-1, 0)$ and $(1, 0)$ on a grid in random order (i.e., the first agent could be placed to either the left or the right of the origin).
At each time step, one of nine actions (corresponding to the 3$\times$3 grid surrounding the agent) was sampled from a uniform distribution, and each of the agents was translated along this trajectory.
This process was repeated for 20 time steps (see Figure \ref{fig:toy}(a) for a sample).

\subsection{Basketball dataset}

We used the same National Basketball Association (NBA) dataset\footnote{\url{https://github.com/linouk23/NBA-Player-Movements}} employed by \citet{alcorn2021baller2vec}, whom we paraphrase here:

\begin{quote}
    The NBA dataset consists of trajectories from 631 games from the 2015-2016 season, which were split into 569/30/32 training/validation/test games, respectively.
    During training, each sequence was sampled using the following procedure: (1) randomly select a training game, (2) randomly select a starting time from the game, (3) take the following four seconds of data and downsample it to 5 Hz from the original 25 Hz, and then (4) randomly (with a probability of 0.5) rotate the court 180\degree.
    This sampling procedure gave us access to on the order of $\sim$82 million different (albeit overlapping) training sequences.
    For both the validation and test sets, $\sim$1,000 different, \textit{non-overlapping} sequences were selected for evaluation by dividing each game into $\lceil \frac{1,000}{N} \rceil$ non-overlapping chunks (where $N$ is the number of games), and using the starting four seconds from each chunk as the evaluation sequence.
\end{quote}

\subsection{Model}\label{sec:model}

Our \texttt{baller2vec++}{} and \texttt{baller2vec}{} models for the basketball dataset closely followed \cite{alcorn2021baller2vec}, and so largely resemble the original Transformer architecture \cite{vaswani2017attention}.
Specifically, the Transformer settings were: $d_{\text{model}} = 512$ (the dimension of the input and output of each Transformer layer), eight attention heads, $d_{\textrm{ff}} = 2048$ (the dimension of the inner feedforward layers), six layers, no dropout, and no positional encoding.
Each MLP (i.e., $g_{z}$, $g_{u}$, and $g_{r}$) had 128, 256, and 512 nodes in its three layers, respectively, and a ReLU nonlinearity following each of the first two layers.
The player embeddings \cite{alcorn20182vec} had 20 dimensions, and $h_{t,k}$ was a binary variable indicating the side of the frontcourt for player $p_{k}$ (i.e., the direction of his team's hoop) at time step $t$.
Lastly, the $11 \text{ ft} \times 11 \text{ ft}$ 2D Euclidean trajectory space was binned into 121 $1 \text{ ft} \times 1 \text{ ft}$ squares.

We used the Adam optimizer \cite{kingma2014adam} with an initial learning rate of $10^{-6}$, $\beta_{1} = 0.9$, $\beta_{2} = 0.999$, and $\epsilon = 10^{-9}$ to update the model parameters, of which there were $\sim$19 million.
The learning rate was reduced to $10^{-7}$ after 20 epochs of the validation loss not improving.
Models were implemented in PyTorch and trained on a single NVIDIA GTX 1080 Ti GPU for $\sim$650 epochs (seven days) where each epoch consisted of 20,000 training samples, and the validation set was used for early stopping.

For the toy dataset, we used scaled down versions of the basketball models with $d_{\text{model}} = 128$, four attention heads, $d_{\textrm{ff}} = 512$, and two layers in the Transformer.
Additionally, each MLP had two layers with 64 and 128 nodes, respectively.
The models were trained for 50 epochs of 500 samples per epoch ($\sim$10.5 minutes) using a single learning rate of $10^{-5}$.

\subsection{\texttt{baller2vec++}{} can effectively model coordinated agents in both simulated and real settings}\label{sec:effective}

For the toy dataset, the training loss for \texttt{baller2vec}{} converged to $\sim$2.2 $\approx -\ln(\frac{1}{9})$, i.e., the model was simply independently guessing the trajectories for both agents at every time step.
In contrast, the training loss for \texttt{baller2vec++}{} converged to $\sim$1.1 $\approx -\ln(\frac{1}{9}) \div 2$, which is the expected loss for a model that perfectly learns the deterministic relationship between the agents' trajectories (because the prediction for the second agent will always contribute $-\ln(1.0) = 0$ to the loss).
\begin{wrapfigure}{r}{0.7\linewidth}
\centering
\subfigure[Training sample.]{\includegraphics[width=0.2\columnwidth]{train_toy}}
\hskip 0.2cm
\subfigure[\texttt{baller2vec}{}.]{\includegraphics[width=0.2\columnwidth]{b2v_toy}}
\hskip 0.2cm
\subfigure[\texttt{baller2vec++}{}.]{\includegraphics[width=0.2\columnwidth]{b2v++_toy}}
\caption{When trained on a dataset of perfectly coordinated agent trajectories (a),  the trajectories generated by \texttt{baller2vec}{} are completely \textit{uncoordinated} (b) while the trajectories generated by \texttt{baller2vec++}{} are perfectly coordinated (c).
Animated versions can be found in the code repository.
}
\vskip -0.1in
\label{fig:toy}
\end{wrapfigure}
When generating trajectories with \texttt{baller2vec}{}, the agents are completely uncoordinated, with each agent following an independent random walk around the grid (Figure \ref{fig:toy}(b)).
In contrast, trajectories generated by \texttt{baller2vec++}{} display the same coordinated agent behavior as the training data (Figure \ref{fig:toy}(c)).

For the basketball dataset, \texttt{baller2vec++}{} achieved an average NLL of 0.472 on the test set, 8.9\% better than the average NLL for \texttt{baller2vec}{} (0.518) (see Figure \ref{fig:basketball} for trajectories generated by \texttt{baller2vec++}{} and \texttt{baller2vec}{}).
As was observed in \cite{alcorn2021baller2vec}, the trajectory bin distributions for \texttt{baller2vec}{} become much more certain after observing a portion of the sequence (Figure \ref{fig:plots}), which suggests \texttt{baller2vec}{} may be inferring some of the shared hidden variables (e.g., plays) influencing the players.
If that hypothesis was true, the performance gap between \texttt{baller2vec++}{} and \texttt{baller2vec}{} should be largest at the beginning of the sequence (before any shared hidden variables can be inferred by \texttt{baller2vec}{}).
Indeed, the average NLL for \texttt{baller2vec++}{} in the \textit{first time step} of each test set sequence (1.567) is 16.1\% better than the average NLL for \texttt{baller2vec}{} (1.869), while the average NLL for \texttt{baller2vec++}{} in the \textit{last time step} of each test set sequence (0.420) is only 9.7\% better than the average NLL for \texttt{baller2vec}{} (0.465) (see Figure \ref{fig:plots}).

\begin{figure}[h]
\centering
\subfigure{\includegraphics[width=\columnwidth]{plots}}
\caption{\textbf{Left}: when modeling the trajectories of professional basketball players, the performance gap between \texttt{baller2vec++}{} and \texttt{baller2vec}{} is largest at the beginning of the sequence, before shared unobserved variables can be inferred by \texttt{baller2vec}{}.
Each bar indicates a model's average NLL over the entire test set for that particular time step.
For full sequences, \texttt{baller2vec++}{} outperforms \texttt{baller2vec}{} by 8.9\%.
\textbf{Right}: the joint probability assigned to a sequence by \texttt{baller2vec++}{} is approximately permutation invariant with respect to the order of the agents.
For each point, its $x$ value indicates \texttt{baller2vec++}{}'s average NLL for a test set sequence using the \textit{original} order of the agents in the sequence, while its $y$ value indicates \texttt{baller2vec++}{}'s average NLL \textit{for the same sequence} with the order of the agents \textit{shuffled}.
The shuffled average NLLs are highly correlated with their corresponding unshuffled average NLLs.
}
\vskip -0.2in
\label{fig:plots}
\end{figure}

\subsection{\texttt{baller2vec++}{} makes better predictions when conditioned on concurrent trajectory information from other agents}\label{sec:conditioning}

Implicit in much of our discussion has been the intuition that providing a model with additional (relevant) information will improve its performance.
To empirically test this conjecture, we compared the performance of \texttt{baller2vec++}{} when predicting the trajectory of a specific basketball player placed in the \textit{first position} of the player order (i.e., when $k = 1$) vs. predicting the trajectory for that \textit{same player} placed in the \textit{last position} (i.e., when $k = 10$).
Specifically, for each player in each test sequence, we calculated the NLL of the player's trajectory in the first time step\footnote{Because, as previously discussed, the benefits of \texttt{baller2vec++}{} were most pronounced in the first time step.} with the player in the \textit{first position} of the player order.
Next, we moved the player to the \textit{last position} of the player order, and then randomly shuffled the remaining nine players 10 times, calculating the NLL for the player in the last position each time.
Finally, we calculated the average percent change in the last position NLLs relative to their corresponding first position NLLs.
On average, moving a player from the first to the last position improved the NLL for the player's trajectory by 14.6\%.

\subsection{The joint probability assigned to a sequence by \texttt{baller2vec++}{} is approximately permutation invariant with respect to the order of the agents}\label{sec:permutation}

To determine whether or not \texttt{baller2vec++}{} respects the fact that any ordering of a chain rule decomposition of a joint probability produces the same value, we measured how much the average NLL for each test sequence in the basketball dataset varied when the order of the agents changed.
Specifically, for each test set sequence, we shuffled the order of the agents 10 times.
Then, for each permuted sequence, we calculated the percent error\footnote{See: \url{https://en.wikipedia.org/wiki/Approximation_error\#Formal_Definition}.} in the average NLL relative to the original, \textit{unshuffled} sequence.
Across all test sequences, the average percent error was only $\pm1.5$\%.
Further, as can be seen in Figure \ref{fig:plots}, the shuffled average NLLs are highly correlated with their corresponding unshuffled average NLLs (Pearson correlation coefficient = 0.997), i.e., the joint probability assigned to a sequence by \texttt{baller2vec++}{} is approximately permutation invariant with respect to the order of the agents.

\section{Introduction and Related Work}
\input{Introduction}

\section{Background}\label{sec:background}
\input{Background}

\section{Model Architecture}\label{sec:baller2vec++}
\input{Architecture}

\section{Experiments}
\input{Experiments}

\section{Results}
\input{Results}

\section{Conclusion and Future Work}\label{sec:conclusion}
\input{Conclusion}

\section{Author Contributions}
MAA conceived and implemented the architecture, designed and ran the experiments, and wrote the manuscript.
AN partially funded MAA, provided the GPUs for the experiments, and commented on the manuscript.

\section{Acknowledgements}
We would like to thank Jan Van Haaren for his helpful suggestions on how to improve the manuscript.


\section{Introduction}

At present the old problem of boundary regularity of proper or biholomorphic mappings between strictly pseudoconvex domains  in $\mathbb C^n$ is almost completely clarified due to the  contribution of several authors. Theorem of Ch. Fefferman \cite{F}  asserts that a biholomorphic mapping betweeen strictly pseudoconvex domains with boundaries of class $C^\infty$ extends as a $C^\infty$ diffeomorphism between their closures. The original  proof is based on the study of asymptotic behavior of the Bergman kernel near the boundary and is difficult to adapt to the case of finite smoothness. Later several authors developed different approaches.  One of them is due to L. NIrenberg, S.Webster, P. Yang \cite{NWY} and uses a smooth version of the reflection principle. Pushing further their approach,  S.Pinchuk and S.Khasanov \cite{PiKh} proved that a proper holomorphic mapping between strictly pseudoconvex domains with $C^s$ boundaries, with $s > 2$  real, extends to the boundary as a mapping of class $C^{s-1}$ if $s$ is not an integer, and of class $C^{s-1-\varepsilon}$, with any $\varepsilon > 0$, when $s$ is an integer. Similar results are obtained by L. Lempert \cite{Le1,Le2} by quite different techniques of extremal discs for the Kobayashi metric. Y. Khurumov \cite{Kh} proved that an even a better  result still remains true   with the loss of regularity on $1/2$.
However, a natural question on the precise regularity in the case where  the boundaries of class exactly $C^2$ (i.e. the minimal possible regularity of boundaries), remains open. Y.Khurumov announced without further details that his result remains true also in this case, but, to the best of my knowledge, a detailed  proof is not available. The well-known result  in this case was obtained independently by  G.Henkin \cite{Khen} and  S.Pinchuk \cite{Pi1}) and states that a mapping extends to the boundary as a H\"older $1/2$-continuous mapping.

Our  main result is the following

\begin{thm}
\label{MainTheo1}
 Let $f:\Omega_1 \to \Omega_2$ be a proper holomorphic mapping between  bounded strictly pseudoconvex domains  $\Omega_j \subset \mathbb C^n$, $j = 1,2$,  with   boundaries  $b\Omega_j$ of class $C^2$. Then $f$ extends to a mapping of the H\"older class $C^{\alpha}(\overline{\Omega}_1)$
for each $\alpha \in [0,1[$.
\end{thm}
In particular, this result means that the above mentioned theorem of Lempert-Pinchuk-Khasanov still remains true for boundaries of class $C^2$. This result is obtained in \cite{Su1} under the additional assumption that the boundary $b\Omega_1$ is of class $C^{2 + \varepsilon}$, $\varepsilon > 0$. From this point of view the present paper is a continuation and the complement of \cite{Su1}. The proof consists of two main parts. The first one is based on estimates of the Kobayashi metric in a tube
neighborhood of a totally real manifold (see \cite{ChSu}). These estimates allow to obtain uniform  H\"older estimates on analytic discs glued to a prescribed totally real manifold which in turn allows to obtain the H\"older regularity of holomorphic mappings between wedge-type domains with totally real edges.
This argument is presented in details in \cite{Su1}. The present paper is devoted to the second main step of the proof. We study geometric properties of analytic discs attached  along an arc to a totally real manifold of class $C^1$. This  {\it gluing disc argument}, which  is often quite useful for the study of totally real submanifolds,  was introduced in \cite{Pi2}.  In the case where the regularity of a totally real manifold is higher than $C^1$, this construction was developped by several authors. The $C^1$ case was  considered by E.Chirka \cite{KhenCh} and Y.Khurumov \cite{Kh2}. However, for our goals we need some additional properties of such discs which are not explicitely stated there. This is the reason why  I present some details for completeness of exposition. I stress that I do not claim any originality here. Only applications of this construction are new.

\section{Terminology and notations}

We briefly recall some well-known definitions and basic notations.

 Let $\Omega$ be a domain in $\mathbb C^n$. For a positive integer $k$, denote by  $C^k(\Omega)$  the space of $C^k$-smooth complex-valued functions in~$\Omega$. Also $C^k(\overline\Omega)$ denotes the class of functions whose partial derivatives up to order $k$ extend as continuous functions on
$\overline\Omega$. Let  $s > 0$ be a real noninteger and let $k$ be its integer part. Then  $C^s(\Omega)$ denotes the space of functions of class $C^k(\overline\Omega)$ such that their partial derivatives of order $k$ are (globally)  $(s-k)$-H\"older continuous in $\Omega$; these derivatives  automatically satisfy the  $(s-k)$-H\"older condition on
$\overline\Omega$ so the notation $C^s(\overline\Omega)$ for the same space of functions is also appropriate. We also use the space $L^p(\Omega)$ of Lebesgue $p$-integrable functions with the usual norm
$\parallel f \parallel_{L^p(\Omega)}$; if $f = (f_1,...,f_m)$ is a vector valued function, we set $\parallel f \parallel_{L^p(\Omega)} = \sum_{j=1}^m \parallel f_j \parallel_{L^p(\Omega)}$.
We also denote by $W^{k,p}(\Omega)$  (resp. $W^{k,p}_{loc}(\Omega)$) the Sobolev spaces of (vector) functions with generalized derivatives of order up to $k$ which are Lebesgue $p$-integrable (resp. locally) in $\Omega$.

Denote by $\mathbb D = \{ \zeta \in \mathbb C: \vert \zeta \vert < 1 \}$ the unit disc of $\mathbb C$. Recall that in this special case the Sobolev embedding theorem
asserts that for $\alpha = 1 - 2/p$ and $p > 2$ the natural inclusion  $W^{1,p}(\mathbb D) \to C^{\alpha}(\mathbb D)$  is a linear bounded compact operator.

A (closed) real submanifold $E$ of a domain $\Omega \subset \mathbb C^n$ is of class $C^s$ (with real $s  \ge 1$) if for every point $p \in E$ there exists an open neighbourhood $U$ of $p$ and a map $\rho: U \longrightarrow \mathbb R^d$ of the maximal rank $d<2n$ and of class $C^s$  such that $E \cap U = \rho^{-1}(0)$; then $\rho$ is called a local defining (vector-valued ) function of $E$. The positive integer $d$ is the real codimension of $E$. In the most important special case $d=1$ we obtain the class of real hypersurfaces.

Let $J$ denotes the standard complex structure of $\mathbb C^n$. In other words, $J$ acts on a vector $v$ by multiplication by $i$ that is $J v = i v$. For every $p \in E$ the {\it holomorphic tangent space}  $H_pE:= T_pE \cap J(T_pE)$ is the maximal complex subspace of the tangent space $T_pE$ of $E$ at $p$. Clearly $H_pE =   \{ v \in \mathbb C^n: \partial \rho(p) v = 0 \}$. The complex dimension of $H_pE$ is called the CR dimension of $E$ at $p$; a manifold $E$ is called a {\it CR  (Cauchy-Riemann) manifold} if its CR dimension is independent of $p \in E$.

A real submanifold $E \subset \Omega$ is called {\it generic} (or generating) if the complex span of $T_pE$ coincides with $\mathbb C^n$ for all $p \in E$. Note that every generic manifold of real codimension $d$ is a CR manifold of CR dimension $n-d$. A function $\rho = (\rho_1,...,\rho_d)$ defines a generic manifold if $\partial\rho_1 \wedge ...\wedge \rho_d \neq 0$.
Of special importance are the so-called {\it totally real manifolds}, i.e., submanifolds $E$ for which $H_pE = \{ 0 \}$ at every $p \in E$. A totally real manifold in $\mathbb C^n$  is generic if and only if its real dimension is equal to $n$; this is the maximal possible value for the dimension of a totally real manifold.

 Let $\Omega$ be a bounded domain in~$\mathbb C^n$. Suppose that its boundary $b\Omega$ is a (compact) real hypersurface of class $C^s$ in $\mathbb C^n$. Then there exists a $C^s$-smooth real
function $\rho$ in a neighbourhood $U$ of the closure $\overline\Omega$ such that $\Omega = \{ \rho < 0 \}$ and
$d\rho|_{b\Omega} \ne 0$. We call such a function $\rho$ a global defining function. If $s \ge 2$ one may consider
{\it the Levi form} of $\rho$:
\begin{eqnarray}
\label{Leviform}L(\rho,p,v) = \sum_{j,k = 1}^n \frac{\partial^2\rho}{\partial z_j\partial\overline{z}_k}(p)v_j \overline v_k.
\end{eqnarray}
A bounded domain $\Omega$ with $C^2$ boundary is called  {\it strictly pseudoconvex}
if $L(\rho,p,v) > 0$  for every nonzero  vector $v \in H_p(b\Omega)$.

Consider a wedge-type domain

\begin{eqnarray}
\label{wedge}
W = \{ z \in \mathbb C^n: \phi_j(z) < 0, j = 1,...,n \}
\end{eqnarray}
with the edge

\begin{eqnarray}
\label{edge}
E = \{ z \in \mathbb C^n: \phi_j(z) = 0, j = 1,...,n \}
\end{eqnarray}
We assume that the defining functions $\phi_j$ are of class $C^{1}$. Furthermore, as usual we suppose that $E$ is a generic manifold that is
$\partial \phi_1 \wedge ... \wedge \partial \phi_n \neq 0$ in a neighborhood of $E$.

Given $\delta > 0$ (which is supposed to be small enough) we also define a shrinked wedge
\begin{eqnarray}
\label{shrinked}
W_{\delta} = \{ z \in \mathbb C^n: \phi_j - \delta \sum_{l \neq j} \phi_l < 0, j= 1,...,n \} \subset W
\end{eqnarray}
It has the same edge $E$. Note that there exists a constant $C > 0$ such that for every point $z \in W_\delta$ one has
\begin{eqnarray}
\label{dist}
C^{-1} dist(z, b W) \le dist(z, E) \le C dist (z,b W)
\end{eqnarray}

\section{Gluing complex discs to $C^1$ totally real manifolds}
In this section we present the main technical tool for the proof of Theorem \ref{MainTheo1}.
Consider a wedge-type domain (\ref{wedge})  with the edge (\ref{edge}).

We need the well-known construction of filling a wedge $W$ (or, more precisely, the wedge $W_\delta$) by complex discs gluing to $E$ along an open arc.

A complex (or analytic, or holomorphic) disc is a holomorphic map $h: \mathbb D \to \mathbb C^n$ which is at least continous on the closed disc $\overline\mathbb D$.
We say that such a disc is glued (or attached) to a subset $K$ of $\mathbb C^n$ along an (open, nonempty) arc $\gamma \subset b\mathbb D)$, if $f(\gamma) \subset K$.

Our presentation consists of   several steps.

\subsection{The generalized Bishop equation, the existence  and the regularity of discs}

Let $E$ be an $n$-dimensional totally real manifold of class $C^1$ in a neighborhood of $0$ in $\mathbb C^n$; we assume $0 \in E$. After a linear change of coordinates, using the implicit function theorem  we also may assume that in a neighbourhood $\Omega$ of the origin the manifold $E$ is defined by the (vector) equation

\begin{eqnarray}
\label{edge2}
y = h(x)
\end{eqnarray}
 where a  vector function $h = (h_1,...,h_n)$ of class $C^1$ in a neighborhood of $0$ in $\mathbb R^n$ and satisfies the conditions

 \begin{eqnarray}
 \label{edge3}
 h_j(0) = 0, \,\,\,\nabla h_j(0) = 0 , \, \, j = 1,...,n.
\end{eqnarray}
Here and below $\nabla$ denotes the gradient.

\bigskip

Fix a positive noninteger $s$. Consider  the Hilbert transform $T: u \to Tu$, associating to  a real function $u \in C^s(b\mathbb D)$  its harmonic conjugate function vanishing at the origin. In orther words, $u + iTu$ is a trace on $b\mathbb D$ of a function, holomorphic on $\mathbb D$ and of class $C^s(\mathbb D)$, and $Tu(0) = 0$.

Recall that explicitely the Hilbert transform  is given  by

\begin{eqnarray*}
Tu (e^{i\theta}) = \frac{1}{2\pi} v. p. \int_{-\pi}^{\pi} u(e^{it}) \cot \left ( \frac{\theta - t}{2} \right ) dt
\end{eqnarray*}

 This is a classical linear singular integral operator; it is  bounded on the space $C^s(b\mathbb D)$ for any non-integer $s > 0$. Furthermore, for $p > 1$ the operator $T:L^p(b\mathbb D) \to \L^p(b\mathbb D)$ is a bounded linear operator as well; we denote by $\parallel T \parallel_p$ its norm.

 Let $b\mathbb D^+ = \{ e^{i\theta}: \theta \in [0,\pi] \}$ and $b\mathbb D^- = \{ e^{i\theta}: \theta \in ]\pi, 2 \pi[ \}$ denote the upper and the lower semicircles respectively. Fix a $C^\infty$-smooth real functions $\psi_j$ on $b\mathbb D$ such that $\psi_j\vert b\mathbb D^+ = 0$ and  $\psi_j\vert b\mathbb D^- < 0$, $j=1,...,n$ (one may take the same function independent of $j$). Set $\psi = (\psi_1,...,\psi_n)$. Consider the {\it generalized Bishop equation}
\begin{eqnarray}
\label{BishopEq}
u(\zeta) =  -Th(u(\zeta)) - tT\psi(\zeta) + c, \,\, \zeta \in b\mathbb D ,
\end{eqnarray}
where $c \in \mathbb R^n$ and $t = (t_1,...,t_n) \in \mathbb R^n$, $t_j \ge 0$, are real parameters. We will prove that for any $p > 2$, and for any $c$, $t$ close enough to the origin, this singular integral equation admits a unique solution $u(c,t)(\zeta)$ in the  Sobolev class $W^{1,p}(b\mathbb D)$. Such a solution is  of class $C^{\alpha}(b\mathbb D)$, $\alpha = 1 - 2/p$,  by the Sobolev embedding.

\bigskip

Before proceed the proof, we explain how such a solution is related to complex discs glued to $E$ along $b\mathbb D^+$.
Indeed, consider the function $$U(c,t)\zeta) = u(c,t)(\zeta) + i h(u(c,t)(\zeta)) + i t\psi(\zeta).$$
Since $T^2 = -Id$ and $u$ is a solution of (\ref{BishopEq}), the function $U$ extends holomorphically on $\mathbb D$ as a function
\begin{eqnarray}
\label{Discs}
H(c,t)(\zeta) = P U(c,t), \,\,\zeta \in \mathbb D
\end{eqnarray}
 of class $C^\alpha(\mathbb D)$.  Here $P$ denotes the Poisson operator of harmonic extension to $\mathbb D$:

\begin{eqnarray}
\label{Poisson}
PU(c,t)(\zeta) =  \frac{1}{2\pi} \int_{-\pi}^{\pi} \frac{1- \vert \zeta \vert^2}{\vert e^{it} - \zeta \vert^2} U(c,t)(e^{it})dt
\end{eqnarray}

The function  $\psi$ vanishes on $b\mathbb D^+$, so by (\ref{edge2}) we have $H(c,t)(b\mathbb D^+) \subset E$ for all $(c,t,d)$.

\bigskip

It is convenient to  extend the equation (\ref{BishopEq}) on the whole space. Fix a $C^\infty$ smooth
function $\lambda: \mathbb R^{n} \to \mathbb R^+$ equal to $1$ on the unit ball $\mathbb B^{n}$ and vanishing on
$\mathbb R^{n} \setminus 2\mathbb B^{n}$. For $\delta > 0$ small enough  the function
$h_\delta(x) = \lambda(x/\delta)h(x)$  naturally extends (by $0$) on the whole $\mathbb R^{n}$.
Fix $\tau > 0$ small enough (it will be choosen later). Then in view of (\ref{edge3}) we can choose $\delta = \delta(\tau) > 0$
such that the gradient $\nabla h_\delta(x)$ is small on the whole $\mathbb R^{n}$:

\begin{eqnarray}
\label{edge4}
\parallel \nabla h_\delta \parallel_{L^\infty(\mathbb R^{n})} \le \tau
\end{eqnarray}

First we study the global equation

\begin{eqnarray}
\label{BishopEq2}
u(\zeta) =    -Th_\delta(u(\zeta)) - tT\psi(\zeta) + c, \,\, \zeta \in b\mathbb D ,
\end{eqnarray}
We prove that its solutions depend continuously on parametrs $(c,t)$; this allows to localize  the  solutions and  to conclude with the
initial equation (\ref{BishopEq}).

\bigskip

Let $V$ be a domain in $\mathbb R^m$ and $f \in L^p(V \times b\mathbb D)$. Then by the Fubini theorem $Tf \in
L^p(V \times b\mathbb D)$ (the variables in $V$ are viewed as parameters when the operator $T$ acts). Hence, keeping the same notation, we obtain
a bounded linear operator $T: L^p(V \times b\mathbb D) \to L^p(V \times b\mathbb D)$  with the same norm as in $L^p(b\mathbb D)$.
We again denote its norm by $\parallel T \parallel_p$.

Fix a domain   $V\subset \mathbb R^{2n}$  of the parameters $(c,t)$.

\begin{lemma}
\label{DiscExistence}
Under the above assumptions, for any $p > 1$,  one can choose $\tau > 0$ in (\ref{edge4}), and $\delta = \delta(\tau) > 0$,  such that   the equation (\ref{BishopEq2}) admits a unique solution $u(c,t)(\zeta) \in L^p(V  \times b\mathbb D)$.
\end{lemma}
\proof  Consider the operator $\Phi: L^p(V \times  b\mathbb D) \to L^p(V \times  b\mathbb D)$ defined by
$$\Phi: u \mapsto  -Th_\delta(u(\zeta)) - tT\psi(\zeta) + c.$$ In view of (\ref{edge4}) it follows by the finite differences
theorem that for all $u^1$, $u^2$ from $L^p(V \times  b\mathbb D)$ one has
\begin{eqnarray*}
\parallel \Phi(u^1) - \Phi^2(u^2) \parallel_{L^p(V  \times b\mathbb D)} \le \parallel T \parallel_p \parallel h_\delta(u^1) - h_\delta(u^2) \parallel_{L^p(V  \times b\mathbb D)} \le (1/2) \parallel u^1 - u^2 \parallel_{L^p(V \times  b\mathbb D)}
\end{eqnarray*}
when $\tau$ in (\ref{edge4}) is fixed small enough.  Hence  $\Phi$ is a contracting map and lemma is proved.
\bigskip

Since $V$ is arbitrary we conclude that the equation (\ref{BishopEq2}) admits a unique solution $u \in L^p_{loc}(\mathbb R^{2n}  \times b\mathbb D)$. By this
space we mean the space of $L^p$ functions on $K \times b\mathbb D$ for each (Lebesgue) measurable compact subset $K \subset \mathbb R^{2n}$.

\bigskip

Next we study the regularity of solutions of (\ref{BishopEq2}) in the Sobolev scale.  Let $\Omega$ be a domain in $\mathbb R^k$ and let  $f $ be a function in $\Omega$.
Denote by $e_j$, $j=1,...,k$ the canonical basis of $\mathbb R^k$.
Given $j = 1,...,k$  and $\Delta x_j \in \mathbb R^*$ consider the finte differences
$$\Delta f / \Delta x_j = (f(x + e_j \Delta x_j) - f(x))/\Delta x_j.$$
Recall two well-known properties of the Sobolev spaces (see, for example, \cite{Ni}).

\begin{itemize}
\item[(1)] If a function $f$ is in  $W^{1,p}_{loc}(\Omega)$, then for every subdomain $\tilde\Omega \subset \Omega$, the finite differences $\Delta f/\Delta x_j$ converge
in $L^p(\tilde \Omega)$ as $\Delta x_j \to 0$ to the generalized derivative $\partial f/\partial x_j$.
\item[(2)] $f \in W^{1,p}_{loc}(\Omega)$ if and only if the finite differences $\Delta f/\Delta x_j$ are bounded
in $L^p(\tilde \Omega)$ uniformly in $\Delta x_j$.
\end{itemize}

\begin{lemma}
\label{DiscRegularity}
Every solution $u$ of (\ref{BishopEq2}) is of class $W^{1,p}_{loc}(\mathbb R^{2n} \times b\mathbb D)$.
\end{lemma}
\proof  Let $x_j$ denote one of the variables $c_j$, $t_j$  or $\zeta \in b\mathbb D$. We estimate the finite difference $\Delta u / \Delta x_j$. It follows from
(\ref{edge4}) that $h_\delta$ satisfies the Lispchitz condition  with the Lipschitz constant $\tau$. Hence from (\ref{BishopEq2}) we have:
$$\parallel\Delta u/ \Delta x_j \parallel_{L^p(V \times  b\mathbb D)}\le C_1 \tau \parallel \Delta u/\Delta/x_j\parallel_{L^p(V \times b\mathbb D)} + C_2$$
where $C_j > 0$ are constants. When $\tau > 0$ (and so $\delta > 0$) are small enough, we obtain
$$\parallel\Delta u/\Delta x_j\parallel_{L^p(V \times  b\mathbb D)} \le C_3$$
for some constant $C_3 > 0$, and therefore $u \in W^{1,p}_{loc}(\mathbb R^{2n} \times b\mathbb D)$. Lemma is proved.
\bigskip

It follows by the Sobolev embedding that a solution $u$ belongs to $C^{1-(2n+1)/p}(V\times b\mathbb D)$, where $V$ is an open subset in $\mathbb R^{2n}$.
In particular, the constructed family of discs
is continuous in all variables for $p$ big enough. Now we note that for $t = 0$ the equation (\ref{BishopEq2}) admits a constant solution $u(c,0)(\zeta) = c$. When $c$ is close enough to
the origin in $\mathbb R^n$, this solution gives a point $c + ih(c) \in E$. By continuity and uniqueness of solutions, there exists a neighborhood $V$ of the origin in $\mathbb R^{2n}$, such that for  $(c,t) \in V$ any solution of (\ref{BishopEq2}) is a solution of (\ref{BishopEq}). We have proved the following

\begin{lemma}
\label{BishopLemma1}
Given $p > 2$ the exists a neighborhood $V$ of the origin in $\mathbb R^{2n} $ such that the Bishop equation (\ref{BishopEq}) admits a unique solution
$u(c,t)(\zeta) \in W^{1,p}(V \times  b\mathbb D)$.
\end{lemma}
Since $p$ is arbitrary, we obtain that our equation admits solutions in the Holder class $C^{\alpha}(V \times  b\mathbb D)$ with $\alpha = 1-(2n+1)/p$. Note that here $V$ depends on $p$ (and hence, on $\alpha$). Nevertheless, it follows from \cite{ChSu} that for each  $(c,t)$ fixed, the map $\zeta \mapsto u(c,t)(\zeta)$ is of class $C^\alpha(b\mathbb D)$ for every $\alpha < 1$.

\subsection{Stability of discs} Until now we did not study any geometric properties of the family (\ref{Discs}). Here we consider some of them which will be useful for our applications.

We represent the family (\ref{Discs}) as a small perturbation in the $W^{1,p}$ norm of some model family.
The model case arises when $E = \mathbb R^n$ that is $h = 0$ in (\ref{edge}). Then the equation (\ref{BishopEq}) takes the from

\begin{eqnarray}
\label{BishopEq7}
u(\zeta) =   - tT\psi(\zeta) + c, \,\, \zeta \in b\mathbb D ,
\end{eqnarray}
where as usual $c \in \mathbb R^n$ and $t = (t_1,...,t_n)$, $t_j \ge 0$, are real parameters. This equation is already solved with respect to $u$; one can view (\ref{BishopEq7}) as its general solution. In this case the family (\ref{Discs}) becomes

\begin{eqnarray}
\label{Discs2}
H(c,t)(\zeta) = P U(c,t), \,\,\zeta \in \mathbb D
\end{eqnarray}
where

\begin{eqnarray}
\label{FlatDiscs}
U(c,t)\zeta) = - tT\psi(\zeta) + c +  i t\psi(\zeta).
\end{eqnarray}
This family arises  from the family of complex lines intersecting $\mathbb R^n$ along
real lines; it is simply obtained by a biholomorphic reparametrization of arising half-lines by the unit disc. These lines are given by $l(c,t): \zeta \mapsto t\zeta + c$, $\zeta \in \mathbb C$. The conformal map $-T \psi + i\psi$ takes the unit disc into a smoothly bounded
domain in the lower half-plane, gluing $b\mathbb D^+$ to the real axes. In what follows  we refer this case as the flat case.

When $h$ is of class $C^s$ with $s > 1$,  it follows easily by the implicit function theorem that the general family (\ref{Discs}) is obtained from the flat family (\ref{FlatDiscs}) by a small $C^s$-perturbation and hence has the similar geometric properties. The $C^1$ case requires some additional technical analysis.

 Let  $E$ be a totally real manifold given by (\ref{edge2}), (\ref{edge3}).  Given $d \in I \setminus \{ 0 \}$, where $I \ni 0$ is an open interval  small  enough, consider the manifolds $E_d$ given by

\begin{eqnarray}
\label{edge.d}
y = d^{-1}h(d x)
\end{eqnarray}
Note that for every $d  \neq 0$ the manifold $E_d$ is biholomorphic to $E$ via the isotropic dilation $z \mapsto  d^{-1}z$. Set $h(x,d) = d^{-1}h(dx)$.
Thus, we consider the $1$-parameter family $E_d$ of totally real manifolds defined by the equation
\begin{eqnarray}
\label{edge2.d}
y = h(x,d)
\end{eqnarray}

 \begin{eqnarray}
 \label{edge3.d}
 h_j(0,d) = 0, \,\,\,  \nabla_x h_j(0,d) = 0,  \, \, \ d \in I,  \, \, j = 1,...,n.
 \end{eqnarray}
(we consider the gradient $\nabla_x$ with respect to $x$).
 Hence for each $(c,t,d)$ we have the discs
$H(c,t,d)$  defined by (\ref{Discs}). If $d = 0$, we have $E_0 = \{ y = 0 \} = \mathbb R^n = T_0(E)$ that is, the flat case. By the uniqueness of the solution of the Bishop equation,
the family $H(c,t,0)( \zeta)$ coincides with the family ( \ref{FlatDiscs}).

 \begin{lemma}
  \label{LemDeformation}
 For any $p > 1$  one has
  $$ \parallel H(c,t,d)( \zeta) - H(c,t,0)( \zeta) \parallel_{W^{1,p}(V  \times  \mathbb D)}  \to 0$$
  as $d  \to 0$.
   \end{lemma}
 \proof Let $u^0$ be a solution of the flat Bishop equation ( \ref{BishopEq7}). For any $d$ let $u(c,t,d)$ be a solution of the Bishop equation
  \begin{eqnarray}
  \label{BishopEq8}
 u = -T h(u,d) - tT \psi + c
  \end{eqnarray}
 It follows from (\ref{edge3.d}) that previous estimates on the norm of $u$ are uniform in $d$ that is $\parallel u \parallel_{W^{1,p}(V \times \mathbb D)} \le C$
 where a constant $C > 0$ is independent of $d$. Indeed, in the above estimates $\tau > 0$ and $\delta(\tau) > 0$ may be choosen ondependent of $d$.
 Since $u \in W^{1,p}(V \times \mathbb D)$ for any $d$ and $h(\bullet,d)$ is of class $C^1(\mathbb R^n)$ for every $d$, the composition $h(u,d)$ is of class $W^{1,p}(V \times \mathbb D$ as well and the chain rule can be applied for generalized derivatives: $h(u)_{x_j} = Dh(u) u_{x_j}$.

  Therefore we obtain

 \begin{eqnarray*}
 \label{BishopEq9}
  \parallel  u - u^0  \parallel_{W^{1,p}(V \times \mathbb D)}  =  \parallel Th(u,d) \parallel_{W^{1,p}(V \times \mathbb D)}  \le    C   \parallel h(\bullet,d)  \parallel_{C^1_x}
  \end{eqnarray*}
 where   $\parallel h(\bullet,d) \parallel_{C^1_x}$ denotes the $C^1$ norm of $h(x,d)$  with respect to the variable $x$, and $C > 0$ is a constant.
 But $\parallel h(\bullet,d)  \parallel_{C^1_x}  \to 0$ as $d  \to 0$,  and Lemma follows.

 \section{Proof of Theorem  \ref{MainTheo1}}

The following proposition is the key technical result.

\begin{prop}
\label{MainProp}
Let $M$ be an $m$-dimensional totally real manifold of class $C^1$ in  $\mathbb C^m$. Assume that $\Omega \subset \mathbb C^n$ is a pseudoconvex domain defined by $\Omega = \{ \phi < 0 \}$ where $\phi$ is a plurisubharmonic function of class $C^2$. and $d \phi \neq 0$ near $\Gamma = b\Omega$. Let also $W \subset \Omega$
 be a wedge (\ref{wedge})  with the edge $E \subset \Gamma$ of type (\ref{edge}).  Consider a holomorphic map   $f:  \Omega \to \mathbb C^m$ such that $f$ is continuos on $\Omega \cup E$ and $f(E) \subset M$.  Then,  for every $\delta > 0$,  and every $\alpha < 1$, the map $f$ extends to a H\"older
$\alpha$-continuous mapping on $W_{\delta} \cup E$.
\end{prop}
\proof Consider the family of discs constructed in the former section and attached to $E$ along $b\mathbb D^+$. It follows by the maximum principle that all discs belong to $\Omega$.
Indeed, the flat discs fill a prescribed wedge of type (\ref{wedge}) with the edge $E_0 =  \mathbb R^n$.  More precisely, we can fix an open convex  cone $K$ in
$W^0 = \{ (x,y)  \in  \mathbb R^{2n}: y_j < 0, j = 1,...,n  \}$ with the vertex at the origin and such that $ \overline K \cap r\mathbb B^n$ is contained in $W^0   \cup  \{ 0  \}$, for some $r > 0$ small
enough. Clearly, the flat discs fill a neighborhood of $ \overline K  \cap r  \mathbb B^n$.  The same remains true for the cone $K_z$ obtained by the parallel translation of $K$ to the vertex at $z  \in  \mathbb R^n$.
  Since the family $H(c,t,d)(\zeta)$ is a small perturbation of the flat discs in  $C^s(V \times \overline \mathbb D)$ (with any $0 < s < 1$), by continuity for $d $ small enough the family $H(c,t,d)$ also fills a presribed edge of type (\ref{shrinked}) with the edge $E_d$. By the holomorphic equivalence, the same is true for the initial edge $E_\delta$ with any $\delta > 0$.

 Applying the strong version of the Hopf lemma (see \cite{PiTs}) to the subharmonic function $\phi \circ H$ on $\mathbb D$, we
obtain that $$\vert \phi \circ H(c,t)(\zeta) \vert \ge C (1 - \vert \zeta \vert)$$ with $C > 0$ independent of discs (i.e. on $(c,t)$) . Recall that
$$C^{-1} dist (z, \Gamma) \le \vert \phi(z) \vert \le C dist(z,\Gamma)$$
Since $E \subset \Gamma$, we have
$$dist(z, \Gamma) \le dist(z,E)$$

This implies the estimate

\begin{eqnarray}
\label{dist}
 1- \vert \zeta \vert \le C dist(H(c,t)(\zeta),E)
\end{eqnarray}

Recall that $M$ can be defined by  $M = \rho^{-1}(0)$, where $\rho$ is a nonegative strictly psh function of class $C^2$ (see \cite{Ch2,HW}).
Now we apply the previous argument to each disc $f(z)$ with $z = H(c,t)(\zeta)$ and obtain

$$\rho(f(z))  \le C(1- \vert \zeta \vert) \le C dist(z,E)$$
Note that here the left estimate  is obtained in  \cite{Su1}.

Hence we obtain the key estimate

\begin{eqnarray}
\rho(f(z)) \le C dist(z,E)
\end{eqnarray}
for all $z \in W_\delta$.

With this estimate the argument  from \cite{Su1} (based on estimates of the Kobayashi metric in a tube neighborhood of $M$) literally goes through and gives that the mapping $f$ is
 $\alpha$-H\"older on $W_\delta$ for each $\alpha < 1$. This proves Proposition.

\bigskip
\bigskip

Now Theorem \ref{MainTheo1} follows exactly as in \cite{Su1}.

{\footnotesize