text:
\section{Introduction}
\input{1_intro}
\section{Related Work}
\input{2_related}
\section{Methods}
\input{3_methods}
\section{Experiments}
\input{4_experiments}
\section{Discussion}
\input{5_discussion}
\FloatBarrier
\section{Conclusion}
\input{6_conclusion}

\paragraph{\bf Aknowledgements} We thank Hugo Lecomte and Nicolas David for their help in the preparation of the PASTIS-R dataset. This work was partly supported by \href{https://www.asp-public.fr}{ASP}, the French Payment Agency.

\bibliographystyle{elsarticle-num-names}

\section{Introduction}
\input{Fusion/1_intro}
\section{Related Work}
\input{Fusion/2_related}
\section{Methods}
\input{Fusion/3_methods}
\section{Experiments}
\input{Fusion/4_experiments}
\section{Discussion}
\input{Fusion/5_discussion}
\section{Conclusion}
\input{Fusion/6_conclusion}

\subsection{Fusion Strategies}
The methods reviewed previously can be categorised into three main strategies: \emph{early}, \emph{late}, and \emph{decision} fusion, all represented in \figref{fig:fusion_schemes}. We also present  \emph{mid}-fusion, a novel fusion scheme specifically adapted for multimodal time sequences.
Certain terms---such as ``features''---have seen their accepted meaning evolve with the gradual adoption of the deep learning paradigm, leading to ambiguity in terms such as ``early'' or ``late'' feature fusion.
We propose redefining the terminology of fusion schemes unambiguously for analyzing temporal sequences of images in the following.

\paragraph{\bf Early Fusion}
This approach combines the different modalities  at the raw feature level. In our context, this amounts to concatenating the modalities \emph{channel-wise} at each observation date. If the different acquisitions are simultaneous, and since the resolutions are identical, this is a straightforward step.
However, when the modalities are captured at different times, {a} preprocessing step is required to interpolate all modalities to a standard temporal sampling.
We denote by $T^\dagger$ the number of time steps in the chosen temporal sampling and by $\tX$ the resulting aggregated tensor of size $T^\dagger \times C^\dagger \times H \times W$ with $C^\dagger=\sum_m C\m$ as defined in \equaref{eq:interpolate}.

This interpolation step can be costly in terms of computation and memory. Furthermore, the relevance of temporal interpolation for fast-changing processes such as plant growth and harvesting is questionable, and this is only made worse by clouds obstructing the optical modalities.
However, an advantage of this approach is the simplicity of encoding $X^\dagger$: a single spatio-temporal encoder $\spatiotemporal$
can be used to learn a truly cross-modal representation, and a unique decoder $\cD$ produces the final prediction:
\begin{align} \label{eq:interpolate}
  \tX = &
  \merge{C}{
  \left\{
    \text{interpolate}(X\m)
    \;\text{to}\; T^\dagger
  \right\}_{m=1}^M}\\
  y^{\text{early}} = &\cD\comp\spatiotemporal(\tX)~.
\end{align}

\paragraph{\bf Late Feature Fusion}
This fusion scheme starts by encoding each modality $m$ separately with dedicated spatio-temporal encoders $\spatiotemporal\m$ into embeddings of size $F\m$. These vectors are then concatenated for all modalities along the channel dimension into a vector of size $\sum_m F\m$, which is ultimately mapped to a prediction $y^{\text{late}}$ by a unique decoder $\cD$:
\begin{align}
  y^{\text{late}} & =
  \cD\comp
  \merge{C}{
    \left\{
      \spatiotemporal\m
      \left(
        X\m
       \right)
    \right\}_{m=1}^M}~,
\end{align}
with $\text{merge}^{(C)}$ the channelwise concatenation operator. While each latent features are derived from a single modality, this method allows the decoder to make decisions taking all modalities into account simultaneously.
\paragraph{\bf Decision Fusion}
This approach ignores the interplay between modalities and makes a prediction for each modality independently. A set of $M$ spatio-temporal encoder $\spatiotemporal\m$ maps each sequence of size $T\m \times C\m \times H \times W$ to a latent space of size $F\m$. {Then, a} set of $M$ decoders $\cD\m$ maps each spatio-temporal feature into a prediction. {Finally, an} aggregation rule {is} applied to combine {all} $M$ predictions into {a} final prediction $y^{\text{decision}}$. Typically, predictions are averaged across all available modalities :
\begin{align}
 y^{\text{decision}} & =
 \frac1M \sum_{m=1}^M
 \cD\m
 \comp
 \spatiotemporal\m
     \left(
       X\m
      \right)
 ~.
\end{align}

\paragraph{\bf Mid-Fusion}

Specific network architectures used to process temporal sequences such as SITS can be broken down into a spatial and a temporal encoder. In such cases, the spatial features can be interwoven, \ie temporally stacked, into a single multimodal sequence, see \figref{fig:stack}.
This approach can be seen as a compromise between early and late fusion and combines three of their advantages:
(i) the temporal encoder can leverage all modalities simultaneously,
(ii) only one temporal encoder is needed,
(iii) no heavy preprocessing is necessary to merge the feature sequences as they are stacked.

\begin{figure*}[h]
  \begin{center}
    \includegraphics[width=0.32\textwidth]{gfx/mid_v2.png}
  \end{center}
  \caption{{\bf Mid-Fusion.} A dedicated spatial encoder processes each modality, and the resulting features are stacked into a single sequence of features.}
  \label{fig:stack}
\end{figure*}

Each modality $m$ has a dedicated spatial encoder $\spatial\m$ mapping images to a feature vector of size $F\m$. These vectors are then concatenated chronologically along the temporal dimension into a unique sequence of length $\sum_m T\m$. A unique temporal encoder $\temporal$ maps this sequence of features into a unique vector, which is in turn classified by a unique decoder $\cD$:
\begin{align}
  y^{\text{mid}} &=
  \cD
 \comp
   \temporal
     \comp
       \merge{T}
       {
         \left\{
           \spatial\m
             \left(
               X\m
             \right)
         \right\}_{m=0}^{M}
         }
  ~,
\end{align}
with $\text{merge}^{(T)}$ the operator concatenating a set of tensors along {the} temporal dimension.
\subsection{Auxiliary Supervision}
\label{sec:aux}
We denote by $\text{criterion}(\cdot\,,\,\cdot)$ the function used to compare the prediction $y$ with the target signal $\hat{y}$. This is typically the cross-entropy for parcel or pixel classification and can be more complex for panoptic or instance segmentation \citep{garnot2021panoptic}.
The resulting function $\mathcal{L}_\text{obj}$ is called the {objective} loss and supervizes the {prediction $y$ of the }network to realize the {sought task}:
\begin{align}
\mathcal{L}_\text{obj}=\text{criterion}(y,\hat{y})
\end{align}

A common problem in deep feature fusion is encountered when most (but not all) discriminative information is concentrated among a reduced number of modalities.
In this case, the other modalities yield predictions and less relevant features for the considered task. Consequently, the final decision taken by the multimodal network focuses on the \emph{better} modalities, and the parts of the network operating on the \emph{lesser} modalities receive a weaker supervisory signal. This results in a network that may not fully leverage the inter-modal patterns that would otherwise allow the multimodal prediction to outperform the \emph{best} modality. This is typically the case for Sentinel SITS, as multispectral optical acquisitions are {often} more conductive to capture phenological patterns than SAR information. Sentinel-1 signal is indeed affected by local terrain angle \citep{kaplan2021S1localinc}, humidity \citep{garkusha2017research}, and is subject to speckle \citep{abramov2017speckle}.

To mitigate this issue, we can add auxiliary losses to supervise each modality independently on top of the {objective} loss $\mathcal{L}_\text{obj}$. \citet{ienco2019combining} has shown this strategy to help to combine optical and radar imagery.
To this end, we associate a prediction $y^m$ to each modality, which is supervised by the auxiliary loss {$\mathcal{L_\text{aux}}$}:
\begin{align}\label{eq:aux}
    \mathcal{L_\text{aux}} =  \sum_{m=1}^M  \lambda^m \: \text{criterion} (y^m,  \hat{y})  ~,
\end{align}
with $\lambda_m$ the strength associated with each modality. Note that, depending on the chosen fusion scheme, computing the single-modality prediction $y^m$ may imply adding new modules {to the backbone network}.
This requires $M$ decoders $\cD\m$, in the case of late fusion. For {mid-}fusion, we must add $M$ {temporal} encoders $\temporal\m$ as well. No additional modules are necessary for decision fusion as single-modality predictions $y^m$ are already necessary to produce the final prediction $y$.
In contrast, auxiliary supervision in the case of early fusion would amount to duplicating the entire network, making it both fruitless and costly.

Since the added modules do not participate in the multimodal decision, only the gradients they propagate are beneficial to the training and not their predictions.
Hence, auxiliary supervision only benefits the modules that  receive gradients from both the main and auxiliary losses. Consequently, auxiliary loss with early supervision would not affect the performance. By the same reasoning, we can expect auxiliary supervision to be beneficial for late and decision fusion than {mid-}fusion for which only the spatial decoders are affected by the auxiliary losses.

\subsection{Temporal Dropout}
We propose a simple data augmentation strategy called temporal dropout to promote a multimodal model that leverages all available modalities.
Inspired by the classic dropout strategy \citep{srivastava2014dropout}, we {randomly} drop observations from the input sequences. The idea is to prevent the network from over-relying on a single modality since its presence is never assured.
Formally, we associate a dropout probability  $p^m \in [0,1]$ for each modality $m \in [1, M]$. During training, each observation of the sequence is dropped with probability $p^m$. At inference time, the network can use all available observations. Note that this technique can also be used on models operating on a single modality by randomly dropping some acquisitions.

\subsection{{Implementation.}}
\label{sec:implem}
{As we set out to evaluate the benefit of multimodality for several tasks, we detail how our fusion schemes can be integrated into temporal attention-based, state-of-the-art networks.}

\paragraph{\bf Parcel-based classification}
We first implement the different fusion strategies for parcel-based crop type classification. In this setting, the contour of parcels is {known}, and the task is to classify the cultivated crop {from} a corresponding yearly SITS.
The spatial and temporal encoding modules we selected are the Pixel-Set Encoder (PSE) and Lightweight Temporal Attention Encoder (L-TAE), whose accuracy and computational efficiency have been solidified in recent studies  \citep{schneider2020re, kondmann2021denethor, garnot2020lightweight, garnot2020satellite}, and whose implementations are available. \footnote{\url{VSainteuf/lightweight-temporal-attention-pytorch}}
All spatio-temporal encoders $\spatiotemporal$ are a combination of a PSE encoding all images of the time series simultaneously and an L-TAE processing the resulting sequence of embeddings, in the manner of \citet{garnot2020satellite}. All decoders $\cD$ are simple Multi-Layer Perceptrons (MLP). All models are trained with {the} cross-entropy loss.
Since we are using these networks in a straightforward manner and without {parameter} alteration, we refer the reader to the \cite[Sec.~3]{garnot2020satellite} for more details on their configuration.
{As explained in \secref{sec:aux}, auxiliary supervision does not affect the early supervision beyond an increased memory load, and we do not evaluate it.

\paragraph{\bf Semantic segmentation} In this setting, the contours of the parcels are unknown, and the model predicts a crop type for each pixel of a {given} patch {from the corresponding yearly SITS}. For this task, we use the state-of-the-art U-TAE architecture \citep{garnot2021panoptic} as {spatio-temporal encoder \footnote{ \url{github.com/VSainteuf/utae-paps}}. The fact that this network's spatial and temporal encoders are intertwined prevents us from applying the mid-fusion scheme.} We use a 2-layer convolutional neural net as decoder $\cD$. The models are trained with cross-entropy loss.

\paragraph{\bf Panoptic segmentation}
{Panoptic segmentation amounts to retrieving both the boundary and crop type of each agricultural parcel. In practice, we predict a set of non-overlapping instance masks and their semantic labels
 \citep{kirillov2019panoptic}.
In our setting, pixels that do not belong to a predicted parcel are classified as background. For this task, we also use U-TAE for spatio-temporal encoding, combined with the instance segmentation module Parcel-as-Points (PaPs) and its associated loss function for supervision \citep{garnot2021panoptic}.$^{2}$ {Averaging instance masks predicted by different modality-specific modules is not straightforward {and costly in terms of memory}, hence }
we do not evaluate the decision-fusion scheme for this task.
 }

\subsection{Pastis-R}
To evaluate the benefit of multimodality, we extend the open-access PASTIS dataset \citep{pastis} with corresponding Sentinel-1 observations.
PASTIS is composed of $2433$ time series of  multi-spectral patches sampled in four different regions of France. Each patch has a spatial extent of $1.28$km$\times1.28$km and contains all available Sentinel-2 observations for the $2019$ season for a total of $115$k images.
Note that PASTIS does not filter out  observations with high cloud cover, hence, certain patches can be partially or entirely obstructed by clouds.

We use Sentinel-1 in Ground Range Detected format processed into { $\sigma_0$ backscatter coefficient } in decibels, {orthorectified at a } $10$m spatial resolution with Orfeo Toolbox \citep{christophe2008orfeo}. We do not apply any spatial or temporal speckle filtering, {nor radiometric terrain correction}: {following the deep learning paradigm, we limit data preprocessing to the minimum.} We assemble each Sentinel-1 observation into a $3$-channel image: vertical polarization (VV), horizontal polarisation (VH), and the ratio of vertical over horizontal polarization (VV/VH). We separate observations made in ascending and descending orbit into two distinct time series. Indeed, the incidence angle of space-borne radar can significantly influence the return signal \citep{singhroy1999effects}. As represented in \figref{fig:pastis}, each time series comprises around $70$ radar acquisitions { for each of the $2433$ patches. This amounts to } a total of $339$k added radar images.
We use the annotations of PASTIS: semantic class and instance identifier for each pixel, allowing us to evaluate models for parcel-based classification, semantic segmentation, and panoptic segmentation. We make the PASTIS-R dataset \citep{pastis-r} publicly available at: \url{github.com/VSainteuf/pastis-benchmark} .

\begin{figure}
    \centering
    \includegraphics[width=\textwidth, trim=0cm 12.5cm 0cm 0cm,clip]{gfx/pastis_figv2.002.jpeg}
    \caption{\textbf{{Pastis-R.}} We extend the PASTIS dataset with radar time series corresponding to ascending and descending orbits of Sentinel-1. For each square patch of $1.28$km$\times1.28$km, PASTIS-R thus provides the image time series of $3$ different modalities, along with semantic and instance annotation for each pixel.}
    \label{fig:pastis}
\end{figure}

\subsection{Implementation details}
As detailed in \secref{sec:implem}, we use the official PyTorch implementations of PSE+LTAE and U-TAE with default hyperparameters.
We  use the official $5$ cross-validation folds of PASTIS \citep{garnot2021utae} to evaluate the performance of the different models. We train our models using the Adam optimizer \citep{kingma2014adam} with default parameters $\text{lr}=0.001$, $\beta=(0.9, 0.999)$ unless specified otherwise, and train all networks on a single TESLA V100 GPU with $32$Gb of VRAM.

\paragraph{\bf Multimodality Configuration}  We consider the two orbits of Sentinel-1 as separate modalities to account for their difference in incident angle, which corresponds to $M=3$. When using auxiliary loss terms, we set $\lambda^m=0.5$ for all modalities. When using temporal dropout, we set $p_0=0.4$ for the optical modality and $p_1=p_2=0.2$ for the radar time series. For early fusion, we interpolate the Sentinel-1 observations to the dates of the Sentinel-2 time series. Indeed, the opposite interpolation strategy would imply tripling the temporal length of the Sentinel-2 time series, which would significantly increase memory usage. Interpolation is computed on the fly when loading dataset samples.

\paragraph{\bf Parcel Classification} For this problem, we train the models for $100$ epochs in batches of $128$ parcels. We use the $K=18$ class nomenclature of PASTIS and report the classification Intersection over Union macro-averaged over the class set (mIoU) to evaluate the parcel-level predictions.

\paragraph{\bf Semantic Segmentation} We train the semantic segmentation models for $100$ epochs in batches of $4$ multi-temporal patches. In this setting, the models also predict \emph{background} pixels, resulting in a $K=19$ class nomenclature. We report the mIoU of the pixel-level predictions:
\begin{align}
    \text{mIoU} = \frac{1}{K} \sum_{k=1}^{K} \frac{TP_k}{TP_k + FP_k + FN_k}~,
\end{align}
with $TP_k$,$FP_k$, and $FN_k$ the count of true positives, false positives, and false negatives for the binary class predictions defined by a class $k$.
\paragraph{\bf Panoptic Segmentation} We follow the training procedure recommended by \citet{garnot2021utae} to train the PaPs network: the learning rate starts at $0.01$ for the first $50$ epochs, and decreases to $0.001$ for the last $50$ epochs. We report the class-averaged panoptic metrics introduced in \citet{kirillov2019panoptic}: Segmentation Quality (SQ), Recognition Quality (RQ), and Panoptic Quality (PQ).
The RQ corresponds to the $F_1$ score for the problem of combined detection and classification: to be counted as a true positive, a parcel must be both detected (the intersect over union of the predicted and true instance masks is above $0.5$) and have its crop type correctly classified.
The SQ corresponds to the intersect over union between the true and predicted masks for correctly detected and classified parcels. Finally, the PQ is the product of both values, thus simultaneously combining information on the quality of detection, classification, and delineation. We report the unweighted classwise average of the three quality measurements. We refer the reader to \citet{kirillov2019panoptic} for more details on these metrics.

\subsection{Parcel Classification Experiment}
We first implement and evaluate the different fusion schemes and their training enhancements for parcel classification. In this setting, the contour of parcels is known in advance, and the model predicts the type of crop cultivated during the period covered by the SITS.

\begin{table}[ht!]
\caption{\textbf{Parcel Classification.} We evaluate the performance of models operating on a single modality (top) and for different fusion strategies for parcel-based classification (bottom). We evaluate each model's baseline performance and the impact of the temporal dropout and auxiliary classifiers enhancements, when applicable. We report the 5-fold cross-validated classification scores in terms of mean classwise Intersect over Union, the base model's parameter count, and, when relevant, of the model with auxiliary classifiers.}
\label{tab:xp:parcelbased}
\begin{tabular}{lccccccc}
\toprule
         & \multicolumn{2}{c}{\multirow{2}{*}{Base}} &\,& Temp. & Auxiliary   & Auxiliary \&     &      Parameter      \\
         &       &&                  & dropout  & supervision & Temp. dropout &      Count      \\  \cline{2-3} \cline{5-7}
         &OA&mIoU& \multicolumn{4}{c}{mIoU} \\
         \midrule
S2       & 91.7 &73.9                  && 74.5     & -           & -                &       114k    \\
S1D      & 87.0&64.5                  && 64.7     & -           & -                &       114k     \\
S1A      & 86.4&63.3                  && 62.9     & -           & -                &       114k    \\\midrule
Early Fusion  &91.8  &74.9                  && 76.5       & -           & -                &       117k     \\
Mid Fusion   & 92.0 &75.1                  && 75.9     &      75.0       & 76.5             &       152k/185k    \\
Late Fusion  & 91.1&73.0                  &&   73.6       & 76.1        & \textbf{77.2}             &       254k/287k    \\
Decision Fusion & 91.0 &72.5                  &&   72.8       & 75.2        & 75.8            &        259k    \\ \bottomrule
\end{tabular}
\end{table}

\paragraph{\bf Analysis}
In \tabref{tab:xp:parcelbased}, we report the performance of all fusion schemes with and without  enhancements.
We first observe that the optical satellite S2 significantly outperforms the two radar time series by a margin of almost $10$ points of mIoU, confirming the relevance of Sentinel-2 for crop type mapping.
We remark that, without enhancement (first column), multimodal models trained with early or mid-fusion schemes improve the performance {compared to} single optical modality network{s}, {while} decision and late fusion perform slightly worse consistently with the results of \citet{pelletier2021fusion}. This highlights the benefit of learning {to mix} modality features early on.
In contrast, auxiliary supervision and temporal dropout improve the later models. This shows that these enhancements can encourage attention-based models to combine features and decisions efficiently from different modalities, as observed in \citet{ienco2019combining} for recurrent networks.
All things considered, late fusion with both enhancements performs best with $+3.3$ mIoU compared to  {a network operating purely on the optical modality}, see \figref{fig:perclass_parcel} for {a classwise comparison}. {Mid-}fusion without enhancement provides good performances with a lower parameter count and none of the preprocessing necessary for early fusion. In practice, the {mid-fusion scheme} is $20$\% faster at inference time than late fusion, making it a valid choice when operating with limited computational resources.

\begin{figure}
    \centering
    \includegraphics[width=.9\linewidth]{gfx/Fusion_per_class_parcel.pdf}
    \caption{{\bf Classwise Performance for Parcel Classification.} We report the IoU of the late fusion model with auxiliary supervision and temporal dropout and {of} the model traine{d} purely on the optical modality. {Multimodality brings a consistent benefit across all classes, which is more notable for some of the most challenging classes such as \emph{Potatoes}, or \emph{Winter triticale}.}}
    \label{fig:perclass_parcel}
\end{figure}
\paragraph{\bf Auxiliary Supervision and Gradient Flow}
Motivated by the impact of auxiliary supervision on the performance of the late fusion approach, we propose to study its effect on the learning process further.
Specifically, we wish to evaluate the different spatio-temporal encoders' contribution to the reduction of the objective loss $\Lmain$, with and without auxiliary supervision, and for the parcel classification task.
Note that, as auxiliary decisions are not computed at inference time, we only consider the decrease of $\Lmain$: a decrease in the auxiliary losses does not directly affect the model's performance.

Following the insights of  \citet{wang2020picking}, we consider the following first-order approximation of the decrease of $\Lmain$ incurred by taking a gradient step:
\begin{align}\label{eq:flow}
    \Delta \Lmain = \eta \langle \nabla \mathcal{L} , \nabla  \Lmain\rangle~,
\end{align}
with $\eta$ the current learning rate.The term $\nabla \mathcal{L}$ of the scalar product in \eqref{eq:flow} corresponds to the step size in the gradient descent and the term $\nabla  \Lmain$ to the slope of the objective loss. Their scalar product approximates the decrease in objective loss when taking a single gradient step. Note that this approximation, called gradient flow, is only valid when using stochastic gradient descent (SGD) and does not hold for momentum or adaptive optimization schemes {such as ADAM \citep{kingma2014adam}}. We thus retrain the late fusion model with SGD for parcel classification. By considering each term in the scalar product in \equaref{eq:flow}, we can estimate the contribution of each parameter of the network to the decrease of the objective loss $\Lmain$.
\begin{figure}[h]
    \centering
    \begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=\linewidth, trim=0cm 1cm 0cm 0cm, clip]{gfx/gflow_base.pdf}
    \caption{Training without auxiliary supervision: $\mathcal{L} = \Lmain$~.}
    \label{fig:gflow:base}
    \end{subfigure}
    \vfill
    \begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=\linewidth]{gfx/gflow_aux.pdf}
    \caption{Training with auxiliary supervision: $\mathcal{L} = \Lmain + \Laux$~.}
    \label{fig:gflow:aux}
    \end{subfigure}
    \caption{{\bf Gradient Flow.} Evolution of the gradient flow for different modules of the late fusion model. The contribution of each modality is plotted as a fraction of the total flow, without auxiliary loss terms (top) and with the additional $\Laux$ term (bottom). We report the flow for the spatial encoders (PSE), temporal encoders (LTAE), and the MLP-based decoders.}
    \label{fig:gflow}
\end{figure}

In \figref{fig:gflow}, we represent the evolution of the gradient flow for different modules of our architecture by summing the contribution of their corresponding parameters.
We observe that, as expected, the gradient flow is concentrated in the modules dedicated to the optical modality. Interestingly, the spatial encoders contribute as much or even more than the temporal encoders despite having four times fewer parameters.

We remark that auxiliary losses lead the model to a different training regime.
While auxiliary supervision results in an increase of the proportion of gradient flow in some radar modules such as PSE-S1A, the flow also increases in proportion in some optical modules as well. We conclude that auxiliary supervision affects all modalities, not only the weaker modalities.

\subsection{Semantic Segmentation}
In this section, we evaluate the performance of the late fusion scheme compared to single modality baselines for semantic segmentation. While the  {mid-fusion} scheme yields promising results on parcel-based experiments, its implementation into a semantic segmentation architecture is not trivial. Indeed,  the state-of-the-art network for this task \citep{garnot2021utae} relies on a U-Net architecture with temporal encoding. In this architecture, spatial and temporal encoding are performed conjointly. After several unsuccessful attempts, we limit our study to the other fusion schemes for this task.

\begin{table}[h]
\caption{\textbf{Semantic Segmentation Experiment}. We evaluate the  semantic segmentation performance of models operating on a single modality and multimodal models trained with early, late, and decision fusion strategies. We evaluate each model's baseline performance and the impact of temporal dropout and auxiliary classifiers, when applicable. We report the 5-fold cross-validated classification scores in mean classwise Intersect over Union (- not applicable). Note that temporal dropout is necessary for the late and decision fusion models to fit in memory.}
\label{tab:semantic}
\begin{center}
\begin{tabular}{lcccc}
\toprule
         & \multirow{2}{*}{Base} & Temporal   & Auxiliary \&     &      Parameter      \\
         &                       & dropout  & Temporal dropout &      Count      \\ \midrule
S2       & 63.1                  & 63.6            & -                &       1\;087k   \\
S1D      &     54.9             &   54.7           & -                &      1\;083k     \\
S1A      & 53.8                  & 53.3          & -                &       1\;083k    \\
Early Fusion        & 64.9        & 65.8         & -     & 1\;602k\\
Late Fusion  & -                 &    65.8          & \textbf{66.3}             &       1\;709k     \\
Decision Fusion  &-                &   64.7          & 64.3          &       1\;742k     \\
\bottomrule
\end{tabular}
\end{center}
\end{table}

\paragraph{\bf Analysis} We report the performance of the different models in \tabref{tab:semantic}. In our experimental setup, the late fusion model with {over} $\sim200$ total {multimodal} observations did not fit in {the $32$Gb of memory of our GPU} with a batch size of $4$ image time series. {By reducing the size of the input sequences, temporal dropout allowed us to  train this memory-intensive model.}
The late fusion model improves the performance of the unimodal models by $2.7$ mIoU points. The performance is further improved by {another} $0.5$ point with the addition of auxiliary supervision. {The early fusion model performs slightly below late fusion, even with temporal dropout. }
As represented in \figref{fig:qualisem}, the radar {modality allows for prediction with crisper contours, in particular between adjacent or nearly adjacent parcels}. This suggests that {the image rugosity}  {of the radar} acquisitions is {can be} valuable to detect inter-parcel zones. {These areas, often of sub-pixel extent,} may display optical reflectances {similar to their} neighboring parcels {but often present surfaces such as fences or groves with a volumetric scatter and thus a distinct radar response . }

Note that the performance of our models on semantic segmentation is around $10$pts below that for parcel classification. This result was expected as {the semantic segmentation task prevents us from exploiting knowledge about the contour of parcels and has the supplementary class \emph{background}, corresponding to non-agricultural land.}

\begin{figure}[th!]
    \centering
\begin{tabular}{c}
    \begin{tabular}{rlrlrlrl}
 \definecolor{tempcolor}{rgb}{0,0,0}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Background}
           &
           \definecolor{tempcolor}{rgb}{0.6823529411764706, 0.7803921568627451, 0.9098039215686274}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Meadow}
           &
           \definecolor{tempcolor}{rgb}{1.0, 0.4980392156862745, 0.054901960784313725}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Soft W. wheat}
           &
           \definecolor{tempcolor}{rgb}{1.0, 0.7333333333333333, 0.47058823529411764}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Corn}
           \\
           \definecolor{tempcolor}{rgb}{0.17254901960784313, 0.6274509803921569, 0.17254901960784313}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{W. barley}
           &
           \definecolor{tempcolor}{rgb}{0.596078431372549, 0.8745098039215686, 0.5411764705882353}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{W. rapeseed}
           &
           \definecolor{tempcolor}{rgb}{0.8392156862745098, 0.15294117647058825, 0.1568627450980392}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Spring barley}
           &
           \definecolor{tempcolor}{rgb}{1.0, 0.596078431372549, 0.5882352941176471}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Sunflower}
           \\
           \definecolor{tempcolor}{rgb}{0.5803921568627451, 0.403921568627451, 0.7411764705882353}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Grapevine}
           &
           \definecolor{tempcolor}{rgb}{0.7725490196078432, 0.6901960784313725, 0.8352941176470589}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Beet}
           &
           \definecolor{tempcolor}{rgb}{0.5490196078431373, 0.33725490196078434, 0.29411764705882354}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{W. triticale}
           &
           \definecolor{tempcolor}{rgb}{0.7686274509803922, 0.611764705882353, 0.5803921568627451}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{W. durum wheat}
           \\
           \definecolor{tempcolor}{rgb}{0.8901960784313725, 0.4666666666666667, 0.7607843137254902}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Fruits, veg., flow.}
           &
           \definecolor{tempcolor}{rgb}{0.9686274509803922, 0.7137254901960784, 0.8235294117647058}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Potatoes}
           &
           \definecolor{tempcolor}{rgb}{0.4980392156862745, 0.4980392156862745, 0.4980392156862745}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Leguminous fodder}
           &
           \definecolor{tempcolor}{rgb}{0.7803921568627451, 0.7803921568627451, 0.7803921568627451}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Soybeans}
           \\
           \definecolor{tempcolor}{rgb}{0.7372549019607844, 0.7411764705882353, 0.13333333333333333}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Orchard}
           &
           \definecolor{tempcolor}{rgb}{0.8588235294117647, 0.8588235294117647, 0.5529411764705883}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Mixed cereal}
           &
           \definecolor{tempcolor}{rgb}{0.09019607843137255, 0.7450980392156863, 0.81176470588235291}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Sorghum}
           &
           \definecolor{tempcolor}{rgb}{1,1,1}
           \tikz \fill[fill=tempcolor, scale=0.3, draw=black] (0,0) rectangle (1,1);
           & \footnotesize{Void label}
    \end{tabular}
 \\
\begin{tabular}{ccccc}
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_1_RGB.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_1_RAD.png}
     &
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_1_S2.png}
     &
     \begin{tikzpicture}
        \node[anchor=south west,inner sep=0] (image) at (0,0) {  \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_1_FUS.png}};
        \begin{scope}[x={(image.south east)},y={(image.north west)}]
            \draw[cyan,ultra thick] (0.52,0.34) circle (0.12);
        \end{scope}
     \end{tikzpicture}

     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_1_GT.png}
     \\
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_2_RGB.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_2_RAD.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_2_S2.png}
     &
     \begin{tikzpicture}
        \node[anchor=south west,inner sep=0] (image) at (0,0) {   \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_2_FUS.png}};
        \begin{scope}[x={(image.south east)},y={(image.north west)}]
            \draw[magenta,ultra thick] (0.85,0.85) circle (0.12);
            \draw[cyan,ultra thick] (0.25,0.3) circle (0.10);
        \end{scope}
     \end{tikzpicture}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_2_GT.png}
     \\
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_4_RGB.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_4_RAD.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_4_S2.png}
     &
      \begin{tikzpicture}
        \node[anchor=south west,inner sep=0] (image) at (0,0) {       \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_4_FUS.png}};
        \begin{scope}[x={(image.south east)},y={(image.north west)}]
            \draw[magenta,ultra thick] (0.82,0.43) circle (0.08);
            \draw[cyan,ultra thick] (0.66,0.45) circle (0.08);
        \end{scope}
     \end{tikzpicture}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/VIZ_4_GT.png}
     \\
     \begin{subfigure}{.19\textwidth}
    \caption{Optical}
    \label{fig:qualisem:s2}
    \end{subfigure}
    &
    \begin{subfigure}{.19\textwidth}
    \caption{Radar}
    \label{fig:qualisem:s1}
    \end{subfigure}

    &
    \begin{subfigure}{.19\textwidth}
    \caption{S2 Prediction}
    \label{fig:qualisem:mono}
    \end{subfigure}
    &
    \begin{subfigure}{.19\textwidth}
    \caption{Fusion Prediction}
    \label{fig:qualisem:late}
    \end{subfigure}
    &
     \begin{subfigure}{.19\textwidth}
    \caption{Ground Truth}
    \label{fig:qualisem:gt}
    \end{subfigure}
    \end{tabular}
  \end{tabular}
    \caption{{\bf Qualitative Results for Semantic Segmentation.} We show one observation from the optical time series in \Subfigref{fig:qualisem:s2} and {one} from the radar time series in \Subfigref{fig:qualisem:s1}. The prediction for the unimodal optical model is represented in \Subfigref{fig:qualisem:mono}, a{our late fusion multimodal model} in \Subfigref{fig:qualisem:late}, and finally the ground truth in \Subfigref{fig:qualisem:gt}. We observe that the multimodal model produces results with clearer and more distinct borders between close parcels (cyan circle \protect\tikz \protect\node[circle, thick, draw = cyan, fill = none, scale = 0.7] {};). The multimodal model also displays fewer errors for hard and ambiguous parcels, showing the benefit of learning intermodal features (magenta circle \protect\tikz \protect\node[circle, thick, draw = magenta, fill = none, scale = 0.7] {};). Crop types are represented according to the color code above (W. stands for Winter). The same legend is {used} in all subsequent figures representing crop labels. }
    \label{fig:qualisem}
\end{figure}

\paragraph{ \bf Varying Cloud Cover Experiment}
One of the motivations for using both optical and radar images in the context of crop type mapping is to exploit the imperviousness of radar signals to cloud cover. This potentially allows our model to rely on the radar signal when optical observations are obstructed by clouds, which is particularly crucial in countries with pervasive cloud cover, such as subtropical regions \citep{orynbaikyzy2020crop}. The parcel-based and semantic experiments allow for a first exploration of this capacity, but remain {bound} to the specific cloud conditions of the French metropolitan territory and the {year} of acquisition ({2019}). We propose to further investigate this benefit of multimodality by artificially simulating increased cloud obstruction {on the test set}. To do so, we evaluate the performance of models when removing random optical acquisitions while leaving the radar time series unchanged. We report the performance of the models in Figure \ref{fig:london}, for different ratios of remaining optical observations, corresponding to different levels of cloud obstruction.

\begin{figure}[h!]
    \centering
    \begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=.8\linewidth]{gfx/LondonFogv5.pdf}
    \caption{Parcel-based classification}
    \label{fig:london:parcel}
    \end{subfigure}
    \vfill
    \begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=.8\linewidth]{gfx/LondonFog_semsegv2.pdf}
    \caption{Semantic segmentation}
    \label{fig:london:semantic}
    \end{subfigure}
    \caption{\textbf{Varying Cloud Cover Experiment.} We {evaluate} the different models with {varying} ratios optical observations {remaining}. In both parcel-based classification \Subfigref{fig:london:parcel} and semantic segmentation \Subfigref{fig:london:semantic}, the fusion models prove more robust to a reduced number of optical observations.  }
    \label{fig:london}
\end{figure}

As expected, the performance of the S2-only model drops drastically as the number of available optical observations decreases for both parcel classification and semantic segmentation, performing worse than unimodal radar models for {a ratio of} $70$\% of artificial occlusion.
Multimodal fusion models can maintain an almost constant level of performance for up to $50$\% missing optical acquisitions. For more extreme ratios, the performances of the multimodal models eventually drop. The magnitude of the drop seems to be related to the amount of interplay between modalities in the network.
{Early fusion proves the least robust to missing optical observations.} Mid-fusion, and to a lesser extent, the late fusion are also affected by obstruction. These models rely on multimodal encoders and decoders, which are likely to be affected by a severe decrease in the quality of the optical sequence. In contrast, the decision fusion scheme comprises independent classifiers and proves to be the most resilient: even with $90$\% of optical images removed, it still outperforms the radar modality by several mIoU points.
We conclude that such models should be favored in regions with pervasive or inconsistent cloud cover.

We also observe that auxiliary supervision and temporal dropout make both unimodal and multimodal models  more resilient to missing optical acquisitions for semantic segmentation. The same phenomenon can be observed for parcel classification but was not represented for clarity.

\subsection{Panoptic segmentation experiment}
In this section, we evaluate the performance of the {early and} late fusion schemes compared to single modality baselines for panoptic segmentation. We do not assess auxiliary losses on the late fusion model  as the use of auxiliary decoders in this setting comes at a prohibitive computational cost. Indeed, the auxiliary decoders would be PaPs instance segmentation modules which already significantly impact training times on single modality architectures. Decision fusion is not evaluated here for the same reason.  Like in the semantic segmentation experiment, temporal dropout proved necessary to train the late fusion model.

\begin{table}[]
\centering
\caption{\textbf{Panoptic Segmentation Experiment}. We evaluate the panoptic segmentation performance of models operating on a single modality and multimodal models trained with the early and late fusion strategy with temporal dropout.}
\label{tab:xp:pano}
\begin{tabular}{lcccc}
\toprule
                    & SQ & RQ & PQ & Parameter count\\\midrule
S2                  &  81.3  & 49.2   & 40.4 & 1\;318k  \\
S1D                 &  77.0  &  39.3  &   30.9 & 1\;318k  \\
S1A                 &  77.4  &  38.8   &  30.6 & 1\;318k  \\
Early Fusion + Tdrop        &   \textbf{82.2 }&  \textbf{50.6} &  \textbf{42.0} &1\;791k \\
Late Fusion + Tdrop        &   81.6 &  50.5 &  41.6 & 2\;390k \\\bottomrule
\end{tabular}
\end{table}

\paragraph{\bf Analysis} We report the results of this experiment on \tabref{tab:xp:pano}. Overall, the early and late fusion schemes increase the panoptic quality by $1.6$pt and $1.2$pt, respectively, compared to the optical baseline. This improvement is mostly driven by an increase in recognition quality, while the segmentation quality remains almost unchanged. This suggests that the radar modality helps correctly detect additional agricultural parcels rather than refining the delineation of their boundaries. Although modest, this improvement is valuable for this notoriously complex task.

We show on \figref{fig:qualipano} the qualitative evaluation of the panoptic fusion model compared to the optical baseline. In practice, the fusion model seems to retrieve more agricultural parcels successfully and manages to retrieve small parcels missed by the optical model. We also display the predictions made by the unimodal models and the predictions of the fusion model in \figref{fig:qualipanomod}. These qualitative results show how the radar modality helps detect more parcels than the optical baseline or improve the fusion model's semantic predictions. Additionally, given the relative noisiness of radar observations, the radar-only models retrieve surprisingly well the parcel boundaries. {As mentioned previously, this could be attributed to the distinct volumetric radar response on parcel boundaries.}  We report the per-class performances on \figref{fig:perclass_pano}.

Regarding robustness to clouds, when performing inference on only $30$\% of the optical observations, the S2 baseline model drops to $33.0$ PQ. In contrast, our late fusion model maintains a score of $37.6$ PQ. Consistently with the previous experiments, the addition of the radar modality helps improve the panoptic predictions with reduced availability of optical observations.

\begin{figure}[h]
\centering
\begin{tabular}{ccccc}
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_1_RGB.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_1_RAD.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_1_S2.png}
     &
   \begin{tikzpicture}
        \node[anchor=south west,inner sep=0] (image) at (0,0) {       \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_1_FUS.png}};
        \begin{scope}[x={(image.south east)},y={(image.north west)}]
            \draw[magenta,ultra thick] (0.1,0.35) circle (0.08);
            \draw[magenta,ultra thick] (0.26,0.14) circle (0.08);
        \end{scope}
    \end{tikzpicture}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_1_GT.png}
     \\
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_3_RGB.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_3_RAD.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_3_S2.png}
     &
    \begin{tikzpicture}
    \node[anchor=south west,inner sep=0] (image) at (0,0) {       \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_3_FUS.png}};
    \begin{scope}[x={(image.south east)},y={(image.north west)}]
        \draw[cyan,ultra thick] (0.25,0.75) circle (0.2);
    \end{scope}
    \end{tikzpicture}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_3_GT.png}
     \\
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_5_RGB.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_5_RAD.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_5_S2.png}
     &
    \begin{tikzpicture}
    \node[anchor=south west,inner sep=0] (image) at (0,0) {       \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_5_FUS.png}};
    \begin{scope}[x={(image.south east)},y={(image.north west)}]
        \draw[green,ultra thick] (0.8,0.75) circle (0.18);
        \draw[green,ultra thick] (0.8,0.4) circle (0.15);
    \end{scope}
    \end{tikzpicture}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/pano/PANOVIZ_5_GT.png}
     \\
     \begin{subfigure}{.19\textwidth}
    \caption{Optical}
    \label{fig:qualipano:s2}
    \end{subfigure}
    &
    \begin{subfigure}{.19\textwidth}
    \caption{Radar}
    \label{fig:qualipano:s1}
    \end{subfigure}

    &
    \begin{subfigure}{.19\textwidth}
    \caption{S2 Prediction}
    \label{fig:qualipano:mono}
    \end{subfigure}
    &
    \begin{subfigure}{.19\textwidth}
    \caption{Fusion Prediction}
    \label{fig:qualipano:late}
    \end{subfigure}
    &
     \begin{subfigure}{.19\textwidth}
    \caption{Ground Truth}
    \label{fig:qualipano:gt}
    \end{subfigure}
    \end{tabular}

\caption{{\bf Qualitative Results for Panoptic Segmentation.}  We show one observation from the optical time series in \Subfigref{fig:qualipano:s2} and {one} from the radar time series in \Subfigref{fig:qualipano:s1}. The prediction for the unimodal optical model is represented in \Subfigref{fig:qualipano:mono} and the multimodal model in \Subfigref{fig:qualipano:late}, and finally the ground truth in \Subfigref{fig:qualipano:gt}, with the same colormap as in \figref{fig:qualisem}. The fusion model {retrieves} more parcels (cyan circle
\protect\tikz \protect\node[circle, thick, draw = cyan, fill = none, scale = 0.7] {};), and even {small parcels that were missed by the purely optical model} (magenta circle
\protect\tikz \protect\node[circle, thick, draw = magenta, fill = none, scale = 0.7] {};). We also note that the fusion model seems to handle parcels with internal subdivisions (green circle
\protect\tikz \protect\node[circle, thick, draw = green, fill = none, scale = 0.7] {};) better than the optical model.}
\label{fig:qualipano}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=.9\linewidth]{gfx/Fusion_per_class_pano.pdf}
    \caption{{\bf Classwise Performance for Parcel Classification.} We report the Panoptic Quality of the late fusion model with temporal dropout and the model trained purely on the optical modality. The classes are ordered as in \figref{fig:perclass_parcel}. In the panoptic setting, the radar modality is also specifically beneficial for hard classes such as \emph{Winter triticale}.}
    \label{fig:perclass_pano}
\end{figure}

\begin{figure}[h]
\centering
\begin{tabular}{ccccc}
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_6_S1A.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_6_S1D.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_6_S2.png}
     &
     \begin{tikzpicture}
    \node[anchor=south west,inner sep=0] (image) at (0,0) {           \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_6_FUS.png}};
    \begin{scope}[x={(image.south east)},y={(image.north west)}]
        \draw[green,ultra thick] (0.15,0.23) circle (0.12);
        \draw[red,ultra thick] (0.8,0.62) circle (0.12);

    \end{scope}
    \end{tikzpicture}
     &
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_6_GT.png}
 \\
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_7_S1A.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_7_S1D.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_7_S2.png}
     &
         \begin{tikzpicture}
    \node[anchor=south west,inner sep=0] (image) at (0,0) {               \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_7_FUS.png}};
    \begin{scope}[x={(image.south east)},y={(image.north west)}]
        \draw[cyan,ultra thick] (0.15,0.43) circle (0.09);
    \end{scope}
    \end{tikzpicture}

     &
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_7_GT.png}
 \\
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_8_S1A.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_8_S1D.png}
     &
    \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_8_S2.png}
     &
    \begin{tikzpicture}
    \node[anchor=south west,inner sep=0] (image) at (0,0) {               \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_8_FUS.png}};
    \begin{scope}[x={(image.south east)},y={(image.north west)}]
        \draw[green,ultra thick] (0.3,0.7) circle (0.2);
    \end{scope}
    \end{tikzpicture}
     &
     \includegraphics[width=.18\textwidth, trim=1cm 0.5cm 0cm 0cm, clip]{gfx/panomod/PANOMODVIZ_8_GT.png}
    \\
     \begin{subfigure}{.19\textwidth}
    \caption{S1A Prediction}
    \label{fig:qualipano:s2}
    \end{subfigure}
    &
    \begin{subfigure}{.19\textwidth}
    \caption{S1D Prediction}
    \label{fig:qualipano:s1}
    \end{subfigure}

    &
    \begin{subfigure}{.19\textwidth}
    \caption{S2 Prediction}
    \label{fig:qualipano:mono}
    \end{subfigure}
    &
    \begin{subfigure}{.19\textwidth}
    \caption{Fusion Prediction}
    \label{fig:qualipano:late}
    \end{subfigure}
    &
     \begin{subfigure}{.19\textwidth}
    \caption{Ground Truth}
    \label{fig:qualipano:gt}
    \end{subfigure}
\end{tabular}

\caption{{\bf Qualitative Results for Panoptic Segmentation.} We compare the predictions made by unimodal models operating on S1A (a), S1D (b), S2 (c), and the predictions made by the late fusion model (d). We also show the ground truth annotations (e). We observe cases where the optical model does not detect parcels but successfully predicted by the radar-only models and by the fusion model as well (green circle
\protect\tikz \protect\node[circle, thick, draw = green, fill = none, scale = 0.7] {};). We also note that the optical model detects some parcels, but the crop type is corrected by the addition of the radar modality
(red circle
\protect\tikz \protect\node[circle, thick, draw = red, fill = none, scale = 0.7] {};).
Conversely, some parcels are detected by the radar-only model with an incorrect crop type and not detected by the optical model. Combining both modalities in the fusion model leads to a correct prediction. (cyan circle
\protect\tikz \protect\node[circle, thick, draw = cyan, fill = none, scale = 0.7] {};)}

\label{fig:qualipanomod}
\end{figure}
\subsection{Recommendations.}
Our experiments showed that each fusion scheme has advantages and limitations influencing when its use is most relevant:
\begin{itemize}
    \item {\bf Early Fusion.} It is the most compact of the fusion models and shows competitive performance on all three tasks. The main drawback of this approach is the necessity of an expensive interpolation.  As reported in \tabref{tab:inftime:fusion}, this preprocessing makes the early fusion scheme slower than late fusion despite relying on a smaller network for parcel classification and semantic segmentation. Early fusion is the least robust fusion scheme to cloud cover.

    \item {\bf Mid Fusion.} Of all methods without preprocessing, this strategy leads to the fastest run time and the lowest memory requirement. It yields the second-best performance for parcel-based classification but suffers more than late and decision fusion when the cloud cover is extensive. Its dependence on separate spatial and temporal encoders prevents its straightforward adaptation to pixel-based tasks. We recommend using this scheme for parcel classification in areas without extensive cloud cover and when inference speed is critical.
    \item {\bf Late Fusion.} This fusion method, when combined with enhancement schemes, leads to the best performance and the highest adaptability, as well as excellent resilience to even extreme cloud cover. This method is our default recommendation when using temporal attention methods with multimodal time series.
    \item {\bf Decision Fusion.} Despite having the highest parameter count, this method lags in terms of performance and is prohibitively costly for panoptic segmentation. However, it is the most resilient to cloud cover. We recommend using decision fusion when it is expected that only a few optical observations may be available for inference.
\end{itemize}
We also have evaluated the influence of two enhancement schemes:
\begin{itemize}
    \item {\bf Auxiliary Supervision.} This method consists in adding alongside the main prediction auxiliary predictions based on one modality alone. The rationale is to help each specialized module to learn meaningful features regardless of the interplay with other modalities. We observe a strong effect in precision for late and decision fusion, which have dedicated encoding modules for each modality.
    \item {\bf Temporal Dropout.}  This simple method consists in randomly dropping acquisitions of the time series considered. Its effect was beneficial to all fusion schemes and the optical baseline across our experiments. Another benefit of this scheme is that it reduces the memory footprint of networks during training.
\end{itemize}

\subsection{Limitations.}

Our study hinged on the PASTIS dataset, which contains annotated agricultural parcels from four different regions of the French metropolitan territory. In this regard, our results are most relevant for crop mapping applications with the same meteorological context, terrain conditions, and crop types as this region. Certain crop types not observed in PASTIS could benefit even more from the radar modality than our results show. For instance, rice fields are often filled with water and thus have a distinctive SAR response but are not represented in PASTIS.

Furthermore, our evaluation of cloud robustness focused on assessing the effect of a reduced number of optical observations \emph{at inference time}.
This corresponds to artificially increasing the cloud cover in the test set without affecting crop growth. A more rigorous approach would  constitute a dataset comprising truly observed cloud coverage by varying the regions and years of acquisition. This is complicated by the lack of harmonization between LPIS across different countries in nomenclature and open-access policy.
Lastly, we only used backscattering coefficients from the SAR data in our experiments, as is commonly done in the crop type mapping literature \citep{orynbaikyzy2019crop}. \citet{mestre2020time} found that the addition of interferometric radar features is beneficial to crop classification when using only radar inputs. Further work is needed to  assess the benefit of interferometric radar features in a fusion setting with optical imagery.
Moreover, we chose to prepare the SAR inputs with limited preprocessing. We do not apply speckle filtering or  radiometric terrain correction to compensate for the effect of the local incident angle. Interestingly, our experiments showed that this does not prevent the radar modality from benefiting crop mapping models.
However, further studies could evaluate the benefit of adding speckle filtering, elevation information, or meteorological context to networks using radar images for crop mapping.

\section{Introduction}

\file{elsarticle.cls} is a thoroughly re-written document class
for formatting \LaTeX{} submissions to Elsevier journals.
The class uses the environments and commands defined in \LaTeX{} kernel
without any change in the signature so that clashes with other
contributed \LaTeX{} packages such as \file{hyperref.sty},
\file{preview-latex.sty}, etc., will be minimal.
\file{elsarticle.cls} is primarily built upon the default
\file{article.cls}.  This class depends on the following packages
for its proper functioning:

\begin{enumerate}
\item \file{natbib.sty} for citation processing;
\item \file{geometry.sty} for margin settings;
\item \file{fleqn.clo} for left aligned equations;
\item \file{graphicx.sty} for graphics inclusion;
\item \file{txfonts.sty} optional font package, if the document is to
  be formatted with Times and compatible math fonts;
\item \file{hyperref.sty} optional packages if hyperlinking is
  required in the document;
\item \file{endfloat.sty} optional packages if floats to be placed at
 end of the PDF.
\end{enumerate}

All the above packages (except some optional packages) are part of any
standard \LaTeX{} installation. Therefore, the users need not be
bothered about downloading any extra packages.  Furthermore, users are
free to make use of \textsc{ams} math packages such as
\file{amsmath.sty}, \file{amsthm.sty}, \file{amssymb.sty},
\file{amsfonts.sty}, etc., if they want to.  All these packages work in
tandem with \file{elsarticle.cls} without any problems.

\section{Major Differences}

Following are the major differences between \file{elsarticle.cls}
and its predecessor package, \file{elsart.cls}:

\begin{enumerate}[\textbullet]
\item \file{elsarticle.cls} is built upon \file{article.cls}
while \file{elsart.cls} is not. \file{elsart.cls} redefines
many of the commands in the \LaTeX{} classes/kernel, which can
possibly cause surprising clashes with other contributed
\LaTeX{} packages;

\item provides preprint document formatting by default, and
optionally formats the document as per the final
style of models $1+$, $3+$ and $5+$ of Elsevier journals;

\item some easier ways for formatting \verb+list+ and
\verb+theorem+ environments are provided while people can still
use \file{amsthm.sty} package;

\item \file{natbib.sty} is the main citation processing package
  which can comprehensively handle all kinds of citations and
works perfectly with \file{hyperref.sty} in combination with
\file{hypernat.sty};

\item long title pages are processed correctly in preprint and
  final formats.

\end{enumerate}

\section{Installation}

The package is available at author resources page at Elsevier
(\url{http://www.elsevier.com/locate/latex}).
It can also be found in any of the nodes of the Comprehensive
\TeX{} Archive Network (\textsc{ctan}), one of the primary nodes
being
\url{http://tug.ctan.org/tex-archive/macros/latex/contrib/elsarticle/}.
Please download the \file{elsarticle.dtx} which is a composite
class with documentation and \file{elsarticle.ins} which is the
\LaTeX{} installer file. When we compile the
\file{elsarticle.ins} with \LaTeX{} it provides the class file,
\file{elsarticle.cls} by
stripping off all the documentation from the \verb+*.dtx+ file.
The class may be moved or copied to a place, usually,
\verb+$TEXMF/tex/latex/elsevier/+,
or a folder which will be read
by \LaTeX{} during document compilation.  The \TeX{} file
database needs updation after moving/copying class file.  Usually,
we use commands like \verb+mktexlsr+ or \verb+texhash+ depending
upon the distribution and operating system.

\section{Usage}\label{sec:usage}
The class should be loaded with the command:

\begin{vquote}
 \documentclass[<options>]{elsarticle}
\end{vquote}

\noindent where the \verb+options+ can be the following:

\begin{description}

\item [{\tt\color{verbcolor} preprint}]  default option which format the
  document for submission to Elsevier journals.

\item [{\tt\color{verbcolor} review}]  similar to the \verb+preprint+
option, but increases the baselineskip to facilitate easier review
process.

\item [{\tt\color{verbcolor} 1p}]  formats the article to the look and
feel of the final format of model 1+ journals. This is always single
column style.

\item [{\tt\color{verbcolor} 3p}] formats the article to the look and
feel of the final format of model 3+ journals. If the journal is a two
column model, use \verb+twocolumn+ option in combination.

\item [{\tt\color{verbcolor} 5p}] formats for model 5+ journals. This
is always of two column style.

\item [{\tt\color{verbcolor} authoryear}] author-year citation style of
\file{natbib.sty}. If you want to add extra options of
\file{natbib.sty}, you may use the options as comma delimited strings
as arguments to \verb+\biboptions+ command. An example would be:
\end{description}

\begin{vquote}
 \biboptions{longnamesfirst,angle,semicolon}
\end{vquote}

\begin{description}
\item [{\tt\color{verbcolor} number}] numbered citation style. Extra options
  can be loaded with\linebreak \verb+\biboptions+ command.

\item [{\tt\color{verbcolor} sort\&compress}] sorts and compresses the
numbered citations. For example, citation [1,2,3] will become [1--3].

\item [{\tt\color{verbcolor} longtitle}] if front matter is unusually long, use
  this option to split the title page across pages with the correct
placement of title and author footnotes in the first page.

\item [{\tt\color{verbcolor} times}] loads \file{txfonts.sty}, if
available in the system to use Times and compatible math fonts.

\item [{\tt\color{verbcolor} reversenotenum}] Use alphabets as
author--affiliation linking labels and use numbers for author
footnotes. By default, numbers will be used as author--affiliation
linking labels and alphabets for author footnotes.

\item [{\tt\color{verbcolor} lefttitle}] To move title and
author/affiliation block to flushleft. \verb+centertitle+ is the
default option which produces center alignment.

\item [{\tt\color{verbcolor} endfloat}] To place all floats at the end
of the document.

\item [{\tt\color{verbcolor} nonatbib}] To unload natbib.sty.

\item [{\tt\color{verbcolor} doubleblind}] To hide author name,
affiliation, email address etc. for double blind refereeing purpose.

\item[] All options of \file{article.cls} can be used with this
  document class.

\item[] The default options loaded are \verb+a4paper+, \verb+10pt+,
  \verb+oneside+, \verb+onecolumn+ and \verb+preprint+.

\end{description}

\section{Frontmatter}

There are two types of frontmatter coding:
\begin{enumerate}[(1)]
\item each author is
connected to an affiliation with a footnote marker; hence all
authors are grouped together and affiliations follow;
\pagebreak
\item authors of same affiliations are grouped together and the
relevant affiliation follows this group.
\end{enumerate}

An example of coding the first type is provided below.

\begin{vquote}
 \title{This is a specimen title\tnoteref{t1,t2}}
 \tnotetext[t1]{This document is the results of the research
    project funded by the National Science Foundation.}
 \tnotetext[t2]{The second title footnote which is a longer
    text matter to fill through the whole text width and
    overflow into another line in the footnotes area of the
    first page.}
\end{vquote}

\begin{vquote}
\author[1]{Jos Migchielsen\corref{cor1}%
  \fnref{fn1}}
\ead{J.Migchielsen@elsevier.com}

\author[2]{CV Radhakrishnan\fnref{fn2}}
\ead{cvr@sayahna.org}

\author[3]{CV Rajagopal\fnref{fn1,fn3}}
\ead[url]{www.stmdocs.in}

 \cortext[cor1]{Corresponding author}
 \fntext[fn1]{This is the first author footnote.}
 \fntext[fn2]{Another author footnote, this is a very long
   footnote and it should be a really long footnote. But this
   footnote is not yet sufficiently long enough to make two
   lines of footnote text.}
 \fntext[fn3]{Yet another author footnote.}

 \affiliation[1]{organization={Elsevier B.V.},
                 addressline={Radarweg 29},
                 postcode={1043 NX},
                 city={Amsterdam},
                 country={The Netherlands}}

 \affiliation[2]{organization={Sayahna Foundation},
                 addressline={JWRA 34, Jagathy},
                 city={Trivandrum}
                 postcode={695014},
                 country={India}}
\end{vquote}

\begin{vquote}

 \affiliation[3]{organization={STM Document Engineering
                               Pvt Ltd.},
                 addressline={Mepukada, Malayinkil},
                 city={Trivandrum}
                 postcode={695571},
                 country={India}}
\end{vquote}

The output of the above \TeX{} source is given in Clips~\ref{clip1} and
\ref{clip2}. The header portion or title area is given in
Clip~\ref{clip1} and the footer area is given in Clip~\ref{clip2}.

\deforange{blue!70}
\src{Header of the title page.}
\includeclip{1}{130 612 477 707}{1psingleauthorgroup.pdf
\deforange{orange}

\deforange{blue!70}
\src{Footer of the title page.}
\includeclip{1}{93 135 499 255}{1pseperateaug.pdf
\deforange{orange}

Most of the commands such as \verb+\title+, \verb+\author+,
\verb+\affiliation+ are self explanatory.  Various components are
linked to each other by a label--reference mechanism; for
instance, title footnote is linked to the title with a footnote
mark generated by referring to the \verb+\label+ string of
the \verb=\tnotetext=.  We have used similar commands
such as \verb=\tnoteref= (to link title note to title);
\verb=\corref= (to link corresponding author text to
corresponding author); \verb=\fnref= (to link footnote text to
the relevant author names).  \TeX{} needs two compilations to
resolve the footnote marks in the preamble part.
Given below are the syntax of various note marks and note texts.

\begin{vquote}
  \tnoteref{<label(s)>}
  \corref{<label(s)>}
  \fnref{<label(s)>}
  \tnotetext[<label>]{<title note text>}
  \cortext[<label>]{<corresponding author note text>}
  \fntext[<label>]{<author footnote text>}
\end{vquote}

\noindent where \verb=<label(s)>= can be either one or more comma
delimited label strings. The optional arguments to the
\verb=\author= command holds the ref label(s) of the address(es)
to which the author is affiliated while each \verb=\affiliation=
command can have an optional argument of a label. In the same
manner, \verb=\tnotetext=, \verb=\fntext=, \verb=\cortext= will
have optional arguments as their respective labels and note text
as their mandatory argument.

The following example code provides the markup of the second type
of author-affiliation.

\begin{vquote}
\author{Jos Migchielsen\corref{cor1}%
  \fnref{fn1}}
\ead{J.Migchielsen@elsevier.com}
 \affiliation[1]{organization={Elsevier B.V.},
                 addressline={Radarweg 29},
                 postcode={1043 NX},
                 city={Amsterdam},
                 country={The Netherlands}}

\author{CV Radhakrishnan\fnref{fn2}}
\ead{cvr@sayahna.org}
 \affiliation[2]{organization={Sayahna Foundation},
                 addressline={JWRA 34, Jagathy},
                 city={Trivandrum}
                 postcode={695014},
                 country={India}}

\author{CV Rajagopal\fnref{fn1,fn3}}
\ead[url]{www.stmdocs.in}
 \affiliation[3]{organization={STM Document Engineering
                               Pvt Ltd.},
                 addressline={Mepukada, Malayinkil},
                 city={Trivandrum}
                 postcode={695571},
                 country={India}}
\end{vquote}

\vspace*{-.5pc}

\begin{vquote}
\cortext[cor1]{Corresponding author}
\fntext[fn1]{This is the first author footnote.}
\fntext[fn2]{Another author footnote, this is a very long
  footnote and it should be a really long footnote. But this
  footnote is not yet sufficiently long enough to make two lines
  of footnote text.}
\end{vquote}

The output of the above \TeX{} source is given in Clip~\ref{clip3}.

\deforange{blue!70}
\src{Header of the title page..}
\includeclip{1}{119 563 468 709}{1pseperateaug.pdf
\deforange{orange}

Clip~\ref{clip4} shows the output after giving \verb+doubleblind+ class option.

\deforange{blue!70}
\src{Double blind article}
\includeclip{1}{124 567 477 670}{elstest-1pdoubleblind.pdf
\deforange{orange}

\vspace*{-.5pc}
The frontmatter part has further environments such as abstracts and
keywords.  These can be marked up in the following manner:

\begin{vquote}
 \begin{abstract}
  In this work we demonstrate the formation of a new type of
  polariton on the interface between a ....
 \end{abstract}
\end{vquote}

\vspace*{-.5pc}
\begin{vquote}
 \begin{keyword}
  quadruple exiton \sep polariton \sep WGM
 \end{keyword}
\end{vquote}

\noindent Each keyword shall be separated by a \verb+\sep+ command.
\textsc{msc} classifications shall be provided in
the keyword environment with the commands
\verb+\MSC+. \verb+\MSC+ accepts an optional
argument to accommodate future revisions.
eg., \verb=\MSC[2008]=. The default is 2000.\looseness=-1

\subsection{New page}
Sometimes you may need to give a page-break and start a new page after
title, author or abstract. Following commands can be used for this
purpose.

\begin{vquote}
  \newpageafter{title}
  \newpageafter{author}
  \newpageafter{abstract}
\end{vquote}

\begin{itemize}
\leftskip-2pc
\item [] {\tt\color{verbcolor} \verb+\newpageafter{title}+} typeset the title alone on one page.

\item [] {\tt\color{verbcolor} \verb+\newpageafter{author}+}  typeset the title
and author details on one page.

\item [] {\tt\color{verbcolor} \verb+\newpageafter{abstract}+}
typeset the title,
author details and abstract \& keywords one one page.

\end{itemize}

\section{Floats}
{Figures} may be included using the command, \verb+\includegraphics+ in
combination with or without its several options to further control
graphic. \verb+\includegraphics+ is provided by \file{graphic[s,x].sty}
which is part of any standard \LaTeX{} distribution.
\file{graphicx.sty} is loaded by default. \LaTeX{} accepts figures in
the postscript format while pdf\LaTeX{} accepts \file{*.pdf},
\file{*.mps} (metapost), \file{*.jpg} and \file{*.png} formats.
pdf\LaTeX{} does not accept graphic files in the postscript format.

The \verb+table+ environment is handy for marking up tabular
material. If users want to use \file{multirow.sty},
\file{array.sty}, etc., to fine control/enhance the tables, they
are welcome to load any package of their choice and
\file{elsarticle.cls} will work in combination with all loaded
packages.

\section[Theorem and ...]{Theorem and theorem like environments}

\file{elsarticle.cls} provides a few shortcuts to format theorems and
theorem-like environments with ease. In all commands the options that
are used with the \verb+\newtheorem+ command will work exactly in the same
manner. \file{elsarticle.cls} provides three commands to format theorem or
theorem-like environments:

\begin{vquote}
 \newtheorem{thm}{Theorem}
 \newtheorem{lem}[thm]{Lemma}
 \newdefinition{rmk}{Remark}
 \newproof{pf}{Proof}
 \newproof{pot}{Proof of Theorem \ref{thm2}}
\end{vquote}

The \verb+\newtheorem+ command formats a
theorem in \LaTeX's default style with italicized font, bold font
for theorem heading and theorem number at the right hand side of the
theorem heading.  It also optionally accepts an argument which
will be printed as an extra heading in parentheses.

\begin{vquote}
  \begin{thm}
   For system (8), consensus can be achieved with
   $\|T_{\omega z}$
   ...
     \begin{eqnarray}\label{10}
     ....
     \end{eqnarray}
  \end{thm}
\end{vquote}

Clip~\ref{clip5} will show you how some text enclosed between the
above code\goodbreak \noindent looks like:

\vspace*{6pt}
\deforange{blue!70}
\src{{\ttfamily\color{verbcolor}\expandafter\@gobble\string\\ newtheorem}}
\includeclip{2}{1 1 453 120}{jfigs.pdf}
\deforange{orange}

The \verb+\newdefinition+ command is the same in
all respects as its\linebreak \verb+\newtheorem+ counterpart except that
the font shape is roman instead of italic.  Both
\verb+\newdefinition+ and \verb+\newtheorem+ commands
automatically define counters for the environments defined.

\vspace*{6pt}
\deforange{blue!70}
\src{{\ttfamily\color{verbcolor}\expandafter\@gobble\string\\ newdefinition}}
\includeclip{1}{1 1 453 105}{jfigs.pdf}
\deforange{orange}

The \verb+\newproof+ command defines proof environments with
upright font shape.  No counters are defined.

\vspace*{6pt}
\deforange{blue!70}
\src{{\ttfamily\color{verbcolor}\expandafter\@gobble\string\\ newproof}}
\includeclip{3}{1 1 453 65}{jfigs.pdf}
\deforange{orange}

Users can also make use of \verb+amsthm.sty+ which will override
all the default definitions described above.

\section[Enumerated ...]{Enumerated and Itemized Lists}
\file{elsarticle.cls} provides an extended list processing macros
which makes the usage a bit more user friendly than the default
\LaTeX{} list macros.   With an optional argument to the
\verb+\begin{enumerate}+ command, you can change the list counter
type and its attributes.

\begin{vquote}
 \begin{enumerate}[1.]
 \item The enumerate environment starts with an optional
   argument `1.', so that the item counter will be suffixed
   by a period.
 \item You can use `a)' for alphabetical counter and '(i)' for
   roman counter.
  \begin{enumerate}[a)]
    \item Another level of list with alphabetical counter.
    \item One more item before we start another.
\end{vquote}

\deforange{blue!70}
\src{List -- Enumerate}
\includeclip{4}{1 1 453 185}{jfigs.pdf}
\deforange{orange}

Further, the enhanced list environment allows one to prefix a
string like `step' to all the item numbers.

\begin{vquote}
 \begin{enumerate}[Step 1.]
  \item This is the first step of the example list.
  \item Obviously this is the second step.
  \item The final step to wind up this example.
 \end{enumerate}
\end{vquote}

\deforange{blue!70}
\src{List -- enhanced}
\includeclip{5}{1 1 313 83}{jfigs.pdf}
\deforange{orange}

\section{Cross-references}
In electronic publications, articles may be internally
hyperlinked. Hyperlinks are generated from proper
cross-references in the article.  For example, the words
\textcolor{black!80}{Fig.~1} will never be more than simple text,
whereas the proper cross-reference \verb+\ref{tiger}+ may be
turned into a hyperlink to the figure itself:
\textcolor{blue}{Fig.~1}.  In the same way,
the words \textcolor{blue}{Ref.~[1]} will fail to turn into a
hyperlink; the proper cross-reference is \verb+\cite{Knuth96}+.
Cross-referencing is possible in \LaTeX{} for sections,
subsections, formulae, figures, tables, and literature
references.

\section[Mathematical ...]{Mathematical symbols and formulae}

Many physical/mathematical sciences authors require more
mathematical symbols than the few that are provided in standard
\LaTeX. A useful package for additional symbols is the
\file{amssymb} package, developed by the American Mathematical
Society. This package includes such oft-used symbols as
$\lesssim$ (\verb+\lesssim+), $\gtrsim$ (\verb+\gtrsim+)  or
$\hbar$ (\verb+\hbar+). Note that your \TeX{}
system should have the \file{msam} and \file{msbm} fonts installed. If
you need only a few symbols, such as $\Box$ (\verb+\Box+), you might try the
package \file{latexsym}.

Another point which would require authors' attention is the
breaking up of long equations.  When you use
\file{elsarticle.cls} for formatting your submissions in the
\verb+preprint+ mode, the document is formatted in single column
style with a text width of 384pt or 5.3in.  When this document is
formatted for final print and if the journal happens to be a double column
journal, the text width will be reduced to 224pt at for 3+
double column and 5+ journals respectively. All the nifty
fine-tuning in equation breaking done by the author goes to waste in
such cases.  Therefore, authors are requested to check this
problem by typesetting their submissions in final format as well
just to see if their equations are broken at appropriate places,
by changing appropriate options in the document class loading
command, which is explained in section~\ref{sec:usage},
\nameref{sec:usage}. This allows authors to fix any equation breaking
problem before submission for publication.
\file{elsarticle.cls} supports formatting the author submission
in different types of final format.  This is further discussed in
section \ref{sec:final}, \nameref{sec:final}.

\enlargethispage*{\baselineskip}

\subsection*{Displayed equations and double column journals}

Many Elsevier journals print their text in two columns. Since
the preprint layout uses a larger line width than such columns,
the formulae are too wide for the line width in print. Here is an
example of an equation  (see equation 6) which is perfect in a
single column preprint format:

In normal course, articles are prepared and submitted in single column
format even if the final printed article will come in a double column
format journal. Here the problem is that when the article is typeset by
the typesetters for paginating and fit within the single column width,
they have to break the lengthy equations and align them properly. Even
if most of the tasks in preparing your proof is automated, the equation
breaking and aligning requires manual judgement, hence this task is manual.
When there comes a manual operation that area is error prone. Author
needs to check that equation pretty well.

However if authors themselves break the equation to the single column
width typesetters need not want to touch these area and the proof authors
get will be without any errors.

\setlength\Sep{6pt}
\src{See equation (6)}
\deforange{blue!70}
\includeclip{4}{105 500 500 700}{1psingleauthorgroup.pdf}
\deforange{orange}

\noindent When this document is typeset for publication in a
model 3+ journal with double columns, the equation will overlap
the second column text matter if the equation is not broken at
the appropriate location.

\vspace*{6pt}
\deforange{blue!70}
\src{See equation (6) overprints into second column}
\includeclip{3}{59 421 532 635}{elstest-3pd.pdf}
\deforange{orange}
\vspace*{6pt}

\noindent The typesetter will try to break the equation which
need not necessarily be to the liking of the author or as it
happens, typesetter's break point may be semantically incorrect.
Therefore, authors may check their submissions for the incidence
of such long equations and break the equations at the correct
places so that the final typeset copy will be as they wish.

\section{Bibliography}

Three bibliographic style files (\verb+*.bst+) are provided ---
\file{elsarticle-num.bst}, \file{elsarticle-num-names.bst} and
\file{elsarticle-harv.bst} --- the first one can be used for the
numbered scheme, second one for numbered with new options of
\file{natbib.sty}. The third one is for the author year
scheme.

In \LaTeX{} literature, references are listed in the
\verb+thebibliography+ environment.  Each reference is a
\verb+\bibitem+ and each \verb+\bibitem+ is identified by a label,
by which it can be cited in the text:

\verb+\bibitem[Elson et al.(1996)]{ESG96}+ is cited as
\verb+\citet{ESG96}+.

\noindent In connection with cross-referencing and
possible future hyperlinking it is not a good idea to collect
more that one literature item in one \verb+\bibitem+.  The
so-called Harvard or author-year style of referencing is enabled
by the \LaTeX{} package \file{natbib}. With this package the
literature can be cited as follows:

\begin{enumerate}[\textbullet]
\item Parenthetical: \verb+\citep{WB96}+ produces (Wettig \& Brown, 1996).
\item Textual: \verb+\citet{ESG96}+ produces Elson et al. (1996).
\item An affix and part of a reference:
\verb+\citep[e.g.][Ch. 2]{Gea97}+ produces (e.g. Governato et
al., 1997, Ch. 2).
\end{enumerate}

In the numbered scheme of citation, \verb+\cite{<label>}+ is used,
since \verb+\citep+ or \verb+\citet+ has no relevance in the numbered
scheme.  \file{natbib} package is loaded by \file{elsarticle} with
\verb+numbers+ as default option.  You can change this to author-year
or harvard scheme by adding option \verb+authoryear+ in the class
loading command.  If you want to use more options of the \file{natbib}
package, you can do so with the \verb+\biboptions+ command, which is
described in the section \ref{sec:usage}, \nameref{sec:usage}.  For
details of various options of the \file{natbib} package, please take a
look at the \file{natbib} documentation, which is part of any standard
\LaTeX{} installation.

In addition to the above standard \verb+.bst+ files, there are 10
journal-specific \verb+.bst+ files also available.
Instruction for using these \verb+.bst+ files can be found at
\href{http://support.stmdocs.in/wiki/index.php?title=Model-wise_bibliographic_style_files}
{http://support.stmdocs.in}

\section[Graphical ...]{Graphical abstract and highlights}
A template for adding graphical abstract and highlights are available
now. This will appear as the first two pages of the PDF before the
article content begins.

\pagebreak
Please refer below to see how to code them.

\begin{vquote}
....
....

\end{abstract}

\begin{graphicalabstract}
\end{graphicalabstract}

\begin{highlights}
\item Research highlight 1
\item Research highlight 2
\end{highlights}

\begin{keyword}
....
....
\end{vquote}

\section{Final print}\label{sec:final}

The authors can format their submission to the page size and margins
of their preferred journal.  \file{elsarticle} provides four
class options for the same. But it does not mean that using these
options you can emulate the exact page layout of the final print copy.

\lmrgn=3em
\begin{description}
\item [\texttt{1p}:] $1+$ journals with a text area of
384pt $\times$ 562pt or 13.5cm $\times$ 19.75cm or 5.3in $\times$
7.78in, single column style only.

\item [\texttt{3p}:] $3+$ journals with a text area of 468pt
$\times$ 622pt or 16.45cm $\times$ 21.9cm or 6.5in $\times$
8.6in, single column style.

\item [\texttt{twocolumn}:] should be used along with 3p option if the
journal is $3+$ with the same text area as above, but double column
style.

\item [\texttt{5p}:] $5+$ with text area of 522pt $\times$
682pt or 18.35cm $\times$ 24cm or 7.22in $\times$ 9.45in,
double column style only.
\end{description}

Following pages have the clippings of different parts of
the title page of different journal models typeset in final
format.

Model $1+$ and $3+$  will have the same look and
feel in the typeset copy when presented in this document. That is
also the case with the double column $3+$ and $5+$ journal article
pages. The only difference will be wider text width of
higher models.  Here are the specimen single and double column journal
pages.

\begin{comment}
\begin{center}
\hypertarget{bsc}{}
\hyperlink{sc}{
{\bf [Specimen single column article -- Click here]}
}

\hypertarget{bsc}{}
\hyperlink{dc}{
{\bf [Specimen double column article -- Click here]}
}
\end{center}
\end{comment}

\vspace*{-.5pc}

\enlargethispage*{\baselineskip}

\src{}\hypertarget{sc}{}
\deforange{blue!70}
\hyperlink{bsc}{\includeclip{1}{88 120 514 724}{elstest-1p.pdf}}
\deforange{orange}

\src{}\hypertarget{dc}{}
\deforange{blue!70}
\hyperlink{bsc}{\includeclip{1}{27 61 562 758}{elstest-5p.pdf}}
\deforange{orange}

~\hfill $\Box$

\end{document}

\subsection{Segmentation architecture}
Segmentation tasks such as semantic or panoptic segmentation require a specific architecture from the simpler parcel classification problem. We integrate our fusion scheme into the recent U-Net with Temporal Attention Encoder (U-TAE)  \cite{garnot2021utae}, which generalize the L-TAE approach to learning multi-scale pixel features which can be used in turn for semantic or panoptic segmentation.
In this section, we drop the assumption that the patch covered by $X$ contains necessarily only one parcel. Our goal is to produce a set of feature maps at different resolutions.

\paragraph{U-TAE architecture.} The U-TAE architecture for encoding a single modality image time series follows a five-step procedure:
\begin{itemize}
    \item[(i)] \textbf{Spatial Encoding.} Compute for each image independently a set of feature maps of decreasing spatial resolution with a shared multi-stage convolutional encoder. The resulting maps are concatenated in temporal dimension into a sequence of feature maps for each resolution.
    \item[(ii)] \textbf{Temporal Encoding.} Run a L-TAE network with $G$ heads for each pixel of the lowest resolution sequence independently. This produces $G$ low-resolution temporal attention maps, encoding which acquisitions contain the most relevant information for each head.
    \item[(iii)] \textbf{Attention Up-sampling.} Up-sample the attention maps to the resolution of all feature maps up to full resolution using bilinear interpolation. Each resolution level now has $G$ temporal attention maps associating to all pixels of all acquisitions their importance in the time series.
    \item[(iv)] \textbf{Temporal Averaging.} Compute the temporal average of each pixel across the image time series using the temporal attention maps as weights. The attention of a given head attends to a dedicated group of channels defined in advance. This results in a single feature map for each resolution: the \emph{innermost map} and the \emph{skip connections}.
    \item[(v)] \textbf{Spatial Decoding.}  A convolutional decoder takes the resulting feature maps and combines them in a classic U-Net fashion to produce refined feature maps at all resolutions.
\end{itemize}
We also refer the reader to the \cite[Sec.~3.1]{garnot2021utae} for more details on the design choices of this algorithm.
The next paragraph explains how each of these five steps should be modified to accommodate multi-modal image time series.
The first two steps can be adapted straightforwardly in the same manner than explained in the previous section: we spatially encode all modalities independently and merge the resulting feature sequences at the innermost level.
However, the next steps require special attention. Indeed, while radar images contain precious and cloud-robust information for determining the content of parcels, their high noise levels make them ill-suited to derive high-resolution information.
For this reason, we add two mechanisms to help prevent unwanted noisy information corrupting the high resolution feature maps: resolution-specific modality gating (RSMG) and modality-specific temporal attention maps (MSTAM).

For simplicity's sake, the subsequent explanation and equations are given for a L-TAE with a single head $G=1$. The equations for an arbitrary number of heads can be found in our appendix.

\paragraph{Spatial Encoding.} In a similar fashion to the parcel classification approach, each modality $m$ has its own dedicated multi-stage convolutional encoder $\spatial\m$. Such network is composed of $L$ levels $\spatial^{m,1},\cdots,\spatial^{m,L}$ producing feature maps $e^{m,l}$ with $F\m_l$ channels and of resolution $H_l=H/2^l$ and $W_l=W/2^l$ for $l\in[1,L]$. The maps are computed in parallel \wrt modalities and time and sequentially \wrt resolution levels:
\begin{align}
    e^{m,l}_t = \spatial^{m,1} \left(e^{m,l-1}_t \right), \forall m \in [1,M],\, t \in [1,T_m],\, l \in [1,L]~,
\end{align}
with $e^{m,0}_t=X\m,t$. The resulting maps at each level are concatenated temporally into a set of temporal sequences of feature maps $e^{m,l}$ of size $T\m \times F\m_l \times H_l \times W_l$.

\paragraph{Temporal Encoding.} Each pixel $h,w \in [1, H_L] \times [1,W_L]$ of the innermost sequence of feature maps is associated with $M$ time series $e^{m,l}_{h,w}$ of shape $T\m \times F\m_l$ for each modality $m$. Similarly to the parcel classification approach, we merge the different time series of spatial features into a single one to be processed with a L-TAE network.
This network produces a temporal sequence of \emph{key-query compatibility} for each pixel and each modality, which can be rearranged into $M$ compatibility maps $z^{m,L}$ of size $T_m \times H_L \times W_L$.

The compatibility maps are used to compute modality-specific attention masks for each pixels
 $T^\star \times H_L \times W_L$

\paragraph{Methods Segmentation WIP}

\begin{align}
    e^{m,l} &= [\mathcal{E}_l^m(e^{m, l-1}_t)]_{t=0}^{T}\;\;\text{for}\;\; l \in [1, L]\\
    e^L &= \text{merge}^{(T)}\left\{ e^{m,L}\right\}_{m=0}^M \;, T^* = \sum_m T_m\\
    &z^{L,1}, \cdots, z^{L,G} = \text{LTAE}(e^L)\;\text{, applied pixelwise.} \\
    &a^{L,1}, \cdots, a^{L,G} = \text{softmax}_{T^*}(z^{L,1}), \cdots, \text{softmax}_{T^*}(z^{L,G}) \\
    &f^{L} = \text{Conv}_{1 \times 1}^L\left(\left[\sum_{t=1}^{T^*}a^{L,g}_t \odot e^{L,g}_t\right]_{g=1}^G\right)~\\
    &a^{m,l,1}, \cdots, a^{m,l,G} = \text{softmax}_{T_m}(z^{l,1}), \cdots, \text{softmax}_{T_m}(z^{l,G}) \\
    &a^{m,l,g} = \text{resize}\; a^{m,L,g} \;\text{to} \; H_l \times W_l\\
    &f^{l} = \sum_m \alpha_{m,l} * \text{Conv}_{1 \times 1}^{m,l}\left(\left[\sum_{t=1}^{T_m}a^{m,l,g}_t \odot e^{m,l,g}_t\right]_{g=1}^G\right)~\\
    d^{l}  &= \mathcal{D}_l([ \mathcal{D}^{\text{up}}_l(d^{l+1}), f^{l}]) \;\text{for}\; l \in [1, L-1]\\
\end{align}

\paragraph{Attention Up-sampling.}

start from sota architecture U-TAE , also based on l-TAE in a U-Net like architecture.

redpajama_set_name: RedPajamaArXiv


After working tirelessly to build his empire over nearly seven decades, 89-year-old Hong Kong tycoon Li Ka-Shing announced his retirement last Friday. He will be stepping down as chairman of Hong Kong conglomerate CK Hutchison Holdings Ltd. and property giant CK Asset Holdings Ltd. after the annual general meeting on May 10.
Li will continue to serve as senior advisor to the group after his retirement, and his eldest son Victor Li is expected to take over as chairman.
Perhaps Li's business acumen and charisma can be attributed to his zodiac sign, the dragon, which is typically associated with ambitious personalities. True to form, Li grew his personal fortune from a young age as a skilled salesman, outearning his colleagues and impressing his superiors. Li is also a Leo, which could explain his bold, trail-blazing persona.
Spotify. Facebook. Airbnb. Siri. Google's DeepMind. Li has invested in all of them and is a fan of game-changing innovation. His latest investments include a startup looking to replace eggs with plant substitutes. Li even founded Li Ka Shing Foundation to invest in disruptive technologies that will benefit the education and healthcare sectors, calling the foundation his "third son".
During WW2, Li was forced to leave school before he was 16 years old in order to support his family after his father passed away. Today, he is a benefactor of several schools around the world including in New Zealand and Israel.
In the late 1990s, Hong Kong's high profile families were often targets for kidnappings and Li's family was no exception. In 1996, the infamous Chinese gangster Cheung Tze-keung, nicknamed "Big Spender", kidnapped Li's son at gunpoint, demanding HK$1 billion in ransom. According to Li, after Cheung received his payment, he called Li for investment tips.
With an estimated net worth of US$35.4 billion, it may come as a surprise that Li's trademark accessory for several decades was a US$50 Citizen electric watch. Less than one week into retirement, the 89-year old shared in his retirement speech that he is most grateful for his good health.
Li Ka-Shing His storied business prowess with Hutchison spans property, retail and telecommunications.


text:
Jaden Smith's Batman Song Is The Weirdest Thing You Will Hear Today
There are some things you have'ta see but wish you could unsee. Jaden Smith's Batman is one of those things. Will Smith's offspring channels Bruce Wayne in a new music video. It's all sorts of strange.
"Batman, Batman, Batman, Joker just put me on acid…"
"Batman, Batman, Batman, I heard you ain't got a mattress.."
"Batman, Batman, Batman, I heard you married a Porsche…"
These are just three lines from the bizarre Batman song by Jaden Smith. Dressed in a white Batman suit, the one he wore to prom and Kanye West and Kim Kardashian's wedding, he raps and fights against Spider-Man and Superman somewhere in Hollywood Blvd.
What Is Happening with the New Batman Video Game?
Zdravstvuyte! Superman Is Russian In Superman: Red Son Trailer
It's Michael Keaton Versus Christopher Reeve In This Fan-Made Batman v Superman Trailer
Who Would Win In A Fight Between Ben Affleck's Batman And Christian Bale's Dark Knight?
Detective Comics #1015 Review – Great Twists And Turns
When asked why he wore the suit to the celebrity wedding, he replied, "I felt as though I needed to protect everyone there and needed to have the proper gear to do so."
The music video was directed and edited by Moises Arias, most famously known as Rico from Hannah Montana. It currently has 135,383 views, with 1834 likes and 1443 dislikes (make that 1444).
https://www.youtube.com/watch?v=A4hLVnL3DnM
Tags: BatmanMusic
Shake It Baby! Duke Nukem Is Getting A Movie
The Conjuring Universe Is Expanding With A Crooked Man Spin-Off
Jacques Joubert says:
And then daddy said "it's all guud son, this will be great!" Hahahahahaha
Domnique Eliza Zeiler says:
Why though??? Good grief. That dude will do anything to be "different" lol
Ross McEwan says:
Ha ha ha he's clearly lost the plot, poor Will probably shook his head when he saw this!!
Ashley Sams says:
What is this hell….
Linda Nel says:
Nick Mckay says:
Andrew Lapierre
Chris Pantz says:
Great. Batman was my favorite an now i want nothing to do with him JUS cuz of this garbage
Ahmed Jaffer says:
Ahmed-Faraaz Sparx Parker Muhammad Sanglay
Michael Bell says:
Jay Amar Gounder says:
WTF is this sh!t!? My eyes!! My ears!! AAARRRGG!!! Lol
Danny Rains says:
It sounds like every other piece of over-produced, high hat saturated "urban" bullshit that's released these days.
Seija Mistiaen says:
why did I click, why did I watched this, why did I ignored the warnings ….? damn you internet!!!!!
ok grandpa
I guess? I appreciate the days when musicians actually played real instruments that exist in the real world and not at the behest of a few mouse clicks.
This kid is a talentless total fucking weirdo. He's only given any attention because of his daddy's fame in an era where Janis Joplin would be laughed out of any record label because she's "ugly".
technology is its own instrument man, rejecting it is like people who rejected electronic keyboards in the 70's
you can't fight the future, all you can do is embrace it
Naw, im good. I'm happy to be the "grandpa" who listens to classic rock and blues :)
nothing wrong with that, but as Beck said, "you can't write if you can't relate…."
I'm not sure I could ever relate to an out of touch millionaire spoiled brat with schizophrenia, heh.
I think what he was saying is no one cares about what you create if it's not relatable to your audience
J Baas says:
It's like a car accident! I know it's completely awful but I can't help but to watch!
I now need therapy to have this erased from
my mind!
Kenneth Cortez Halcromb says:
Disturbing.
I gotcha man, and agree.
Deborah Dodier Wells says:
I love Batman too much to watch this. Thanks for the heads up (you can't unsee this).
Thomas Groulx says:
Vincent Hébert master bruce
tade says:
why did this make me feel bad, it is disturbing as hell

redpajama_set_name: RedPajamaCommonCrawl