text:
\section{Introduction}

Low Mass X-ray Binaries (LMXB) are binary systems in which one of the
two stars is a neutron star (NS) with low magnetic field ($< 10^9
\mathrm{Gauss}$) which accretes matter from a low-mass ($< 1 \,
M_\odot$) companion star. According to the so-called recycling
scenario \citep[see for a review][]{Bhatta_91} millisecond radio
pulsars originate from LMXBs, where the accretion torques and the
relatively weak magnetic fields are able to spin up the NSs up to
millisecond periods. When the companion star stops transferring matter
to the NS, the NS can switch on as millisecond radio pulsar.

A striking confirmation of this scenario was the discovery in 1998 of
millisecond X-ray pulsars in transient LMXBs. The first LMXB observed
to show coherent pulsations at a frequency of $\sim 400$ Hz was the
well studied SAX~J1808.4--3658\ \citep{Wijnands_98, Chakra_98}.  Due to the weak
magnetic field of these sources, the chance to see a pulsed emission
from a LMXB is quite low. However, to date 8 LMXBs were discovered to
be accreting millisecond pulsars \citep{Wijnands_05}, and all of them
are in transient systems. They spend most of the time in a quiescent
state, with very low luminosities (of the order of $10^{31} - 10^{32}$
ergs/s) and rarely we go into an X-ray outburst with luminosities in
the range $10^{36} - 10^{37}$ ergs/s.  Indeed, of all these sources
only SAX~J1808.4--3658, which shows more or less regular outbursts every two
years, has been observed in outburst more than once with RXTE. All the
other sources have shown just one outburst in the RXTE era.

This fact makes the study of the timing properties of these sources
even more difficult, given that the duration of the observations is
not a matter of choice, but is conditioned by the duration of the
outbursts which, in turns, puts a constrain on the precision of the
parameters that we can derive. And this is also the reason why we have
to obtain all the information and the precision of the parameters we
need just using the available data.  In the case of accreting
millisecond pulsars, among the parameters of interest there are, of
course, the timing parameters, that are the orbital parameters and the
spin parameters. The orbital parameters can give us important
information on the binary system, its evolution, and even on the
nature (e.g. degenerate or not) of the companion star.  Also, a
precise orbital solution will be important for a precise determination
of the spin parameters, the spin period evolution and the accretion
torques acting onto the NS.

As already mentioned above, the knowledge with the maximum possible
precision of the orbital parameters is of fundamental importance in
itself and for a successive study of the spin and the spin variations.
The study of the frequency Doppler shift due to orbital motion of a
millisecond pulsar in a binary system is the first step to obtain an
estimate of the set of orbital parameters.  To refine this estimate
the next step is the study of the pulse phase shifts in order to
obtain differential corrections to the orbital parameters and
therefore a finer orbital solution. However, in some cases, not all
the data in which the coherent X-ray pulsations are visible can be easily
used to obtain the differential corrections. The pulse phase shifts
are frequently affected by intrinsic long-term variations and/or
fluctuations (probably caused by the accretion torques) which are
superimposed to the modulation due to the orbital motion of the
source, making the fit of the residual sinusoidal modulation much more
complicated. Clear examples of these complex behaviors of the pulse phase
shifts in accreting millisecond pulsars can be found in
\citet{Burderi_06}, who analyze SAX~J1808.4--3658\ and find a big jump in the
pulse phase shifts of the fundamental harmonic of the pulse, and in
\citet{Papitto_07}, who analyze XTE~J1814--338 finding a modulation of
the pulse phase shifts, anti-correlated to the X-ray flux, superposed
on a general spin-down trend.

Of course, the presence of non negligible long-term variations of the
pulse phase shifts with time, make it very difficult to fit a long
dataset with differential correction as reported in
Eq.\ref{eq:phase_orb_doppler} in order to obtain a precise estimate of
the orbital parameters using all the available time-span. Using the
classical technique it is then necessary, in order to decouple the
orbital modulation from the proper fluctuations of the pulse phases,
to take into account
the latter in some way.  This is often impossible to obtain by fitting
with a simple model, due to the observed complex behaviors and/or our poor
knowledge of the physics of the accretion torques.  In such cases we
are forced to fit differential corrections of the orbital parameters
on restricted time intervals, in which the proper
fluctuations or variations of the phase shifts can be safely
approximated with a simple model, e.g.\ a parabola. In these cases,
therefore, the precision of the orbital solution is limited by
the limited used timespan or by our ability to model intrinsic
phase variations.

In this paper we describe a simple method which permits, under certain
conditions, to remove from the pulse phase shifts all the effects not
due to differential orbital parameters corrections.  We apply this
method to the source XTE~J1807--294, obtaining for the first time a complete
set of orbital parameters with a precision at least one order of
magnitude higher with respect to the previously available orbital
solution.

\section{Observations}

The millisecond X-ray pulsar XTE~J1807--294~was spotted by RXTE on February
21$^{st}$, 2003 \citep{Markward_atel_03a}. The source was observed
with PCA (Proportional Counter Array) and HEXTE (High Energy X-ray
Timing Experiment), the principal instruments on-board of RXTE
\citep{Jahoda_96}, from February 28 to June 22, 2003. XTE~J1807--294~was also
observed with other satellites such as XMM-Newton \citep{Campana_03,
Kirsch_04, Falanga_05}, Chandra \citep{Markwardt_iauc_03b} and
Integral \citep{Campana_03}. No optical or radio counterpart has been
reported.  \citet{Linares_05} have reported the presence of twin kHz
QPOs analyzing RXTE observation.

In literature several attempts have been done in order to derive the
orbital parameters of this source. \citet{Markward_atel_03a} give only
source position and the orbital period. \citet{Kirsch_03}, analyzing an
XMM-Newton observation during the outburst, give an estimate of the
semi-mayor axis. The first complete set of orbital parameters was
reported by \citet{Campana_03}, and successively by \citet{Kirsch_04}
and by \citet{Falanga_05}, analyzing the same XMM-Newton observation.
All these authors assumed as orbital period the period reported by
\citet{Markward_atel_03a}.

Here we analyze all the archival RXTE observations of this source available
to date.  In particular, we use data from the PCA (proportional counter array)
instrument on board of the satellite RXTE. We use data collected in
GoodXenon packing mode, with maximum time and energy resolution
(respectively $1 \mu s$ and 256 energy channels). In order to improve
the signal to noise ratio we select photon events from PCUs top layer
and in the energy range 3-13 keV. Using the \textit{faxbary} tool
(DE-405 solar system ephemeris) we corrected the photon arrival times
for the motion of the earth-spacecraft system and reported them to
barycentric dynamical times at the Solar System barycenter. We use the
source position reported by \cite{Markwardt_iauc_03b} using the
\textit{Chandra} observation of the same outburst.

In order to test the goodness of the available orbital solution, we
correct the photon arrival times with the formula:
\begin{equation}
  \label{eq:time_orb_doppler}
  t_{em} \simeq t_{arr} - A \qpth{\Sin{m(t_{arr}) + \omega} +
    \frac{\varepsilon}{2} \Sin{2m(t_{arr}) + \omega} -
    \frac{3\varepsilon}{2}\Sin{\omega}},
\end{equation}
where $t_{em}$ is the photon emission time, $t_{arr}$ is the photon
arrival time, $A$ the projected semi-major axis in light seconds,
$m(t_{arr}) = 2\pi (t_{arr} - T^\star)/P_{orb}$ is the mean anomaly,
$P_{orb}$ the orbital period, $T^\star$ is the time of ascending node
passage, $\omega$ is the periastron angle and $\varepsilon$ the
eccentricity. We used the orbital parameters reported by
\cite{Kirsch_04}, adopting an eccentricity $\varepsilon = 0$ (see
Tab.~\ref{table1} for details).

We divided the whole observation in time intervals of 1/6 $P_{orb}$
length each and epoch-folded each of these data intervals with respect
to the spin period we reported in Tab.~\ref{table1}. The pulse phase
delays are obtained fitting each pulse profile with two sinusoidal
components, since higher-order harmonics were detectable in the folded
light curve.  We fixed the period of the sinusoids to 1 and 0.5 times
the spin period, respectively, and we used the phase of the
fundamental harmonic to infer the pulse phase shifts. In
Fig.~\ref{fig:fig1} we show the pulse phase delays obtained in this
way, where we have plotted only the pulse phase delays corresponding
to the folded light curves for which the statistical significance for
the presence of the X-ray pulsation was $> 3\sigma$.

In Fig.~\ref{fig:fig1} a residual orbital modulation it is clearly
visible, superimposed to an intrinsic long-term variation of the
phases, possibly similar to the erratic spin changes mentioned by
\cite{Markwardt_04}.
It can be seen that this behavior has characteristic time scales of the
order of several $P_{orb}$. In the next section we describe a method able
to temporarily eliminate any long-term phase variations or fluctuations,
in order to easily fit the residual modulations of the phases at the
orbital period and to find a revised, more precise orbital solution.

\section{Differential corrections of the orbital parameters}

We propose here a simple method of analysis which allows to (temporarily)
eliminate, or at least strongly reduce, the long-term variation and
erratic behavior of the pulse phase shifts in order to derive a precise
orbital solution.
The residuals in the phase delays due to a non-perfectly corrected
orbital parameters is given by the expression:
\begin{eqnarray}
  \label{eq:phase_orb_doppler}
  \phi_{orb}(t) = P_{spin}^{-1} ( \quad & &\nonumber \\
  (\Sin{m(t) + \omega} + \frac{\varepsilon}{2} \Sin{2m(t) + \omega} - \frac{3\varepsilon}{2}\Sin{\omega} ) &dA \; &- \nonumber\\
  \frac{2\pi A}{P_{orb}} (\Cos{m(t) + \omega} + \varepsilon \Cos{2m(t) + \omega}) &dT^\star &- \nonumber\\
  \frac{m(t) A}{P_{orb}} (\Cos{m(t) + \omega} + \varepsilon \Cos{2m(t) + \omega}) &dP_{orb} &+ \nonumber\\
  A (\frac{1}{2} \Sin{2m(t) + \omega} - \frac{3}{2}\Sin{\omega}) &d\varepsilon &+ \nonumber\\
  (A \Cos{m(t) + \omega} + \frac{\varepsilon}{2}
  \Cos{2m(t) + \omega} - \frac{3\varepsilon}{2}\Cos{\omega}) &d\omega &)
\end{eqnarray}
where $P_{spin}$ is the spin period with respect to the light curves
are folded and $dA$, $dT^\star$, $dP_{orb}$, $d\varepsilon$, and $d\omega$
are the differential corrections to the orbital parameters (the
projected semi-major axis, the time of ascending node passage, the
orbital period, the eccentricity, and the periastron angle, respectively).

If we compute difference between the phase of two adjacent folded
light curves we obtain, for $\Delta\phi_{orb}(t_i)$, the expression:
\begin{eqnarray}
  \label{eq:orb_phase_diff}
  \Delta\phi_{orb}(t_i) = \phi_{orb}(t_{i+1}) - \phi_{orb}(t_i) = P_{spin}^{-1} (\quad & &\nonumber \\
  (\;2\Cos{m_i + \omega + m_\Delta/2}\Sin{m_\Delta/2} + \varepsilon \Cos{2m_i + \omega + m_\Delta} \Sin{m_\Delta}\;) & dA &+ \nonumber  \\
  \frac{4\pi A}{P_{orb}} (\;\Sin{m_i + \omega + m_\Delta/2}\Sin{m_\Delta/2} + \varepsilon \Sin{2m_i + \omega + m_\Delta} \Sin{m_\Delta}\;) & dT^\star  &+ \nonumber \\
  (\;\frac{2 m_i A}{P_{orb}} (\Sin{m_i + \omega + m_\Delta/2}\Sin{m_\Delta/2} + \varepsilon \Sin{2m_i + \omega + m_\Delta} \Sin{m_\Delta}) - & & \nonumber \\
  \quad\frac{m_\Delta A}{P_{orb}} (\Cos{m_i + m_\Delta + \omega} + \varepsilon \Cos{2m_i + 2m_\Delta + \omega})\;) &dP_{orb} &+ \nonumber \\
  A \Cos{2m_i + \omega + m_\Delta}\Sin{m_\Delta} & d\varepsilon &- \nonumber \\
  (\;2\Sin{m_i + \omega + m_\Delta/2}\Sin{m_\Delta/2} +
  \varepsilon \Sin{2m_i + \omega + m_\Delta} \Sin{m_\Delta}\;) & d\omega &\;\;)
\end{eqnarray}
where we pose for simplicity $m(t_i) = m_i$ and $2\pi \Delta t / P_{orb} =
m_{i+1} - m_i = m_\Delta$.

In this way, that is calculating the phase differences between two
consecutive intervals instead of the phases, we apply a linear
filter to the pulse phase delays, for which we illustrate the
fundamental properties. We now use the term input to indicate the
original signal, that is the pulse phase delays vs.\ time, and output
to indicate the signal we obtain plotting the phase difference of each
interval with the following vs.\ time. When the input signal is a
sinusoid of period $P$, the output is another sinusoid with same
period but with different phase and amplitude. In Fig.~\ref{fig:fig2}
we report the gain $G$, that is the ratio of the amplitudes of the
output to the input signal, for a sinusoidal signal of period $P$.
The analytical expression for $G$ is: $G = 2\Sin{\pi\,\Delta t/ P}$.  As
can be seen in Fig.~\ref{fig:fig2}, $G$ has the maximum for $P = 2\Delta t$
coincident with the Nyquist frequency. For $P >> \Delta t$ we have $G
\propto P^{-1}$. This filter is then a band-pass filter, limited at
high frequencies by the Nyquist frequency and at low frequencies we
can fix a limit at the period $P \simeq 12 \Delta t$, at which the
amplitude is reduced to a half.  For period of $P \simeq 60 \Delta t$ the
amplitude is reduced by a factor ten.

In particular for the case of XTE~J1807--294, instead of
plotting the obtained pulse phases as a function of time, we consider
the phase difference between each interval and the following one,
$\Delta\phi(t_i) = \phi(t_{i+1}) - \phi(t_i)$, in the hypothesis that
for each $i$ we have $t_{i+1} - t_i = \Delta t$, where $\Delta t$ is constant
during all the observation. In this way we obtain the phase shifts
shown in Fig.~\ref{fig:fig3} (the same of Fig.~\ref{fig:fig1} but plotting
the phase differences instead of the phases), where, as it is easy to see,
the orbital modulation is still visible, but any long-term variation of
the pulse phases is completely smoothed out. To produce this figure we
divided each pulse profile in 6 time bins (in other words we chose
$\Delta t = P_{orb} / 6$) in order to maximize the signal to noise ratio.
In fact, in this case the effect of the filter does not change the amplitude
of the orbital modulation of the output with respect to the input
(from Fig.~\ref{fig:fig2} the Gain = 1 for $P_{orb} / \Delta t = 6$).

Due to its linearity the application of this filter to a signal which
is the sum of several signals is equal to the sum of each filtered
signal. We can then analyze separately the response to the filter of
the Doppler shift due to the orbital motion without fear that the
erratic behavior of the source can alter the result.
We note that, in cases like the one considered in this paper, where
the orbital period is much shorter than long term variations of the
phases (probably caused by accretion torques onto the neutron star), the
phase variations induced by the orbital modulation can be studied
independently of the phase variations induces by the spin evolution
(see e.g. \citet{Burderi_07}, \cite{Papitto_07}). Therefore, our analysis does
not introduce any error or approximation in the determination of the
orbital solution.

\section{Results and Discussion}

We have applied the technique described above to the PCA data of
XTE~J1807--294. In particular, we have used the phase delays of
Fig.~\ref{fig:fig1} in order to calculate for each time interval the
phase difference with respect to the following interval, and these are
plotted vs.\ time in Fig.~\ref{fig:fig3}. We consider only phase
differences between contiguous time intervals and exclude the phase
differences between interval separated by gaps in time. The errors on
the phase differences are propagated summing in quadrature the errors
on the phases from which the difference is calculated, that is
$\sigma_{\Delta\phi(t_i)}^2 = \sigma_{\phi(t_{i+1})}^2 +
\sigma_{\phi(t_i)}^2$.  From the figure it is apparent that the long
term variation and the erratic behavior of the phase delays is now
completely smoothed out. We can therefore proceed to fit with
Eq.~\ref{eq:orb_phase_diff} these phase differences over the whole
period in which the coherent pulsation was detectable (about 90 days).
In this way we
obtain a very good fit of the data. To show the goodness of the fit we
plot in Fig.~\ref{fig:fig4} (top panel) the phase differences between days
10 and 11 from the start of the outburst; the dashed line is the best fit
sinusoidal modulation obtained from Eq.~\ref{eq:orb_phase_diff}. In the
bottom panel of the figure we show the post-fit residuals with respect to
the best fit sinusoidal modulation.

Eq.~\ref{eq:orb_phase_diff} is essentially a sum of sinusoidal terms
with period equal to $P_{orb}$ and $P_{orb}/2$. The latter are due only to
the eccentricity. Then, to test if the orbit shows an eccentricity we
epoch folded the light curves on a time interval $\Delta t = P_{orb}/10$; this
reduces by about 40\% the Gain of the filter but gives the possibility
to have a sufficient number of points to sample each period in order to
avoid aliasing phenomena. In fact, if we use, as before, intervals of
length $\Delta t = P_{orb}/6$ this means that we sample the modulation at
$P_{orb}/2$ (eventually due to a non negligible eccentricity) with only
three points, and this can produce ambiguities in the results of the
fit. Using instead time intervals of length $\Delta t = P_{orb}/10$, we
sample the modulation with period $P_{orb}$ with 10 points and the
modulation with period $P_{orb}/2$ with 5 points, which is, as we have
verified, a good compromise to get precise estimates of all the
orbital parameters.

To check that the long-term trend visible in Fig.~\ref{fig:fig1} has
been indeed eliminated by the technique described above we add to the
best-fit sinusoid a parabolic function to describe possible residuals
caused by the long-term phase variations. Hence we fit the phase
differences with the expression:
\begin{equation}
  \label{eq:fit_formula}
  \Delta\phi(t) = a + b\,t + c\,t^2 + \Delta\phi_{orb}(t),
\end{equation}
where $a$, $b$ and $c$ are the coefficients of the
parabola. These coefficient can be expressed in terms of
$\Delta\nu$, $\dot{\nu}$ and $\ddot{\nu}$, respectively, since
$a \simeq -\Delta\nu \;
\Delta t$, $b \simeq -\dot{\nu}/2 \; \Delta t$ and $c \simeq -
\ddot{\nu} / 3 \; \Delta t$. There is no evidence of residuals due
to the long-term behavior, and in the fit the $a$, $b$ and $c$
parameters result largely compatible with zero. This is due to
two factors: the first is that both $\Delta\nu$ and $\dot{\nu}$ are
attenuated by a factor $\Delta t\; (\simeq 5 \times 10^{-3})$ s, the
second is that the filter reduces the time dependence on these terms.
Moreover the orbit does not show an appreciable eccentricity,
$\varepsilon$, for which we find an upper limit (at 95\% c.l.) of $3.6
\times 10^{-3}$.  We also find that $dT^\star$ and $d\omega$ result
perfectly correlated, as expected for a circular orbit.

Due to these results we can safely make two assumptions: 1) the orbit
is circular; 2) we can safely describe the residuals simply with a
constant. We therefore epoch folded the light curves on a time interval
$\Delta t = P_{orb}/6$ in order to have a better statistics, and fitted the
phase differences with the simpler formula:
\begin{equation}
  \label{eq:fit_simp_formula}
  \Delta\phi(t) = a + \Delta\phi_{orb}(t),
\end{equation}
where we fixed $d\varepsilon = d\omega = 0$.  We iterate this process
until no residual are observed. In this way we find a good fit,
corresponding to a $\chi^2 / \textit{d.o.f.}$ of $864.2 / 790$;
the best fit parameters are reported in Tab.~\ref{table1}.
In Fig.~\ref{fig:fig5}) we show the phase differences obtained correcting
the time series with our best-fit orbital solution. No orbital modulation
is visible in this plot, and the amplitude of the oscillation is now much
reduced with respect to that visible in Fig.~\ref{fig:fig3} corresponding
to the orbital solution given by \citet{Kirsch_04}.

To verify that our orbital solution is indeed better than the orbital
solution given by \citet{Kirsch_04} even during the times of the XMM
observation we performed the following check.
We looked in the RXTE observations for a time interval close in time to
the time of the XMM observation; unfortunately there is not a complete
superposition between the RXTE and the XMM observation (that starts
at 52720.57 MJD and stops at 52720.68, for an exposure time of about
9.3 ks). We therefore took two RXTE observations ($80145-01-04-08$ and
$80145-01-05-01$, the closest continuous observations to the XMM observation,
which cover the time interval from 52718.94 MJD to 52719.11 MJD), corrected
them alternatively with the Kirsch solution and our solution, respectively,
and then performed a folding search around the expected value of the spin
period. While the Kirsch solution gives a  peak in the $\chi^2$ curve of
about $150 - 200$, our solution gives a peak in the $\chi^2$ curve
of about 1000, demonstrating that the periodic signal revealed on the
time series corrected with our orbital solution is much stronger even
in a time interval as close as possible to the XMM observation.

The method described above that we used to determine a precise orbital
solution for XTE~J1807--294\ does not allow any study of the spin frequency and
its derivative, since the long-term phase variations, which indeed give
information on possible variations of the spin frequency during the
outburst, are eliminated when we calculate the phase differences.
Therefore, in order to perform a timing study of the spin frequency,
we have to correct the entire time series of the RXTE observations
with our best-fit orbital solution, and ri-calculate the phase delays,
which are shown in Fig.~\ref{fig:fig6}.
As it can be seen from the figure, the strong sinusoidal modulation visible
in Fig.~\ref{fig:fig1} is no more present. Moreover, our more precise orbital
solution allows us to clearly detect the coherent pulsations up to 104
days since the start of the observation (June 12) with a detection
confidence level of $3.6\; \sigma$, while between May 26 and June 10,
although the source is still detectable, the pulsations are no more
significantly detected. Long-term variations (a parabolic trend on which
erratic fluctuations are superimposed) of the phases are again visible in
this figure and are probably determined by the presence of a spin frequency
derivative and/or an error in the spin frequency used to obtain the folded
pulse profile, as well as by fluctuations of the phases on shorter
timescales possibly related to phase shifts in the neutron star surface caused
by variations of the X-ray flux. The discussion of these effects is beyond
the scope of this paper and will be presented elsewhere \citep{Riggio_07}.

\section{Conclusions}

We have described a simple technique that permits to drastically
reduce the presence of erratic behavior and long-term intrinsic
variations of the pulse phase delays of the source, thus allowing to
fit the residual orbital modulation of these phase delays, caused by errors
in the previously reported orbital parameters, on a very long time-span and
to obtain a much more precise measure of the orbital parameters. We applied this
technique to the source XTE~J1807--294, which shows the longest X-ray outburst
observed by RXTE from an accreting millisecond pulsar. In this way we
can fit the residual modulation of the phase differences over the
whole time-span in which the coherent pulsations were significantly
detected (about 100 days from the start of the outburst), obtaining a
set of orbital parameters with a precision that is at least one order
of magnitude better than the previously published orbital solutions
for this source.

Once a good orbital parameters set is known, a detailed discussion of the
spin period and its derivative is possible. However, this source also shows
erratic fluctuations of the phases that are anticorrelated to variations
in the X-ray flux, in a way that is similar to what is found by \cite{Papitto_07}
for the source XTE~J1814--338. A detailed discussion of these effects and
a determination of the spin frequency and its derivative in XTE~J1807--294\ will
be presented elsewhere \citep{Riggio_07}.

\thanks{We acknowledge the use of RXTE data from the HEASARC public archive.
  This work was supported by the Ministero della Universit\`a e della Ricerca (MiUR)
  national program PRIN2005 2005024090$\_$004.}


text:
\section{introduction}
The availability of multimodal data enables us to perform many downstream tasks with cross-modal information, such as conversation generation, multimodal sentiment analysis, etc. In the field of sentiment analysis (MSA), recently researchers leverage the rich information contained in different modalities (e.g., audio, visual, language) to design multimodal models, and existing works mainly focus on exploring cross-modal dynamics and designing sophisticated fusion methods \cite{ARGF,MCTN,Poria2017A,MISA,mai2021unimodal}.

While existing MSA models are mostly optimized by multimodal loss, the design towards the optimization of unimodal networks in MSA models is often neglected. However, the reach of optimal unimodal networks determines the lower limit of the whole MSA models, which should specifically addressed for the higher performance of the models. Besides, an optimal solution for each modality also ensures the performance of MSA models even with the absence of any modality.

Moreover, even with satisfactory unimodal networks, it is not always the case that multimodal models reach higher performance than the unimodal ones \cite{TASLP}. The reason may be that, a modality may not contain useful information in some utterances and may even carry noises, which hinders the learning of correct multimodal embedding. Some attention-based methods leverage attention mechanism to determine modality importance \cite{CIA, MTL}, which can filter out noise information in a certain degree, but those methods introduce a large amount of parameters and increase the risk of overfitting. Moreover, despite the attention on informative modalities, the noisy modalities cannot be explicitly filtered out.

Based on the aforementioned problems, we mainly concern about two questions: how to obtain an optimal unimodal network; which modality is informative and how to filter out noisy modalities. We hold the intuition that each modality carries modality-specific information, whose importance varies from one another. Moreover, the role of the same modality also varies (the amount of useful and noisy information varies in different utterances).
To address these concerns, we propose a novel \textbf{M}odulation \textbf{M}odel for \textbf{M}ultimodal \textbf{S}entiment \textbf{A}nalysis {$ M^3SA $} to modulate the training of different modalities.

Specifically, modulation loss and modality filter module are designed to identify import modalities and reduce the negative impact of noisy information. To learn an optimal unimodal network, modulation loss is proposed to modulate the training of each unimodal network. The core idea is that during the training stage, the modulation function manages to modulate the loss contribution of each modality according to the confidence of all the modalities \cite{focal}, which enables the model to balance multi-modal information and identify the importance of each modality at each utterance. In this way, the model can dynamically adjust the contribution from different modalities so as to better leverage the importance information hidden within each modality to update the unimodal networks. With our proposed modulation loss, the training of individual unimodal networks is modulated and they can be better optimized by reducing the inference of the noisy modalities at each utterance.

Besides, to obtain correct multimodal embedding, we design a modality filter module (MFM) to identify modality importance and explicitly filter out noisy modalities. We present two possible candidates of the filter of MFM, i.e., a hard-filter and a soft-filter, where the hard-filter provides a binary choice $\{0, 1\}$ to retain or filter out individual modalities, while the soft-filter outputs a number between [0, 1] to filter out noisy information based on the noise level. Moreover, instead of directly removing the noisy modalities or tokens \cite{Chen2018Multimodal,rl2}, inspired by \cite{mask}, we innovative to train a baseline embedding for each modality and replace the noisy embedding with it, such that our method can be fitted into any fusion mechanisms and compensate for the loss of unimodal information.

In brief, the contributions can be summarized as:

\begin{itemize}
\item We propose a novel framework $ M^3SA $ to modulate the training of MSA models, which aims to explore optimal solution for unimodal networks and multimodal embedding.
\item A cross-modal modulation loss is devised to modulate the contribution of each modality based on the confidence of individual modalities during the training stage, and it can reduce the interference from noisy modalities so that unimodal networks can be better optimized, which is often neglected in existing works.
\item A modality filter module (MFM) is designed to identify noisy modalities and filter them out where soft-filter, hard-filter and unimodal embedding baselines are proposed, so as to minimize the negative impact of noisy information and obtain correct multimodal embedding. Compared with attention-based methods, MFM introduces much less parameters and can explicitly filter out noisy modalities.
\item Our proposed method is compared with several models on public datasets and achieves state-of-the-art performance, which demonstrates its effectiveness and superiority.
\end{itemize}

\section{Related Work}

In the field of MSA, each sample is an utterance that captures different views with complementary information. Most previous works focus on elaborately designing various fusion strategies so that the model can explore inter-modal dynamics to sufficiently learn a joint embedding, including simple ways like early fusion and late fusion \cite{Wollmer2013YouTube, Rozgic2012Ensemble, Poria2017Convolutional, Poria2017Context}, and more advanced fusion strategies like tensor-based fusion \cite{Liu2018Efficient, Zadeh2017Tensor, HFFN}, graph fusion \cite{ARGF,MOSEI,mai2020analyzing}, factorization methods \cite{MFM,MMB}, fine-tuning BERT \cite{MAG-BERT, CM-BERT} etc.

The above-mentioned methods focus on exploring more advanced fusion strategies, and optimize the whole network mostly based on multimodal loss so as to achieve higher performance for MSA task. While more attention is paid on the optimization of multimodal networks, specifically designed method for optimizing individual unimodal networks is neglected.
We hold that apart from the learning of cross-modal dynamics, it is also important to reach an optimal solution for the optimization of unimodal networks. To achieve this goal, we specifically design a modulation loss to modulate the loss contribution of unimodal networks based on their confidence. We train all unimodal networks with the modulation loss across all data points with the aim to reaching optimal parameters on the corresponding dataset.

\begin{figure*}[h]
  \centering
  \includegraphics[width=0.95\linewidth]{framework.png}
  \caption{\label{diagram}The diagram of our proposed $M^3SA$
  }
\end{figure*}

Another problem in the field of MSA is the interference between modalities.
Noisy modalities can interfere the learning of other modalities and the correct multimodal embedding. Some \textbf{attention-based fusion} methods such as Context-aware Interactive Attention (CIA) \cite{CIA}, Multi-Task Learning (MTL) \cite{MTL} and Multilogue-Net \cite{Multilogue-Net} that apply cross-modal attention mechanism consider the importance of different modalities and assign different weights to them. But they focus on identifying and highlighting important modalities, and can not explicitly filter out noisy modalities. Although these models have considered modality importance, we format it from a different perspective instead of learning attention weights. Specifically, we focus on identifying and filtering out noisy modalities with a modality filter module (MFM), which introduces much few parameters than attention mechanisms and can explicitly filters out noisy information. Actually, there also exists works that aim to filter out the noisy modalities or the tokens within modality using reinforcement learning (RL) \cite{Chen2018Multimodal,rl2}. However, RL is unstable in training and suffers from high variants and control variates that requires auxiliary models or multiple evaluations of the network \cite{hard1,hard3}. Moreover, they provide a binary choice to retain or filter out the whole noisy modality, and modality-specific information may be lost. Unlike it, our proposed MFM is much more easier to train, and at the same time MFM considers the baseline embedding to compensate the loss of modality-specific unimodal information.

\section{Algorithm}

\subsection{Notations and Problem Formulation}
Our task is to perform multimodal sentiment analysis with multimodal data by scoring the sentiment intensity. The input to the model is an utterance \cite{Olson1977From} (i.e., a segment of a video bounded by pauses and breaths), each of which has three modalities, i.e., acoustic ($a$), visual ($v$), and language ($l$). The sequences of acoustic, visual, and language modalities are denoted as $\bm{u^a} \in \mathbb{R}^{T_a \times d_a}$, $\bm{u^v} \in \mathbb{R}^{T_v \times d_v}$, and $\bm{u^l} \in \mathbb{R}^{T_l \times d_l}$, where $T_a$, $T_v$ and $T_l$ represent the length of the audio, visual and language modality, respectively, and $d_a$, $d_v$ and $d_l$ denote the dimensionality of the audio, visual and language modality, respectively.

\subsection{Overall Algorithm}

Formally, a traditional multimodal learning system can be formulated as:
\begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
  \bm{x}^{m}=\bm{F}^m(\bm{u}^m; \theta_m), m \in \{ l, a, v\}
\end{equation}
\begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
  y_M = \bm{F}^M(\bm{x}^{l}, \bm{x}^{a}, \bm{x}^{v}; \theta_M)
\end{equation}
where $y_M$ is the prediction, $\bm{F}^m$ parameterized by $\theta_m$ and $\bm{F}^M$  parameterized by $\theta_M$ refer to the unimodal and multimodal network, respectively. $\bm{U}^m\in \mathbb{R}^{T_m \times d_m}$ is the input raw feature of modality $m$ where $T_m$ is the sequence length.
To update the parameters of the multimodal system, we have the following equation:
\begin{equation}
\setlength{\abovedisplayskip}{3pt}
\setlength{\belowdisplayskip}{3pt}
 \ell = ||y-y_M||_1,\ \theta\!\leftarrow\! \theta - \alpha \frac{\partial\ell}{\partial \theta}
\end{equation}
where $y$ is the ground truth label, $\theta \in \{\theta_a,\theta_v,\theta_l,\theta_M \}$, $\alpha$ is the learning rate, and $\ell$ is mean absolute error (MAE).

Unlike the traditional multimodal learning system which mostly focuses on optimizing the whole multimodal framework, we decouple the learning procedure of unimodal and multimodal networks, introduce modulation losses to specifically optimize the unimodal networks for learning better unimodal representations, and design modality filter module (MFM) for identifying and filtering out noisy modalities. As illustrated in Fig.~\ref{diagram}, given an input utterance of three modalities, we first obtain the unimodal representations via unimodal networks. Modulation loss is specifically designed to train individual unimodal networks by modulating the loss contributions of each modality
Besides, the output of each unimodal network will be sent to the MFM, and in this way, noisy modalities can be identified and filtered out. With our proposed method, we can modulate the learning of correct unimodal and multimodal dynamics, and minimize the negative impact of noisy information. In a word, our multimodal learning system is formulated as:
\begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eq4}
  \bm{x}^{m}=\bm{F}^m(\bm{u}^m; \theta_m), m \in \{ l, a, v\}
\end{equation}
\begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eq5}
  y_{m}=\bm{C}(\bm{x}^{m}; \theta_c), \ \ell^m = \left|y_{m}-y\right|
 \end{equation}
 \begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eq5555}
  l^{m}_2=\text{Modulation}(\ell^a, \ell^v, \ell^l; \ell^m)
 \end{equation}

  \begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eq555}
  \bm{x}^{m}_2=\text{MFM}(\bm{x}^{l}, \bm{x}^{a}, \bm{x}^{v}; \bm{x}^{m})
 \end{equation}
 \begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
  \bm{x^M} = \bm{F}^M(\bm{x}^{l}_2, \bm{x}^{a}_2, \bm{x}^{v}_2; \theta_M)
\end{equation}
\begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eq6}
  y_M=\bm{C}(\bm{x^M}; \theta_c), \ \ell_M = \left|y-y_M\right|
 \end{equation}
where $\bm{C}$ is the classifier that  takes encoded representation as input and outputs the sentiment prediction, which  is shared across unimodal and multimodal networks to force the learned unimodal and multimodal representations to have approximately same distributions. As illustrated in Eq.~\ref{eq5555}, the unimodal losses are adjusted by a Modulation function, which helps to identify the contribution of each modality of the current utterance to the optimization of the respective unimodal network. $ l^{m}_2$ is used to update the respective unimodal network. Moreover, in Eq.~\ref{eq555}, MFM is introduced to identify and replace the uninformative modalities with the learned unimodal baseline embeddings to filter out the noisy information that interferes the learning of the cross-modal interactions. The detailed introduction of the modulation function and the MFM is shown in Section 3.3 and 3.4, respectively.

Unlike most existing works which need sophisticated designed fusion methods to sufficiently explore cross-modal dynamics, our proposed $M^3SA$ can leverage simple fusion method to reach the state-of-the-art performance with  better generalization ability. Also note that our algorithm is model-agnostic, and we can integrate any sequence learning networks into our unimodal networks $\bm{F}^m$.  In this paper, we apply Transformer-based \cite{transformer} architectures to build up the unimodal networks. As for the multimodal network $\bm{F}^M$, we introduce different fusion mechanisms to evaluate the algorithm. Please refer to Appendix for the details about the unimodal and multimodal networks.

\subsection{Modulation Loss}
The cross-modal modulation function is proposed to modulate the loss contribution of
each modality as a function of the confidence of individual
modalities. This is based on the assumption that each modality carries various modality-specific information, whose importance varies from one modality to another modality. And in different utterances, the role of the same modality also varies (in some utterances, this modality is important, while in other utterance, it contains only the noisy information). Instead of learning the fixed attention weight for each modality as the previous methods do \cite{RAVEN,ARGF}, we seek to dynamically adjust the contribution from different modalities so as to better leverage the important information hidden within each modality to update the network, and effectively reduces the interference of the noisy utterances. Compared to the attention mechanism, the modulation loss directly has influence on the optimization procedure, which is more straightforward and non-parametric.

How do we dynamically determine the contribution of each modality during training? A intuitive idea is that we can estimate the value of the unimodal loss, under the assumption that the smaller the value of the unimodal loss, the more discriminative it is for the task, and a higher weight shall be assigned so as to better leverage the discriminative information hidden in this modality to update the network. More importantly, when assigning weight to each unimodal loss, we should have a global view on all the modalities to consider the value of the other unimodal losses to estimate the relative importance and adjust the weight for this modality accordingly. The modulation loss can be formulated as (taking language modality as an example):

\begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eq5}
  \ell^l_2 =\text{Modulation}(\ell^l, \ell^a, \ell^v; \ell^l)
 \end{equation}
where $\ell^l_2$ is the modulation loss for language modality.
The Modulation function aims to learn the weight for unimodal loss by estimating the discriminative information in all the modalities (this is why we call it modulation). The formulation of the Modulation function could have many choices. In practice, we formulate
it as:
\begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eqmean}
\alpha =\frac{1}{\frac{1}{3} \sum_{m}^{m\in\{l,a,v\}} \frac{1}{l^{m}}}=\frac{3}{\sum_{m}^{m\in\{l,a,v\}} \frac{1}{l^{m}}}
 \end{equation}
\begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eqmodulation}
  \alpha_l = \alpha \times \ell^a \times \ell^v
 \end{equation}
 \begin{equation}
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\label{eq5}
  \ell^l_2 = \ell^l \times \alpha_l
 \end{equation}
where $\alpha$ is the harmonic mean of the three unimodal losses which performs a kind of scale on the weight of unimodal losses,  and $\alpha_l$ is the weight for the language loss.
By using the loss values of other modalities to compute weights for the current modality, the weight of the current modality reduces when the other modalities obtain relatively low losses (i.e., other modalities have high confidence for prediction). In other words, the modality that has a relatively high loss obtains a low weight when updating the corresponding unimodal network, which dynamically reduces the influence of noisy modalities to the network. This simple operation is shown to be very effective (see experiment).

\subsection{Modality Filter Module}

The problem of noisy modalities negatively affects the learning of other informative modalities and hinders higher performance of existing MSA models.
Many existing works try to identify modality importance with attention mechanisms \cite{ARGF,RMFN}, which can highlight useful tokens or modalities and filter certain noisy information out. However, those methods cannot completely filter out the noisy information and only tend to assign high weight to the informative modalities.
\citet{Chen2018Multimodal} leverage reinforcement learning (RL) to learn a gate controller for each modality, which can shut off noisy modalities. But RL suffers from high variance and introduces more parameters and optimization objective \cite{hard1}, which is unstable in training.

Unlike previous methods, we propose a modality filter module (MFM) to selectively filter noisy modalities out, in which way the negative impact of noisy information can be minimized. Unlike \cite{Chen2018Multimodal} which only considers non-lexical modalities as the possible noisy modalities, we aim to identify if the three modalities in each utterance contain noisy information, and if they should contribute to the final prediction.

Mathematically, the deployment of MFM firstly takes the feature embeddings of all the modalities as inputs, and calculates a feature shift of the overall multimodal embedding to each specific unimodal embedding, which can be formulated as:

  \begin{equation}
  \begin{aligned}
   & \bm{x^M} = \bm{x}^l\oplus\bm{x}^a\oplus\bm{x}^v \\
   & \bm{x'} = \text{Linear}(\bm{x^M}; \theta_L) \\
    \bm{x_{shift}^m} =& \text{ReLU}(\bm{x'-x^m}), m \in \{ l, a, v\}
    \end{aligned}
\end{equation}
where $\bm{x^M}$ denotes a multimodal representation by the concatenation of the embeddings of the three modalities, $\bm{x'}$ represents the processed multimodal representation which preserves the same dimensionality as individual modalities by a linear transformation, and $\bm{x_{shift}^m}$ is the feature shift of modality $m$ compared to $\bm{x'}$. By using all the unimodal embeddings to modulate and determine the noisy level of each specific modality, the model can have a global view on all the modalities and determine which is informative and which is not.

With the obtained feature shift of each modality, MFM filters out noisy information by a Filter:
  \begin{equation}
  \setlength{\abovedisplayskip}{3pt}
\setlength{\belowdisplayskip}{3pt}
    s^m= \text{Filter}(\bm{x_{shift}^m}; \theta_f), m \in \{ l, a, v\}
\end{equation}
where \text{Filter} parameterized by $\theta_f$ outputs $s^m$, which determines whether to filter the  modality $m$ out based on its noise level. The \text{Filter} is trained across all utterances, and it can identify and filter out noisy modality. The realization of \text{Filter} has many possibilities, and we put forward two candidates in Section~\ref{soft} and Section~\ref{hard}. After obtaining the output $s^m$ from the \text{Filter}, the final embedding of the modality $m$ can be determined:
  \begin{equation}
  \label{eqbb}
  \setlength{\abovedisplayskip}{3pt}
\setlength{\belowdisplayskip}{3pt}
    \bm{x_2^m}= s^m \cdot \bm{x^m} + (1-s^m) \cdot \bm{b^m}

\end{equation}
where $\bm{x_{out}^m}$ represents the final embedding of the  modality $m$, which  contains much less noisy information. $\bm{x_{out}^m}$ of individual modalities is then leveraged to learn a correct multimodal embedding for MSA task. Besides, we assume that filtering out too much information of the noisy modality may degrade the performance, for the model may lose modality-specific information. To compensate the modality-specific information of noisy modalities, we learn a baseline embedding $\bm{b^m}$ for each modality. The unimodal baseline embedding $\bm{b^m}$ is a critical part of our MFM, which is trained across multiple data points in the dataset. $\bm{b^m}$ is assumed to integrate the general distributions and properties of each modality, and therefore it can compensate the modality-specific information for fusion. Moreover, instead of directly removing the noisy modalities or tokens \cite{Chen2018Multimodal,rl2}, the unimodal baseline embedding enables our model to fit into any fusion mechanism such as tensor fusion or element-wise multiplication, providing more generalization ability.

With our proposed MFM, our model is capable of identifying and filtering out noisy modalities. In this way, our proposed model can dynamically retain informative modalities to modulate the learning of correct multimodal embedding for each utterance. Besides, to minimize the negative impact of the absence of modality-specific information, the learned baseline embedding $\bm{b^m}$ of each modality helps to sufficiently learn cross-modal dynamics.

\subsubsection{Soft Filter}
\label{soft}
To realize the Filter function, we first consider the soft filter mechanism whose output value is not binary. The procedure for soft filter is shown below:
\begin{equation}
  \setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
    \bm{z^m} = \text{FC}(\bm{x_{shift}^m}; \theta_{fc})
\end{equation}
\begin{equation}
  \setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}

   s_i^m = \frac{e^{\lambda\cdot z_i^m}}{\sum_{j=1}^2e^{\lambda\cdot z_j^m}},\ \ \bm{s^m} = [s_1^m, s_2^m]
\end{equation}
\begin{equation}
  \setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
    l^p =  1 - (s_1^m -s_0^m)^2
\end{equation}
where $\lambda$ is the scale factor to widen the distance between the elements in $s$, $FC$ is the fully-connected network activate by ReLU, and $\bm{s^m}\in \mathbb{R}^{2}$ is the assignment vector that determines the noisy level of modality. $l^p$ is the penalty loss that encourages the elements of $\bm{s^m}$ to be close to 0 or 1. Nevertheless, the elements of $\bm{s^m}$ are not likely to be binary because they are continuous. But via the soft filter, the model can learn to estimate how much information in the modality can be filtered out instead of directly filtering out all the information, providing more fine-grained filtering effect. Since the output of soft-filter is a 2-dimensional vector, Eq.~\ref{eqbb} should be rewritten as:
  \begin{equation}
  \setlength{\abovedisplayskip}{3pt}
\setlength{\belowdisplayskip}{3pt}
    \bm{x_2^m}= s^m_1 \cdot \bm{x^m} + s^m_2 \cdot \bm{b^m}  ,\ \ s^m_1 + s^m_2 = 1
\end{equation}
Soft filter differs from attention mechanism in following aspects: 1)  introducing scale factor $\lambda$ and penalty loss $l^p$ to reach better filtering effect; 2) introducing the unimodal baseline embedding to compensate the filtered modality-specific information; 3) merely modifying the unimodal embedding and can be integrated with any fusion mechanisms.
\subsubsection{Hard Filter}
\label{hard}
The output of the hard filter, i.e., $s^m$, is a scalar that is either $0$ or $1$. However, due to the discrete nature of $s^m$, training this kind of framework using gradient-based optimization algorithm is intractable. To resolve this problem, we follow \cite{hard1} to use reparameterization trick \cite{hard2} to compute the unbiased and low variance gradients. Specifically, we utilize the Hard Concrete distribution introduced in \cite{hard1}, which is a mixed discrete-continuous distribution on the interval [0, 1]. Hard Concrete assigns a continuous probability to exact zeroes or ones, and meanwhile it allows continuous outcomes in the unit interval such that the gradient can be computed via the reparameterization trick. The computation of $s^m$ for hard filter is illustrated as follows:
\begin{equation}
\begin{aligned}
&z^m = \text{FC}(\bm{x_{shift}^m}; \theta_{fc})\\
 &  \hat{s}^m =\operatorname{Sigmoid}((\log \frac{u}{1-u} + z^m ) / \beta)\\
 &\bar{s}^m =\hat{s}^m \times (\zeta-\gamma)+\gamma  \\

 & s^m = 1\  \text{iff}\ \bar{s}^m>0.5\ \text{else}\ s^m = 0
\end{aligned}
\end{equation}
where $\beta$ is the temperature, $\zeta$ and $\gamma$ are the hyper-parameter to scale $s^m$, and $u \sim \mathcal{U}(0,1)$ ($\mathcal{U}$ denotes Gaussian distribution).
Compared to using RL \cite{Chen2018Multimodal,rl2} to obtain the exact binary weight, using the Hard Concrete distribution is much more simple and stable in training, with no additional optimization objectives or components introduced. Via the hard filter, the model can completely filter out the noisy modalities which cannot be realized by the attention mechanisms.  For more details about Hard Concrete distribution, please refer to \cite{hard1}.

\section{Experiment}

\begin{table}[t]
\centering
 \caption{ \label{t1}\textbf{ The comparison with baselines on CMU-MOSI.} Note that QMF and MISA do not provide the code so we present the result from their papers.}
\resizebox{.95\columnwidth}{!}{\begin{tabular}{c|c|c|c|c|c}
 \hline
    & Acc7 & Acc2 & F1 & MAE & Corr \\
 \hline
 EF-LSTM  & 31.6 & 75.8 & 75.6 & 1.053 & 0.613  \\
 LF-LSTM  & 31.6 & 76.4 & 75.4 & 1.037 & 0.620  \\
TFN \cite{Zadeh2017Tensor} & 32.2 & 76.4  & 76.3 & 1.017 & 0.604 \\
 LMF \cite{Liu2018Efficient} & 30.6 & 73.8  & 73.7 & 1.026 & 0.602 \\
 MFN \cite{Zadeh2018Memory} & 32.1 & 78.0  & 76.0 & 1.010 &0.635  \\
RAVEN \cite{RAVEN} & 33.8 & 78.8  & 76.9 & 0.968 & 0.667 \\
 MULT \cite{MULT} & 33.6 & 79.3  & 78.3 & 1.009 & 0.667 \\
 QMF \cite{Quantum} & 35.5 & 79.7  & 79.6 & 0.915 & 0.696 \\

 MAG-BERT \cite{MAG-BERT} & 42.9 & 83.5 & 83.5 & 0.790 & 0.769 \\
 \hline
 $M^3SA$ (Hard) & 45.5 & 85.3  & 85.3 & 0.730 & 0.793 \\
$M^3SA$ (Soft) & \textbf{46.4} & \textbf{85.7}  & \textbf{85.6} & \textbf{0.714} & \textbf{0.794} \\
 \hline
 \end{tabular}}
 \vspace{-0.3cm}
\end{table}%

\begin{table}[t]
\centering
 \caption{ \label{t2}\textbf{ The comparison with baselines on CMU-MOSEI.} Note that IMR cannot perform regression task so that MAE and Corr are not available. }

\resizebox{.95\columnwidth}{!}{\begin{tabular}{c|c|c|c|c|c}
 \hline
    & Acc7 & Acc2 & F1 & MAE & Corr \\
 \hline
 EF-LSTM  & 46.7 & 79.1 & 78.8 & 0.665 & 0.621\\
 LF-LSTM  & 49.1 & 79.4 & 80.0 & 0.625 & 0.655 \\
 TFN \cite{Zadeh2017Tensor} & 49.8 & 79.4  & 79.7 & 0.610 & 0.671 \\
 LMF \cite{Liu2018Efficient} & 50.0 & 80.6  & 81.0 & 0.608 & 0.677 \\
 MFN \cite{Zadeh2018Memory} & 49.1 & 79.6  & 80.6 & 0.618 &0.670 \\
 RAVEN \cite{RAVEN} & 50.2 & 79.0  & 79.4 & 0.605 & 0.680 \\
 MULT \cite{MULT} & 48.2 & 80.2  & 80.5 & 0.638 & 0.659 \\
 IMR \cite{MRM} & 48.7 & 80.6  & 81.0 & - & - \\
 QMF \cite{Quantum} & 47.9 & 80.7  & 79.8 & 0.640 & 0.658 \\

 MAG-BERT \cite{MAG-BERT} & 51.9 & 85.0 & 85.0 & 0.602 & 0.778 \\
 \hline
 $M^3SA$ (Hard) & \textbf{52.7} & \textbf{85.6}  & \textbf{85.5} & 0.587 & \textbf{0.789} \\
$M^3SA$ (Soft) & 52.5 & 85.2  & 85.1 & 0.599 & 0.781 \\
 \hline
 \end{tabular}}
  \vspace{-0.3cm}
\end{table}%

\subsection{Experimental Setting}
We use the CMU-MOSI  \cite{Zadeh2016MOSI} and CMU-MOSEI  \cite{MOSEI} datasets to evaluate the model. We provide details about the datasets, evaluation protocols, baseline methods, and other experimental details in Appendix.

During the training stage, we first update individual unimodal sub-networks with the modulated unimodal losses, after which the whole model is updated with the multimodal loss derived from MFM.

\subsection{Experimental Results}

\subsubsection{\textbf{Comparison with Baselines}}

In this section, we compare our proposed model with other baselines on two datasets CMU-MOSI \cite{Zadeh2016Multimodal} and CMU-MOSEI \cite{MOSEI}. As shown in Table~\ref{t1} and ~\ref{t2}, although \textbf{MAG-BERT} outperforms other existing methods and sets up a high baseline due to the effectiveness of BERT \cite{BERT}, it can be seen that both of our proposed $M^3SA$ (Hard) and $M^3SA$ (Soft) significantly outperform all baselines in most cases. Specifically, on CMU-MOSI dataset, our method achieves the best results on all metrics, and $M^3SA$ (Soft) outperforms MAG-BERT by 3.5\% on Acc7, 2.2\% on Acc2 and 2.1\% on F1 score. On CMU-MOSEI dataset, our proposed $M^3SA$ (Hard) yields 0.8\% improvement on Acc7, and 0.6\% on Acc2 and 0.5\% on F1 score compared with MAG-BERT. These results demonstrate the superiority of our proposed model, indicating the effectiveness of reaching optimal unimodal network and filtering out noisy modalities.

\subsubsection{\textbf{Ablation Study}}

In this section, we perform ablation studies to verify the effectiveness of each component by removing it from the model.

Aiming to verify the effectiveness of the designed modulation loss, we conduct experiments where \textbf{modulation loss} is removed (see the cases of `$M^3SA$ (Hard) (W/O ML)' and `$M^3SA$ (Soft) (W/O ML)' in Table~\ref{t3}). From the experimental results, it can be seen that removing the modulation loss degrades the performance of the model. Specifically, performance on Acc7, Acc2 and F1 score has seen a great drop. It is obvious that our proposed contrastive learning method is effective and can greatly boost the performance.

Meanwhile, we design two ablation experiments to investigate the contribution of MFM (see the cases of `$M^3SA$ (Hard) (W/O MFM)' and `$M^3SA$ (Soft) (W/O MFM) in Table~\ref{t3}). We can observe that without MFM, our model sees a greater drop in performance, which may be due to the reason that noisy information interferes the learning of other useful modalities. The results suggest the necessity to identify and filter out noisy modalities for a correct multimodal embedding, and in this way informative modalities can also be highlighted.

We also perform ablation study on the design of considering \textbf{baseline embedding} in MFM (see the cases of `$M^3SA$ (Hard) (W/O BE)' and `$M^3SA$ (Soft) (W/O BE) in Table~\ref{t3}). We can see from the results that removing the compensation of baseline embedding in MFM degrades the performance of $M^3SA$ severely compared to other cases. Specifically, the performance drops even greater than the cases W/O MFM. It may be because, despite the removal of noisy information, modality-specific information of the noisy modality is lost. The results indicate that \textbf{the learning of baseline embedding in MFM is of necessity}, for it compensates the filtered modality-specific information.

\begin{table}[t]
\centering
 \caption{ \label{t3}\textbf{ Ablation studies on the CMU-MOSI dataset.} The `ML', `MFM' and `BE' refer to our proposed modulation loss, modality filter module and baseline embedding, respectively.}
\resizebox{.95\columnwidth}{!}{\begin{tabular}{c|c|c|c|c|c}
 \hline
    & Acc7 & Acc2 & F1 & MAE & Corr \\
 \hline
 $M^3SA$ (Hard) (W/O ML) & 44.9 & 84.2 & 84.2 & 0.743 & 0.786  \\
 $M^3SA$ (Soft) (W/O ML) & 46.2 & 85.0 & 84.9 & 0.729 & 0.794  \\
 $M^3SA$ (Hard) (W/O MFM) & \textbf{47.0} & 84.8 & 84.8 & 0.725 & 0.791  \\
 $M^3SA$ (Soft) (W/O MFM) & 44.2 & 83.9 & 83.9 & 0.737 & 0.794  \\
 $M^3SA$ (Hard) (W/O BE) & 46.1 & 84.2 & 84.2 & 0.728 & 0.788  \\
 $M^3SA$ (Soft) (W/O BE) & 46.1 & 83.9 & 83.9 & 0.733 & 0.794  \\
 \hline
 $M^3SA$ (Hard) & 45.5 & 85.3  & 85.3 & 0.730 & 0.793 \\
 $M^3SA$ (Soft) & 46.4 & \textbf{85.7}  & \textbf{85.6} & \textbf{0.714} & \textbf{0.794} \\

 \hline
 \end{tabular}}
\end{table}%

\begin{table}[t]
\centering
 \caption{ \label{t4}\textbf{ Discussion on the fusion strategies.} Graph fusion \cite{ARGF} regards each unimodal, bimodal, and trimodal interaction as one node, and explicitly models their relationship. Tensor fusion \cite{Zadeh2017Tensor} applies outer product to explore interactions, which introduces a large amount of parameters and has high space complexity. The defaulted fusion method is addition.
 }
\resizebox{.95\columnwidth}{!}{\begin{tabular}{c|c|c|c|c|c}
 \hline
    & Acc7 & Acc2 & F1 & MAE & Corr \\
 \hline
 Concatenation+FC (Hard) & \textbf{48.0} & 84.0 & 83.9 & 0.744  &   0.783 \\
 Addition (Hard)  & 45.5 & 85.3  & 85.3 & 0.730 & 0.793  \\
 Tensor Fusion (Hard)  & 43.1 & 84.3 & 84.3 & 0.772 & 0.786 \\
 Graph Fusion (Hard)  & 45.7 & 84.6 & 84.6 & 0.759 &  0.772  \\
  Concatenation+FC (Soft) & 45.4 & 84.4 & 84.4 & 0.740 &  0.790  \\
 Addition (Soft)  & 46.4 & \textbf{85.7}  & \textbf{85.6} & \textbf{0.714} & \textbf{0.794}  \\
 Tensor Fusion (Soft)  & 43.8 & 84.7 & 84.7 & 0.742 &    0.787\\
 Graph Fusion (Soft)  & 46.6 & 84.7 & 84.6 & 0.748 &  0.775\\
 \hline
 \end{tabular}}
\end{table}%

\begin{figure}
\setlength{\abovecaptionskip}{0.05cm}
\setlength{\belowcaptionskip}{-0.3cm}
\centering
\includegraphics[scale=0.2]{mask.png}
\caption{\label{tsne}\textbf{Visualization of the Mask Values of the Three Modality Learned by Soft Filter.}
}
\end{figure}

\subsubsection{\textbf{Analysis of Generalization Ability}}
We also conduct experiments to verify that our proposed $M^3SA$ is generalized to be applied with different fusion strategies. Previous work mostly rely on sophisticated fusion methods to sufficiently learn cross-modal dynamics to reach satisfactory results. Unlike them, our proposed model can achieve state-of-the-art performance with simple fusion strategies. As shown in Table~\ref{t4}, \textbf{even with simple and direct fusion methods like concatenation and element-wise addition of unimodal representations, $M^3SA$ still outperforms all baselines in most cases}. Note that despite the choice of $M^3SA$ (Hard) or $M^3SA$ (Soft), \textbf{all the variants of our model reach the state-of-the-art performance compared to baselines}. A conclusion can be reached that our designed modulation loss and MFM is effective and of satisfactory generalization ability. Also note that our proposed modulation loss and MFM can be applied to any cross-modal scenarios.

As shown in the Table, combining all the evaluation metrics, the simple fusion method, i.e., Addition performs best. We argue that apart from the modulation loss which can help to learn better unimodal representation, it is partly because we use the same classifier $C$ to regularize the feature distributions of unimodal and multimodal representations which forces them to have the same distribution, such that direct addition is strong enough to explore the complementary information and interactions between modalities.  Instead, the high-complex learnable fusion methods may introduce noise to the distribution, which degrades the performance. Specifically, we can observe that tenser fusion \cite{Zadeh2017Tensor} gets a relatively unfavorable results. The reason for it could be that tensor fusion implements the outer product on vectors of all modalities, which may change the distribution of high-level features and exhaust the deep network for introducing a lot of computation and parameters.

\subsubsection{\textbf{Analysis on the Modality Importance}}
We provide a visualization for the learned mask value of the soft filter for the testing utterances, aiming to verify the effectiveness of MFM to identify and filter out noisy modalities.
Note that the value of 'Mask1' and 'Mask2' represents the percentage of the preserved information and filtered information of the corresponding modality. We can infer from Fig.~\ref{tsne} that, the language modality is the most informative modality that is rarely filtered out (and this conclusion is consistent with other works \cite{TASLP}). Contrary to it, the acoustic modality is frequently identified as noisy and filtered out which is the most uninformative modality. It can be seen that our MFM is capable to identify and filter out noisy modalities, which can also highlight the role of informative modalities when noisy information is filtered. Notably, the mean mask value is 0.998, 0.012, 0.088 for language, acoustic, and visual modalities, respectively.

Also, from the visualization results we can observe that the learned mask value approximates the 0-1 distribution (i.e, a modality is identified as either very informative or very noisy), which differs from existing attention mechanisms and the difference is mostly due to our defined scale factor $\lambda$ and penalty loss $l^p$. Apart from highlighting important modalities as in attention mechanisms, our MFM can reach better filtering effect and can be integrated with any fusion mechanisms.
The visualization of $M^3SA$ (Hard) is similar, which is not presented due to the page limitations.

\section{Conclusions}
We propose novel MSA framework to modulate the learning of unimodal and cross-modal dynamics, which is capable of exploring an optimal solution for unimodal networks and filtering out noisy modalities. Specifically, modulation loss can  modulate the learning of unimodal networks based on their confidence of prediction, while modality filter module can filter out noisy modalities for a correct multimodal embedding.
Experiments demonstrate that our model outperforms state-of-the-art methods in two datasets.

redpajama_set_name: RedPajamaArXiv