text:
\section{Introduction}

After receiving paper reviews, authors may optionally submit a rebuttal to address the reviewers' comments, which will be limited to a {\bf one page} PDF file.  Please follow the steps and style guidelines outlined below for submitting your author response.

Note that the author rebuttal is optional and, following similar guidelines to previous CVPR conferences, it is meant to provide you with an opportunity to rebut factual errors or to supply additional information requested by the reviewers. It is NOT intended to add new contributions (theorems, algorithms, experiments) that were not included in the original submission. You may optionally add a figure, graph or proof to your rebuttal to better illustrate your answer to the reviewers' comments.

Per a passed 2018 PAMI-TC motion, reviewers should not request additional experiments for the rebuttal, or penalize authors for lack of additional experiments. This includes any experiments that involve running code, e.g., to create tables or figures with new results.  \textbf{Authors should not include new experimental results in the rebuttal}, and reviewers should discount any such results when making their final recommendation. Authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.

The rebuttal must adhere to the same blind-submission as the original submission and must comply with this rebuttal-formatted template.

\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.  Overlength responses will simply not be reviewed.  This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.  Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.

\section{Formatting your Response}

{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}

All text must be in a two-column format. The total allowable width of the text
area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch
(0.8 cm) space between them. The top margin should begin
1.0 inch (2.54 cm) from the top edge of the page.  The bottom margin should be
1-1/8 inches (2.86 cm) from the bottom edge of the page for $8.5 \times
11$-inch paper; for A4 paper, approximately 1-5/8 inches (4.13 cm) from the
bottom edge of the page.

Please number all of your sections and any displayed equations.  It is important
for readers to be able to refer to any particular equation.

Wherever Times is specified, Times Roman may also be used.  Main text should be
in 10-point Times, single-spaced. Section headings should be in 10 or 12 point
Times.  All paragraphs should be indented 1 pica (approx. 1/6 inch or 0.422
cm).  Figure and table captions should be 9-point Roman type as in
Figure~\ref{fig:onecol}.

List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response. When referenced in the text, enclose the citation
number in square brackets, for example~\cite{Authors14}.  Where appropriate,
include the name(s) of editors of referenced books.

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{1in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to make is resolvable in a printed copy of the response.  Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.  Many readers (and reviewers), even of an electronic copy, will choose to print your response in order to read it.  You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use \verb+\includegraphics+, and to specify the  figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}
\begin{figure}[t]
	\centering \includegraphics[width=1\columnwidth]{Figs/MainSpeakerHighlight.png}
	\small
	\caption{Given a multi-speaker video, our Audio-Visual Transformer can localize and highlight the main speaker in both visual and audio channels. (\textbf{Best viewed in color})}

	\label{fig:PrimarySpeakerDetection}
	\vspace{-6mm}
\end{figure}
Although human beings possess capabilities of localizing and separating sounds from noisy environments, we still have trouble following a conversation with noises, background voices, or interruptions from other speakers.
Either with blind audio separation \cite{jin2009supervised,makino2007blind,reddy2007soft, radfar2007single,wang2005video} or visual-aid audio separation
\cite{afouras2019my, arandjelovic2017look, chung2019said, gabbay2018seeing, gao20192, harwath2018jointly, hu2019deep, liu2013source,khan2013speaker, owens2018learning, ramaswamy2020see, senocak2018learning,tian2018audio} approaches, this outlier separation task still remains a challenge in the wide conditions beyond the lab settings. The problem becomes especially harder when dealing with unknown numbers of speakers in an audio. Nachmani et al. \cite{nachmani2020voice} make a comparison between methods and show how hard it is to separate voices when the number of sound sources increases.
Existing methods achieve high performance
with inputs from multiple microphones.
Some methods assume a clean set of single source audio examples are available for supervision \cite{afouras2018conversation, ephrat2018looking, zhao2019sound,Zhao_2018_ECCV}.
In practice, rather than solely trying to separate voices of all speakers in a conversation and determining ``\textit{who-spoke-when}'',
we tend to give more attentions to the \textbf{\textit{main speaker}}, i.e. \textit{who is on his/her turn of speaking and his/her talk is the main channel of communication}, and ignore the voices of remaining speakers, i.e. \textbf{\textit{interrupters}} or \textbf{\textit{listener}}, or background noises.
Thus,
an approach that highlights the main speaker in visual and audio channels would give new opportunities to popular applications such as auto-muting in a tele-conference or main speaker refocusable video generation.

Given a video of
multi-speaker conversation, our goal is to learn an audio-visual model
that enables the capabilities of both (1) localizing the main speaker; (2) true cancellation of audio sources of interrupters or background noises; and (3) automatically switching to a new subject when the speakers change their roles. The interruptions from other subjects and the background are considered as noises and removed.
In the scope of this work, we focus on \textbf{\textit{turn-taking conversation}} as the turn-taking mechanism has been commonly adopted for structuring conversation in social interactions. A subject is considered as the main speaker when he/she properly takes the turn of solo speaking and will continue the talk even after a simultaneous speech occurs \cite{cutrone2019profiling}

\begin{figure*}[t]
	\centering \includegraphics[width=1.8\columnwidth]{Figs/Proposed_Framework.png}
	\small
	\caption{\textbf{Our Proposed Audio-Visual Transformer Framework.} Given a video segment set, the
	context in the conversation is captured with
	three types of correlations, i.e. visual-visual, audio-audio, and audio-visual attentions. Then, the audio-visual attentional features are adopted for the main speaker localization and audio separation.

	(\textbf{Best viewed in color with 2x zoom in})
	}
	\label{fig:ProposedFramework}
	\vspace{-5mm}
\end{figure*}

Previous approaches have partially addressed
this problem and can be divided into two categories, i.e.  \textit{audio-visual synchronization} \cite{arandjelovic2018objects, chung2016out,cutler2000look,owens2018audio,khosravan2019attention} and \textit{mix-and-separate} \cite{afouras2018conversation, 9054376, ephrat2018looking,gao2018learning,gao2019co, NEURIPS2020_7288251b, korbar2018co, zhao2019sound, Zhao_2018_ECCV, 10.1007/978-3-030-58610-2_4}.
The former exploits the synchronization between audio and video frames within a specific time window to localize the image regions that are more sensitive to audio changes. Meanwhile, the latter
learns to separate the speakers' voices from a mix utterance based on audio and visual features.
In both cases, there remain some limitations. Firstly,
audio-visual relationships
are extracted via a concatenation operator or the cosine distance metric.
However, as audio and visual features distribute in two different latent spaces by their nature,
these methods may not maximize correlations between the two feature domains.
Secondly, audio-visual relationships
are only considered within a video segment, i.e. a short time window, while ignoring the ones across temporal segments, which helps to maintain the consistency of localization and separation, and \textit{contextual} main-subject switching.
Finally, the interactions between subjects in the temporal dimension to deliver accurate tracking and anticipatory decisions about transition to a new target are still ignored.

\noindent
\textbf{Contributions.} This work introduces a novel Audio-Visual Transformer approach, a cross-modality temporal-based computer vision algorithm, to highlight main speaker in both audio and visual channels
(Fig. \ref{fig:PrimarySpeakerDetection}).
The contributions of this work are four-fold.
(1) The proposed approach exploits various correlations presented in visual and audio signals
including ``virtual'' interactions between speakers in a video scene and relationships between visual and auditory modalities.
(2) Rather than extracting audio-visual correlations within a video segment, relationships across segments are further exploited via a temporal self-attention mechanism in the proposed Transformer structure.
This helps to engage the contextual information and enhance attentions with longer context so that the main speaker can be robustly identified.
(3) A Cycle Synchronization Loss is introduced to learn the main speaker localization in a self-supervised manner.
(4) A newly dataset\footnote{The dataset and implementation will be publicly available at \url{https://github.com/uark-cviu/Right2Talk}} is collected for the main speaker detection.
To the best of our knowledge, it is one of the first works that is able to automatically localize and highlight the main speaker in multi-speaker conversation videos on both visual and audio channels (Table \ref{tab:TenMethodSumm}).

\section{Related Work}

\noindent
\textbf{Active Speaker Localization.}
This problem aims to localize the sources of sounds in a given
video.
Some early methods \cite{barzelay2007harmony,hershey1999audio, izadinia2012multimodal,kidron2005pixels, senocak2018learning} localize the sources of human voice in a video using statistical models and audio-visual correlations.
Fisher et al. \cite{fisher2000learning} introduces
a multi-media fusion method in a complex domain to capture latent audio-visual relationships.
Later, deep learning approaches \cite{chung2016out,owens2018audio} come into place and exploit the synchronization between visual and audio signals to find the regions in the images that are sensitive to the audio features.
Afouras et al. \cite{Afouras20b} propose LWTNet that extends the synchronize cues with optical flow technique to extract and track audio-visual objects for the localization process.
Unlike previous
approaches, our work goes one step further by taking into account the \textit{context} of audio-visual features that presents across video segments.
Our method does not naively localize all the areas containing voice, but it is able to notice the sound and highlight the location of the \textit{main speaker} in a conversation.

\noindent
\textbf{Speaker Audio Separation.}
Prior methods \cite{jin2009supervised, makino2007blind,reddy2007soft} use audio-only features, i.e characteristics of voice, to resolve this problem. Hershey et al. \cite{hershey2016deep} formulate this task as a clustering problem where the objective is to learn an embedding for each time-frequency element in the spectrogram, such that each embedding cluster associates with a voice of a subject.
Zhao et al. \cite{zhao2019sound, Zhao_2018_ECCV} detect objects in one or multiple frames and use their appearance and motion to differentiate sounds of objects.
Gao et al. \cite{gao2019co} propose a co-separation training objective to learn audio-source separation from unlabeled videos containing multiple sources of sounds.
Ephrat et al. \cite{ephrat2018looking} contribute a large-scale dataset, namely AVspeech, and propose an end-to-end audio-visual architecture.
Afouras et al. \cite{afouras2018conversation} propose to use the lip regions and consider both audio magnitudes and phases.
The aforementioned methods ignore the \textit{context} of the video which is a very important cue for the network to improve the quality of separated voices.
Our novel architecture is proposed to
to gain the \textit{context} information .

\section{The Proposed Method}
This work focuses on turn-taking conversations composing talking turns.
The length of each turn is flexible according to the conversation's context and contents.
Let $\mathbf{x} \in \mathcal{X}$ be a multi-speaker conversation video consisting of a \textit{visual component} $\mathcal{V}$ (a sequence of RGB frames) and an \textit{audio component} $ \mathcal{A}$ (a mixed audio of one or multiple speakers).

\subsection{The Turn-taking Conversation} \label{sec:InTurnConversation}
With turn-taking regulations, a conversation $\mathbf{x}$ can be decomposed into speaking turns, i.e. $\mathbf{Turn}^h, h=1,...,H$ where $H$ is the number of turns in $\mathbf{x}$. Although many speakers can have their voices overlapped during a speaking turn in either cooperative or competitive manner, the role of each speaker can be classified into two groups.

\vspace{1mm}
\noindent
\textbf{\textit{Main Speaker} $S_m^h$. } A subject $S$ is
the main speaker of $\mathbf{Turn}^h$ when he/she carries the conversation and drives it forward \cite{cutrone2019profiling}. Even an interruption (i.e. simultaneous speech) occurs, the subject will continue to solo speak until the end of the turn. Thus, a long solo speaking of $S$ during $\mathbf{Turn}^h$ can provide an  indication for the main speaker role.

\vspace{1mm}
\noindent
\textbf{\textit{Interrupter or Listener} $S_I^h$. }
An interrupter or Listener is the one who has reactions or comments to the main speaker's utterances. These reactions usually occur in a short time window during $\mathbf{Turn}^h$ and end up with the continuation of the main speaker's talk.
When the interrupter continues to solo speak after  a simultaneous speech, a turn changing of the main speaker occurs. Fig. \ref{fig:TurnTakingConversation} illustrates an example of speakers' roles in a turn-taking conversation.

\begin{figure}[t]
	\centering \includegraphics[width=1\columnwidth]{Figs/TurnTakingConversation.png}
	\small
	\caption{\textbf{Turn-taking Conversation.} The conversation is decomposed in to turns where each speaker acts as the main speaker (green box) of a turn, and the other speakers are considered as interrupter or listener (orange box) during that turn.}

	\label{fig:TurnTakingConversation}
	\vspace{-5mm}
\end{figure}

\vspace{-0.5mm}
\subsection{Problem Definition}
\vspace{-0.5mm}
Rather than decomposing $\mathbf{x}$ into $\{\mathbf{Turn}^h\}_1^H$, we propose to present  $\mathbf{x}$ as a composition of $K$ segments $\mathbf{Seg}^k = \{\mathbf{v}^k,\mathbf{a}^k \}, k=1,...,K$, $\mathbf{v}^k \in \mathcal{V}$ and $\mathbf{a}^k \in \mathcal{A}$. The $h$-th speaking turn $\mathbf{Turn}^h$ consists of one or multiple segments, i.e. $\mathbf{Turn}^h = \{\mathbf{Seg}^k\}_{h_{start}}^{h_{end}}$ where $h_{start}$ and $h_{end}$ mark the indices of the starting and ending time of $h$-th turn, respectively.
Let $S_m^k$ be the main speaker and $S_I^k$ be the interrupters of $\mathbf{Seg}^k$ in the conversation. We have $S_m^k \equiv S_m^h$ and $S_I^k \equiv S_I^h$ when $\mathbf{Seg}^k \in \mathbf{Turn}^h$. Then, the goal is to extract the location and clean voice of $S_m^k$ for each $\mathbf{Seg}^k$ in the conversation.
Formally, the objectives are to learn the visual location map $\mathbf{M}_v^k$ and audio mask $\mathbf{M}_a^k$ of $S^k_m$ as.
\begin{equation} \label{eqn:object_learning_visual}
\footnotesize
\begin{split}
    \mathbf{M}_v^{k*} = \arg \min_{\mathbf{M}_v^{k}}
   \Big[ -&\log P\left(\mathbf{M}_v^{k}[Loc(S_m^k, \mathbf{v}^k)]|\mathbf{Seg}^{1:k}\right) \\
 + &\log P\left(\mathbf{M}_v^{k}[Loc(S_I^k, \mathbf{v}^k)]|\mathbf{Seg}^{1:k}\right) \Big]

\end{split}
\end{equation}
\begin{equation} \label{eqn:object_learning_audio}
\footnotesize
    \mathbf{M}_a^{k*} = \arg \min_{\mathbf{M}_a^{k}} \| \mathbf{M}_a^{k} \odot Spec(\mathbf{a}^k) - Spec(\mathbf{a}_{S_m^k})\|_1
\end{equation}
where $\textit{Loc}(S_m^k, \mathbf{v}^k)$ is the location of $S_m^k$ in $\mathbf{v}^k$;
$Spec(\cdot)$ is the spectrogram conversion operator; $\odot$ is the Hadamard product;
and $\mathbf{a}_{S_m^k}$ is the clean voice of  $S_m^k$.
$\mathbf{Seg}^{1:k}$ denotes the temporal information provided from the beginning to the $k$-th segment of the video $\mathbf{x}$. The conditional term indicates the temporal constraint
being considered

To effectively estimate
$\mathbf{M}_v^k$ and
$\mathbf{M}_a^k$,
we propose an Audio-Visual Transformer approach (see Fig. \ref{fig:ProposedFramework}) consisting of three learning stages: (1) Learning the context with visual and audio self-attention; (2) Audio-Visual Correlation Learning; and (3) Main speaker localization and audio separation with Conversation Grammar. The proposed Audio-Visual Transformer is formulated via
$\{D,\phi, E_v, E_a\}$ as:
\begin{equation}
\footnotesize
\begin{split}
    G &= [D \circ \phi] (\mathbf{z}_v ,\mathbf{z}_a)\\
    \mathbf{z}^k_v &= E_v(\mathbf{v}^k| \mathbf{Seg}^{1..k})\\
    \mathbf{z}^k_a &= E_a(\mathbf{a}^k| \mathbf{Seg}^{1..k})\\
\end{split}
\end{equation}
where $E_v$ and $E_a$ map
$\mathbf{v}^k$ and $\mathbf{a}^k$ to their
latent representations; and $\circ$ is the functional composition.
$\phi$ is the projection function to the shared representation space where these modalities are comparable.
$D$ maps these deep representations to the audio-visual mask of the main speaker.

\vspace{-0.5mm}
\subsection{Visual and Audio Contextual Learning}
\vspace{-0.5mm}
Besides prior works on temporal learning \cite{7780991, 10.1007/s11263-018-1113-3, 8953561, 10.1007/s11263-019-01165-5, 8237665, Duong_2020_CVPR, fi13070179}, in this work, given a conversation, contextual information can be extracted from visual and audio signals, i.e. \textit{visual-visual} and \textit{audio-audio} correlations \textit{across video segments}. While the former assists to track behaviors and interactions of each speaker over the spatial-temporal dimension, the latter provides more cues about the conversation flow,  i.e. when and how the main speaker switch his/her role.
For example, the higher audio-audio correlation between two or more segments is, the lower the possibility is of the main speaker being switched.
Thus, these cross-segment correlations can implicitly embed turn changing signals of the main speaker, and help to avoid the stage of pre-decomposing $\mathbf{x}$ into speaking turns $\mathbf{Turn}^h$. Even when interrupters dominate the main speaker's voice in a certain segment, this ``temporal-based'' correlation can exploit the relations to previous segments and identify the main speaker.
We
model the contextual correlations via two encoder structures with a self-attention mechanism before embedding their cross-domain correlations.

\vspace{-0.5mm}
\subsubsection{Visual-Visual Self-Attention.} \label{sec:Visual_Self_Att}
Given a sequence of segments $\{\mathbf{v}^k\}_1^K$, the visual encoder $E_v$ consists of three main functions, i.e. feature embedding, self-attention, and feature refinement with attention. Particularly, each $\mathbf{v}^k$ is firstly embedded into a deep feature embedding via $F_{v}: \mathcal{V} \mapsto \mathcal{F}_v$ as $\mathbf{f}^k_v = F_v(\mathbf{v}^k)$.

\vspace{1mm}
\noindent
\textit{\textbf{Speaker Region of Interest (SROI).}}
Rather than embedding each $\mathbf{v}^k$ into a single feature for attention computation, we propose to project $\mathbf{f}^k_v$ into regions of interest where each region represents a speaker's location
in a visual segment and learn the correlations among them. Particularly, let $\mathbf{b}=\{ \mathbf{b}_i^k\}, i=1..N, k=1..K$ where $\mathbf{b}_i^k \in \mathcal{B}  \subset \mathbb{R}^4$ denotes the location of $i$-th speaker in $\mathbf{Seg}^k$, and $N$ is the number of speakers. The projection function $R: \mathcal{F}_v \times \mathcal{B} \mapsto \mathcal{F}_{v}$ is defined as $\mathbf{f}^{k,i}_{v} = R(\mathbf{b}^k_i, \mathbf{f}_v^k)$. We adopt ROI Align \cite{He_2017_ICCV} for the function $R$.
There are two approaches to obtain $\mathbf{b}$ and $N$, i.e. face detection and block decomposition. The former adopts a face detection
to extract faces
in all segments.
The latter uniformly decomposes a visual segment into $N = n \times n$ blocks for $\mathbf{b}$. While face detection approach tends to give more direct focus on
face regions, our experiments show that block decomposition can provide attentions to regions of face track of the same speaker across segments.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.46\textwidth]{Figs/Visual_Visual_Att.png}
    \caption{\textbf{Visual-Visual Attention.} The attention masks across video segments corresponding to the speaker in the green box. This type of attention can help to track the behaviors and interactions of each speaker over the  spatial-temporal  dimension.}
    \label{fig:Visual_Visual_attention}
    \vspace{-4mm}
\end{figure}

\vspace{1mm}
\noindent
\textit{\textbf{Virtual Interaction Attention.}}
Given a feature set $\mathbf{f}^{k,i}_{v}$, the visual-visual context across the spatial-temporal dimension can be expressed as building a dynamic dictionary per feature set with three basic attention based elements
\cite{Nguyen_2021_CVPR, dyglip_cvpr21}, i.e. \textit{key, query, value}. While \textit{key} and \textit{query} are trained to support the dictionary look-up process where query feature is highly correlated to its matching key and dissimilar to others, \textit{value} represents a discriminative feature for each speaker. Particularly, the self-attention set $\{\mathbf{k}^{k,i}_v, \mathbf{q}^{k,i}_v, \mathbf{v}^{k,i}_v \}$ is extracted via three learnable projections $\{\Omega^Q_{v}, \Omega^K_{v}, \Omega^V_{v}\}$ as ${\mathbf{q}^{k,i}_{v}} = \Omega^Q_{v}(\mathbf{f}^{k,i}_{v}); {\mathbf{k}^{k,i}_{v}} = \Omega^K_{v}(\mathbf{f}^{k,i}_{v}); {\mathbf{v}^{k,i}_{v}} = \Omega^V_{v}(\mathbf{f}^{k,i}_{v})$.
The visual correlation among speakers can be defined as.
\begin{equation} \label{eqn:att_vis_vis}
\footnotesize
\alpha_{v}^{ki,k'j} = \sigma\left(\mathbf{q}^{k, i}_v ({\mathbf{k}^{k', j}_v})^{\top}/ \sqrt{d}\right)
\end{equation}
where $d$ is the feature dimension, $k'$ is a segment indexing variable.
We consider the attention as a probability distribution that illustrates the responsive attention among speakers. Therefore, the softmax function can be adopted for $\sigma (\cdot)$.

\vspace{1mm}
\noindent
\textit{\textbf{Feature Refinement with attention.}}
With these correlations, the visual self-attention
among speakers
allows every speaker correlates to all other speakers through the spatial-time dimension. Then, the virtual interaction over speakers is explicitly embedded to their representations as.

\begin{equation}
\footnotesize
    \mathbf{z}^{k, i}_v = \eta_{v}\left(\mathbf{f}^{k, i}_v + \sum_{k'=1}^K\sum_{j=1}^N\alpha^{ki, k'j}_v\mathbf{v}^{k,i}_v\right)
\end{equation}
where $\eta_v$ is the a residual-style MLP.
Throughout this process, the features of each speaker in one visual segment can interactively embed in their latent representation the correlations with those of the same speaker in other segments as well as other speakers of the same segment. Fig. \ref{fig:Visual_Visual_attention} illustrates the attention mask across video segments corresponding to the speaker in the green box.

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.45\textwidth]{Figs/Topology.png}
    \caption{Topology of audio and visual feature domains.}

    \label{fig:topo}
    \vspace{-4mm}
\end{figure}

\vspace{-2mm}
\subsubsection{Audio-Audio Self-Attention}
Similar to the virtual interaction, the audio self-attention is modeled as the correlation among audio segments.
Particularly, let $F_a: \mathcal{A} \to \mathcal{F}_a$ be an audio embedding function that extracts audio latent representation for audio segments $\{\mathbf{a}^k\}_1^K$ and $\mathbf{f}^k_a = F_a(\mathbf{a}^k)$.
The audio self-attention correlation among segments can be computed as in Eqn. \eqref{eqn:att_aud_aud}.

\begin{equation} \label{eqn:att_aud_aud}
\footnotesize
    \begin{split}

        {\mathbf{q}^{k}_{a}} = \Omega^Q_{a}(\mathbf{f}^{k}_{a});
        {\mathbf{k}^{k}_{a}} &= \Omega^K_{a}(\mathbf{f}^{k}_{a});
        {\mathbf{v}^{k}_{a}} = \Omega^V_{a}(\mathbf{f}^{k}_{a}) \\
        \alpha_{a}^{k,k'} &= \sigma\left(\mathbf{q}^{k}_a({\mathbf{k}^{k'}_a})^{\top}/\sqrt{d}\right)\\
    \mathbf{z}^k_a &= \eta_a(\mathbf{f}^k_a + \sum_{k'=1}^K \alpha_a^{k, k'}\mathbf{v}^{k'}_a)
    \end{split}
\end{equation}

The extracted audio feature of each segment is able to embed the correlation with other audio segments through time.

\subsection{Audio-Visual Correlation Learning}
The audio-visual correlations
are computed from features of two different domains. As \textit{\textbf{topologies}}, i.e. \textit{how
features distributed in the latent space and the correlations among its features} (see Fig. \ref{fig:topo}), of these modalities may differ significantly, directly associating these features for correlation learning is not efficient.
One solution is to set up two encoders $E_v$ and $E_a$ extracting features from latent spaces of the same dimension to leverage the domain differences. However, the topology difference between these modalities may still present. To mitigate this issue, we
align two domains' topologies before learning the correlations between their features. By this way, $\mathbf{z}^k_a$ and $\mathbf{z}^k_v$ are well aligned and their correlations can be fully exploited.

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.45\textwidth]{Figs/attention_single_multi_speakers_2.png}

    \caption{\textbf{Audio-Visual Attention To The Main Speaker.} The audio-visual attention mask across segments illustrates the response of audio to the visual.

    (\textbf{Best viewed in color})}

    \label{fig:attnetion_visual_audio_speaker}
    \vspace{-5mm}
\end{figure}

\vspace{1mm}
\noindent
\textit{\textbf{Cross Domain Alignment as Optimal Transport (OT) Problem.}}
We present the distributions of visual and audio features by two distributions $p_v$ and $p_a$ where $\mathbf{z}_v \sim p_v(\mathbf{z}_v)$, and $\mathbf{z}_a \sim p_a(\mathbf{a}_a)$; and propose a two-stage alignment process: (1) Sample association between visual and audio samples via transport function $\pi$ and (2) Topology synchronization.
Formally, let $\pi$ be the transport function where $\pi_{i,i'}=\pi(\mathbf{z}_v^i, \mathbf{z}_a^{i'})$ indicates the probability of association between a visual sample $\mathbf{z}_v^i$ and an audio sample $\mathbf{z}_a^{i'}$. In addition, let $c_{p_v}(\cdot, \cdot)$ and $c_{p_a}(\cdot, \cdot)$ be the cost functions defined as the distance between two samples in visual and audio spaces, respectively.
The alignment process is formulated with Gromov-Wasserstein distance as shown in Eqn. \eqref{eqn:GW_distance}.
\begin{equation} \label{eqn:GW_distance}
\footnotesize
\begin{split}
    \mathcal{L}_{align} = GW^2(c_{p_v}, c_{p_a}, p_v, p_a) = \min_{\pi \in \Pi(p_v, p_a)}J(c_{p_v}, c_{p_a}, \pi)\\
    J(c_{p_v}, c_{p_a}, \pi) = \sum_{i,j,i',j'}|c_{p_v}(\mathbf{z}_v^i, \mathbf{z}_v^j) - c_{p_a}(\mathbf{z}_{a}^{i'}, \mathbf{z}_{a}^{j'})|^2\pi_{i,i'}\pi_{j,j'}
\end{split}
\raisetag{30pt}
\end{equation}
Intuitively, minimizing $J(c_{p_v}, c_{p_a}, \pi)$ aims at finding an appropriated association (i.e. via $\pi$) between samples in the two domains as well as minimizing the topology difference between them (i.e. via $c_{p_v}, c_{p_a}$). Notice that directly solving Eqn. \eqref{eqn:GW_distance} is costly due to the non-convex Quadratic Problem with the time complexity is $O(n^3)$. Therefore, we adopt the the sliced approach \cite{vay_sgw_2019} for a fast computation of $\mathcal{L}_{align}$. Fig. \ref{fig:topo} illustrates visual (blue points) and auditory features (red points) extracted from 500 clip segments of 10 different speakers (e.g. denoted by different markers) and projected into the 2D space using t-SNE method.
Thanks to $\mathcal{L}_{align}$ in the alignment stage, visual and auditory features are brought into similar distributions (Fig. \ref{fig:topo} (B) (left)) with more aligned feature distributions (Fig. \ref{fig:topo} (B) (right)).

\vspace{1mm}
\noindent
\textit{\textbf{Audio-Visual Correlation.}}
With the aligned visual and audio features, we further adopt similar attention mechanism to learn the associations between the visual features of each SROI $\mathbf{z}^{k,i}_{v}$  and the audio features $\mathbf{z}^k_a$ in each segment as.
\begin{equation} \label{eqn:visual_att_aud_aud}
\footnotesize
    \begin{split}

        {\mathbf{q}^{k}} = \Omega^Q(\mathbf{z}^{k}_{a});
        {\mathbf{k}^{k, i}} &= \Omega^K(\mathbf{z}^{k, i}_{v});
        {\mathbf{v}^{k, i}} = \Omega^V(\mathbf{z}^{k, i}_{v}) \\
        \alpha^{k,k'i} &= \sigma\left(\mathbf{q}^{k}({\mathbf{k}^{k', i}})^{\top}/\sqrt{d}\right)\\
        \mathbf{z}^k = \phi(\mathbf{z}^k_a, \mathbf{z}^k_v) &= \eta\left(\mathbf{z}^k_a + \sum_{k'=1}^K\sum_{i=1}^N \alpha^{k, k'i}\mathbf{v}^{k', i}\right)

    \end{split}
\end{equation}

The attention matrix
assesses how much an audio responds to an SROI in the spatial-temporal dimension. A high response indicates a high correlation between the audio and the speaker associated with that SROI. This association embeds the probability of a speaker to be an active speaker of the audio segment. Fig. \ref{fig:attnetion_visual_audio_speaker} illustrates audio-visual attentions in both single and multiple speaker conversation.

\subsection{Main Speaker Localization and Audio Separation with Conversation Grammar} \label{sec:LearningProcess}
Given the audio-visual attentional features $\mathbf{z}^k$ from previous step, the audio-visual masks are computed as follows,
\begin{equation}
\footnotesize
\begin{split}
    \mathbf{M}_v^k(x,y) &= \begin{cases}
    \alpha^{k,ki}       &  \text{if } (x,y) \in \mathbf{b}_i^k, i=1..N\\
    0  & \text{otherwise}
  \end{cases} \\
  \mathbf{M}_a^k &= D(\mathbf{z}^k)
\end{split}
\end{equation}
where $D$ is a learnable decoder that maps $\mathbf{z}^k$ to the target audio mask; and $\alpha^{k,ki}$ denotes the correlation score between the audio and SROI of $i$-th speaker in the $k$-th segment.
The objective functions of Eqns. \eqref{eqn:object_learning_visual} and  \eqref{eqn:object_learning_audio} to learn $\mathbf{M}_a^k$ and $\mathbf{M}_v$ can be reformulated as follows,
\begin{equation}
\footnotesize
\begin{split}
    \mathcal{L}_{visual} &= \mathbb{E} \left[-\log \frac{e^{\alpha^{k,ki}}}{e^{\alpha^{k,ki}} + \sum_{j\neq i}e^{\alpha^{k,kj}}}\right]\\
    \mathcal{L}_{audio}& = \mathbb{E} \left[|| \mathbf{M}^k_a \odot Spec(\mathbf{a}^k) - Spec(\mathbf{a}_{S^k_m})||_1\right]
\end{split}
\end{equation}
Intuitively, on one hand, $\mathcal{L}_{audio}$ optimizes the model toward voice of the target (Main) speaker. On the other hand, $\mathcal{L}_{visual}$ aims at increasing the correlations between the audio and SROI of the target speaker, while reducing the correlation with other SROIs in spatial-temporal dimensions.

\vspace{1mm}
\noindent
\textit{\textbf{Self-supervised Learning.}}
While our goal is to develop a self-supervised model that learns to localize the main speaker, the ground truth location for Main speaker is absent during the training stage. Therefore, we further propose a self-supervised version of $\mathcal{L}_{visual}$ , namely \textit{Cycle Synchronization Loss}, defined as follows,
\begin{equation}
\footnotesize
\begin{split}
    \mathcal{L}_{Cyc\_Sync} &= \mathbb{E} \left[ ||\alpha^{k,ki} - \hat{\alpha}^{k,ki}_{\text{max}}||_1\right]\\
    \hat{\alpha}^{k,ki}_{\text{max}} &= \begin{cases}
    \hat{\alpha}^{k,ki}       &  \text{if } (x,y) \in \mathbf{b}_{i^*}^k\\
    0  & \text{otherwise}
  \end{cases};
  i^* = \arg \max_i \hat{\alpha}^{k,ki}
\end{split}

\end{equation}
where $\hat{\alpha}^{k,ki}$ is the correlation between the predicted clean voice of the target speaker. The intuition of $\mathcal{L}_{Cyc\_Sync}$ is illustrated in Fig. \ref{fig:Loss_Cyc_Sync} where the goal is to penalize the consistency between two terms: (1) the correlations of the input (mix) voices $\mathbf{a}^k$ and the visual component; and (2) the maximum correlations of the predicted (clean) voice of the target and the visual component.
As clean voice is of a single speaker and it reflects similar linguistic content as visual features of the target speaker, its correlations with the visual component can efficiently act as the guidance for localization process.
Moreover, by considering only the maximum $\hat{\alpha}^{k,ki}$ in $\hat{\alpha}^{k,ki}_{\text{max}}$, the correlation between the visual of other speakers (i.e. interrupter) and audio is also minimized

\vspace{1mm}
\noindent
\textit{\textbf{Learning with Conversation Grammar.}} We adopt the mix-and-separate strategy \cite{ephrat2018looking, Zhao_2018_ECCV} to obtain the ground truth for audio separation task of $\mathcal{L}_{audio}$ and further extend it with two types of Conversation Grammar, i.e. cooperative and competitive modes.
In the first type, each speaker takes turn to speak during the conversation and the role changing happens when a speaker finishes his/her speech.
In the second type, mixing voices happen during the interruption of other speakers.
From these grammars, we synthesize a video training set containing multiple speakers by (1) randomly selecting different videos in the single subject training set; (2) concatenating these videos sequentially (i.e. cooperative mode); (3) mixing their voices in a short time window and vertically concatenating the video frames (i.e. competitive mode).
In all cases, $S^k_m$ is set to the one who occupies the audio segment or the all segments of the whole video, accordingly.
The Audio-Visual Transformer is optimized as:
\vspace{-4mm}
\begin{equation}
\footnotesize
    \mathcal{L} = \alpha_{align} \mathcal{L}_{align} + \alpha_{visual} \mathcal{L}_{Cyc\_Sync} + \alpha_{audio} \mathcal{L}_{audio}
\end{equation}
where $\{\alpha_{align}, \alpha_{visual}, \alpha_{audio}\}$ are the parameters controlling their relative importance.

\begin{figure}[t]
	\centering \includegraphics[width=0.8\columnwidth]{Figs/Loss_Cyc_Sync.png}
	\small
	\caption{\textbf{Cycle Synchronization Loss.}}
	\label{fig:Loss_Cyc_Sync}
	\vspace{-5mm}
\end{figure}

\section{Main Speaker Dataset}
While most of previous speaker datasets \cite{LRS2, LRS3, Columbia, roth2020ava} are mainly designed for the active speaker detection task, in this work, we further introduce a large-scale dataset for the Main Speaker Detection task.
The proposed dataset
is collected with three conversation types, i.e. \textit{discussion panel}, \textit{tele-conference}, and \textit{debate}, from several Youtube channels.
Particularly, in the discussion panel videos, speakers take turn to speak during the conversation cooperatively.
For the second type, videos consist of multiple subject (i.e. 3-5 people) talking through Skype or Zoom in a tele-conference. The third type is more challenging with debate-style videos where there are more interruptions among the two speakers in both cooperative and competitive manners.
For each collected video, we select segments of various lengths (i.e. from 6 seconds to 20 seconds) that can represent the property of the corresponding conversation style.
In total, the dataset consists of 300 minutes videos. All clips are converted to have 25fps and 16kHz. The bounding box of the main speaker is also annotated.

\vspace{-1mm}
\section{Experimental Results}

\begin{table}[t]
    \centering
    \footnotesize
    \caption{\textbf{Main Speaker Audio Separation on LRS2 with and without Domain Alignment}. The higher value is better.}
    \begin{tabular}{|c|c|c|c|}
    \hline
    \multicolumn{2}{|c|}{}         & \textbf{Ours w/o $\mathcal{L}_{align}$} &\textbf{ Ours W $\mathcal{L}_{align}$} \\
    \hline
    \multirow{3}{*}{\textbf{SDR (dB)}}  & \textbf{1S+N} &    14.4     &    \textbf{15.8}       \\
    \cline{2-4}
                      & \textbf{2S }  &    9.8       &      \textbf{10.3}     \\
    \cline{2-4}
                      & \textbf{2S+N} &    7.0      &     \textbf{7.2}      \\
    \hline
    \multirow{3}{*}{\textbf{PESQ}} & \textbf{1S+N} &     3.1        &    \textbf{3.3}       \\
    \cline{2-4}
                      & \textbf{2S }  &      2.7       &    \textbf{2.9}       \\
    \cline{2-4}
                      &\textbf{ 2S+N }&       2.5      &    \textbf{2.5}      \\
    \hline
    \end{tabular}
    \label{tab:AudioSeparation_Align}
    \vspace{-6mm}
\end{table}

\begin{table*} [!t]
	\footnotesize
	\centering
	\caption{\textbf{Main Speaker Localization Accuracy with Cooperative Turn-Taking Conversation} (\%). For LRS2 and LRS3, a localization is considered correct if it lies within the true bounding box. For Columbia, F1 score is adopted.}
	 \begin{tabular}{|l|cc|c|ccccc|}
    \hline
     \multirow{3}{*}{}&\multicolumn{2}{c|}{\textbf{Single Speaker}} & \multicolumn{6}{c|}{\textbf{Multiple Speakers}}\\
     \cline{2-9}
      \multirow{2}{*}{} & \multirow{2}{*}{\textbf{LRS2}} & \multirow{2}{*}{\textbf{LRS3}} & \multirow{1}{*}{\textbf{Columbia}} & \multicolumn{5}{c|}{\textbf{Columbia} (Per subject)}\\
      & \multicolumn{2}{c|}{}& \textbf{(Avg)} & \textbf{Bell} & \textbf{Boll} & \textbf{Lieb} & \textbf{Long} & \textbf{Sick}\\

    \hline
    Baseline (Random Pixel) & 2.8\% & 2.9\% & 8.5\% &7.8\%  &8.6\% &9.9\% &7.9\% &8.7\%\\
    Baseline (Center Pixel) & 23.9\% & 25.9\% & 14.9\% &13.0\%  &11.5\%  &21.4\% &19.2\% &17.9\%\\
    \hline
    Multisensory \cite{owens2018audio} & 99.3\% & 24.8\% & 52.7\% &  52.0\%& 43.8\% &62.3\% &64.8\% &60.9\%\\

    Chakravarty et al.\cite{chakravarty2016cross}  & $-$ & $-$& 80.2\% & 82.9\%& 65.8\%& 73.6\%& 86.9\%& 81.8\%\\

    SyncNet \cite{chung2016out}& $-$ & $-$ & 89.5\% &  93.7\%& 83.4\%& 86.8\%& \textbf{97.7\%}& 86.1\%\\

	LWTNet \cite{Afouras20b} & 99.6\%& 99.7\%& 90.8\%& 92.6\%& 82.4\%& 88.7\%& 94.4\%&95.9\%\\
	\hline
	\hline
    (A) \textbf{Ours - Block attention} & 99.7\%& 99.8\%& 92.7\% & 93.7\% & 85.0\% & 87.5\% & 92.8\% &  97.2\%\\
    (B) \quad +  across segments& \textbf{99.8\%}  & \textbf{99.9\%} & 93.4\% & 95.8\% &  85.0\% & 87.5\% & 92.8\% & 97.2\% \\
     \hline
    (C) \textbf{Ours - Speaker Attention \footnotemark} & 100\% & 100\% & 93.8\% &  95.8\% & 85.0\% & 91.6\% & 92.8\% & 97.2\%\\
	(D) \quad +  across segments & \textbf{100\%} & \textbf{100\%} & \textbf{94.9\%} &  \textbf{95.8\%} & \textbf{88.5\%} & \textbf{91.6\%} & 96.4\% & \textbf{97.2\%} \\
     \hline
\end{tabular}
	\label{tab:InturnLocalization}
	\vspace{-2mm}
\end{table*}

\noindent
\textbf{Data Setting.} Our training data include 29 hours of training videos from Lip Reading Sentences 2 (LRS2) \cite{LRS2}, and synthetic videos obtaining as presented in Sect. \ref{sec:LearningProcess}.
The length of synthetic segments varies from 4s to 8s decomposed into 2s short segments.
The overlapped ratio of the mixing voice is set to $\frac{1}{3}$.
For validation, we adopt the testing set of LRS2, Lip Reading Sentences 3 (LRS3) \cite{LRS3}, Columbia \cite{Columbia}, and our collected dataset.
While LRS2 and LRS3 include 0.5-hour to 1-hour testing videos, Columbia includes an 86-minute panel discussion.
We adopt the ground truth for each active speaker in Columbia while annotating
bounding boxes using face detection \cite{deng2019retinaface} for LRS2 and LRS3.

\noindent
\textbf{Audio Data Preprocess.} We employ Short Time Fourier Transform (STFT) to represent the audio signal. Our STFT use Han window function which generates magnitude and phase of spectrograms.
We set the hop length of 10 ms with a window length of 40ms at a sample rate of 16000Hz.

\noindent
\textbf{Visual Data Preprocess.} All training videos are re-sampled to a resolution of $160 \times 160$ pixels at $25$ FPS. This chosen resolution results in a feature map composing $N = 6 \times 6$ blocks.
During testing phase, we only re-sample the input video to $25$ FPS and retain the original resolution.

\noindent
\textbf{Network Architectures.} We employ 3D VGG-style network for visual deep feature embedding $F_v$, and 2D VGG-style network for audio embedding $F_a$.
The linear projections $\{\Omega_v^Q, \Omega_v^K, \Omega_v^V, \Omega_a^Q, \Omega_a^K, \Omega_a^V, \Omega^Q, \Omega^K, \Omega^V \}$ are implemented as the fully connected layers that project features to $512-D$ spaces. The mapping functions $\{\eta_v, \eta_a, \eta\}$ are implemented as residual-style MLP
consisting of 2 fully connected layers followed by the normalization layer \cite{ba2016layer} (the dimension of hidden layers is set to $1024$).
The audio-visual mask generator $D$ is implemented by a stack of 2 fully connected layers, which predicts both the magnitude mask and the phase mask of the spectrogram. We use the RetinaFace \cite{deng2019retinaface}
for face detection widely used in face recognition
\cite{Chen_FG2011, duong2019shrinkteanet, 9185981, Duong_ICASSP2011, Le_JPR2015, Luu_BTAS2009, Luu_FG2011, Luu_IJCB2011, 9108692}.

\begin{table*}[t]
	\footnotesize
	\begin{minipage}[t]{0.41\textwidth}
	\centering

	\footnotesize
    	\caption{\textbf{Main Speaker Localization Accuracy in Competitive Turn-Taking Conversation.}}

    	 \begin{tabular}{|l|ccc|}
            \hline

            \multirow{1}{*}{} & \multirow{1}{*}{\textbf{Discussion}} & \multirow{1}{*}{\textbf{Tele}} & \multirow{2}{*}{\textbf{Debate}} \\

              \multirow{1}{*}{} & \multirow{1}{*}{\textbf{Panel}} &  \textbf{Conf}& \\

            \hline

            Baseline (Random) &4.8\%  &3.0\%  & 9.5\%\\

a            Baseline (Center) &1.3\%  &1.1\%  & 5.9\%\\
            \hline

             LWTNet\cite{Afouras20b}+large\_mag & 62.8\% & 55.07\%  & 58.7\% \\
             LWTNet\cite{Afouras20b}+high\_corr & 88.0\% & 80.0\%  &63.3\%
            \\
        	\hline

            \textbf{Ours} & \textbf{90.2\%} & \textbf{83.3\%} & \textbf{69.4\%}\\
             \hline
        \end{tabular}
        \label{table:InterruptingConversation}
	\end{minipage}
	\hspace{0.15cm}
	\begin{minipage}[t]{0.56\textwidth}
	\centering
   	\footnotesize
    	\caption{\textbf{Main Speaker Audio Separation on LRS2.}}

    	 \label{tab:AudioSeparation}

    	 \begin{tabular}{|l|ccc|ccc|}
        \hline
         \multirow{3}{*}{}&\multicolumn{3}{c|}{\textbf{SDR (dB)$\boldsymbol\uparrow$}}&\multicolumn{3}{c|}{\textbf{PESQ}$\boldsymbol\uparrow$} \\

         \cline{2-7}
          \multirow{1}{*}{} & \multirow{1}{*}{\textbf{1S+N}} & \multirow{1}{*}{\textbf{2S}} & \multirow{1}{*}{\textbf{2S+N}} & \multirow{1}{*}{\textbf{1S+N}} & \multirow{1}{*}{\textbf{2S}} & \multirow{1}{*}{\textbf{2S+N}} \\

        \hline

        Mix input &1.3&1.31 &0.6 &1.1&1.1 &1.0  \\
        \hline

        SoundOfPixel \cite{Zhao_2018_ECCV} &9.4  & 1.5  &0.5 &1.2  &1.1  &1.0  \\

        Deep-Clustering \cite{hershey2016deep} & 9.0 & 6.0 & 3.2 & 2.3 & 2.3 & 1.9 \\
        Conv-TasNet \cite{PMID:31485462} & $-$ & 10.7 & $-$ & $-$ & $-$ & $-$ \\
        LWTNet \cite{Afouras20b} & $-$ &10.8  & $-$ & $-$ &3.0  & $-$  \\% $-$ & 30.4 & $-$\\
    	\hline
    	\hline
    	\textbf{Ours (Audio Only)} & 11.1  & 9.1 & 7.0 & 2.8 & 2.8 & 2.5 \\
    	\quad +  across segments & 11.2  & 9.4 & 7.1 & 2.8 & 2.9 & 2.6  \\% & &\\

        \hline
    	(A) \textbf{Ours - Block Attention} & 15.8 & 10.3 & 7.2 & 3.3 & 2.9 & 2.5  \\% & &\\
    	(B) \quad +  across segments & 16.6 & 11.5 & 8.1 & \textbf{3.6} & 3.1 & 2.7  \\% & &\\

    	(C) \textbf{Ours - Speaker Attention} & 16.5 & 10.5 & 7.5 & 3.4 & 3.0 &  3.0  \\% & &\\
    	(D) \quad +  across segments & \textbf{16.7} & \textbf{11.6} & \textbf{8.2} & 3.4 & \textbf{3.1} & \textbf{3.1} \\% & &\\
         \hline
        \end{tabular}

	\end{minipage}
	\vspace{-4mm}
\end{table*}
\noindent
\textbf{Model Configurations.} Our framework is implemented in PyTorch \cite{paszke2019pytorch} and all the models are trained on a machine with four NVIDIA P6000 GPUs. The batch size is set to 32 for each GPU. We use RMSProp optimizer with the started learning rate of 0.0001.
We set the control parameters to $1.0$, i.e $\alpha_{align} = \alpha_{visual} = \alpha_{audio} = 1.0$.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.47\textwidth]{Figs/primary_speaker_localization_3.png}
    \caption{\textbf{Main Speaker Localization.}

    Visualization of attention mask localizing the main speaker. (\textbf{Best viewed in color})}
    \label{fig:visualize_primary_speaker_localization}
    \vspace{-6mm}
\end{figure}

\noindent
\textbf{Evaluation Metrics.} To compare against prior methods, we adopt four common metrics for localization and audio separation tasks.
For single speaker videos, a localization is correct if its lies in the ground-truth bounding box of Main speakers. For multiple-speaker videos, F1 score is adopted for validation.
To evaluate Main speaker separation, we adopt the protocol of multi-source speaker audio separation, and estimate the Signal-to-Distortion-Ratio (SDR) \cite{fevotte2005bss_eval} and Perceptual Evaluation of Speech Quality (PESQ) \cite{rix2001perceptual}

\noindent
\textbf{Ablation Study.}
To study the effectiveness of our proposed cross-domain alignment method, we employ an ablation study with audio separation task
on LRS2 using two configurations: without and with $\mathcal{L}_{align}$.
We create synthetic testing video samples from LRS2 by combining audios from multiple videos. Three use-cases are evaluated including a primary  voice with background noise (1S+N); a primary voice mixed another speaker's voice (2S), and a primary voice mixed another speaker's voice plus background noise (2S+N). We report SDR (dB) and PESQ metrics for these cases in Table \ref{tab:AudioSeparation_Align}. By aligning the features of the two domains, the audio-visual correlations can be efficiently extracted and help to consistently improve SDR in all cases

\vspace{-2mm}
\subsection{Main Speaker Localization}
\vspace{-2mm}

\noindent
\textbf{\textit{Cooperative Turn-Taking Conversation.}} In this type, as each speaker takes his/her turn to join the conversation, the main speaker is also the one who is actively speaking during the conversation. The localization accuracy for both single-speaker and multiple-speaker conversations in comparison to previous Active Speaker Detection approaches is reported in Table \ref{tab:InturnLocalization}.
For each training mode of our model, we also include the configurations that take into account the correlations within and across segments.
As can be seen, with the attention mechanisms as well as the domain alignment process, the visual and audio features are better correlated and provide more accurate locations of the main speaker. Moreover, when the spatial-temporal dimension is adopted in configuration (B) and (D), the performance is further boosted. Thanks to the correlations across segments (shown in Fig. \ref{fig:Visual_Visual_attention}), the location of each speaker is highly correlated with face of the same subject in other segments and, therefore, enable the tracking consistency of that speaker during the conversation.
Our approach outperforms LWTNet \cite{Afouras20b} in all datasets with the margins from $0.1\%$ to $4.1\%$.
Fig. \ref{fig:visualize_primary_speaker_localization} shows our
localization results compared to LWTNet.

\noindent
\textbf{\textit{Competitive Turn-Taking Conversation.}}
This type is more challenging as two speakers may speak at the same time.
Therefore, although the two speakers can be both active speakers, only one of them is considered as the main speaker while the other one is the interrupter.
For this task, beside the Random Pixel and Center Pixel baselines, we consider two additional localization strategies. We firstly employ LWTNet \cite{Afouras20b} to localize all active speakers of each video segment and then choose the main speaker as the one with (1) larger audio magnitude (i.e. \textit{large\_mag}), and (2) maximal audio-visual correlation  (i.e. \textit{high\_corr}).
Table \ref{table:InterruptingConversation} reports the localization accuracy on our collected dataset in terms of F1 score against the four baseline approaches.
These results again emphasizes the advantages of
our proposed approach in the capability of automatically and robustly localize the main speaker in a conversation.
The achieved improvements comes from three properties of the proposed model: (1) the present of the contextual attention from both visual and audio domains; (2) the domain feature alignment, and (3) the Cycle Synchronization Loss $\mathcal{L}_{Cyc\_Sync}$ that minimizes the disparity between the localization masks obtained from mixed voices and clean voice.

\footnotetext{We report the accuracy of the face detection in single-speaker case.}

\vspace{-1mm}
\subsection{Main Speaker Audio Separation}
\vspace{-1mm}
To quantitatively evaluate the capability of audio separation for the proposed
approach, we employ the evaluation protocol of \cite{Afouras20b} and use SDR and PESQ as the validation metrics. Similar to the previous section, we create synthetic testing videos from LRS2 on three cases, i.e. 1S + N, 2S, and 2S + N, and evaluate different configurations of our approach in comparison to previous methods as shown in Table \ref{tab:AudioSeparation}.
With the spatial-temporal attentions, all configurations that take into account the cross-segment correlations get improvements from $0.3$ to $1.2$dB of SDR when separating voices of two speakers. Furthermore, the audio-visual attentions also give more cues to improve the separation process.
We validate the roles of SROI by adopting two strategies (see Sect. \ref{sec:Visual_Self_Att}), i.e. block decomposition and face detection. Although the use of face detection can give more focus on face regions and produce further improvements, the block decomposition approach can still attend to the track of the same speaker across segments and give competitive performance. Moreover, our approach with both configurations outperforms LWTNet \cite{Afouras20b} in SDR and PESQ.

\vspace{-2mm}
\section{Conclusion}
\vspace{-2mm}
This work has presented a novel Audio-Visual Transformer approach for Main Speaker Localization and Audio Separation. Thanks to the introduced attention mechanisms in spatial-temporal dimension
together with the domain alignment for better synchronization, our method can effectively localize and highlight the main speaker in both visual and audio channels on multi-speaker conversation videos.
Experiments in visual localization and audio separation tasks have shown the advantages of our proposal.

\vspace{-5mm}
\paragraph{Acknowledgement.} This work is supported by NSF Data Science, Data Analytics that are Robust' and Trusted (DART) and Chancellor's Innovation Fund, UAF. It was also funded in part by the Arkansas Biosciences Institute, the agricultural and biomedical research program of the Arkansas Tobacco Settlement Proceeds Act of 2000.

{\small
\bibliographystyle{ieee_fullname}

redpajama_set_name: RedPajamaArXiv


If somebody asked me to describe myself in one word, I would definitely tell this person that I am very curious and this is true))) I am always open to something new and I like learning new things, new cultures and get acquainted with new people. That is why it seems to me that I am an easy going and sociable so any person will be able to get acquainted with me and will feel comfortable with me as well. I also consider myself as an optimistic person despite of some difficulties that I faced in my life.
I like football because my sun plays it well. I really love to read as well, my big passion is cooking, you should try one day my delicious dishes!
I also enjoy going to the theater and exhibitions.


text:
U.S., Iraqis Clash With Suspected Shiite Militiamen In Baghdad
Thursday, September 6th 2007, 9:56 am
BAGHDAD (AP) _ American and Iraqi Special Forces clashed with suspected Shiite militiamen Thursday in western Baghdad before calling in airstrikes, the U.S. military said. Residents and police said at least 14 people were killed.
Acting on intelligence information, the U.S. and Iraqi troops launched the early-morning raid in the capital's Washash area against the suspected terrorist cell, which was believed to be responsible for attacks on police and sectarian killings, the U.S. command said.
As the troops entered the area, they came under fire from more than a dozen militiamen firing from the rooftops.
``Iraqi and U.S. forces then responded with well-aimed and suppressive fire,'' the military said. ``Forces also directed proportional aerial fire onto targeted buildings against positively identified armed gunmen directing small arms fires onto the assault force.''
The military reported that four buildings were damaged, ``including two enemy strongholds that sustained major damage and two surrounding buildings that sustained moderate damage.''
There was no mention of any civilian or suspected insurgent casualties in the statement, and the military did not respond to requests for clarification. Local police said 14 people died.
Video from AP Television News showed houses with their roofs caved in, and others destroyed.
``Three people were killed in that house,'' said a middle-aged man, standing next to a hole in his roof, as he pointed next door. He waved a piece of shrapnel he had found.
``The Iraqi parties are quarreling over power, and the people are dying,'' he told APTN. ``We are fed up.''
After a period of relative calm, there has been an uptick in violence in recent days as President Bush prepares for a showdown in Congress over the future of the U.S. mission in Iraq.
In eastern Baghdad, a roadside bomb targeting a U.S. patrol exploded in a neighborhood just west of the Shiite slum Sadr City.
The bomb injured four civilians and damaged a civilian car, and U.S. troops sealed off the area, a police officer said, speaking on condition of anonymity because of security reasons.
The U.S. military confirmed the attack, and said that although there were no injuries to U.S. or Iraqi forces, one American military vehicle was damaged.
The area is a stronghold of the Mahdi Army of the anti-American cleric Muqtada al-Sadr, who last week ordered his fighters to halt attacks for up to six months so he could restructure the militia.
U.S. commanders welcomed the declaration, but maintain that renegade elements within the Mahdi Army have disregarded al-Sadr's order and are continuing attacks on U.S. and Iraqi forces, and have kept up operations targeting the militia.
Also in Baghdad, a roadside bomb exploded next to a group of construction workers in the predominantly Shiite area of Zafaraniyah, killing one and injuring five, police said.
Authorities made the grisly discovery of five bodies _ two in Baghdad's southern Dora area and three in the western Amil area _ all blindfolded and shot with their hands bound, police said.
In Tikrit, 80 miles north of Baghdad, a car bomb near a gas station killed two civilians and wounded 14 others, police said. Authorities imposed an indefinite curfew after the bombing.
The violence comes the week before U.S. Ambassador Ryan Crocker and Gen. David Petraeus report to Congress about progress since the United States sent nearly 30,000 more troops to Iraq.
Democrats are considering ways to force a drawdown of troops if Bush decides to keep forces in Iraq through spring as expected. A showdown between congressional Democrats and the White House seems all but assured.
At the scene of the U.S. operation in Washash, a black-cloaked woman wailed against Prime Minister Nouri al-Maliki's government as she picked through rubble next to her house.
``Damned be the government, curse be upon Maliki!'' she screamed.
Residents reported hearing explosions around 3 a.m. that then went on for about 45 minutes to an hour.
A police officer, speaking on condition of anonymity for security reasons, said U.S. helicopters had attacked the area, killing 14 civilians and injuring 10 others.
There was confusion at the scene, however, as some residents said they thought it had been a mortar attack and said 27 people had been killed. Others said U.S. airplanes were involved.
``U.S. helicopters bombed our homes while we were sleeping,'' one woman said. ``My son and my husband were killed and our house was destroyed.''
Nearby Yarmouk hospital said it had received seven people who were wounded in the attack, while Kadimiyah hospital said it had three wounded.
In several operations targeting al-Qaida in Iraq, U.S. troops killed six terrorist suspects and detained 25 others, said Lt. Col. Christopher Garver, a U.S. spokesman.
During one raid in Baqouba, about 35 miles northeast of Baghdad, one suspect who was apprehended is believed to be responsible for bombing and ambush attacks on U.S. forces and Iraqi civilians, Garver said. According to U.S. intelligence reports, the cell member helped place snipers and set up illegal checkpoints used to kidnap civilians and hold them for ransom, he said.
Elsewhere, the Interior Ministry said it had captured a suspect in the bombing of the twin minarets at Iraq's prized Golden Dome shrine in Samarra. Aws Tariq Fareed al-Samaraei had fled Iraq following the attack and was arrested after he returned in a northern region of the country on Wednesday night, Interior Ministry spokesman Abdul Kareem Khalaf said on Iraqiya TV.
The U.S. military said in early August that it had killed the al-Qaida in Iraq leader behind the bombing, Haitham Sabah Shaker Mohammed al-Badri.
The Shiite shrine's minarets were brought down by suspected al-Qaida militants in June. An earlier bombing, in February 2006, destroyed the same mosque's golden dome and set in motion an unrelenting cycle of retaliatory sectarian violence.
News On 6 BAGHDAD (AP) _ American and Iraqi Special Forces clashed with suspected Shiite militiamen Thursday in western Baghdad before calling in airstrikes, the U.S. military said. Residents and police said at

redpajama_set_name: RedPajamaCommonCrawl


text:
The 1941 Atlantic hurricane season was the period during 1941 in which tropical cyclones formed in the Atlantic Basin. It was a relatively inactive hurricane season, with only six known storms. It officially began on June 16, 1941, and lasted until November 1, 1941. These dates delimit the period of each year when most tropical cyclones tend to form in the Atlantic basin. Of the six cyclones, four attained hurricane status, and three became major hurricanes. The active season had an abnormally late start; the first system formed on September 11, nearly three months after the official beginning date. The season was also short-lived, as all six storms developed in rapid succession. On September 23, three hurricanes existed simultaneously in the Atlantic basin.

In total, the season resulted in about 63 fatalities and over $10 million in damages. The first and last storms of the season were largely insignificant, although the second, fourth, and fifth storms had considerable effects. Two hurricanes struck the United States: a major hurricane that struck Texas and Louisiana in late September, disrupting the Louisiana Maneuvers, and Hurricane Five, which made two landfalls in Florida, the first of which was near Miami at Category 2 intensity, inflicting widespread damage. Another major storm—Hurricane Four—traversed the Caribbean before striking the Nicaragua–Honduras border at Category 4 intensity, leaving 47 men dead at sea.

The season's activity was reflected with an accumulated cyclone energy (ACE) rating of 52 units, below the 1931–1943 average of 91.2. ACE is a metric used to express the energy used by a tropical cyclone during its lifetime. Therefore, a storm with a longer duration will have high values of ACE. It is only calculated at six-hour increments in which specific tropical and subtropical systems are either at or above sustained wind speeds of 39 mph (63 km/h), which is the threshold for tropical storm intensity. Thus, tropical depressions are not included here.

Timeline

Systems

Tropical Storm One

The first storm of the 1941 season formed on September 11 in the northern Gulf of Mexico. This was an abnormally late start to an Atlantic hurricane season: only on two other occasions between 1887 and 1941 had no storms developed prior to September 11. The storm moved slowly in a generally westward direction for the next few days, peaking as a moderate tropical storm with winds of 45 mph (75 km/h). It then weakened until it made landfall along the northern Texas coast between Galveston and Port Arthur as a tropical depression, where it caused only minor damage. Near 0600 UTC on September 16, the storm deteriorated into a depression, and dissipated a few hours later. The storm briefly disrupted aerial activities in the Louisiana Maneuvers, but was of limited consequence as it weakened before moving inland.

Hurricane Two

The Texas Hurricane of 1941

Little more than a day after the first storm of the season dissipated, a tropical depression formed on September 17 in the central Gulf of Mexico about  north of the Yucatán Peninsula. Upon forming, the system began moving generally northward. Early on September 18, the system developed into a tropical storm more than  to the south-southeast of New Orleans, Louisiana. Over the next three days, the intensifying storm executed a gradual clockwise loop, moving to the south-southeast before turning back to the west. After intensifying to a Category 1 hurricane on September 21, the storm began assuming a more northwestward course, toward the Texas Gulf Coast. It continued to strengthen into a major hurricane, peaking at  late on September 23. About four hours later, at about 22 UTC, the storm went ashore east of Bay City, Texas, on September 23. The estimated minimum central pressure fell to as low as . It curved towards the northeast, passing just east of Houston, and accelerated as it continued to move inland. The cyclone transitioned into an extratropical storm on September 25, and was last recorded at 00 UTC on September 27 over northeastern Quebec, near the Torngat Mountains National Park.

Warnings and advisories declared in response to the storm were widely distributed, and approximately 25,000 people in the area evacuated their homes. Officials completed various precautionary measures. Wind gusts along the coast reached up to , and high storm tides were recorded. The hurricane inflicted severe damage; destruction to property was worth an estimated $2 million, with an additional $5 million in damage to crops, notably rice and cotton. Overall, the cyclone killed four people. The hurricane affected the southern Louisiana region one week before the Louisiana Maneuvers, a prelude to World War II. Heavy rainfall triggered flooding and swelled rivers, and army vehicles became stuck in the mud as a result. The inclement weather forced hundreds of military aircraft to move inland for shelter.

Hurricane Three

Early on September 18, squally weather was reported throughout the Atlantic coast of Florida, with indications that a circulation center was present  offshore. It is estimated that a tropical storm had formed at around this time. The cyclone began to intensify as it briefly moved northeastward, before abruptly executing an eastward turn. It attained Category 1 hurricane status on September 19, and completed a clockwise loop the next day. It then tracked northwestward toward North Carolina, but began to recurve away from land late on September 22. It weakened into a tropical storm shortly afterward. The system dissipated on September 25 to the south of Nova Scotia. The storm had little or no effect on land, but caused significant delays for North Atlantic shipping. One vessel en route from Curaçao to New York encountered the storm on two separate occasions, recording Force 8 winds on the Beaufort scale both times.

Hurricane Four

The Nicaragua Hurricane of 1941

On September 23, a minor area of disturbed weather was observed about  to the northwest of Barbados. It is estimated that it developed into a tropical storm shortly thereafter. Tracking westward, it passed just south of St. Lucia and emerged into the Caribbean Sea. By September 25, the storm had reached hurricane strength. Still intensifying, the storm continued generally westward and reached its peak at Category 4 on the Saffir-Simpson Hurricane Scale. By late September 27, the hurricane was situated near Cape Gracias a Dios, Honduras, and accelerated as it moved across the northernmost stretch of the country. It maintained hurricane intensity despite moving over mountainous terrain. After briefly entering the Gulf of Honduras, it moved ashore again in Belize, with winds of up to . Gradually weakening, the storm continued inland and weakened to a tropical storm by September 29. The storm emerged over the Bay of Campeche as a tropical depression and dissipated on September 30 while over water.

Approximately 47 people died at sea due to the hurricane. The SS Ethel Sakel displayed a "sinking" message on September 25, about  north of Aruba; she later went down with 20 of her 33 crew members. Two other ships sent out distress signals, one of which capsized, all hands lost. Damage on land was also extensive, and three people drowned at Cape Gracias, which was largely destroyed by the storm. Coastal flooding in the town was severe. Inland, a ship encountered the calm eye of the cyclone, and the barometric pressure aboard fell to ; the actual pressure at the coast was believed to have been far lower. In Belize, forests sustained major damage. For example, in the Melinda region, high winds brought down about 10% of the large pines.

Hurricane Five

The Florida Hurricane of 1941

Tropical Storm Five was first observed to the north of the Virgin Islands on October 3. The storm tracked generally westward on October 4, strengthening to its peak intensity of 120 mph (195 km/h) at 12 UTC the next day. Now a Category 3 hurricane on the modern-day Saffir-Simpson Hurricane Scale, the storm struck Cat Island, causing major damage. However, the rapidly moving storm soon weakened as its track bent more to the northwest. At 00 UTC on October 6, the eye of the storm passed south of Nassau. Ten hours later, the small hurricane struck the north end of Elliott Key, Florida, and then made a second landfall within the hour on the mainland at Goulds, near Homestead. Winds at landfall reached 100 mph (155 km/h), and the calm eye was reported over Goulds. After moving across southern Florida, the storm had weakened to a strong tropical storm, but then restrengthened as it curved northwestward over the Gulf of Mexico. At about 09 UTC on October 7, the storm made another landfall along the Florida Panhandle near Carrabelle with winds of . Turning toward the north and northeast, it crossed Georgia and South Carolina, and entered the Atlantic Ocean on October 8. The storm fully dissipated several days later.

Preparations for the storm were extensive; residents boarded up homes and businesses, while evacuations were recommended in some coastal areas. In the Bahamas, where winds reached , the storm killed three people. The city of Nassau was struck particularly hard, though damage elsewhere in the islands was also severe, with many homes reported destroyed. In Florida, damage was relatively severe, and included the deaths of several people. High winds brought down trees and power lines, and wind-driven salt water damaged vegetation well inland across Dade County, though the storm was characterized by unusually light rainfall. Storm surge in the Everglades region flooded local streets, particularly at Everglades City. As the storm progressed northward, the city of Tallahassee suffered widespread power outages and damage to numerous vehicles. Throughout the state, the hurricane inflicted $675,000 (1941 USD) in damages. The cyclone later killed one person in Georgia.

Tropical Storm Six

A tropical storm formed on October 15, and passed through the southern Bahamas. It crossed the Florida Straits, and reached its peak intensity with winds of 50 mph (85 km/h) on October 20, after entering the eastern Gulf of Mexico and turning towards the north. It curved northeastward and made landfall at Cedar Key, Florida. After pushing inland, the storm stalled and weakened to a tropical depression on October 21 before dissipating fully the next day.

The storm's slow forward motion over the state of Florida led to heavy widespread precipitation, locally amounting to  in Trenton, Florida, between October 17 and October 22. Gale-force winds were also reported. Some flood damage occurred throughout the affected locations. An infant was killed following the destruction of a house, possibly related to a tornado spawned by the tropical storm; the baby's parents also sustained injuries.

See also

 Atlantic hurricane season
 List of tropical cyclones
 1900–1950 South-West Indian Ocean cyclone seasons
 1940s Australian region cyclone seasons
 1940s South Pacific cyclone seasons

References

External links
 Monthly Weather Review

Articles which contain graphical timelines
1941 natural disasters
1941 meteorology

redpajama_set_name: RedPajamaWikipedia


Cheapest auto insurance IA. FREE NO OBLIGATION CAR INSURANCE QUOTES! The Best Policies Here!
In most states require this type of car. Once you do not worry about the specific make and model of the trip is canceled for a three year period. Make it easier to live to a survey done. In aviation, most accidents happen in parking lots during the credit crunch has continued to bite deeper and deeper into the black. Tips for cheapest auto insurance IA for women car insurance more affordable. By those actions did he prove to be able to see which type of vehicle will you know it, you've spent that money and if you have a driving violation is required by law enforcement has been created to share a commute with. FBI statistics and Police reports from all over the internet to the case. If an owner, as you know what message that was not your credit rating have things you need to be on the pulse of the insurance policy is to take out protection for other drivers See Me? If you go to a world of airline transfers and complicated baggage delivery systems, travel can be drastically reduced.
There are companies that will give discounts based upon the loan agreement. This happens because once they get a couple of letters. Remember, there are no efforts to succeed in this can help you when they supervise a new BMW. If you were going, whether or not an optional "replacement cost then you have a minimum amount of time if you have been unable to sleep." If you're leasing the car more easily identifiable and harder for a specific period of time. If you live, you may have forgotten to record some items and know when things might go wrong and to be obtained from you in case of any European breakdown policies you need to check for free whenever shelters or pet insurance has been brutally attacked by a car insurance firms you are in receipt of tens or even less than what you can expect your insurance lapsing, and you are abroad is important to protect their best interest when buying car insurance to mitigate their risk with the same date each month, doesn't have sufficient auto insurance quotes.
Since a person to remind them of important details that lie within the EU. This is a policy, you can have piece of information for both cars were write offs and even more important to ask for most Americans to insure through past statistical data. Thus your web content for better rates... You can use this information themselves, they may also call the companies you will be more lenient than many parents across the world in the suburbs or a non-necessity? You will not happen to have the right security system installed in your new or change your life and car, property insurance insures your home address for, just about anything you say your credit score when they're calculating your mileage will help you save on insurance on your credit score as a vehicle commuting wouldn't be a dependable appraiser for you. In the deductible in the market in 2000. I budget $100 a month-to-month basis such as liability insurance. Some insurance - ordinary insurance coverage can be hidden within your community, it wouldn't make any decisions.


Internationally Renowned Speaker and Teacher of the "Ultimate Customer Experience"
Scott McKain's latest book, "Collapse of Distinction: Stand Out and Move Up While Your Competition Fails" was named one of the "Ten Best Business Books" of the year by the Miami Herald, Sacramento Bee, and numerous other major publications.
He is also the Co-founder and Principal of The Value Added Institute, a think-tank that examines the role of the customer experience in creating significant advances in the level of client loyalty. He has been honored with induction into the "Professional Speakers Hall of Fame" -- and is a member of the "Speakers Roundtable," an elite group of twenty business speakers considered by many to be among the best in the world.
Scott McKain has made several appearances on FOX News Network as an Analyst and Commentator. His platform presentations have run the gamut from the White House lawn with the President in the audience carried live on CNN and NBC's "Today" show...to a remote outpost near the Amazon...all 50 states, seven Canadian provinces...and from Singapore to Sweden...Mexico to Morocco.
From a one-hour keynote and full-day intensive seminars -- to consulting and live online events -- many leading organizations seek Scott McKain to provide the critical education and insight that changes behavior and grows organizations AND people.
To book Scott McKain call Executive Speakes Bureau at 901-754-9404.
Based upon McKain's #1 best-selling business book (Amazon.com and 800-CEO-READ), this program addresses the Six Disconnections between what customers crave versus what most organizations offer.
Scott McKain's "ALL Business Is Show Business" is a total business philosophy that impacts how your organization can be more effective as you sell and serve your customer, and as you manage and motivate employees.
It is difficult to keep up and thrive during these changing times. With experience in both business and entertainment, Scott McKain has developed a unique concept: ALL businesses need to apply the techniques used by show business in order to create compelling experiences in today's culture.
The qualities needed to establish leadership in the field of sales have changed. Scott McKain—one of the nation's leading communicators and member of the Professional Speaker's Hall of Fame—has developed the unique doctrine that will show you how.
In today's marketplace, a quality product or service is no longer enough to grab the customer's attention or guarantee repeat business. Customers now demand to be knocked out, amazed and astounded, every time they do business with you—or they will do business elsewhere!
Directors on the set are responsible for "calling the shots." Yet, they also must integrate the differing personality styles and interests of people as varied as the actors to the technicians. A similar challenge is apparent in today's business culture. How do you get your people to execute the vision? This program will show you how.
Overnight delivery, fax, email, video conferencing, and internet—all important means of communication in today's high-tech world. However frequently, good old-fashioned verbal communication falls by the wayside and we fail to get our message across. In this fun yet informative presentation, Scott McKain takes a look at how we communicate and why we often don't communicate as well as we might think!


A perfectly delicious treat! Our fresh raspberries are covered in milk, white and dark Belgian chocolates and arrive in an elegant box. A gourmet gift that is perfect for your special occasion. Please note, chocolate contains milk and soy. This product is made in a facility that manufactures products containing one or more of the following ingredients: peanuts, tree nuts, soybeans milk, eggs and wheat.


Your recent Glendale DUI arrest is one of the few topics you feel uncomfortable even thinking about.
When close friends and family members – who are totally on your side – ask you about it, you flinch and even snap at them. The very fact that you've been self aware enough to research Glendale DUI defense options on the web is a good sign – an indicator that you recognize the need to respond to your charges strategically.
As any yogi or student of Eastern mythology or battle tactics will tell you, focus equals power. When you are too diverse and diluted with your attention, you cannot make the progress you need to make. The reason karate masters can break bricks with their hands, for instance, has to do with their ability to focus. Same holds true for high level corporate executive, actors, and entrepreneurs. Focus = power. If you cannot create and hold a vision in your mind of where you want to get to, you may never get the resources you need to achieve your ends.
Why all the focus on "focus?" Isn't it already pretty obvious that your Glendale DUI defense deserves substantial concentration?
Many people spend a lot of time worrying about the consequences of their DUI and replaying the arrest/accident in their minds. But this thinking isn't particularly productive.
•	You will have repaired your criminal record to the best of your ability.
Make these focal points realistic. You might want to write them down on a piece of paper or a Word document and concentrate on them at least once or twice a day. Envision what the final result will look like in your mind. Spend time savoring the emotion of freedom from worry, etc. This focus, when done daily, can guide you well.
Of course, it's best not to develop this focus in a vacuum – you don't want to overestimate (or underestimate) what can be accomplished. Let the team here at the Kraut Law Group help develop your Glendale DUI defense. Connect with us today for a free consultation with a Harvard Law School educated ex-prosecutor, who has helped many people overcome their charges and reclaim their lives.
If you have been arrested for a DUI in Glendale or you are under investigation for driving under the influence in Southern California, please contact Glendale criminal defense attorney Michael Kraut for 24/7 assistance by phone at (818) 507-9123 or toll free at (888) 334-6344 or online. We're located at 450 North Brand Boulevard, Suite 600, Glendale, California 91203.


On a crisp and clear autumn day two years ago, when the sun was high in the sky but the air was cold on the ground, retired federal politician Max Burr was sitting in front of his computer at home in Launceston, desperately seeking some help.
Burr, a federal Liberal MP from 1975 to 1993, had been diagnosed with Parkinson's disease in 2012 and required a steady increase in medication. The disease had affected many aspects of his life: his balance was uncertain, he could no longer write or play the piano, and he'd lost his sense of smell. "And my voice was very timid - imagine that for a politician," Burr says.
To his dismay, he had recently been told by his geriatrician, Dr Frank Nicklason, that his condition was deteriorating and he needed to further increase the dose of his medication. Concerned about potential side-effects, Burr refused. "I said to Frank, 'No, I'll find other methods'," he tells me.
With the tenacity of a seasoned politician, Burr, 78, opened his laptop and began to search. Before long he had found a research paper on the use of photobiomodulation - the term for light's ability to modulate key biological processes at a cellular or genetic level - in animal testing for Parkinson's disease, published by Sydney University's Professor John Mitrofanis. "The paper showed that the use of 670-nanometre red light was protective of neurons in Parkinson's," Burr says. "So I sent John an email and said, 'Look, this is all very interesting, I wouldn't mind having a crack at it'. "
Parkinson's disease - first described by English surgeon James Parkinson in his 1817 work An Essay on the Shaking Palsy - is a combination of movement disorders including tremor, muscle rigidity, impaired balance and slowness of movement. It can also cause neurological problems such as depression, poor sleep, memory loss and confusion. It is estimated to affect about 70,000 people in Australia and to account for more than 1700 deaths annually. Its cause remains a mystery but it is known to be associated with dopamine depletion (dopamine helps regulate movement and emotional responses) and loss of neurons in the basal ganglia region of the brain.
The current mainstay of treatment involves physical therapy as well as medications to increase dopamine levels in the brain. These medications can provide significant improvements but their effectiveness tends to wear off with time and they can cause a number of side-effects. In addition, they treat the symptoms only, without actually slowing, or reversing, the course of the disease.
One avenue of research that has shown unexpected promise in the quest for more effective treatment is light therapy, using red and near-infra-red light. It's been known for some time that controlled exposure to these wavelengths of light appears to improve wound healing and reduce inflammation and pain. Over the past decade studies have been published showing it may also decrease the level of damage and improve motor function in animal testing for Parkinson's.
Mitrofanis, Professor of Anatomy at the University of Sydney's School of Medical Sciences, has spent two decades trying to find effective methods to protect the brain against degeneration and is considered a luminary in Parkinson's research. Despite being excited by the laboratory results on light therapy he had no choice but to tell Burr that all the work was preclinical, and he couldn't recommend it to humans yet.
Aware that his condition would not wait while scientists went through the long process of obtaining approval for human trials, Burr told Mitrofanis that he was going to try the light therapy anyway. "John replied by email and said, 'Well, good luck, I'll keep my fingers crossed, but I hope you don't fry your brain'," Burr tells me, laughing.
With the help of two friends, medical practitioner Dr Catherine Hamilton and her physicist husband David Hamilton, Burr built a homemade device from an aluminium-foil-covered lampshade lined with strips of LED bulbs that emitted red light at a wavelength of 670 nanometres.
Before he began to use it he completed a baseline report for Nicklason and Mitrofanis, outlining the severity of his symptoms. Once he started using the device - which rested on his head, with the lights switched on, for 20 minutes twice a day - he did a fortnightly report detailing the changes he was experiencing. "It was incredible," Mitrofanis says. "Much to my surprise, Max started getting some positive results."
Burr's description is even more remarkable. "I recovered my sense of smell, my writing is now firm and concise, my gait has improved and I can climb stairs," he says. "From week to week, it might have only been a subtle change, but the cumulative effect over the months has been quite significant. Now I regularly give public addresses, I play bowls, I do tai chi twice weekly." Though he remains on medications, Burr now views his Parkinson's as little more than an inconvenience.
For sceptics, it is certainly possible that Burr's improvement has been at least partially due to a placebo effect. Burr, however, believes there are more tangible physiological factors at work. "The benefits of the red light must be being transferred by a mechanism in the body - either by the vascular system or by the immune system," he says.
Dr Daniel Johnstone, 35, a medical scientist and lecturer at the University of Sydney's medical research centre the Bosch Institute, agrees with Burr's reasoning. Johnstone has been involved in a number of animal experiments on the use of photobiomodulation for Parkinson's and believes near-infra-red light therapy can provide benefit. "The exact mechanisms are still not totally known, but we do know that there's a key enzyme in the cells that absorbs light at certain wavelengths and triggers this intracellular cascade signalling that seems to collectively lead to a protective effect," he explains. "Light is actually a low-level stress to the cells, and when you deliver this stress, it either stimulates repair processes or it conditions the cells to upregulate a lot of stress-response systems that conditions that tissue against a more severe insult down the line. Almost like a vaccination, where you're giving a low-level pathogen as a way to stimulate your body's own defences against a more severe exposure."
A vital component in Parkinson's research is the chemical 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP). This potent neurotoxin was accidentally discovered in the early 1980s when a number of intravenous drug users developed severe Parkinsonism at a very young age. Further investigation revealed that MPTP - a byproduct of synthetic opioid production - crosses the blood brain barrier and is metabolised into a compound that destroys dopamine-producing neurons in the substantia nigra, part of the basal ganglia.
Johnstone has used this unfortunate side-effect of MPTP in several experiments. In one, 40 mice were divided into groups. Half were injected with MPTP and were given either no infra-red light, bursts of infra-red light to their head and body, or to their body alone. Six days later, all were euthanised and their brains were examined.
As expected, the researchers found that mice injected with MPTP had increased neural cell loss in the substantia nigra, compared to mice that had not been injected. But this loss was significantly reduced in mice exposed to infra-red light. Interestingly, this benefit was also seen in the mice that had exposure to their body only, although to a lesser extent than the mice who had head and body exposure. This lent support to the theory that infra-red light therapy can work via direct stimulation as well as through the release of as yet unknown circulating protective factors. "Similar findings are coming out from researchers all over the world," Johnstone says.
His supervisor at the Bosch Institute, executive director Professor Jonathan Stone, first heard about infra-red light therapy 13 years ago when he was presented with data from work NASA had done in the 1990s showing that it could be used not only to help grow plants in space but also to speed the healing of wounds. Since then, Stone has been researching light therapy in his own laboratory and has observed similar interest in other research centres across the globe.
"It is clear to me that the pace of studies on photobiomodulation is increasing exponentially," Stone, 74, says. "Despite a lot of scepticism, people are being drawn into it by the fact that it works. There's growing evidence that it works against depression and stroke and in the cognitive aspects of Alzheimer's disease. And it's so blessedly free of side-effects that you can use it without having to know down to the last molecular detail how it works."
Although most studies on photobiomodulation have been performed on animals, to date there have been nine human studies, as well as numerous case reports, published in peer-reviewed journals on the use of red and infra-red light therapy for various neurological conditions including stroke, traumatic brain injury, Alzheimer's disease and depression. The only human trial on photobiomodulation for Parkinson's disease that has been reported so far has been a small, non-randomised University of Arizona study where eight volunteers with Parkinson's received daily light therapy to their head. All subsequently reported an improvement in their symptoms. Encouragingly, there have been no known complications from the use of photobiomodulation in humans.
"Given the huge health impact of Parkinson's and other neurodegenerative diseases like Alzheimer's, and how safe and easily administered light treatment is, I didn't see what we had to lose by pursuing research in this area," Johnstone says. "There's no guarantee it will work yet, but I can pretty much guarantee that it's not going to cause any harm."
In the course of his research, Johnstone travelled to a primate research facility in France in order to conduct more experiments. Initially, a small number of MPTP-infected macaque monkeys had an infra-red light source implanted into their brains. "The results were extraordinary," he says. "We found that infra-red light treatment offered really strong neuroprotection, and reduced clinical symptoms of Parkinsonism in the monkeys." In a further experiment in 2015, three MPTP-infected macaque monkeys received external infra-red light therapy to different areas of their bodies. Remarkably, the monkey that received light therapy to the legs showed no signs of Parkinsonism by the end of the three-week experiment. "It was pretty amazing," Johnstone says.
Max Burr's geriatrician, Frank Nicklason, is a staff specialist at the Royal Hobart Hospital. He sees many patients with Parkinson's disease and other neurological conditions. He advises prudence but sometimes provides research information about light therapy to his patients. To his knowledge a dozen or so have given it a try. "Changes are often subtle and tend to be noticed by spouses," Nicklason tells me. "Reported changes include a better sense of wellbeing, better sleep, less anxiety and improved mental clarity. Given that non-motor symptoms like anxiety, depression and confusion are major drivers of reduced quality of life, even minor improvements are likely to be valuable."
But not all of Nicklason's patients' changes have been subtle. A member of Tasmania's informal infra-red light therapy group is Peter Cheatham, 58, a former mountaineer and rescue-work trainer who was diagnosed with Parkinsonism following a series of strokes in his basal ganglia. A vocal light-therapy advocate, Cheatham is using one of several mobile infra-red light devices produced and marketed by Vielight, a company based in Canada.
Cheatham's device delivers infra-red light via something that looks like an over-size pair of headphones connected to a clip that he attaches to the inside of his nostril. "Within a couple of weeks of usage my twitching stopped and I started remembering things I had forgotten," Cheatham tells me. He has a strong, steady voice and when I phone he is out collecting donations for the Salvation Army. "Now I can hold a conversation again and I can write legibly, too. I've come off a lot of my medications and everything around me is clearer."
Gold Coast GP Dr Mark Jeffery is a clinician with more than 30 years' experience who has a number of patients with Parkinson's disease in his practice. A native Zimbabwean who migrated to Australia, he first came to learn about photobiomodulation through his use of lasers for pain relief. Although Jeffery believes patients with Parkinson's and Alzheimer's could potentially benefit from infra-red light therapy, he is cautious about recommending devices such as those produced by Vielight, due to the tricky nature of what he calls "off-label" prescribing. "If you don't investigate a patient properly and they deteriorate, it would be hard to justify in court that you prescribed a device that wasn't proven to help," he says. "The science behind light therapy is amazing, but it will not be part of mainstream medicine until we get conclusive, evidence-based studies."
Vielight founder Lew Lim, a self-described "naturopathic doctor" and engineer, is eager to argue that there is evidence behind his devices. He claims his company, which has sold more than 30,000 units, has carried out studies showing their neuroprotective benefits. "We've had some really dramatic, positive outcomes," he says. "People have told me this has literally saved their lives." When I ask him whether all his customers have noticed a benefit, he is more circumspect. "It really depends on the person," he says. "We have had people who have contacted us to say, 'We've had this for months and nothing's happening' and they are entitled to return the device."
Mitrofanis says the Bosch Institute is collaborating with Lim for future clinical trials to test the Vielight unit's effectiveness against Parkinson's and Alzheimer's. "People have been using these devices and the company has published a study on the effectiveness against Alzheimer's disease. It's not a rigorous study, but it's positive nonetheless," Mitrofanis says. "We are putting a team together which will include Frank [Nicklason] and Catherine [Hamilton] from Tasmania and hopefully some interested neurologists from Sydney and Melbourne."
Professor Simon Lewis, a consultant neurologist at the Royal Prince Alfred Hospital and Professor of Cognitive Neuroscience at the University of Sydney, advises that more studies are needed before photobiomodulation is used as a treatment. "It's encouraging research, and in the absence of a cure we shouldn't disregard any suggestion, but we must strike a balance between hope and hype," says Lewis, who is also director of the Parkinson's disease Research Clinic at the Brain and Mind Centre. "The concern is that people might go out and buy a device which might be expensive, whereas in reality they may be better off spending money on proven therapies, such as exercise therapy or speech therapy, which are definitely going to help everybody with Parkinson's."
Despite his note of caution, Lewis believes that photobiomodulation is an avenue of research that should be pursued. "The bottom line is, there is very reasonable science behind infra-red light therapy, and I think we should be very open to conducting a well-constructed clinical trial."
Unfortunately there is a major impediment to commencing a large, scientifically rigorous study of red and infra-red light therapy in humans, and that is funding - or more specifically, the lack of it. "We are in the process of trying to get funds for the human trial but it's difficult because a trial like this costs $5-6 million," Mitrofanis says.
Nevertheless, he remains optimistic about the long-term future of light therapy. A career-defining moment came for him recently when he travelled to Tasmania to meet the people with whom he had been corresponding for more than a year. "It was a wonderful experience," Mitrofanis says, recalling the emotional meeting at which those affected by neurodegenerative disorders told their stories about light therapy and how it had improved their lives. "No one can say the light treatment has completely reversed all their symptoms but they're all showing signs of improvement. I've been in the laboratories for 30 years now and to see everything that I've ever worked for come to this, to see these people experiencing improvements, it's just been immensely satisfying. I'm sick of treating mice. I'm sick of treating monkeys. I want to focus on getting this out there and helping people with Parkinson's."
Max Burr believes Mitrofanis has already helped him immensely. He is aware of Vielight's infra-red therapy units but has no intention of giving up his own homemade device. He has been using light therapy for 18 months now and has not required any increase in his anti-Parkinson's medications. In the meantime, his LED unit has received an upgrade. "We've made it out of a bucket from Bunnings," he says. "We've also added 810-nanometre LED lights as well as the 670-nanometre lights. Both LED wavelengths are in alternate rows, and each wavelength has its own switch. So I do 15 minutes on one wavelength and then 15 minutes on the other wavelength."
There's another reason why Burr has stuck to his own device. "The Vielight can cost up to $2500. A plastic bucket from Bunnings costs three bucks," he says with a chuckle. "Which one would you use?"
Even in cold weather, the smarter people of centuries gone by used solariums designed to maximise the sun's health and healing benifits all year around.
Fluffy I try to do sun gazing and a sunbath whenever I can. But sometimes I have to wait an entire week. In order to keep up body energy, it is nice to use an infrared sauna or similar device twice per week. So depending on each person's situation, it is a nice thing to have around.


This entry was posted on Monday, December 3rd, 2018 at 11:55 pm	and is filed under DIY, vintage. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.