text:
\section{Introduction}
\label{sec:intro}

Our main goal is to develop a monaural conversation transcription system
that can not only perform automatic speech recognition (ASR) of multiple talkers
but also determine who spoke the utterance when, known as speaker diarization \cite{tranter2006overview,fiscus2007rich}.
For both ASR and  speaker diarization, the main difficulty comes from speaker overlaps.
For example, a speaker-overlap ratio of about 15\% was reported in real meeting recordings \cite{yoshioka2018recognizing}.
For such overlapped speech, neither conventional ASR nor  speaker diarization provides a result with sufficient accuracy.
It is known that mixing two speech significantly degrades ASR accuracy \cite{yu2017recognizing,delcroix2018single,kanda2019two}.
In addition,
no speaker overlaps are assumed with most conventional  speaker diarization techniques,
such as clustering of speech partitions (e.g. \cite{tranter2006overview,meignier2010lium,sell2014speaker,senoussaoui2013study,wang2018speaker}),
which works only if there are no speaker overlaps.
Due to these difficulties,
it is still very challenging to perform ASR and speaker diarization for monaural recordings of conversation.

One solution to the speaker-overlap problem is
applying a speech-separation method such as deep clustering \cite{hershey2016deep}
or deep attractor network \cite{chen2017deep}. However, a major drawback of such a method
is that the training criteria for speech separation do not necessarily maximize
the accuracy of the final target tasks.
For example, if the goal is ASR, it will be better to use training criteria that directly maximize ASR accuracy.

In one line of research using ASR-based training criteria,
multi-speaker ASR based on permutation invariant training (PIT) has
been proposed \cite{yu2017recognizing,chen2018progressive,settle2018end,seki2018purely,chang2019end}.
With PIT, the label-permutation problem is solved by considering all possible permutations
when calculating the loss function \cite{yu2017permutation}.
PIT was first proposed for speech separation \cite{yu2017permutation} and
soon extended to ASR loss with promising results \cite{yu2017recognizing,chen2018progressive,settle2018end,seki2018purely,chang2019end}.
However, a PIT-ASR model produces transcriptions {\it for each utterance} of speakers in an unordered manner,
and it is no longer straightforward to solve speaker permutations {\it across utterances}. To make things worse, a PIT model trained with ASR-based loss normally does not produce separated speech waveforms, which makes speaker tracing more difficult.

In another line of research, target-speaker (TS) ASR, which automatically extracts and transcribes
only the target speaker's utterances given a short sample of that speaker's speech,
has been proposed \cite{zmolikova2017speaker,delcroix2018single}.
{\v{Z}}mol{\'\i}kov{\'a} et al. proposed a target-speaker neural beamformer
that extracts a target speaker's utterances given a short sample
of that speaker's speech \cite{zmolikova2017speaker}.
This model was recently extended to handle ASR-based loss to maximize ASR accuracy
with promising results \cite{delcroix2018single}.
TS-ASR can naturally solve the speaker-permutation problem {\it across utterances}.
Importantly, if we can execute TS-ASR for each speaker correctly,  speaker diarization is solved at the same time
just by extracting the start and end time information of the TS-ASR result.
However, one obvious drawback of TS-ASR is that it cannot be applied
when the speakers in the recordings are unknown because it requires a sample of the target speakers in advance of decoding.

Based on this background, we propose a speech recognition and  speaker diarization method
that is based on TS-ASR but can be applied without knowing the speaker information in advance.
To remove the limitation of TS-ASR,
we propose an iterative method, in which
(i) the estimation of target-speaker embeddings and (ii) TS-ASR based on the estimated embeddings
are alternately executed.
As an initial trial, we evaluated the proposed method
by using real dialogue recordings in the Corpus of Spontaneous Japanese (CSJ).
Although it contains the speech of only two speakers,
the speaker-overlap ratio of the dialogue speech is very high; 20.1\% .
Thus, this is very challenging even for state-of-the-art ASR and  speaker diarization.
We show that the proposed method effectively reduced both word error rate (WER) and diarizaton error rate (DER).

\begin{figure*}[t]
	\centering
	\includegraphics[width=165mm]{overview.pdf}
	\vspace{-4mm}
	\caption{{\it (left) Overview of simultaneous speech recognition and speaker diarization, (right) proposed iterative maximization method.}}
	\label{fig:overview}
	\vspace{-3mm}
\end{figure*}

\section{Simultaneous ASR and  Speaker Diarization}
\label{sec:proposed}

In this section, we first explain the problem we targeted
then the proposed method with reference to Figure \ref{fig:overview}.

\subsection{Problem statement}

The overview of the problem
is shown in Figure \ref{fig:overview} (left).
We assume a sequence of observations $\mathcal{X}=\{{\bf X}_1,...,{\bf X}_U\}$,
where $U$ is the number of observations, and ${\bf X}_u$ is the $u$-th observation consisting
of a sequence of acoustic features.
Such a sequence is naturally generated when we separate a long recording into small segments
based on
voice activity detection
which is a basic preprocess for ASR so as not to generate
overly large lattices.
We also assume a tuple of word hypotheses ${\bf W}_u=(W_{1,u},...,W_{J,u})$ for an observation ${\bf X}_u$
where $J$ is the number of speakers, and $W_{j,u}$ represents the speech-recognition hypothesis
of the $j$-th speaker given observation ${\bf X}_u$.
We assume $W_{j,u}$ contains not only word sequences but also
their corresponding
frame-level time alignments of phonemes and silences.
Finally, we assume a tuple of speaker embeddings $\mathcal{E}=(e_1, ..., e_J)$,
where $e_j\in \mathbb{R}^d$ represents the $d$-dim speaker embedding of the $j$-th speaker.

Then, our objective is to find
the best possible $\mathcal{W}=\{{\bf W}_1,...,{\bf W}_U\}$ given a sequence of observations $\mathcal{X}$ as follows.
\begin{align}
\tilde{\mathcal{W}} &= \mathop{\rm arg~max}\limits_{\mathcal{W}} P(\mathcal{W}|\mathcal{X}) \label{eq:first} \\
 &= \mathop{\rm arg~max}\limits_{\mathcal{W}} \{\sum_{\mathcal{E}} P(\mathcal{W},\mathcal{E}|\mathcal{X})\} \label{eq:second}\\
 &\simeq \mathop{\rm arg~max}\limits_{\mathcal{W}} \{\max_{\mathcal{E}} P(\mathcal{W},\mathcal{E}|\mathcal{X})\} \label{eq:third}
\end{align}
Here, the starting point  is the conventional maximum a posteriori-based decoding
given $\mathcal{X}$ but
for multiple speakers.
We then introduce the speaker embeddings $\mathcal{E}$
as a hidden variable (Eq. \ref{eq:second}).
Finally, we approximate the summation by using a max operation (Eq. \ref{eq:third}).

Our motivation to introduce
$\mathcal{E}$, which is constant across all observation indices $u$, is to
explicitly enforce the order of speakers in $\mathcal{W}$ to be constant over indices $u$.
It should be emphasized that if we can solve the problem,  speaker diarization is solved at the same time
just by extracting the start and end time information of each hypothesis in $\mathcal{W}$.
Also note that there are $J!$ possible solutions
by swapping the order of speakers in $\mathcal{E}$, and it is sufficient to find just one such solution.

\subsection{Iterative maximization}
\label{sec:iterative}

It is not easy to directly solve $P(\mathcal{W},\mathcal{E}|\mathcal{X})$,
so we propose to alternately maximize $\mathcal{W}$ and $\mathcal{E}$.
Namely, we first fix $\underline{\mathcal{W}}$ and find $\mathcal{E}$
that maximizes
$P(\underline{\mathcal{W}},\mathcal{E}|\mathcal{X})$.
We then fix $\underline{\mathcal{E}}$ and find  $\mathcal{W}$
that maximizes
$P(\mathcal{W},\underline{\mathcal{E}}|\mathcal{X})$.
By iterating this procedure,
$P(\mathcal{W},\mathcal{E}|\mathcal{X})$ can be increased monotonically.
Note that
it can be said by a simple application of the chain rule that
finding $\mathcal{E}$
that maximizes $P(\underline{\mathcal{W}},\mathcal{E}|\mathcal{X})$ with a fixed
$\underline{\mathcal{W}}$  is equivalent to
finding $\mathcal{E}$
that maximizes $P(\mathcal{E}|\underline{\mathcal{W}},\mathcal{X})$.
The same thing can be said
for the estimation of
$\mathcal{W}$
with a fixed $\underline{\mathcal{E}}$.

For the $(i)$-th iteration of the maximization ($i\in\mathbb{Z}^{\geq 0}$),
we first find the most plausible estimation of $\mathcal{E}$
given the $(i-1)$-th speech-recognition hypothesis $\tilde{\mathcal{W}}^{(i-1)}$ as follows.
\begin{align}
\tilde{\mathcal{E}}^{(i)}&=\begin{cases}
\mathop{\rm arg~max}\limits_{\mathcal{E}} P(\mathcal{E}|\tilde{\mathcal{W}}^{(i-1)},\mathcal{X}) & (i \geq 1) \label{eq:emb1}\\
\mathop{\rm arg~max}\limits_{\mathcal{E}} P(\mathcal{E}|\mathcal{X}) & (i=0)
\end{cases}
\end{align}
Here, the estimation of $\tilde{\mathcal{E}}^{(i)}$ is dependent on $\tilde{\mathcal{W}}^{(i-1)}$
for $i \geq 1$.
Assume that the overlapped speech corresponds to a ``third person'' who is different from any person in the recording, Eq. \ref{eq:emb1} can be achieved by estimating
the speaker embeddings only from non-overlapped regions (upper part of Figure \ref{fig:overview} (right)).
In this study,
we used i-vector \cite{dehak2011front} as the representation of speaker embeddings,
and estimated i-vector based only
on the non-overlapped region given $\tilde{\mathcal{W}}^{(i-1)}$ for each speaker\footnote{The idea to extract speaker embeddings from non-overlapped regions has been proposed (e.g. \cite{kanda2018hitachi,manohar2019acoustic})}.
Note that, since we do not have an estimation of $\mathcal{W}$ for the first iteration, $\tilde{\mathcal{E}}^{(0)}$ is
initialized only by $\mathcal{X}$.
In this study, we
estimated the i-vector for each speaker given
 the speech region that was estimated by the clustering-based speaker diarization method.
More precicely,
we estimated the i-vector for each ${\bf X}_u$ then applied $J$-cluster K-means clustering.
The center of each cluster\footnote{Using cluster centers does not strictly follow Eq. \ref{eq:emb1}, but we
used them for the procedural simplicity.} was used for the initial
speaker embeddings $\tilde{\mathcal{E}}^{(0)}$.

We then update
$\mathcal{W}$ given speaker embeddings $\tilde{\mathcal{E}}^{(i)}$.
\begin{align}
\tilde{\mathcal{W}}^{(i)}&=\mathop{\rm arg~max}\limits_{\mathcal{W}} P(\mathcal{W}|\tilde{\mathcal{E}}^{(i)},\mathcal{X}) \label{eq:w-update1}\\
 &\simeq \mathop{\rm arg~max}\limits_{{\bf W}_1,...,{\bf W}_U} \prod_u P({\bf W}_u|\tilde{\mathcal{E}}^{(i)},{\bf X}_u) \label{eq:w-update2} \\
 &\simeq \mathop{\rm arg~max}\limits_{{\bf W}_1,...,{\bf W}_U} \prod_u \prod_j P(W_{j,u}|\tilde{e}_j^{(i)},{\bf X}_u) \label{eq:w-update}
\end{align}
Here, we estimate the most plausible hypotheses $\mathcal{W}$ given estimated embeddings $\tilde{\mathcal{E}}^{(i)}$
and observation $\mathcal{X}$ (Eq. \ref{eq:w-update1}).
We then assume the conditional independence  of ${\bf W}_u$ given ${\bf X}_u$ for each segment $u$ (Eq. \ref{eq:w-update2}).
Finally, we further assume  the conditional independence of $W_{j,u}$ given $\tilde{e}_j^{(i)}$ for each speaker $j$ (Eq. \ref{eq:w-update}).
The final equation can be solved by applying TS-ASR for each segment $u$ for each speaker $j$ (lower part of Figure \ref{fig:overview} (right)).
We will review the detail of TS-ASR in the next section.

\section{TS-ASR: Review}
\label{sec:ts-asr}

\subsection{Overview of TS-ASR}

TS-ASR is a technique to extract and recognize only the speech of a target speaker given
a short sample utterance of that speaker \cite{zmolikova2017speaker,vzmolikova2017learning,delcroix2018single}.
Originally, the sample utterance was fed into a special neural network that
outputs an averaged embedding to control the weighting
of speaker-dependent blocks of the acoustic model (AM).
However, to make the problem simpler,
we assume that a $d$-dimensional speaker embedding $e_{\rm tgt}\in \mathbb{R}^d$ is
extracted from the sample utterance.
In this context, TS-ASR can be expressed
as the problem to find the best hypothesis $W_{\rm tgt}$ given
observation ${\bf X}$ and speaker embedding $e_{\rm tgt}$ as follows.
\begin{align}
\tilde{W}_{\rm tgt}=\mathop{\rm arg~max}\limits_{W_{\rm tgt}} P(W_{\rm tgt}|e_{\rm tgt},{\bf X}) \label{eq:ts-asr}
\end{align}
If we have a well-trained TS-ASR, Eq. \ref{eq:w-update} can be solved by simply
applying the TS-ASR for each segment $u$ for each speaker $j$.

\subsection{TS-AM with auxiliary output network}
\subsubsection{Overview}

Although any speech recognition architecture can be used for TS-ASR,
we adopted a variant of the TS-AM that was recently proposed
and has promising accuracy \cite{kanda2019two}. Figure \ref{fig:ts-am} describes the TS-AM
that we applied for this study.
This model has two input branches.
One branch accepts acoustic features ${\bf X}$ as a normal AM
while the other branch accepts an embedding $e_{\rm tgt}$ that
represents the characteristics of the target speaker.
In this study, we used a log Mel-filterbank (FBANK) and i-vector \cite{dehak2011front,saon2013speaker}
for the acoustic features and target-speaker embedding, respectively.

A unique component of the model is in its output branch.
The model has multiple output branches that produce outputs ${\bf Y}^{\rm tgt}$ and ${\bf Y}^{\rm int}$
for the loss functions for the target and interference speakers, respectively.
The loss for the target speaker is defined to
maximize the target-speaker ASR accuracy,
while the loss for interference speakers is defined to
 maximize the interference-speaker ASR accuracy.
We used lattice-free maximum mutual information (LF-MMI) \cite{povey2016purely} for both criteria.

\begin{figure}[t]
	\centering
	\includegraphics[width=80mm]{ts-am.pdf}
	\vspace{-3mm}
	\caption{\textit{Overview of target-speaker AM architecture with auxiliary interference speaker loss \cite{kanda2019two}. A number with an arrow indicates a time splicing index, which forms the basis of a time-delay neural network (TDNN) \cite{peddinti2015time}. The input features were advanced by five frames, which has the same effect as reference label delay.}}
	\label{fig:ts-am}
	\vspace{-2mm}
\end{figure}

The original motivation of the output branch for interference speakers was
the improvement of TS-ASR by achieving
a better representation for speaker separation in the shared layers.
However, it was also shown that
the output branch for interference speakers can be used
for the secondary ASR for interference speakers given
the embedding of the target speaker \cite{kanda2019two}.
In this paper, we found out that
the latter property worked very well for the ASR for dialogue recordings,
which will be explained in the evaluation section.

The network is trained with a mixture of multi-speaker speech
given their transcriptions.
We assume that, for each training sample,
(a) transcriptions of at least two speakers are given,
(b) the transcription for the target speaker is marked
so that we can identify the target speaker's transcription,
and (c) a sample for the target speaker can be used to extract speaker embeddings.
These assumptions can be easily satisfied by artificially generating training data by mixing the speech of multiple speakers.

\subsubsection{Loss function}
The main loss function for the target speaker is defined as
\begin{align}
\mathcal{F}^{\rm tgt}&=\sum_{u} {\mathrm{LFMMI}}({\bf Y}_{u}^{\rm tgt};\mathcal{G}^{\rm tgt}_{u}),  \\
=&\sum_{u} \sum_{{\bf S}} -P({\bf S}|{\bf Y}_{u}^{\rm tgt},\mathcal{G}^{\rm tgt}_{u}) \log P({\bf S}|{\bf Y}_{u}^{\rm tgt},\mathcal{G}^{D}), \label{eq:mmi}
\end{align}
where $u$ corresponds to the index of training samples in this case.
The term $\mathcal{G}^{\rm tgt}_u$ indicates a numerator (or reference) graph
that represents a set of possible correct state sequences for the utterance of the target speaker
of the $u$-th training sample, ${\bf S}$ denotes a hypothesis state sequence
for the $u$-th training sample, and $\mathcal{G}^{D}$ denotes a denominator graph,
which represents a possible hypothesis space and normally consists of a 4-gram phone language model
in LF-MMI training \cite{povey2016purely}.

The auxiliary interference speaker loss is then defined to maximize the interference-speaker ASR accuracy,
which we expect to enhance the speaker separation ability of the neural network. This loss is defined as
\begin{align}
\mathcal{F}^{\rm int}=\sum_{u} {\mathrm{LFMMI}}({\bf Y}_{u}^{\rm int};\mathcal{G}^{\rm int}_{u}),
\end{align}
where $\mathcal{G}^{\rm int}_u$ denotes a numerator (or reference) graph
that represents a set of possible correct state sequences
for the utterance of the interference speaker of the $u$-th training sample.

Finally, the loss function $\mathcal{F}^{\rm comb}$ for training is
defined as the combination of the target and interference losses,
\begin{align}
\mathcal{F}^{\rm comb}=\mathcal{F}^{\rm tgt}+\alpha\cdot\mathcal{F}^{\rm int},
\end{align}
where $\alpha$ is the scaling factor for the auxiliary loss.
In our evaluation, we set $\alpha=1.0$. Setting $\alpha=0.0$, however, corresponds to normal TS-ASR.

\section{Evaluation}
\label{sec:eval}
\subsection{Experimental settings}
\label{sec:settings}

\subsubsection{Main evaluation data: real dialogue recordings}
We conducted our experiments on the CSJ \cite{maekawa2003corpus},
which is one of the most widely used evaluation sets for Japanese speech recognition.
The CSJ consists of more than 600 hrs of Japanese recordings.

While most of the content is lecture recordings by a single speaker,
CSJ also contains 11.5 hrs of
54 dialogue recordings\footnote{We excluded 4 out of 58 dialogue recordings that have speaker duplication with the official E1, E2, and E3 evaluation sets.}
(average 12.8 min per recording)
with two speakers,
which were the main target of ASR and  speaker diarization in this study.
During the dialogue recordings, two speakers sat in two adjacent sound proof chambers divided by a glass window.
They could talk with each other over voice connection through
a headset for each speaker.
Therefore, speech was recorded separately for each speaker,
and we generated mixed monaural recordings by mixing the corresponding speeches of two speakers.
When mixing two recordings, we did not apply any normalization of speech volume.
Due to this recording procedure, we were able to use non-overlapped speech to evaluate the oracle WERs.

It should be noted that, although the dialogue consisted of only two speakers,
the speaker overlap ratio of the recordings was very high due to many backchannels and natural turn-taking.
Among all recordings, 16.7\% of the region was overlapped speech while 66.4\% was spoken by a single speaker.
The remaining 16.9\% was silence.
Therefore, 20.1\% (=16.7/(16.7+66.4)) of speech regions was speaker overlap.
From the viewpoint of ASR, 33.5\% (= (16.7*2)/(16.7*2+66.4)) of the total duration
 to be recognized was overlapped.
These values were even higher than those reported for meetings with more than
two speakers \cite{ccetin2006analysis,yoshioka2018recognizing}.
Therefore, these dialogue recordings are very challenging for both ASR and  speaker diarization.
We observed significantly high WER and DER, which is discussed in the next section.

\subsubsection{Sub evaluation data: simulated 2-speaker mixture}
To evaluate TS-ASR, we also used
the simulated 2-speaker-mixed data by mixing
the three official single-speaker evaluation sets of CSJ, i.e., E1, E2, and E3 \cite{kawahara2003benchmark}.
Each set includes different groups of 10 lectures (5.6 hrs, 30 lectures in total).
The E1 set consists of 10 lectures of 10 male speakers,
and E2 and E3 each consists of 10 lectures of 5 female and 5 male speakers.
We generate two-speaker mixed speech by adding randomly selected speech (= interference-speaker speech)
to the original speech (= target-speaker speech) with
the constraint that the target and interference speakers were different,
and each interference speaker was selected only once from the dataset.
When we mixed the two speeches,
we configured them to have the same power level,
and shorter speech was mixed with the longer speech from a random starting point
selected to ensure the end point of the shorter one did not exceed that of the longer one.

\subsubsection{Training data and training settings}
The rest of the 571 hrs of 3,207 lecture recordings
(excluding the same speaker's lectures in the evaluation sets) were used for AM and language model (LM) training.
We generated two-speaker mixed speech for training data in accordance with the following protocol.
\begin{enumerate}
\setlength{\parskip}{0cm}
\setlength{\itemsep}{0cm}
\item Prepare a list of speech samples (= main list).
\item Shuffle the main list to create a second list under the constraint that the same speaker does not appear in the same line in the main and second lists.
\item Mix the audio in the main and second lists one-by-one with a specific signal-to-interference ratio (SIR). For training data, we randomly sampled an SIR as follows.
\begin{itemize}
\item In 1/3 probability, sample the SIR from a uniform distribution between -10 and 10 dB.
\item In 1/3 probability, sample the SIR from a uniform distribution between 10 and 60 dB. The transcription of the interference speaker was set to null.
\item In 1/3 probability, sample the SIR from a uniform distribution between -60 and -10 dB. The transcription of the target speaker was set to null.
\end{itemize}
\item The volume of each mixed speech was randomly changed to enhance robustness against volume difference.
\end{enumerate}
A speech for extracting a speaker embedding was also randomly selected for each speech mixture from the main list. Note that the random perturbation of volume was applied only for the training data, not for evaluation data.

We trained a TS-AM consisting of a convolutional neural network (CNN),
time-delay NN (TDNN) \cite{waibel1989phoneme},
and long short-term memory (LSTM) \cite{hochreiter1997long}, as shown in \cref{fig:ts-am}.
The input acoustic feature for the network was a 40-dimensional FBANK without normalization.
A 100-dimensional i-vector was also extracted and used for the target-speaker embedding
to indicate the target speaker.
For extracting this i-vector, we randomly selected an utterance of the same speaker.
We conducted 8 epochs of training on the basis of LF-MMI,
where the initial learning rate was set to 0.001 and exponentially decayed to 0.0001 by the end of the training.
We applied $l2$-regularization and CE-regularization \cite{povey2016purely}
with scales of 0.00005 and 0.1, respectively.
The leaky hidden Markov model coefficient was set to 0.1.
A backstitch technique \cite{wang2017backstitch} with a backstitch scale of 1.0 and backstitch interval of 4 was also used.

For comparison, we trained another TS-AM without the auxiliary loss.
We also trained a ``clean AM'' using clean, non-speaker-mixed speech.
For this clean model, we used a model architecture without the auxiliary output branch,
and an i-vector was extracted every 100 msec for online speaker/environment adaptation.

In decoding, we used a 4-gram LM trained using the transcription of the training data.
All our experiments were conducted on the basis of the Kaldi toolkit \cite{povey2011kaldi}.

\subsection{Preliminary experiment with simulated 2-speaker mixture}
\subsubsection{Evaluation of TS-ASR}

We first evaluated the TS-AM
with two-speaker mixture of the E1, E2, and E3 evaluation sets.
For each test utterance, a sample of the target speaker was randomly selected from the other utterances
in the test set.
We used the same random seed over all experiments, so that they could be conducted under the same conditions.

\begin{table}[t]
\caption{\label{tab:first} {\it WERs (\%) for two-speaker-mixed evaluation sets of CSJ.}}
\centering
{\footnotesize
\begin{tabular}{cc|ccc|c} \hline
Model              & Evaluation Data     & E1 & E2 & E3 & Avg. \\  \hline
Clean AM           & 1-spk.                & 8.94  & 7.31  & 7.44  &  7.90 \\
Clean AM           & 2-spk. mixed        & 87.60 & 85.44 & 91.05 &  88.03\\
TS-AM ($\alpha=0.0$) & 2-spk. mixed      & 26.01 & 18.16 & 18.16 & 20.78 \\
TS-AM ($\alpha=1.0$) & 2-spk. mixed      & {\bf 25.68} & {\bf 17.94} & {\bf 17.96} & {\bf 20.53} \\ \hline
\end{tabular}
}
\vspace{-3mm}
\end{table}

The results are listed in Table \ref{tab:first}.
Although the clean AM produced a WER of 7.90\% for the original clean dataset,
the WER severely degraded to 88.03\% by mixing two speakers.
The TS-AM then significantly recovered the WER to 20.78\% ($\alpha=0.0$).
Although the improvement was not so significant
compared with that reported in \cite{kanda2019two},
the auxiliary loss further improved the WER to 20.53\% ($\alpha=1.0$).
Note that E1 contains only male speakers while E2 and E3 contain both female and male speakers.
Because of this,
E1 showed larger degradation of WER when 2 speakers were mixed.

\begin{table}[t]
\caption{\label{tab:second} {\it WERs (\%) for two-speaker-mixed evaluation sets of CSJ. Main output branch was used for target-speaker ASR and auxiliary output branch was used for interference-speaker ASR.}}
\centering
\begin{tabular}{c|cc} \hline
Test set               & Target spk. & Interference spk.  \\ \hline
E1 (10 male)       & 25.68 & 26.91 \\
E2 (5 female, 5 male)      & 17.94 & 18.46 \\
E3 (5 female, 5 male)      & 17.96 & 18.36 \\ \hline
Avg. & 20.53 & 21.24 \\ \hline
\end{tabular}
\vspace{-3mm}
\end{table}

\begin{table*}[t]
\caption{\label{tab:dialogue} {\it WERs (\%) for dialogue speech in CSJ}}
\centering
\begin{threeparttable}
\begin{tabular}{c|cccc|cc|c} \hline
\# &\multicolumn{2}{c}{Speaker Embeddings} & AM        & Evaluation  & \multicolumn{2}{c|}{Gender Pair} & Total \\
& Initialization   & Update   &                                     & Data      & Different & Same &  \\  \hline
1 &-                &  -       & Clean-AM                            & 1-spk.             & 18.49\tnote{$\dagger$} & 21.14\tnote{$\dagger$} & 19.93\tnote{$\dagger$} \\ \hline
2 &Oracle           &  -       & Clean-AM w/ ${\bf e}_1$ \& Clean-AM w/ ${\bf e}_2$ & 2-spk. mixed             & 94.46\tnote{$\dagger$}  &   94.01\tnote{$\dagger$}     &  94.22\tnote{$\dagger$}\\
3 &Oracle           &  -       & TS-AM (tgt) w/ ${\bf e}_1$ \& TS-AM (tgt) w/ ${\bf e}_2$ & 2-spk. mixed & 26.83\tnote{$\dagger$} & 47.33\tnote{$\dagger$} & 37.96\tnote{$\dagger$} \\
4 &Oracle           &  -       & TS-AM (tgt) w/ ${\bf e}_1$ \& TS-AM (int) w/ ${\bf e}_1$ & 2-spk. mixed & 25.99\tnote{$\dagger$} & 53.80\tnote{$\dagger$} & 41.09\tnote{$\dagger$} \\ \hline
5 &K-means          &  ${(i=0)}$       & TS-AM (tgt) w/ ${\bf e}_1$ \& TS-AM (tgt) w/ ${\bf e}_2$ & 2-spk. mixed & 40.99 & 64.97 & 54.01 \\
6 &K-means          &  ${(i=0)}$       & TS-AM (tgt) w/ ${\bf e}_1$ \& TS-AM (int) w/ ${\bf e}_1$ & 2-spk. mixed & 30.00 & 58.61 & 45.54 \\
7 &K-means          &  $i=1$  & TS-AM (tgt) w/ ${\bf e}_1$ \& TS-AM (int) w/ ${\bf e}_1$ & 2-spk. mixed & 26.45 & 53.93 & 41.37 \\
8 &K-means          &  $i=2$  & TS-AM (tgt) w/ ${\bf e}_1$ \& TS-AM (int) w/ ${\bf e}_1$ & 2-spk. mixed & 25.46 & 52.82 & 40.31 \\
9 &K-means          &  $i=3$  & TS-AM (tgt) w/ ${\bf e}_1$ \& TS-AM (int) w/ ${\bf e}_1$ & 2-spk. mixed & {\bf 25.20} & {\bf 52.50} & {\bf 40.03} \\ \hline
\end{tabular}
\begin{tablenotes}
	\item[$\dagger$] Result obtained with some oracle information such as non-overlapped evaluation data or oracle speaker embeddings
\end{tablenotes}
\end{threeparttable}
\vspace{-5mm}
\end{table*}

\begin{table}[t]
\caption{\label{tab:der} {\it DERs (\%) for dialogue speech}}
\centering
\begin{threeparttable}
\begin{tabular}{cccc} \hline
Method               & \multicolumn{2}{c}{Gender Pair} & Total \\
                       & Different & Same &  \\ \hline
i-vector with K-means                           & 25.94 & 37.32     & 32.37 \\
\# 6 of Table \ref{tab:dialogue}        & 15.99 & 37.00    & 27.87 \\
\# 9 of Table \ref{tab:dialogue}        & {\bf 10.76} & {\bf 35.30}    & {\bf 24.63} \\ \hline \hline
i-vector with AHC \cite{sell2018diarization}\tnote{$\ddagger$} & 14.34 & 38.48  &  27.99\\
x-vector with AHC \cite{sell2018diarization}\tnote{$\ddagger$}& 13.77 & 30.02  &  22.96\\ \hline
\end{tabular}
\begin{tablenotes}
	{\scriptsize \item[$\ddagger$] Trained using combination of Switchboard and NIST SRE datasets}
\end{tablenotes}
\end{threeparttable}
\vspace{-5mm}
\end{table}

\begin{table}[t]
\caption{\label{tab:der-detail} {\it Details of DER (\%) for \# 9 of Table \ref{tab:dialogue}}}
\centering
{\footnotesize
\begin{tabular}{c|ccc|c} \hline
                       & Miss & False Alarm & Confusion & DER \\ \hline
Different Gender Pair  & 9.6 & 0.8 & 0.4  & 10.76 \\
Same Gender Pair       & 22.5 & 2.2  & 10.6 & 35.30 \\ \hline
Total      & 16.9 & 1.6  & 6.2 & 24.63 \\ \hline
\end{tabular}
}
\vspace{-3mm}
\end{table}

\subsubsection{Interference-speaker ASR by auxiliary output branch}
Before moving to the evaluation of dialogue recordings,
we also evaluated the use case of the auxiliary output branch for interference speakers
to conduct interference-speaker ASR.
In this experiment, we provided the target speaker's embeddings for the TS-AM
and evaluated the WERs of the ASR results
using the auxiliary output branch.
The results are shown in \cref{tab:second}.
We confirmed that the auxiliary output branch worked very well for the secondary ASR.
This clearly indicates that the shared layers of the neural network were learned to separate speakers.
In addition, we found out that this
secondary ASR can be effectively
incorporated into the proposed method,
which we will explain in the next section.

\subsection{Experiment with dialogue recordings}
\label{sec:dialogue}
Since we confirmed that TS-ASR worked as expected,
we then conducted experiments for dialogue recordings,
which were the main target of this study.

\subsubsection{WER evaluation with oracle non-overlapped speech}

We first evaluated the lower limit of WER for the dialogue recordings
by using the non-overlapped dialogue recordings (see Section \ref{sec:settings} for recording settings).
We used the original non-overlapped recordings with the ground-truth segmentation and
conducted ASR with the clean AM.
The results are shown in the first line of Table \ref{tab:dialogue}.
We observed a WER of 19.93\%, which was the lower limit for the recordings in this experiment.
The WER was worse compared with those for lecture recordings (E1, E2, and E3).
We observed more substitution errors of backchannels for dialogue recordings, which was very short and difficult to recognize.

\subsubsection{WER evaluation using oracle speaker embeddings}

We then conducted experiments for the mixed monaural dialogue recordings.
For preprocessing, we separated each dialogue recording into speech segments
by using simple power-based voice activity detection.
Note that each segment could contain the speech of two speakers.
We counted a recognized word as correct only when it and the recognized speaker both matched the reference label.
Since there was ambiguity in the order of speakers in the reference label,
we calculated the best WER among possible permutations of speakers.

We first conducted an experiment with oracle speaker embeddings to confirm the oracle WER
for two-speaker mixed recordings.
The results are shown in the second to fourth rows of Table \ref{tab:dialogue}.
We extracted the oracle i-vector for each speaker
by using only the non-overlapped region determined from the ground-truth segmentation.

When we used the clean AM with the oracle speaker embeddings, we observed
a very poor WER of 94.22\% (\# 2 of Table \ref{tab:dialogue}).
This was within our expectation because the clean AM was not trained to extract the target speaker's speech.

When we evaluated using the TS-AM
with embeddings $e_1$ and $e_2$,
we observed the best oracle WER of 37.96\% (\# 3 of Table \ref{tab:dialogue}).
This result was the lower limit of WER for two-speaker mixed recordings in this experiment.
As another application of the TS-AM, we also used the auxiliary output branch of the TS-AM
with embedding $e_1$ to recognize speaker $2$.
This result is shown in the fourth row in Table \ref{tab:dialogue}.
It showed a slightly worse WER of 41.09\% compared with the WER with
the TS-AM with embeddings $e_1$ and $e_2$ (\# 4 of Table \ref{tab:dialogue}).
This was also within our expectation because the auxiliary branch produced
a slightly worse result than the main branch according to Table \ref{tab:second}.

\subsubsection{WER evaluation using estimated speaker embeddings with iterative update (proposed method)}

Finally, we evaluated the proposed method
by starting from the estimated speaker embeddings.
In this evaluation, we estimated the i-vector for each speech partition divided by
power-based voice activity detection then applied K-means clustering.
The number of clusters was set to 2 to be the same as the number of speakers.
The center of each cluster was used for the initial set of speaker embeddings.
Note that we denoted the cluster center of the larger cluster as $e_1$ and that of the smaller cluster as $e_2$.

Similar to the comparison of \# 3 and \# 4 of Table \ref{tab:dialogue},
we also compared two methods without and with auxiliary output branch.
The results are shown in the fifth and sixth rows.
Contrary to the experiment with the oracle speaker embeddings,
the method using the auxiliary output branch
with the embedding $e_1$ to recognize the speaker $2$
produced much better WER of 45.54\%
than the method using the main output branch with
the embedding $e_2$ to recognize the speaker $2$.
This is because the K-means-clustering-based speaker embedding estimation
was not sufficient to generate two discriminative embeddings of $e_1$ and $e_2$.
Considering that embedding $e_2$ was selected as the center of the smaller cluster,
it would not be as reliable as embedding $e_1$.
In such a case, using a single embedding $e_1$ with the auxiliary output branch
is better than using an unreliable embedding $e_2$ with the main output branch.

We then evaluated the proposed method of applying speaker-embedding estimation and TS-ASR alternately.
The results are shown in the seventh to ninth rows of Table \ref{tab:dialogue}.
We observed clear improvement in the WER both for different gender pairs and same gender pairs, and achieved a WER of 40.03\%, which differd by only 2.1\% from the oracle WER of 37.96\% with the oracle speaker embeddings.
Note that we observed better results by the
proposed method
than that of the method \# 4 of Table \ref{tab:dialogue}
even though the latter method used the speaker embeddings
obtained from the ground-truth segmentation.
We believe it was because the ground-truth segmentation
contained an unignorable amount of silence frames
that degraded the purity of speaker embeddings,
while strict exclusion of silence frames was achieved by using the TS-ASR results.

\subsubsection{Evaluation of DERs}

Table \ref{tab:der} lists the DERs of three methods.
Note that we set 0.25 sec of the no-score collar according to convention,
and calculated DER including overlapped regions.
It is also noted that we regarded silence frames of less than 0.5 sec as speech regions
for the proposed method because we found the silence information that the ASR produced
was too strict compared to the ground-truth segmentation developed by human transcribers.
The first row is a naive method based on the clustering of i-vectors, which was used for the embedding initialization.
As expected, it produced a very poor DER of 32.37\% due to heavy speech overlaps in the recordings.
 Just by applying TS-ASR, we observed an improvement in the DER to 27.87\%,
especially for different gender pairs.
By using the proposed method, the DER further improved to 24.63\%.

To compare with the state-of-the-art  speaker diarization method,
 we also tested the agglomerative hierarchical clustering (AHC)-based method \cite{garcia2017speaker,sell2018diarization},
the results of which are shown in the last two rows in Table \ref{tab:der}.
 The i-vector and x-vector extractors were trained using about 3,000 hrs of training data
consisting of Switchboard-2 (Phase I, II, III), Switchboard Cellular (Part 1, Part2),
and NIST Speaker Recognition Evaluation (2004, 2005, 2006, 2008) datasets.
Speaker embeddings were extracted every 0.75 sec, and AHC with probabilistic linear discriminant analysis (PLDA)
was used to create a speaker cluster.
Although it is not directly comparable due to the difference in training data,
we confirmed that our method produced a reasonably good DER although it was trained
much smaller training data.
Our method
showed a better DER than that of ``i-vector with AHC'' and a DER close to that of ``x-vector with AHC''.
The proposed method even achieved a better DER than the x-vector-based method
for the different gender pairs although we used an i-vector trained by much smaller data for the proposed method.

Finally, we discuss the detailed error analysis of the
DER for the method with speaker embedding updated three times (Table \ref{tab:der-detail}).
We first found that the main source of the DER came from the miss error.
We also found that the confusion error and false alarm were low even for same gender pairs.
When applying the proposed method for the different gender pairs,
almost no confusion and false alarm were produced.
This means that TS-ASR worked very conservatively, i.e., it tended to output null when
it was not able to find a reliable word hypothesis.
From the results in Tables \ref{tab:der} and \ref{tab:der-detail},
we can expect further improvement in the DER if
 we apply more discriminative speaker embeddings
such as d-vector \cite{wang2018speaker,wan2018generalized} and x-vector \cite{garcia2017speaker,snyder2018x},
which is one important direction for our future work.

\section{Conclusion}
\label{sec:conclusion}

In this paper, we defined the problem of simultaneous ASR and  speaker diarization,
and proposed an iterative method,
in which (i) the estimation of speaker embeddings in the recordings
and (ii) TS-ASR based on the estimated speaker embeddings are alternately executed.
 We evaluated the proposed method by using real dialogue recordings in the
CSJ
in which the speaker overlap ratio was over 20\%.
We confirmed that the proposed method significantly
reduced both the WER and DER.
Our proposed method with i-vector speaker embeddings
ultimately
achieved a WER that differed by only 2.1 \% from the WER of TS-ASR given
 oracle speaker embeddings.
Furthermore, our method
achieved
better DER than that of the conventional clustering-based speaker diarization method based on i-vector.

There are many directions to enhance this research.
First, our proposed method should be examined using recordings with more than two speakers.
Second, the use of more discriminative speaker embeddings,
such as d-vector \cite{wang2018speaker,wan2018generalized} and x-vector \cite{garcia2017speaker,snyder2018x},
will improve performance of both ASR and  speaker diarization.
Third, the initialization of speaker embeddings should be explored with more advanced speaker diarization techniques \cite{wang2018speaker,fujita2019end,fujita2019end2}.
Finally, advanced ASR techniques,
such as data augmentation \cite{jaitly2013vocal,kanda2013elastic,ko2015audio,ko2017study},
model ensemble \cite{tachioka2013generalized,deng2014ensemble,kanda2017investigation}, improved training criterion \cite{kanda-interspeech2018,weng2019comparison}, will also improve overall performance.
We will explore these directions for future work.

\bibliographystyle{IEEEbib}

redpajama_set_name: RedPajamaArXiv


Read Merrell Jungle Moc product reviews or select the size width and color of your choice. Out of stars.
Buy Merrell Mens Jungle Moc Slip On Shoe and other Loafers Slip Ons at.
Maybe you would like to learn more about one of these? Shop a wide selection of Merrell Mens Jungle Moc Brown Gunsmoke Slip on Sneakers. Promotional Results For You. Merrell Mens Jungle Moc Casual Shoes at DICKS Sporting Goods and order online for the finest quality products from the top brands.
This moc creates your ultimate low maintenance shoe featuring a smooth suede upper and Merrell air cushion underfoot for extra support. Jungle Moc by Merrell at. Official Merrell site Shop the full collection of Jungle Moc and find what youre looking for today Carhartt Rugged Flex Belt For Men Black 40.
We did not find results Merrell browngunsmoke. Breaking news more every time you open your browser.
Our wide selection is eligible for free shipping and free returns. Mens Jungle Moc Nubuck Waterproof Wide Width.


text:
A Love Story: Two Marines during WWII
Bataan Death March, courtship, Julie Clem, Marine Corps Women's Reserve, Onnie Clem, prisoners of war, United States history, United States Marine Corps, World War II
By Amanda Gesiorski, Texas Collection graduate assistant and museum studies graduate student
Letter from Onnie E. Clem, Jr. to his fiancée, Julie, during his time at home in Dallas, Texas, circa 1944-1945. Onnie E. Clem, Jr. papers, #3939, Box 1, Folder 2.
As impressive as the recent movie Unbroken may be, a better story about a WWII prisoner of war can be found within the Onnie E. Clem, Jr. papers. Not only do we learn about Onnie's harrowing experience overseas, but we also get a firsthand account of his passionate love affair with his future wife, Julie.
Onnie E. Clem, Jr. was born in Dallas, Texas, and joined the United States Marine Corps in 1938 to see the world. He became a radio operator and was stationed in Peking, China, at the US Embassy. When the United States entered World War II, Onnie took part in the battles of the Bataan and was part of the Bataan Death March. Onnie survived the march only to become a POW at Camp O' Donnell and Cabanatuan for two and a half years.
On August 19, 1944, Onnie was put into the hull of a Japanese "hell ship" with 750 other American prisoners. Nineteen days later, the ship was hit by an American torpedo. As the ship was sinking, the Japanese tossed a grenade into the hull and shot Americans as they tried to escape. Although injured from the torpedo and shot by the Japanese, Onnie managed to escape and swim three miles to shore, where he was rescued by Filipino guerilla fighters. Onnie was one of only 83 known survivors of this incident. By September 1944, Onnie was on his way back home. A transcribed interview with Onnie from 1972 reveals this harrowing experience.
Onnie's story does not end there, though. Things heat up when Onnie arrives in San Diego, California, where he meets Staff Sergeant "Julie" Cecile Lorraine Julian, who was a member of the United States Marine Corps Women's Reserve (USMCWR). She served as secretary to the general in charge of public relations along the west coast in San Diego and later at the office in San Francisco.
This photograph shows Staff Sergeant "Julie" Cecile Lorraine Julian wearing her official United States Marine Corps Women's Reserve uniform, circa 1944-1945. It was enclosed in a letter she sent to her fiancée, Onnie, while he was in Texas. Onnie E. Clem, Jr. papers, #3939, Box 1, Folder 3.
Julie's office coordinated Onnie's war bond tour in California. When Onnie walked into her office one day, there was an instant attraction. The couple had a whirlwind romance, and when Onnie returned home to Texas in November 1944, there was a passionate exchange of letters. In these letters, we learn about Julie and Onnie's relationship as well as their social and work lives. We get a glimpse of Julie's romantic history with other soldiers, and Onnie's attempt to re-assimilate into civilian life. We also learn that upon returning home, Onnie is plagued by a constant stream of inquiries from people wanting to know if he has news about their loved ones. Also woven within these letters is the life of Onnie's fellow escapee, Verks D. Cutter, and Cutter's wife Janet Elliot, including their secret marriage. While Julie, Janet, Onnie, and Verks are friends at the beginning, as the letters continue we learn about a falling out between them. Onnie and Julie marry on July 13, 1945, and move to Texas, where they work and raise two children.
Onnie joined the Marines to see the world, and in that he succeeded—his papers tell that story. The Onnie E. Clem, Jr. papers read like a novel full of passion and drama, but they also provide a unique historical insight into the daily personal, professional, and social lives of two Marines during WWII.
Research Ready: December 2014
Baylor athletics, Baylor football, football, frontier and pioneer life, Grant Teaff, Liberty County, Research Ready, United States Armed Forces, United States Marine Corps, World War II
Each month, we post a processing update to notify our readers about the latest collections that have finding aids online and are primed for research. Here are December's finding aids:
Letter from Onnie Clem Jr. to "Julie" Cecile L. Julian Clem during Onnie's 1945 public relations tour in New York, Chicago, and Washington, D.C. They would marry seven months later. Onnie Clem Jr. papers #3939, box 1, folder 3.
Grant and Donell Teaff Baylor Football collection, 1948-2006, undated (#3835): Contains correspondence, football programs, newspaper clippings, and audiovisual materials relating to Teaff's career as head football coach at Baylor University. As usual, the materials described in the finding aid can be seen at The Texas Collection, but many of them also have been digitized as part of the Baylor University Libraries Athletics Archive in collaboration with the Electronic Library. Check out game films, ribbons, and more in the online collection!
John B. Herpin papers, 1851-1910 (#1636): Letters, tax deeds, and receipts about Herpin's tract of land in Liberty County, Texas.
Onnie Clem Jr. papers, 1944-1948 (#3939): Letters between Marine Corps members Onnie Clem Jr. and "Julie" Cecile L. Julian Clem during World War II. Also included is a transcribed interview with Onnie Clem Jr. about his experience during the Bataan Death March and as a prisoner of war for two and half years.
Tax receipt from 1850 for John Herpin's land claims in Liberty County, Texas. A dispute about ownership of this land was still going on in 1910, according to the collection. John B. Herpin papers #1636, box 1, folder 2.

redpajama_set_name: RedPajamaCommonCrawl


Recommended Android device up to spec with Galaxy S3.
I got it running, but I'm not sure what I'm supposed to be doing. Great music, and it seems quite minimalist indeed!
Works great on my HP Touchpad. Good visuals and music.
Neat idea, horribly awkward controls.
Like the others, I found the presentation to be excellent but the premise completely unclear. The music was great, and the lights and effects were pretty stellar, but I had no clue what I was doing or why.
Fun little game. Neat graphics and music. Had a little trouble figuring out what to do in the beginning though.
Well that was odd, great music, neat idea!


text:
\section{Introduction}

Non-Hermiticity and dissipations are two different effects to describe open systems and the interaction between them, which is an attractive topic, has not been studied detailedly. It is well known that many new features that are inaccessible in Hermitian systems can be observed in non-Hermitian systems. One of the famous examples is the steady states of NH systems. In particular, the notion of pseudo-Hermiticity e.g. parity-time (PT) symmetry provides a general perspective on characterizing a wide class of non-Hermitian Hamiltonians with purely real spectra\cite{Hodaei,Xiao,Chen2,Ozdemir,Ghatak,Chu}.  Recently, a new-type non-Hermitian system with one eigenvalue being purely real and the other being complex has been proposed\cite{Mostafavi}. However, a general tool to explore non-Hermitian systems with part of eigenvalues being purely real is still lacking. Another famous example of NH systems is EPs. Different from degeneracies in Hermitian Hamiltonians where only the eigenvalues coalesce, EPs refer to not only the eigenvalues coalesce, but also the corresponding  eigenstates coalesce. The physics of EPs has been studied extensively, both theoretically and experimentally, because of the observation of novel phenomena\cite{Heiss,Dembowski,Doppler,Cerjan,Yoshida,Dora} and widespread applications, incl. unidirectional invisibility\cite{Regensburger}, enhanced sensors\cite{Chen}, linewidth broadening\cite{Zhang}, topological classification\cite{Kawabata} and optical funneling\cite{Weidemann}.

Indeed, all of these interesting phenomena of NH systems are induced by the emergence of NH Fermi (i-Fermi) arcs corresponding to the contours with purely imaginary(real) energy in the parameter space\cite{Berry}. For example, steady states with pseudo-Hermiticity or PT symmetry are all in NH i-Fermi arcs. And EPs are the endpoints of NH Fermi arcs and NH i-Fermi arcs\cite{Bergholtz}. Besides, NH Fermi(i-Fermi) arcs are also deeply related to the source of novel bulk state\cite{Kozii,Zhou}, new gapless phases\cite{Bessho}, knotted NH nodal structures\cite{Carlstrom} and so on. Based on these developments, two open questions emerge. Firstly, how to find and classify the steady state condition of non-Hermitian Hamiltonians with parts real spectra. Secondly, what is the interesting property of eigenstate in NH Fermi arcs.

In this work, we address non-Hermitian Hamiltonians through the vectorized Lindblad equation, which are usually implemented in the Hermitian Hamiltonians\cite{Agarwal,Prosen,Albert,Macieszczak,Caspel,Arkhipov,Minganti3}. Unlike previous works use the NH systems viewpoint to study the properties of Lindblad master equation\cite{Minganti,Ju,Minganti2}, our work use the vectorized Lindblad equation to explore NH systems and the interaction between non-Hermiticity and dissipations\cite{Havel,Am-Shallem,Keck,Reimer,Scopa,Hatano}. Based on this, we find that there is a new type of steady states with just parts of eigenvalues being real and new EPs due to the interplay between non-Hermiticity and dissipations. Moreover, we demonstrate that there exists a topological phase $|\pi/2|$ of the eigenstates in NH Fermi arcs regime, while the topological phase in EPs is a specialized case of our results. These results are easy to generalize a variety of open systems, e.g. NH lattice systems\cite{Gong}, NH quasicrystals\cite{Longhi}, and high order NH systems\cite{Tang}.

The contents of this paper are as follows. In section II, we introduce the non-Hermitian vectorized Lindblad equation and propose the condition of steady states. In section III, we consider a two-level open system with non-Hermiticity and dissipations and discuss steady states and EPs in different cases. Further, we explore the new topological properties of eigenstates in NH Fermi arcs regime. Discussions and conclusions are given in section IV.

\section{the vectorized Lindblad equation and the condition of steady states for a general open system}

In this section, we extend the Lindblad, namely Lindblad-Gorini-Kossakowski-Sudarshan, master equation of a Hermitian open system\cite{Lindblad,Gorini} to a non-Hermitian case by transforming density matrices into vectors.  We here focus on a general non-Hermitian open system with gains, losses, and dissipations. In the limit of weak Markovian time-independent interaction, the time evolution of an open quantum system can be expressed as\cite{Walls,Breuer}
\begin{equation}
\label{master equation}
\frac{\partial\rho(t)}{\partial t}=-i[\hat{H}\rho(t)-\rho(t)\hat{H}^\dag]+\sum_{\mu}D[\hat{\Gamma}_{\mu}]\rho(t).
\end{equation}
Note that $\hat{H}$ is not limited to Hermiticity. We here consider it essentially is a non-Hermitian Hamiltonian ($\hat{H}\neq\hat{H}^{\dagger}$) so it is quite different from an effective non-Hermitian Hamiltonian $H_{eff}$ stemming from the dissipation of environment, e.g., $H_{eff}=H_{sys}-i\sum_{\mu}\hat{\Gamma}_{\mu}\hat{\Gamma}_{\mu}^{\dag}/2$. $\rho(t)$ is the density matrices of the system and $D[\Gamma_{\mu}]\rho(t)$ is a linear operator and the most general quantum dynamical semigroup form about this can be written as
\begin{equation}
\label{Lindblad operators}
D[\hat{\Gamma}_{\mu}]\rho(t)=\hat{\Gamma}_{\mu}\rho(t)\hat{\Gamma}_{\mu}^{\dag}-\frac{1}{2}\hat{\Gamma}_{\mu}\hat{\Gamma}_{\mu}^{\dag}\rho(t)-\frac{1}{2}\rho(t)\hat{\Gamma}_{\mu}\hat{\Gamma}_{\mu}^{\dag}.
\end{equation}
with the Lindblad operators $\hat{\Gamma}_{\mu}$ describing directly the dissipation effects of environments.

To solve Eq.(\ref{master equation}), one can rewrite the $N\times N$ density matrix $\rho(t)$ into a column vector, e.g., $\hat{\rho}(t)=[\rho(1,:)^T,\rho(2,:)^T,\cdots,\rho(n,:)^T]$ by using the Hilbert-Schmidt expression of the density matrix. After a straightforward algebra, we can rewrite the non-Hermitian Hamiltonian Lindblad equation (1) as a vectorized form
\begin{eqnarray}
\label{vectorized equation}
i\frac{\partial\hat{\rho}(t)}{\partial t}&=&i(\hat{\mathcal{H'}}+\hat{\mathcal{G'}})\hat{\rho}(t)\\ \nonumber
&=&                                                                                i\hat{L'}\hat{\rho}(t),
\end{eqnarray}
where $\hat{\mathcal{H'}}=-i(\hat{H}\otimes I-I\otimes\hat{H}^\dag)$ and $\hat{\mathcal{G'}}=\sum_{n}\hat{\Gamma}_{\mu}\otimes\hat{\Gamma}_{\mu}^{\dag}-\frac{\hat{\Gamma}_{\mu}\hat{\Gamma}_{\mu}^{\dag}}{2}\otimes I-I\otimes\frac{\hat{\Gamma}_{\mu}\hat{\Gamma}_{\mu}^{\dag}}{2}$. $\hat{L'}=\hat{\mathcal{H'}}+\hat{\mathcal{G'}}$ is represented as a $N^2\times N^2$ matrix. It's worth mentioning that $\hat{L'}$ is including two different effects of the open system, one is derived from non-Hermiticity of the system with gains and losses, the other is arising from the dissipation of environments. In this vectorized Lindblad framework, there is an obvious advantage: The analytical expressions of eigenvalues and eigenstates of the open system are easily obtained
\begin{equation}
\label{eigenvalues and eigenstates}
\hat{L'}\hat{\rho}^{i}=\lambda_{i}\hat{\rho}^{i}, (i=1,2,...n)
\end{equation}
where $\lambda_{i}$ and $\hat{\rho}^{i}$ are respectively the eigenvalues and eigenstates of the Liouvillian $\hat{L'}$. Due to non-Hermiticity of $\hat{L'}$, energy spectra become complex in general. For a time-independent Liouvillian $\hat{L'}$, based on the discussions about the steady state in Hermitian case\cite{Albert2,Albert3,Nigro}, we declare there exists a steady state, if at least one of the eigenvalues(suppose it is the $j$th eigenvalue) has zero imaginary part while the remanent eigenvalues have non-positive imaginary parts:
\begin{equation}
\label{steady state}
Im[\lambda_{j}]=0\quad and \quad Im[\lambda_{i\neq j}]\leq 0.
\end{equation}
The first equation ensures that there is at least one eigenstate being steady, and the second equation avoids the fact that the populations of the corresponding eigenstates exponentially grow. Compared to the steady state condition that appeared in Hermitian Lindblad equation, or Pseudo-Hermitian NH systems, our extension provides a more general constraint to find and classify stable open systems.

\begin{figure}[tbp]
\includegraphics[width=5.8cm]{fig1}
 \caption{(Color online) Energy level scheme about our example. There are loss $\gamma_{1}$, gain $\gamma_{2}$, coupling $\Omega$ and dissipations $\Gamma$.
}\label{fig1}
\end{figure}

\section{steady state, NH Fermi arcs and EPs in a two-level system with non-Hermiticity and dissipation}

For simplicity, we consider a two-level system consisting of lower level $|0\rangle$ and higher level $|1\rangle$ coupling to an environment with the loss $\gamma_{1}$ , the gain $\gamma_{2}$ and the dissipation $\Gamma$ that describes decaying from $|1\rangle$ to $|0\rangle$ as shown in Fig\ref{fig1}. This system is described by the  Hamiltonian
\begin{equation}
\label{H2}
\hat{H}=\frac{1}{2}\left(\begin{matrix}-i\gamma_{1}&\Omega\\\Omega&i\gamma_{2}\end{matrix}\right),
\end{equation}
and a dissipative channel
\begin{eqnarray}
\label{decay}
\hat{\Gamma}_-=\Gamma\cdot\hat{\sigma}_{-}=\left(\begin{matrix}0&\Gamma\\0&0\end{matrix}\right).
\end{eqnarray}
Based on the above framework, one can obtain the vectorized superoperator of the non-Hermitian Liouvillian master equation:
 \begin{equation}
\label{vectorized form}
\mathcal{L'}=i\left(\begin{matrix}-\gamma_{1} & \frac{i\Omega}{2} & -\frac{i\Omega}{2} & \Gamma\\ \frac{i\Omega}{2} & \frac{\gamma_{2}-\gamma_{1}-\Gamma}{2}& 0&-\frac{i\Omega}{2}\\ -\frac{i\Omega}{2} & 0 & \frac{\gamma_{2}-\gamma_{1}-\Gamma}{2}&\frac{i\Omega}{2}\\0&-\frac{i\Omega}{2}&\frac{i\Omega}{2}&\gamma_{2}-\Gamma\end{matrix}\right),
\end{equation}
whose one form of eigenvalues is
\begin{equation}
\begin{aligned}
\label{eigenvalues}
\lambda_{1}&=i\frac{1}{2}\eta_-,  \\
\lambda_{2}&=i\frac{1}{2}[\eta_{-}-2i\Lambda(0)], \\
\lambda_{3}&=i\frac{1}{2}[\eta_{-}+i\Lambda(\sqrt{3})],\\
\lambda_{4}&=i\frac{1}{2}[\eta_{-}+i\Lambda(-\sqrt{3})],
\end{aligned}
\end{equation}
and the corresponding eigenstates are given
\begin{equation}
\begin{aligned}
\label{eigenstates}
\hat{\rho}^{1}&\propto\left(\begin{matrix} 0 \\ 1 \\ 1 \\ 0\end{matrix}\right),\\
\hat{\rho}^{2}&\propto\left(\begin{matrix} 1-\frac{[\eta_{+}+2i\Lambda(0)](3\eta_{+}+\Theta^2-12 \Omega^2)}{6 \Omega^2 \Theta} \\ -\frac{i[\eta_{+}+2i\Lambda(0)]}{2 \Omega} \\ \frac{i[\eta_{+}+2i\Lambda(0)]}{2 \Omega} \\ 1\end{matrix}\right), \\
\hat{\rho}^{3}&\propto\left(\begin{matrix} 1+\frac{[\eta_{+}-i\Lambda(\sqrt{3})][-i\Lambda(\sqrt{3})]}{2 \Omega^2} \\ -\frac{i[\eta_{+}-i\Lambda(\sqrt{ 3})]}{2 \Omega} \\ \frac{i[\eta_{+}-i\Lambda(\sqrt{3})]}{2 \Omega} \\ 1\end{matrix}\right),\\
\hat{\rho}^{4}&\propto\left(\begin{matrix} 1+\frac{[\eta_{+}-i\Lambda(-\sqrt{3})][-i\Lambda(-\sqrt{3})]}{2 \Omega^2} \\ -\frac{i[\eta_{+}-i\Lambda(-\sqrt{ 3})]}{2 \Omega} \\ \frac{i[\eta_{+}-i\Lambda(-\sqrt{3})]}{2 \Omega} \\ 1\end{matrix}\right).
\end{aligned}
\end{equation}
Here we define $\Lambda(x)=\frac{(i+x)\Theta}{6}+\frac{(i-x)(\eta_+^2-4\Omega^2)}{2\Theta}$, with
$\eta_{\pm}=\pm \gamma_1+\gamma_2-\Gamma$ and $\Theta=\left[54 \Gamma \Omega^2+\sqrt{2916\Gamma^2 \Omega^4-27(\eta_+^2-4\Omega^2)^3}\right]^{1/3}$. EPs can be found and classified by the above equations and the details will be discussed in the next section.

To further explore and classify the properties of the Liouvillian spectrum of Eq.(\ref{eigenvalues and eigenstates}), we focus on the second and third rows of the eigen-equation governed by Eq.(\ref{vectorized form})
\begin{subequations}
\label{eigen_L_2,3}
\begin{equation}-\Omega\rho_{00}^{j}/2+i(\gamma_{2}-\gamma_{1}-\Gamma)\rho_{01}^{j}/2+\Omega\rho_{11}^{j}/2=\lambda_{j}\rho_{01}^{j}\end{equation}
\begin{equation}\Omega\rho_{00}^{j}/2+i(\gamma_{2}-\gamma_{1}-\Gamma)\rho_{10}^{j}/2-\Omega\rho_{11}^{j}/2=\lambda_{j}\rho_{10}^{j},\end{equation}
\end{subequations}
where $\rho_{mn}$ ($m,n=0,1$) are the elements of eigenstates $\rho^{j}$.
In particular, the diagonal elements $\rho^{j}_{00,11}$ respectively donate probabilities of $|0\rangle$ and $|1\rangle$, while the off-diagonal elements $\rho^{j}_{01,10}$ represent the coherence between $|0\rangle$ and $|1\rangle$. Further, the Eq.(\ref{eigen_L_2,3}) reduces to
 \begin{equation}
\label{the classification of rho}
\rho_{01,10}^{j}=\frac{\Omega(\rho_{00}^{j}-\rho_{11}^{j})}{2\lambda_j-i\eta_-}.
\end{equation}
The coherence elements $\rho_{01,10}^{j}$ are zero if $\rho_{00}^{j}=\rho_{11}^{j}$ or $\Omega=0$ or purely imaginary numbers when $\rho_{00}^{j}\neq\rho_{11}^{j}$, $\Omega\neq0$ and $Re[\lambda_j]=0$. It implies that there is incoherence between $|0\rangle$ and $|1\rangle$ in the first case. In the second case, to ensure that the nondiagonal elements of density matrixes are purely imaginary numbers, the phase between $|0\rangle$ and $|1\rangle$ of the corresponding pure states must be $|\pi/2|$. For mixed states, each coherence part of mixed states also has the above property. Especially, as in the second case, there is not dynamic phase between $|0\rangle$ and $|1\rangle$ when $Re[\lambda_j]=0$. So the phase is geometric, and that is topological because of $|\pi/2|$ being fixed values. In the following, we will discuss the different cases based on the above classification.

\subsection{Systems without coherence}

\begin{figure}[tbp]
\includegraphics[width=8.0cm]{fig2}
\caption{(Color online) The imaginary part of the eigenvalues $\lambda_{1}$ (green dashed line), $\lambda_{2}$ (magenta solid line), $\lambda_{3}$ (red dash-dot line) and $\lambda_{4}$ (blue dotted line) as a function of $\gamma_{2}/\gamma_{1}$ in the system where (a) $\Omega=2\gamma_{1}$ and $\Gamma=\gamma_{1}$; (c) $\Omega=0$ and $\Gamma=2\gamma_{1}$. Probabilities $\rho_{00}$ (red dash-dot line) and $\rho_{11}$ (blue dotted line) as a function of $\gamma_{1}*t$ with the initial state being $(1/4,0,0,3/4)^T$ where the paremeters of (b) and (d) are the same as those of (a) and (c). Support $\gamma_{1}$ is the energy unit in this work.}\label{fig2}
\end{figure}

Considering the case of incoherence between $|0\rangle$ and $|1\rangle$, in other words, $\rho_{01,10}^{j}=0$, the eigen-equation governed by Eq.(\ref{vectorized form}) becomes
 \begin{equation}
\label{eigen_L_rho_2,3=0}
\left(\begin{matrix}i(\Gamma\rho_{11}^{j}-\gamma_{1}\rho_{00}^{j}) \\
\frac{\Omega}{2}(\rho_{11}^{j}-\rho_{00}^{j})  \\
\frac{\Omega}{2}(\rho_{00}^{j}-\rho_{11}^{j})  \\
i(\gamma_{2}-\Gamma)\rho_{11}^{j}\end{matrix}\right)=\lambda_j \left( \begin{matrix}\rho_{00}^{j} \\0\\0\\ \rho_{11}^{j}\end{matrix}
 \right).
\end{equation}
When $\rho_{00}^{j}=\rho_{11}^{j}$ is satisfied, Eq.(\ref{eigen_L_rho_2,3=0}) has a steady state solution with $\Gamma=\frac{1}{2}(\gamma_1+\gamma_2)$. The corresponding eigenvalue and eigenstate are $\lambda_{s1}=0$ and $\rho^{(s1)}=(\frac{1}{2},0,0,\frac{1}{2})^T$, respectively. Due to $Re[\lambda_{j}]=0$, we only focus on the imaginary part of the eigenvalues $\lambda_{j}$ as a function of $\gamma_{2}/\gamma_{1}$ in the case of $\Gamma=\gamma_1$ in Fig.\ref{fig2}(a) and can clearly see that there exists a steady state at $\gamma_{2}/\gamma_{1}=1$. To better understand the consequences of this scenario, we have also plotted the time evolution of the state as shown in Fig.\ref{fig2}(b). Obviously, only $\rho^{(s1)}$ survives and the rest decay in the end. On the other hand, when $\Omega=0$ is satisfied, the system has an other steady state solution with $\Gamma=\gamma_{2}$. In this case, the eigenvalue and eigenstate are $\lambda_{s2}=i(\gamma_{2}-\Gamma)=0$ and $\rho^{(s2)}=(\frac{\Gamma}{\gamma_1+\gamma_2},0,0,\frac{\gamma_1+\gamma_2-\Gamma}{\gamma_1+\gamma_2})$, respectively. The spectra about $\gamma_{2}$ and the time evolution of the state as whown in Fig.\ref{fig2}(c,d). Interestingly, this steady state can be entirely controlled by the parameters of gain $\gamma_{1}$, loss $\gamma_{2}$ and dissipation $\Gamma$. This property is completely different from that of the conventional PT symmetry non-Hermitan or dissipation systems.

What's more, for $\Omega = 0$, the equations $-2\Lambda(0)=\Lambda(\sqrt{3})=\Lambda(-\sqrt{3})=0$ has a solution when $\gamma_{1}+\gamma_{2}=\Gamma$. In this case, all eigenvalues degenerate and all eigenstates coalesce:
\begin{subequations}
\label{NLEPs1}
\begin{equation} \lambda^{NLEPs}=-2i\gamma_{1},\end{equation}
\begin{equation} \hat{\rho}^{NLEPs} \propto(1,0,0,0)^T.\end{equation}
\end{subequations}
Unlike Lindblad exceptional points (LEPs) proposed in Ref.\cite{Minganti} in which LEPs are only induced by the decay channel, this degenerate point is jointly induced by the effects of non-Hermiticity and dissipation, therefore this point is referred as non-Hermitian Lindblad exceptional points(NLEPs).

\begin{figure}[tbp]
\includegraphics[width=8.0cm]{fig3}
 \caption{(Color online) (a) The imaginary part and (b) the real part of the eigenvalues $\lambda_{1}$ (green dashed line),  $\lambda_{2}$ (magenta solid line),  $\lambda_{3}$ (red dash-dot line) and $\lambda_{4}$ (blue dotted line) as a function of $\gamma_{2}/\gamma_{1}$ in the system for $\Omega=2\gamma_{1}$ and $\Gamma=0$. (c) The imaginary part and (d) the real part of the $\rho_{10}^{(2)}$ (magenta solid line), $\rho_{10}^{(3)}$  (red dash-dot line) and $\rho_{10}^{(4)}$  (blue dotted line) as a function of $\gamma_{2}/\gamma_{1}$.}\label{fig3}
 \end{figure}

\subsection{Systems with coherence}

\begin{figure}[tbp]
\includegraphics[width=8.0cm]{fig4}
 \caption{(Color online)(a) The imaginary part and (b) the real part of the eigenvalues $\lambda_{1}$ (green dashed line), $\lambda_{2}$ (magenta solid line), $\lambda_{3}$ (red dash-dot line) and $\lambda_{4}$ (blue dotted line) as a function of $\gamma_{2}/\gamma_{1}$ for $\Omega=2\gamma_{1}$ and $\Gamma=2\gamma_1$.  (c) Probabilities $\rho_{00}$ (blue line), $\rho_{11}$ (red line) and (d) the real part (green line), imaginary part (carmine line) of $\rho_{10}$ as a function of $\gamma_{1}*t$ when the initial state is $(1/2,1/2,1/2,1/2)^T$ for $\gamma_{2}=1.298\gamma_{1}$. (e) The imaginary part and (f) the real part of $\rho_{10}^{(2)}$ (magenta solid line), $\rho_{10}^{(3)}$  (red dash-dot line) and $\rho_{10}^{(4)}$  (blue dotted line) as a function of $\gamma_{2}/\gamma_{1}$ for $\Omega=2\gamma_{1}$ and $\Gamma=2\gamma_1$.}\label{fig4}
\end{figure}

As we previously discussed, there exists coherence if only if $\rho_{00}\neq\rho_{11}$ and $\Omega\neq 0$. We first consider the case of the absence of dissipative channels($\Gamma=0$). In this case, this system reduces to a normal non-Hermitian system and the eigenvalues are shown in Figs.\ref{fig3}(a) and \ref{fig3}(b). So this system exhibits PT-symmetry at $\gamma_{1}=\gamma_{2}$ and the corresponding eigenvalues are purely real numbers if $\Omega \geq\gamma_{1,2}$. On the other hands, there are two exceptional points at $\gamma_{2}/\gamma_{1}=\pm\frac{2\Omega}{\gamma_{1}}-1$ that is also deduced from  $-2\Lambda(0)=\Lambda(\sqrt{3})=\Lambda(-\sqrt{3})=0$. And the corresponding density matrices are $(1,\pm i,\mp i, 1)^T$. More importantly, for $\gamma_{2}/\gamma_{1}\geq\frac{2\Omega}{\gamma_{1}}-1$ and $\gamma_{2}/\gamma_{1}\leq-\frac{2\Omega}{\gamma_{1}}-1$, there are NH Fermi arcs in the purely imaginary eigenvalues regimes. In this case, based on Eq.(\ref{the classification of rho}), the off-diagonal elements of the corresponding density matrices must be purely imaginary numbers. In other words, the phase of eigenstates between $|0\rangle$ and $|1\rangle$ must be $|\pi/2|$. To prove it, the off-diagonal elements $\rho_{01(10)}^{j}$ with $j=2,3,4$ (the eigenstate $\hat{\rho}_{1}$ is trivial) are shown in Figs.\ref{fig3}(c) and \ref{fig3}(d). This result is clearly verified in EPs but have not been theoretically predicted and experimentally demonstrated in NH Fermi arcs regimes.

However, for $\Gamma\neq0$, several distinct features emerge. Firstly, the imaginary parts of the eigenvalues split so it means that there are not PT symmetry steady state solution. This result illustrates in Figs.\ref{fig4}(a) and \ref{fig4}(b). But based on Eq.(\ref{steady state}), there is still a steady state when $Im(E_2)=0$ and $Im(E_j)<0(j=1,3,4)$. And the off-diagonal elements of $\hat{\rho}_{2}$ must be purely imaginary numbers through Eq.(\ref{the classification of rho}). The numerical calculations show that, for a initial state $(1/2,1/2,1/2,1/2)^T$, a steady state emerges and its off-diagonal elements of density matrix are purely imaginary numbers, as plotted in Figs.\ref{fig4}(c) and \ref{fig4}(d). Secondly, allowing for $\Lambda(\sqrt{3})=\Lambda(-\sqrt{3})$, we see that degeneracies occur when
\begin{equation}
\label{condition of EPs}
\eta_+=\pm\sqrt{3(4\Gamma^2\Omega^4)^{1/3}+4\Omega^2}.
\end{equation}
So in this case, NLEPs emerge and the corresponding eigenvalues and eigenstates can be computed to
\begin{subequations}
\label{NLEPs3}
\begin{equation} \lambda^{NLEPs}=i\frac{1}{2}[\eta_{-}-(2\Gamma\Omega^2)^{1/3}],\end{equation}
\begin{equation} \hat{\rho}^{NLEPs} \propto\left(\begin{matrix} 1+\frac{[\eta_{+}+(2\Gamma\Omega^2)^{1/3}](2\Gamma\Omega^2)^{1/3}}{2 \Omega^2} \\ -\frac{i[\eta_{+}+(2\Gamma\Omega^2)^{1/3}]}{2 \Omega} \\ \frac{i[\eta_{+}+(2\Gamma\Omega^2)^{1/3}]}{2 \Omega} \\ 1\end{matrix}\right).\end{equation}
\end{subequations}
Thirdly, NH Fermi arcs also exist, i.e., they are not affected by dissipations. Meanwhile, the off-diagonal elements of the corresponding density matrices must be purely imaginary numbers through Eq.(\ref{the classification of rho}). $\rho_{01(10)}^{j}$ with $j=2,3,4$ (the eigenstate $\hat{\rho}_{1}$ is trivial) as a function of $\gamma_{2}/\gamma_{1}$ is shown in Figs.\ref{fig4}(e) and \ref{fig4}(f). Obviously, NLEPs, which is shown in Eq.(\ref{NLEPs3}), is still a special case of NH Fermi arcs. In other words, although the eigenstates of NH Fermi arcs are mixed states in this case, the phase between $|0\rangle$ and $|1\rangle$ of the pure states parts is still $|\pi/2|$.

\section{discussion and conclusion}

In summary, we have explored an open system with non-Hermitian and dissipations via mapping the non-Hermitian Lindblad equation to a vectorized equation. According to this frame, we propose a general condition of the steady state with open systems. As an illustration, we study a two-level system coupling to an environment with gains, losses, and dissipations. We find that there is a new class of steady states and EPs due to the interplay between non-Hermiticity and dissipations. We further demonstrate that the phase between $|0\rangle$ and $|1\rangle$ of eigenstates of NH Fermi arcs must be $|\pi/2|$ while EPs is a  special case of this. In short, our work brings a new insight into NH systems and provides a new tool to explore the rich variety of open systems.

\acknowledgements
This work is supported by the National Natural Science Foundation of China (Grants No.11747107), the Natural Science Foundation of Guangdong province (Grants No. 2017A030307023), the Scientific Research Project of Hunan Province Department of Education (Grant Nos.19B060). Y N Guo is supported by the Program of Changsha Excellent Young Talents (kq1905005).


text:
Personal Reminiscences
One of the features of aging past ninety is accumulating many stories to tell. Perhaps fewer are left alive to challenge insignificant details.
THREE WAYS TO ACHIEVE FUNDED HEALTH INSURANCE
Since everybody would benefit if the country found a way to pre-fund health insurance, what options are available or doing so? Or put another way, what obstacles, objections, and vested interests obstruct each possible way of doing it; how difficult would it be to overcome them?
Three general approaches might work reincarnation of Individual Retirement Accounts (IRAs) .for this special purpose, the legal encouragement of health insurance companies to sell permanent insurance or the creation of a national trust fund. Each of these approaches will now be changed to make them possible. All of them must somehow overcome the main problem inherent in payment in advance, which is that people must find spare money somewhere. while of course the disposable income could be found by reducing expenditures for non-essential luxuries, this discussion limits itself to examining how seed money might be squeezed out of inefficient practices in the existing health financing system.
Health Banking The first approach to be examined has been known by several names. When Michael J. Smith of Louisiana proposed it, he called it the CHIP proposal (Consumer's Health Investment Plan). When John McClaughry and I independently proposed something similar, it was called the Health Protection Account, Peter J. Ferrara of the Cato Institute, who likewise got the idea independently, had the genius to call it the IRA for Health. The IRA for Health has the great beauty that the title itself does most of the explaining; most people know what an IRA is, or was. An IRA for Health is obviously an IRA whose use is limited to health care costs. Unfortunately, congressman, whose cooperation is vital to the success of the idea, have gone through the experience of repealing the tax exemption of IRA's trying to reduce the national budget deficit. Congressmen have the mindset that IRAs will worsen the deficit, or at least that their constituents might think so. It isn't a correct perception, but the catchy H-IRA title has become a hindrance and must be replaced. Health Banking is here offered as a substitute title for Health Iras because it suggests an interest group who might assist the process of public persuasion because they would benefit from it. The definition of what constitutes a bank is in the process of change; perhaps there is room for health banking in some future compromise revision of the Glass-Steagall Act. For example, the mutual fund industry could become interested in health banking as part of their vision of non-bank banking. Perhaps the moribund savings banks could be entrusted with a trillion-dollar consolation for their regulation-induced troubles.
Whatever institution might aggregate and invest the funds, the main idea is that the tax-sheltered money which employers now spend on health benefits would be deposited in individually named accounts, and invested according to some rule of prudence. Withdrawals from the accounts would only be permitted for specified health purposes. It is absolutely central to understand that nothing is here proposed to be tax sheltered which is not already tax sheltered. For better or worse, employer contributions for health benefits are already tax-sheltered, and will almost surely remain so. Even recent deficit-desperate congresses have not dared to threaten that $50 billion annual revenue cost to the Treasury. Indeed, when Congress does develop the courage to curtail the tax exemption of health benefits (for employed people), perhaps then health banking could be described as costly. Until such time, however, it remains fair to generalize that reestablishment of IRAs, limited to health expenditures, has no Federal cost.
In the next chapter, we will discuss the investment issues which are raised. For the present, we should assume that the institution which becomes custodian or health banker will generate a safe and adequate stream of revenue; what rules must govern the way it is spent? Ideally, a health investor would want to accomplish the following goals:
-He would want to pay for essential health care.
-If he lives a long time and has money to spare, he would want to provide for custodial care in his old age.
If he dies and has money left over, he would want it to go to his estate.
But these are decreasing priorities. The first rule is survival. If he cannot pay for his old age, he must plan to throw himself on the mercy of family, friends, and community. If he has nothing left for his heirs that is no worse than a pity when he has no dependents, but tragic if he does.
To the foregoing self-evident principle should be added two more which are less adequately appreciated. The first is that the federal government has long addressed the problem of aid for dependent children, and ordinary Social Security benefits are quite generous for survivors. No doubt these programs for widows and orphans could be more generous, but they are as generous as twenty congresses have decided to be; it is unrealistic to include provision for dependent survivors in a new program proposal.
The second mitigating principle is nature's trade-off. Nothing is more clear to a doctor than the experience that ten percent of his practice develop ninety percent of the illness. If you get very sick very often, you need not worry greatly about the prospective costs of living for years in a retirement home. Dying young is a tragedy, and outliving your income in a shabby old age are both tragedies, but relatively few people experience both of them. To the extent that some people are needlessly insured against both disasters, it should be possible to create insurance efficiencies by combining both risks in one funding mechanism.
By reflecting on the general principles of prudent provision for the future, we arrive at the following proposed rules for releasing money out of healthy banks (aka IRAs for Health). Vouchers may be issued, or disbursements made directly, to pay health insurance premiums, front-end deductibles, nursing home insurance, entrance fees for life-cares communities, specially designated trade-offs with Medicare, and payment of any residual balance to the individual's estate. Investment income would not be taxed, but the estate tax would apply after age sixty-five. Since this statement of benefits is technical and terse, the following explanatory footnotes are necessary:
Payment of front-end deductibles. If the health bank only paid the present premiums of health insurance there would be nothing left over to invest. It is anticipated that premiums would initially be lowered y purchasing health insurance with at least $500 yearly deductible, and the amount of the "front-end" deductible would later gradually rise as funds accumulate in the account. This process would, in turn, cause a progressive lowering of the premium of the health insurance, accelerating the savings. When expensive illness occurs, the fund would reimburse the subscriber the full amount of the deductible, upon presentation of evidence that his health insurer had paid out amounts in excess of it.
Special trade-offs with Medicare, Nursing home care, Life-care communities. It will be necessary for the reader to be patient for these proposals to coordinate health banking with the entitlements under Medicare and federal catastrophic health insurance. The subject is dealt with in the section of the book devoted to Prescription II: Last Year of Life Insurance.
Payment of Residuals to Estates.No doubt there would be an instinct to examine whether the program could be made less expensive if residuals were forced to revert to the federal treasury. To permit an estate to retain these funds would seem to enrich the heirs out of funds which were largely made possible by federal income tax exemption. True enough, but long experience in my medical practice with life tenants of trust funds has been that they will relentlessly seek ways to spend money on pseudo medical care if there is no other way to get their hands on what they regard as their rightful money. Some incentive must be created to prevent "health insurance companies" from creating benefits like suntan oil and health spas in warm climates, whose only real purpose is to collude with beneficiaries. To attempt to prevent this sort of behavior with regulations is laughable.
That's not all you could do with health banking, however. If individuals should die young or use up all of their funds with repeated expensive illnesses, they might seem to be about where they are now without health banking, minus the extra layer of administrative overhead. But they have achieved portability. They can become unemployed, change employers, or retire early without the same concern that they have lost their health insurance. If they have developed a chronic problem, it has not become a "pre-existing condition" in the eyes of some new insurance carrier. They can quit their jobs without fear that a new employer with company self-insurance would regard them as potentially expensive employees. They need not fear that post-retirement health benefits will be blown away by a corporate raider, or that the company where they worked for years will subsequently go bankrupt and leave them with empty promises instead of health insurance. Portability.
For the vast majority of participants, however, another whole new concept becomes possible after a few years. We are now back to Benjamin Franklin and perpetual insurance. The health bank was just a transitional stage to reach it. The ingredients of perpetual health insurance are:
The fund has grown so large that its investment income is big enough to pay the premium on a high-deductible health insurance policy without any further contribution
And the fund is big enough to pay all of the deductible portions of a major illness
And there is still enough investment income to pay the premium of a reinsurance policy which covers the unusual instance of sustaining two or more major illness.
As a guess, such a fund would be in the neighborhood of thirty thousand dollars. As a further guess, most people would continue to work past the time the fund went perpetual, and further contributions would accumulate. Individuals who reached this happy state of affairs would almost by definition be fairly healthy, and in all likelihood would look forward to a long and dreadful sojourn in a nursing home. For a quick overview of the situation, the next time you go to a high school reunion, take a stroll over to the area where the fifty-year class is meeting. About half of the class will be in attendance, some in wheelchairs, but the other half of the class will be in memoriam. Fifty-fifty at the fiftieth.
What has been briefly described is a mechanism for designating a considerable amount of funds for long term care that are not now available. It was achieved without financial or other barriers to healthcare access except possibly some self-denial since full reimbursement for healthcare is available to every participant. If someone wants to put up with his hemorrhoids in order to have more money for later nursing home care, or if he wants to hold off entering a nursing home in order leave more money in his estate, well, those are his options. If he holds off getting treated for his cancer "for those same reasons", in my medical experience those wouldn't have been the true reasons", in my medical experience those wouldn't have been the true reasons at all and he would have held off no matter what his insurance arrangement might be. So, what's the magic? Where did this money come from?
From the operation of compound interest of between 3% and 8%, depending on the amount of equity risk assumed.
From self-denial of borderline necessary or even frivolous medical expenditures in response to the development of an incentive to save.
From the downward pressure on medical prices created by a more cost-involved public.
From the out-of-pocket contribution of health care expenses which fail to exceed the yearly deductible amount.
From reduced administrative costs of processing small claims through first-dollar coverage rather than paying them out-of-pocket. The reader is invited to consider this issue in more detail in Prescription IV: Insurance Tangles.
Will all this be enough to carry the project? Hard to know. What isn't hard to know is that savings of just 1% of a $500 million annual medical cost, amount to $5 billion per one percent saved.
Health Insurance Companies Unchained.Observers of the economic scene, especially the Washington political-economic scene, will be well aware that health banking is a proposal from the conservative side of politics. It had origins in the Reagan administration and the libertarian Cato Institute. A staunch Virginia conservative, Representative French Slaughter, introduced a bill (HR 536) along these lines, and he has been supported by the Republican Study Group under the chairman of Representative David Dreier of California. Just exactly because the proposal has this sort of appeal, it has a strong weakness. The idea is strongly rooted in the idea that individuals should make their own choices, even to the point of being allowed to make wrong choices. Individual citizens should be allowed to have their own way with their own money, even if their betters think their choices are not in the best interest of society or even best for the particular individual.
Well, that's just fine, but unfortunately in this instance, it comes up against a pretty big problem. Even when the IRA was available, only 14% of the eligible public availed themselves of it, there were vast numbers of people who had never heard of it. I spent endless hours fruitlessly attempting to persuade several of my secretaries to create an IRA, and even some of my children had to be prodded pretty hard. An executive of one of the largest corporations in the country told me of the dispiriting experience at his company with a 401 (k) plan, which is essentially an IRA. The company offered to match contributions dollar for dollar and conducted a large and continuous education program. This man made the estimate that in the whole corporation no more than a dozen unmarried women availed themselves of the opportunity, and the workforce included plenty of women lawyers and accountants. With an experience of only 14% enrollment in the IRA, it certainly isn't fair to point to just single women as having an anti-saving mentality, although I suppose there is something to it if he says so. In doing my little bit to encourage the world to establish IRAs, I noticed that one of the most effective inducements among those who did it was the wicked pleasure of beating the government out of some taxes.
So, we enthusiasts must make a grudging concession that if health banking is to succeed on a major scale any time soon, it must do more than just make itself available and then wait for everybody to get smart enough to buy it. It isn't necessary to share the rhetoric one union president unleashed on me, "You've got to find a way to keep the working man from spending his wages on drink". The paternalism of that degree is too much for me, but I will concede that something must be done to prod people a little. And that missing ingredient just might be a vast army of glib commission-motivated insurance salesmen.
It must not be assumed that health insurance companies are straining at the leash, impatiently lobbying for legislative release from their inability to sell funded health insurance. In fact, most of them do not have much sales force since health insurance is typically marketed in bulk to employer benefits departments. Most of them do not have much investment portfolio management department, especially outside the area of the short-term paper. There is minimal research on the subject of lifetime health costs. Incentives are weak; health insurance is typically regarded as a loser by full-line companies, Blue plans which dominate what is left are non-profit concerns. Insurance companies are fiduciaries, custodians in trust of other people's money. If careless paying obligations they get sued, if they can't pay their obligations they go bankrupt. If they make big profits they get bad press or ruinous competition, it doesn't matter which.
However nothing important is easy, and we want them to peddle our product. What's their problem? Their central problem turns out to be the fact that funded health insurance asks the company to assume the risk, and no one knows how to price the risk. Using the health banking approach, the individual subscriber assumes the twin risks that future health costs will inflate at a predictable rate and that investment results will equal the historical record. Who knows if we will have inflation like a banana republic, a depression like the one in 1933, or the development of medical miracles no one can afford? That's a risk, which the individual holder of a tax-sheltered medical account might assume willingly on the argument that any outcome could be no worse than unfunded health insurance. Put an insurance company's guarantee into the hands of that person, however, and you had better pay your promise if you want to stay out of court. Life insurance companies are just beginning to experiment with best-efforts investing, using the name of variable annuity life coverage, and that sort of approach would surely be necessary for funded health insurance. Predicting the average life expectancy of a large group of healthy people is however a minor mathematical exercise compared with estimating the aggregate average health care costs for the next thirty-five years. The actuary doesn't even know for certain how many people will have any health cost at all; at least the life actuary knows that everyone is going to die. The advent of a new and fatal epidemic like AIDS shows you what these predictions can be worth. The price Dr. Starzl's hospital charges to transplant your liver startled even me when I heard it last year.
Quick reflection on the issues brings the firm conclusion that health insurance companies could only be expected to offer best-efforts investing, without any insurance guarantee at all. They would surely market the product skillfully, as we desire they should, but the cost increment would be quite appreciable. The sales cost for life insurance is typically equal to the entire first year's premium, while an additional profit must be offered the company to induce it to handle the product. Investment experts will have to be paid to manage the money, and they characteristically value their own services highly. It's too bad to have to say it, but if you are too dumb to manage your own money you are going to pay through the nose to have someone else do it.
Perhaps this is an appropriate time to mention the idea that some large unions have toyed with, of becoming self-insured for the health benefits of their members. All that means is they would expect the employer to give them the money, and they would pay it out. In order words, they propose to become insurance companies. Most everything which has been said about other health insurance, therefore, applies to them, too. In addition, it might not hurt to mention Ricardo's principle of relative advantage, which roughly states that people with professional experience in a field often do the best job at it.
Publicly Managed Trust Funds. The third general method for achieving pre-funded health insurance is one which the American Medical Association as a method for basic restructuring of the Medicare program. In a widely discussed report of the AMA proposed the creation of an independent federal agency to receive the health insurance payroll taxes which now flow into the Medicare trust funds, invest them, and issue vouchers to the elderly which would permit them to purchase private health insurance. The agency would also fill the badly needed role of a group of experts charged with overseeing the course of national health inflation, demographic and health practice changes, and the national investment climate.
The AMA employed expert advice in the preparation of this plan, and it is likely their financial estimates are sound when they project that Medicare could become totally pre-funded by 2016, using tax rates achievable close to present taxation. This plan would include provision for working off the present unfunded liability for existing beneficiaries, honoring commitments previously made to the elderly. It is a great testimonial to the flexibility of the AMA in recent years. as well as to the tremendous power of compound interest, that such a monumental problem could be effectively addressed. After all, the AMA in 1965 quite correctly warned the country that the Medicare program would quickly pile up mountains of unfunded obligations if it adopted a "pay as you go" financing method. Not everyone who has been vilified for being right will subsequently adopt a constructive problem-solving approach to repair the accurately predicted damage.
The creation of an independent board of expert overseers supplies a need which health banking through the IRA doesn't, and creations of a government agency create a guarantor of last resort, even though it might be stoutly denied. Lots of Americans trust their government to keep its promises. However unsophisticated that attitude may be m, it helps get national support for the idea of pre-funded insurance in a way no television spot commercials could do, and no insurance salesman could do over the kitchen table. Every government since the Lydians invented coinage might have remorselessly devalued its currency, and it would not matter. A very large proportion of Americans trust government bonds as they trust nothing else in society.
However, a sizeable group of people simply will not have this approach, and a number of them are in Washington. The AMA made a small tactical error in describing the proposed independent agency as roughly comparable to the Federal Reserve Board in its independence and suggested a similar method of appointment. Apparently, the AMA had no idea of the violence with which the Federal Reserve Board is hated by certain influential members of both legislative and executive branches, or of the decades of bitterness between mid-western bankers and the New York banks which control the open market committee of the Fed. In a sense, the Federal Reserve is actually owned by the banks it regulates and tends to defend their interests. The whole investment community is at war (nice polite war, of course) with banks, struggling for turf in some pending revision of that Treaty of Versailles, the Glass-Steagall Act. Every election year, the Federal Reserve gets embroiled in bitter controversy as to whether its policies are slanted to cover up for congressmen who want the Fed to do their dirty work and have it within their power to abolish the whole agency if it does not comply. Independent as the Fed. Foo.
Let a Hundred Flowers Bloom. No doubt this inflammation can be soothed with diplomacy, but there are more serious worries about the investment difficulties which are unique to a trillion dollar fund run y the Federal Government to be addressed in the next chapter. In the meantime, the three funding mechanisms can be summarized as having certain flaws and certain strengths which are peculiar to each. It seems unlikely that the country could agree to permit only one of them to be designated to the exclusion of the other two. To a serious proponent of the general approach, it seems imperative that we authorize all three approaches while forcing them to remain in competition with each other. After all, we are talking about a system which addresses the needs of everyone no matter how different the circumstances. People who take wild chances should be restrained from becoming public charges, ignorant people should be protected against those who would fleece them. As Adlai Stevenson said, "It used to be said, a fool and his money are soon parted. But nowadays, it can happen to anyone."
There is, however, no certain way to know the future even if you go to Princeton to learn it. Let the banks and mutual funds exploit their lower marketing costs, let the insurance companies sell rings around them. Let the private sector shame the mediocre performance of the public trust fund, while the trust fund stands as a tower of strength among self-serving investment pitchmen. So long as the public is allowed to switch around, things should work out, in an investment version of the system of checks and balances.
Originally published: Monday, September 24, 2018; most-recently modified: Thursday, June 06, 2019


TIP! Try using a masticating type of juicer. A masticating juicer is a lot gentler than ordinary juicers.
Do you know what benefits juicing can bring? People who make their own juice enjoy improved health, increased energy and endurance, enhanced metabolism, and a reduced desire to consume snacks, sweets, and other unhealthy foods. Has this information peaked your interest? You've come to the right place!
TIP! Use cucumber as an additive when juicing dark leafy greens. Many of the leafy greens have a distinct and slight unpleasant flavor.
If your juicing recipe calls for leafy green vegetables, add some cucumber to the mix. Some leafy greens do not taste good. Cucumber will put a nice twist on your drink, as well as masking the taste of unpleasant veggies. Cucumber is also very nutritious, especially if you don't peel it beforehand.
TIP! Only use the sweetest and ripest apples for your apple juice. If you only have bruised apples, cut the bruises out.
When you are juicing apples, pick the sweetest and ripest ones you can find. It is okay to get apples that are bruised, just make sure you cut them out. Choose sweeter apple varieties like Red Delicious, Fuji, Rome, and Gala to make the richest and tastiest juices with.
TIP! Once you have completed making your juice, wash the machine right away. Also, some colorful fruits and vegetables can build up dark stains on the juicer components.
To get the most out of juicing, using leafy greens in your juice blends is a great idea. Kale, beet greens, mustard greens and other leafy greens are all great for the task. The idea here is to use leafy greens not for their flavor, but for their health benefits. So use at least 50% leafy greens and then supplement other items for flavor. Fruit juice is full of natural sugar and includes less vitamins and nutrients than vegetables.
Enjoy each drop of this juice. Make sure to get all the flavor possible out of your concoctions. Let your saliva mix with your juice to help start digestion.
TIP! Having fresh juice available in your fridge is a smart idea, however the color change that juice undergoes will have to be prevented. The juice will start out with a really bright color then change to brown or grey, something that is not really appealing to the eye.
Always leave your juicer on the counter in plain sight. This ensures it's available and ready to go at all times. If you have it in sight you will be sure to use it every day.
TIP! Ginger is a great food that can help to aid gastrointestinal problems. Adding it to your juice blends can give them some zest, while helping you to feel better.
Research your fruits and vegetables before you create. Due to the huge variety of market produce available, many differences can exist in the types of nutrients offered. Your best option is to mix items that will provide you with a variety of vitamins and minerals, ones that will meet your daily requirements. By doing this, you will provide your body with some excellent nutrients, plus you might discover some unique, enjoyable new blends.
TIP! If you're finally starting to feel your age through aches and pains, taking up juicing can make you feel years younger. Juice provides many nutrients that can help to ease joint pain, aid memory or even slow down cell death due to free radicals.
If you find pulp getting into your juice, you can use a coffee filter to remove it. Use this if you don't like the pulp created by some juices. Use a cheesecloth, coffee filter, or fine strainer to take out the pulp if this is the case.
TIP! Try out a masticating juicer when you launch a campaign to incorporate juice into your diet. The additional features, such as milling, pureeing, and grinding, are well worth the price.
When you are choosing a juicer, make sure you buy one that is easy to clean. If you need twenty minutes or more to assemble your juicer, make your juice and then clean up, you will quickly tire of the process. Make sure to clean it soon after you juice when any pulp that's inside is still moist.
TIP! When juicing, consider using vegetables or fruits that would not normally appeal to you and use other components to mask their flavor. This will help you to get all the nutrients you need, especially ones you might miss otherwise.