\section{Introduction}
\label{sec:1}

Electrons floating on liquid helium surface have been considered as an alternative physical system to implement quantum computation~\cite{Platzman1967,PhysRevX.6.011031,
PhysRevLett.89.245301,PhysRevB.67.155402,
PhysRevLett.107.266803,PhysRevLett.101.220501,
PhysRevLett.105.040503,Monarkha2019,
PhysRevLett.124.126803,Spin,PhysRevB.86.205408}. Sommer first noticed that there is a barrier at the surface of liquid helium that prevents electrons from entering its interior~\cite{PhysRevLett.12.271}. Therefore, in addition to the attractive force of image charge in liquid helium, the surface electron forms a one-dimensional (1D) hydrogen atom above liquid helium surface~\cite{RevModPhys.46.451,PhysRevLett.32.280,PhysRevB.13.140}.
The two lowest levels of the 1D hydrogen atoms have been proposed to encode
the qubits, which have good scalability due to the
strong Coulomb interactions between electrons~\cite{Platzman1967}. However, the single electron detection is still very difficult in experiments. Recently, E. Kawakami et.al. have used the image-charge approach to experimentally detect the Rydberg states of surface electrons on liquid helium~\cite{PhysRevLett.123.086801}. Compared with the conventional microwave absorption method, this method may be more effective to detect the quantum state of surface electrons~\cite{Elarabi2021,PhysRevLett.126.106802}.

The transition frequency between the lowest two levels of surface electrons is in the regime of Terahertz (THz) and therefore has perhaps applications in the fields of THz detection and generation. The THz spectrum is between the usual microwave and infrared regions, and has many practical applications~\cite{NaturePhotonics}, such as THz wireless communication~\cite{THZ-Wireless Communication}, astronomical observation~\cite{EPJP-astronomy}, Earth's atmosphere monitoring~\cite{watter}, and the food safety~\cite{food}. Specially, in biological and medical fields~\cite{biology,APL}, THz imaging have smaller damages in body than X-ray, but higher resolution than mechanical supersonic waves.

In this paper, we show a frequency-mixed effect of the THz and the Gigahertz (GHz) electromagnetic waves in the system of electrons floating on liquid helium. This effect could be applied to detect THz radiation. In our mixer, the inputting THz wave near-resonantly excites the transition between the lowest two levels of surface state electrons. The GHz wave does not excite the transitions but generates a GHz-varying Stark effect. This Stark effect has no counterpart in the system of natural atoms. The effective coupling between natural atoms and microwaves needs either the atomic high Rydberg states of $n\approx47$~\cite{Jing2020} or the magnetic coupling (with the strong microwave inputting). Here, the surface-bound states of electrons are symmetry broken due to the barrier at the liquid surface, and therefore the GHz wave can cause a GHz-varying energy gap of electrons on liquid helium by the lowest order electric dipole interaction. Specially, we find an effective coupling between the inputting THz and GHz waves, which appears at the frequencies condition that the detuning between electrons and THz wave is equal to the frequency of GHz wave. By this coupling,
the THz and GHz waves cooperatively excite electrons and generate the low-frequency ac currents along the perpendicular direction of liquid helium surface to be experimentally detected by using the image-charge approach~\cite{PhysRevLett.123.086801,Elarabi2021,PhysRevLett.126.106802}.

\section{The Hamiltonian}
\label{sec:2}

Due to the image potential and the barrier at liquid helium
surface, electronic motion along the perpendicular direction of liquid helium surface can be treated as a 1D hydrogen
atom. Its energy level is well known as ~\cite{PhysRevA.61.034901,PhysRevA.80.055801}
\begin{equation}
\begin{aligned}
E_n=-\frac{\Lambda^{2}e^{4}m_e}{2n^{2}\hbar^{2}}\,,
\end{aligned}
\label{E1}
\end{equation}
with $\Lambda=(\varepsilon-1)/4(\varepsilon+1)$. The
$e$ is the charge of an electron, $m_e$ is the electronic mass, and $\varepsilon= 1.0568$ the dielectric constant of liquid helium. Numerically, the transition frequency between the lowest two levels is $\omega_{12}=(E_2-E_1)/\hbar\simeq0.1~$THz (corresponding to a wavelength of $\lambda\simeq3$~mm).
The eigenfunction of the above energy level reads
\begin{equation}
\begin{aligned}
\psi_{n}(z)=2n^{-5/2}r_{B}^{-3/2}z \exp\left(\frac{-z}{nr_B}\right)L_{n-1}^{(1)}\left(\frac{2z}{nr_{B}}\right)\,,
\end{aligned}
\label{E2}
\end{equation}
with $r_{B}=\hbar^{2}/(m_{e}e^{2}\Lambda)\simeq76\rm{\AA}$ being the Bohr radius, and
\begin{equation}
\begin{aligned}
L_{n}^{(\alpha)}(x)=\frac{e^{x}x^{-\alpha}}{n!}\frac{{\rm d}^{n}}{{\rm d}x^{n}}
(e^{-x}x^{n+\alpha})\,,
\end{aligned}
\label{E3}
\end{equation}
the Laguerre polynomial.

Considering only the lowest two levels in Eq.~(\ref{E1}), the Hamiltonian of an electron driven by the classical electromagnetic fields can be written as
\begin{equation}
\begin{aligned}
\hat{H}_0=\hat{H}_{00}+\hat{V}\,,
\end{aligned}
\label{E4}
\end{equation}
with the bare-atomic part
\begin{equation}
\begin{aligned}
\hat{H}_{00}=E_1|1\rangle\langle1|+E_2|2\rangle\langle2|\,,
\end{aligned}
\label{E5}
\end{equation}
and the electric dipole interaction
\begin{equation}
\begin{aligned}
\hat V&=e\textbf{z}\cdot \textbf{E} =(\textbf{u}_{11}\hat{\sigma}_{11}+\textbf{u}_{22}\hat{\sigma}_{22}
+\textbf{u}_{12}\hat{\sigma}_{12}+\textbf{u}_{21}\hat{\sigma}_{21})\cdot\textbf{E}\,.
\end{aligned}
\label{E6}
\end{equation}
Here, $\textbf{u}_{ij}=e\langle i|\textbf{z}|j\rangle=e\textbf{z}_{ij}$ is the electric dipole moment, and $\hat{\sigma}_{ij}=|i\rangle\langle j|$ the two-level operator with $i,j=1,2$. Note that $\langle 1|\textbf{z}|1\rangle \neq 0$ and $\langle 2|\textbf{z}|2\rangle \neq 0$ for the wave function (\ref{E2}). On electric field $\textbf{E}$, we first consider the case of two-wave mixing, i.e.,
\begin{equation}
\begin{aligned}
\textbf{E}(t)&=\textbf{E}_{T}\cos(\omega_{T}t+\theta_{T})
+\textbf{E}_{G}\cos(\omega_{G}t+\theta_{G})\,,
\end{aligned}
\label{E7}
\end{equation}
with $\omega_{T}\simeq 0.1~$ THz and $\omega_{G}\simeq 1~$GHz. The $\textbf{E}_{T}$ and $\textbf{E}_{G}$ are the amplitudes of the inputting THz and GHz waves. The $\theta_{T}$ and $\theta_{G}$ are their initial phases, respectively.

Limiting within the usual rotating-wave approximation, the $\omega_{G}$ works only on the first two terms in Eq.~(\ref{E6}), and $\omega_{T}$ works only on the last two terms in Eq.~(\ref{E6}).
Then, the Hamiltonian (\ref{E4}) in the usual interacting picture [defined by the unitary operator $\hat{U}_0=\exp(-i\hat{H}_{00}t/\hbar)$] can be approximately written as
\begin{equation}
\begin{aligned}
\hat{H}=\hbar\left[\Omega_{11}(t)
\hat{\sigma}_{11}+\Omega_{22}(t)\hat{\sigma}_{22}
+\Omega_{12}(t)\hat{\sigma}_{12}+\Omega_{21}(t)\hat{\sigma}_{21}\right]\,,
\end{aligned}
\label{E8}
\end{equation}
with the time-dependent coupling frequencies $\Omega_{11}(t)=2\Omega_{110}
\cos(\omega_{G}t+\theta_{G})$ ,
$\Omega_{22}(t)=2\Omega_{220}\cos(\omega_{G}t+\theta_{G})$, and
$\Omega_{12}(t)=\Omega_{21}^{\ast}(t)=\Omega_{120}\exp(it\Delta)\exp(i\theta_{T})$. The
$\Omega_{110}=\textbf{u}_{11}\cdot\textbf{E}_{G}/(2\hbar)$ and
$\Omega_{220}=\textbf{u}_{22}\cdot\textbf{E}_{G}/(2\hbar)$ are due to the symmetry-breaking eigenstates of electron floating on liquid helium surface. The
$\Omega_{120}=\textbf{u}_{12}\cdot\textbf{E}_{T}/(2\hbar)$ is the standard Rabi frequency of electric dipole transition, and $\Delta=\omega_T-\omega_{12}$ is the detuning between the THz wave and the resonant frequency $\omega_{12}$ of electronic transition $|1\rangle\rightleftharpoons|2\rangle$.
For the analyzable effects of THz and GHz mixing, we apply another unitary transformation $\hat{U}=\exp[-i\hat{h}(t)/\hbar]$ to the electronic states. Here,
\begin{equation}
\begin{aligned}
\hat{h}(t)
=\hbar[\theta_{11}(t)\hat{\sigma}_{11}+\theta_{22}(t)\hat{\sigma}_{22}]\,,
\end{aligned}
\label{E9}
\end{equation}
\begin{equation}
\begin{aligned}
\theta_{11}(t)=\int_{0}^{t}\Omega_{11}(t){\rm d}t
=\frac{2\Omega_{110}}{\omega_{G}}\sin(\omega_{G}t+\theta_{G})\,,
\end{aligned}
\label{E10}
\end{equation}
and
\begin{equation}
\begin{aligned}
\theta_{22}(t)=\int_{0}^{t}\Omega_{22}(t){\rm d}t
=\frac{2\Omega_{220}}{\omega_{G}}\sin(\omega_{G}t+\theta_{G})\,.
\end{aligned}
\label{E11}
\end{equation}
As a consequence, Hamiltonian (\ref{E8}) in the new interacting picture reads
\begin{equation}
\begin{aligned}
\hat{H}_{I}=\hbar\Omega_{120}
\left[e^{i\phi(t)}\hat{\sigma}_{12}+
e^{-i\phi(t)}\hat{\sigma}_{21}\right]\,,
\end{aligned}
\label{E12}
\end{equation}
with
\begin{equation}
\begin{aligned}
e^{i\phi(t)}=e^{i\theta_T}
e^{i\Delta t}e^{i\xi\sin(\omega_{G}t+\theta_{G})}\,,
\end{aligned}
\label{E13}
\end{equation}
and
\begin{equation}
\begin{aligned}
\xi=\frac{2(\Omega_{110}-\Omega_{220})}{\omega_{G}}\,.
\end{aligned}
\label{E14}
\end{equation}

According to the well-known Jacobi-Anger expansion, we approximately write (\ref{E13}) as
\begin{equation}
\begin{aligned}
e^{i\phi(t)}=&e^{i\theta_T}
e^{i\Delta t}\sum\limits_{n=-\infty}^{\infty}
J_{n}(\xi)e^{in(\omega_{G}t+\theta_{G})}\\
\\
=&J_0(\xi)e^{i\theta_T}e^{i\Delta t}
+J_{-1}(\xi)
e^{i[(\Delta-\omega_G)t+(\theta_T-\theta_G)]}
+J_{1}(\xi)
e^{i[(\Delta+\omega_G)t+(\theta_T+\theta_G)]}\\
\\
&+\cdots\,,
\end{aligned}
\label{E15}
\end{equation}
with $J_{n}(\xi)=(-1)^nJ_{-n}(\xi)$ being the Bessel function of the first kind. Considering the field strength of GHz wave is weak, i.e., $\xi\ll1$, and supposing the frequency condition $\Delta=\omega_G\gg\Omega_{120}$ is satisfied, then the Eq.~(\ref{E15}) reduces to
\begin{equation}
\begin{aligned}
e^{i\phi(t)}\approx&J_{-1}(\xi)
e^{i(\theta_T-\theta_G)}\approx-\frac{\xi}{2}e^{i(\theta_T-\theta_G)}\,,
\end{aligned}
\label{E16}
\end{equation}
with the rotating-wave approximation. As a consequence, Hamiltonian (\ref{E12}) reduces to
\begin{equation}
\begin{aligned}
\hat{H}_{R}=\hbar\Omega_{\rm eff}
\left[e^{i(\theta_T-\theta_G)}\hat{\sigma}_{12}+
e^{-i(\theta_T-\theta_G)}\hat{\sigma}_{21}\right]\,,
\end{aligned}
\label{E17}
\end{equation}
which generates the standard Rabi oscillation at frequency $\Omega_{\rm eff}=\Omega_{120}(\Omega_{220}-\Omega_{110})/\omega_G$.
This Hamiltonian implies that the electrons floating on liquid helium may be applied as a frequency-mixer to generate the low frequency signal with $\Omega_{\rm eff}\ll \omega_G\ll \omega_T$.

\section{The master equation}
\label{sec:3}
We use the standard master equation of classical electromagnetic waves interacting with the two-levels system~\cite{RevModPhys.77.633} to numerically solve the dynamical evolution of surface state electrons on liquid helium. The master equation reads
\begin{equation}
\begin{aligned}
\frac{\rm{d}\hat{\rho}}{\rm{d}t}=-\frac{i}{\hbar}[\hat{H},\hat{\rho}]
+\hat{L}(\hat{\rho})\,,
\end{aligned}
\label{E18}
\end{equation}
with the decoherence operator
\begin{equation}
\begin{aligned}
\hat{L}(\hat{\rho})=&\frac{\Gamma_{21}}{2}
(2\hat{\sigma}_{12}\hat{\rho}\hat{\sigma}_{21}-\hat{\sigma}_{22}\hat{\rho}-\hat{\rho}\hat{\sigma}_{22})\\
&+\frac{\gamma_{1}}{2}(2\hat{\sigma}_{11}\hat{\rho}\hat{\sigma}_{11}-\hat{\sigma}_{11}\hat{\rho}-\hat{\rho}\hat{\sigma}_{11})\\
&+\frac{\gamma_{2}}{2}(2\hat{\sigma}_{22}\hat{\rho}\hat{\sigma}_{22}-\hat{\sigma}_{22}\hat{\rho}-\hat{\rho}\hat{\sigma}_{22})\,.
\end{aligned}
\label{E19}
\end{equation}
Above, $\hat{\rho}=|\psi\rangle\langle\psi|=\sum_{i,j}^2\rho_{ij}|i\rangle\langle j|$, with $\sum_{i=1}^2\rho_{ii}=1$ and $\rho_{ij}=\rho_{ji}^{\ast}$. The $\rho_{ij}$ is the so-called density matrix elements of the two-levels system. The $\Gamma_{21}$ is the spontaneous decay rate of $|2\rangle\rightarrow|1\rangle$. The $\gamma_{1}$ and $\gamma_{2}$ describe the energy-conserved dephasing~\cite{RevModPhys.77.633}.
It is worth to note that the unitary operator $\hat{U}=\exp[-i\hat{h}(t)/\hbar]$ made for the Hamiltonian transformation (\ref{E8}) $\rightarrow$ (\ref{E12}) does not change the formation of master equation. The proofs are detailed  as follows.

Firstly, we rewrite the density operator as
$\hat{\rho}=|\psi\rangle\langle\psi|
=\hat{U}\hat{\rho}_{I}\hat{U}^{\dagger}$
with $\hat{\rho}_{I}=|\psi'\rangle\langle\psi'|$, and rewrite the Hamiltonian (\ref{E8}) as $\hat{H}=\hat{H}_{I0}+\hat{H}_{\rm int}$, with $\hat{H}_{I0}=\hbar\left[\Omega_{11}(t)
\hat{\sigma}_{11}+\Omega_{22}(t)\hat{\sigma}_{22}\right]$ and
$\hat{H}_{\rm int}=\hbar\left[\Omega_{12}(t)\hat{\sigma}_{12}
+\Omega_{21}(t)\hat{\sigma}_{21}\right]$. Our unitary operator
obeys the commutation relation $[\hat{U},\hat{H}_{I0}]=0$, so
\begin{equation}
\begin{aligned}
\frac{d\hat{\rho}}{dt}&=\frac{d\hat{U}}{dt}\hat{\rho}_{I}
\hat{U}^{\dagger}+\hat{U}\frac{d\hat{\rho}_{I}}{dt}\hat{U}^{\dagger}
+\hat{U}\hat{\rho}_{I}\frac{d\hat{U}^{\dagger}}{dt}\\
&=\hat{U}\frac{d\hat{\rho}_{I}}{dt}\hat{U}^{\dagger}
+\frac{-i}{\hbar}[\hat{H}_{I0},\hat{U}\hat{\rho}_{I}\hat{U}^{\dagger}]\,.
\end{aligned}
\label{E20}
\end{equation}
Secondly, according to the original master equation (\ref{E18}), we have
\begin{equation}
\begin{aligned}
\frac{d\hat{\rho}}{dt}&=\frac{-i}{\hbar}[\hat{H},\hat{\rho}]+\hat{L}(\hat{\rho})\\
&=\frac{-i}{\hbar}
[\hat{H}_{\rm int},\hat{U}\hat{\rho}_{I}\hat{U}^{\dagger}]
+\frac{-i}{\hbar}[\hat{H}_{I0},\hat{U}\hat{\rho}_{I}\hat{U}^{\dagger}]
+\hat{L}(\hat{\rho})\\
&=\frac{-i}{\hbar}\hat{U}[\hat{H}_{I},
\hat{\rho}_{I}]\hat{U}^{\dagger}
+\frac{-i}{\hbar}[\hat{H}_{I0},\hat{U}\hat{\rho}_{I}\hat{U}^{\dagger}]
+\hat{L}(\hat{\rho})\,,
\end{aligned}
\label{E21}
\end{equation}
and where $\hat{H}_{I}=\hat{U}^\dagger\hat{H}_{\rm int}\hat{U}$ is nothing but Eq.~(\ref{E12}).
The decoherence operator obeys the following equation
\begin{equation}
\begin{aligned}
\hat{L}(\hat{\rho})&=\hat{U}\hat{L}(\hat{\rho}_{I})\hat{U}^{\dagger}\,,
\end{aligned}
\label{E22}
\end{equation}
because $\hat{U}^{\dagger}\hat{\sigma}_{11}\hat{U}=\hat{\sigma}_{11}$, $\hat{U}^{\dagger}\hat{\sigma}_{22}\hat{U}=\hat{\sigma}_{22}$, and
$\hat{\sigma}_{12}\hat{\rho}\hat{\sigma}_{21}
=\hat{U}\hat{\sigma}_{12}\hat{\rho}_{I}\hat{\sigma}_{21}\hat{U}^\dagger$,
with $\hat{U}^{\dagger}\hat{\sigma}_{12}\hat{U}
=\hat{\sigma}_{12}\exp[i\xi\sin(\omega_{G}t+\theta_{G})]$ and $\hat{U}^{\dagger}\hat{\sigma}_{21}\hat{U}
=\hat{\sigma}_{21}\exp[-i\xi\sin(\omega_{G}t+\theta_{G})]$.

Finally, comparing Eqs. (\ref{E20}) and (\ref{E21}), we find
\begin{equation}
\begin{aligned}
\hat{U}\frac{d\hat{\rho}_{I}}{dt}\hat{U}^{\dagger}
=\frac{-i}{\hbar}\hat{U}[\hat{H}_{I},
\hat{\rho}_{I}]\hat{U}^{\dagger}
+\hat{U}\hat{L}(\hat{\rho}_{I})\hat{U}^{\dagger}\,,
\end{aligned}
\label{E23}
\end{equation}
and get the master equation in interaction picture
\begin{equation}
\begin{aligned}
\frac{d\hat{\rho}_{I}}{dt}
=\frac{-i}{\hbar}[\hat{H}_{I},\hat{\rho}_{I}]
+\hat{L}(\hat{\rho}_{I})\,,
\end{aligned}
\label{E24}
\end{equation}
which has the same form as that of (\ref{E18}). Moreover, using the above approach one can easily find that the usual unitary transformation $\hat{U}_0$ also does not change the form of master equation.
Compared with the original Hamiltonian (\ref{E8}), the Hamiltonian $H_I$ in interacting pictures is more clear to show the frequency-matching condition between electron and microwaves. While, the Hamiltonian $H_I$, as proved above, is also valid to numerically (exactly) solve the master equation.

\section{The results and discussion}
\label{sec:4}
According to master equation (\ref{E24}), we get the following equations for density matrix elements,
\begin{align}
\label{E25}
\frac{{\rm d}\rho_{22}}{{\rm d}t} &
=i\rho_{21}\Omega_{120}e^{i\phi(t)}
-i\rho_{12}\Omega_{120}e^{-i\phi(t)}-\Gamma_{21}\rho_{22}\,,\\
\label{E26}
\frac{{\rm d}\rho_{12}}{{\rm d}t}&=
i(1-2\rho_{22})\Omega_{120}e^{i\phi(t)}
-\frac{1}{2}(\Gamma_{21}+\gamma_{1}+\gamma_{2})\rho_{12}\,.
\end{align}
In the experimental systems~\cite{PhysRevLett.123.086801,Elarabi2021,PhysRevLett.126.106802}, the liquid helium surface (with the surface-state electrons) is set
approximately midway between two plates of a parallel-plate
capacitor, and the induced image charges on one of the capacitor plates is described by $Q_{\rm image}\approx
Q_{e}\langle z\rangle/D$.
The $Q_{e}=eN$ is the charge of $N$ electrons on liquid helium surface,  and $\langle z\rangle$ the average height of these electrons. The $D$ is the distance between the two parallel plates of capacitor.
In terms of density matrix elements, the expectation value $\langle z\rangle$ in Schr\"{o}dinger picture is described by
\begin{equation}
\begin{aligned}
\langle z\rangle
=&\rho_{11}z_{11}+\rho_{22}z_{22}+2 z_{21}{\rm Re}\left[\rho_{12}e^{i\omega_{12}t}
e^{-i\xi\sin(\omega_{G}t+\theta_{G})}\right]\\
=&z_{11}+(z_{22}-z_{11})\rho_{22}\\
&+2 z_{21}{\rm Re}\left[{\rm Re}(\rho_{12})e^{i\omega_{12}t}
e^{-i\xi\sin(\omega_{G}t+\theta_{G})}
+{\rm Im}(\rho_{12})e^{i\omega_{12}t}
e^{-i\xi\sin(\omega_{G}t+\theta_{G})}e^{i\frac{\pi}{2}}\right]\,.
\end{aligned}
\label{E27}
\end{equation}
Numerically, the transition matrix elements are $z_{11}=1.5r_{B}$, $z_{22}=6r_{B}$, and $z_{21}=-0.5587r_{B}$.
Note that, the term $\exp(i\omega_{12}t)$ in Eq.~(\ref{E27}) is due to the standard unitary transformation $\hat{U}_0$ in Sec. \ref{sec:2} applied for the interacting Hamiltonian (\ref{E8}), and $\exp[-i\xi\sin(\omega_{G}t+\theta_{G})]$ is due to the second unitary transformation $\hat{U}$ employed for Hamiltonian (\ref{E12}). Specially, the term $\exp(i\omega_{12}t)=\exp[i(\omega_{T}-\Delta)t]$ rapidly vibrates with the intrinsic frequencies of two-level electrons on liquid helium and generates the THz radiation.

According to our rotating-wave approximated Hamiltonian (\ref{E17}), the density matrix elements $\rho_{22}$ and $\rho_{12}$ vibrate with the Rabi frequency $2\Omega_{\rm eff}\ll\Omega_G\ll\Omega_T$. The rotating-wave terms in Eq.~(\ref{E15}), such as $\exp(i\Delta t)$, vibrate with much higher frequencies than $\Omega_{\rm eff}$, but has small amplitude as shown in Fig.~\ref{fig:1}. The frequency distribution functions in Fig.~\ref{fig:1} are obtained by numerically solving the following Fourier transforms
\begin{equation}
\begin{aligned}
\rho_{22}(T)=\int_{-\infty}^{\infty}\psi_{22}(\omega)e^{-i\omega T}{\rm d}\omega\,,
\end{aligned}
\label{E28}
\end{equation}
\begin{equation}
\begin{aligned}
{\rm Re}(\rho_{12}(T))=\int_{-\infty}^{\infty}\psi_{R12}(\omega)e^{-i\omega T}{\rm d}\omega\,,
\end{aligned}
\label{E29}
\end{equation}
and
\begin{equation}
\begin{aligned}
{\rm Im}(\rho_{12}(T))=\int_{-\infty}^{\infty}\psi_{I12}(\omega)e^{-i\omega T}{\rm d}\omega\,.
\end{aligned}
\label{E30}
\end{equation}
The $\psi_{22}(\omega)$ is the amplitude of a harmonic vibrational component in function $\rho_{22}(T)$, with $T=\Omega_{120}t$ being the dimensionless time. The frequency of this component (harmonic vibration) is $\omega \Omega_{120}$, with $\omega$ being the dimensionless coefficient. The explanations for $\psi_{R12}(\omega)$ and $\psi_{I12}(\omega)$ are similar.

\begin{figure}[tpb]
\centering
\resizebox{1\textwidth}{!}{\includegraphics{FIG1.eps}}
\caption{The numerical solutions of frequency distribution functions $\psi_{22}(\omega)$, $\psi_{R12}(\omega)$, and $\psi_{I12}(\omega)$, based on Eqs.~(\ref{E25}) and (\ref{E26}) of  density matrix elements. The values of the necessary parameters in this figure are $\xi=0.5$, $\Omega_{120}=0.1\omega_{G}$, $\Delta=\omega_G=1$~GHz, $\Gamma_{21}=10$~MHz, and $\gamma_1=\gamma_2=0.5\Gamma_{21}$.
The red lines are obtained from the rotating-wave approximated Hamiltonian (\ref{E17}), and which can be simply explained by the analytical solution $\rho_{ij}$ of  Rabi oscillation.
The blue lines are obtained from the Hamiltonian (\ref{E12}), without the rotating-wave approximation. The blue lines of $\psi_{R12}(\omega)$ and $\psi_{I12}(\omega)$ show that the vibrational frequencies of $\rho_{12}$ are far smaller than THz and thus the off-diagonal term in Eq.~(\ref{E27}) is relate to the THz vibration. The low frequency signal in $\psi_{22}(\omega)$ can be amplified effectively by the low-noise amplifier~\cite{Elarabi2021}, while the high frequency components in $\psi_{22}(\omega)$ could be filtered out by the low-pass filter in experiments.}
\label{fig:1}
\end{figure}

Due to the negligible components of high frequency vibrations in $\rho_{12}$, see Fig.~\ref{fig:1}, the off-diagonal term in Eq.~(\ref{E27}) is still the THz vibration due to the characteristic term $\exp(i\omega_{12} t)$. The THz waves travel in free space, and thus the off-diagonal term in Eq.~(\ref{E23}) is negligible for the  microwave circuit experiments~\cite{PhysRevLett.123.086801,Elarabi2021,PhysRevLett.126.106802}, and then the formula of the vibrational image charge reduces to
\begin{equation}
\begin{aligned}
Q_{\rm image}(t)&=
\frac{Q_{e}(z_{22}-z_{11})}{D}\rho_{22}\,.
\end{aligned}
\label{E31}
\end{equation}
As to was mentioned at the beginning of this paper, the GHz wave can not directly excite the transition $|1\rangle\rightleftharpoons|2\rangle$, the THz wave acts as a trigger to start the system work. This mechanism is showed in Fig.\ref{fig:2}, where the detuning between THz wave and electron is set at $\Delta=\omega_G$. The electric field strength and the frequency of the GHz microwave are set at $E_{G0}=1$~V/cm and $\omega_G=1$~GHz, respectively.

\begin{figure}[htpb]
\centering
\resizebox{0.7\textwidth}{!}{\includegraphics{FIG2.eps}}
\caption{Numerical solutions of $\rho_{22}$ for the case of two-wave mixing.
The black line (closing
to the horizontal axis) describes the driving by GHz wave only, with $\xi=0.5$, $\omega_G=1$~GHz, and $\Omega_{120}=0$, the electron remains at the ground state, and does not generate any signals. The blue line, with $\xi=0$, $\Omega_{120}=0.1\omega_{G}$, and $\Delta=\omega_G=1$~GHz, describes the excitation by THz wave only. The green line, with $\xi=0.5$, $\Omega_{120}=0.1\omega_{G}$, and $\Delta=\omega_G=1$~GHz, describes the cooperative excitation by two waves, the $\rho_{22}$ becomes bigger and has the component of lower frequency vibration, $\Omega_{\rm eff}\ll \omega_G\ll \omega_T$. The values of decay and dephasing are same as that used in Fig.~\ref{fig:1}. Similarly, the red line here is the solution from the rotating-wave approximated Hamiltonian~(\ref{E17}), and the other lines are obtained from the Hamiltonian~(\ref{E12}) without the rotating-wave approximation.}
\label{fig:2}
\end{figure}

In Fig.~\ref{fig:2}, the red line is the solution of the rotating-wave approximated Hamiltonian $\hat{H}_{R}$, i.e., Eq.~(\ref{E17}). This low frequency signal is the damped one because the Hamiltonian $\hat{H}_{R}$ is time-independent. For a time-independent Hamiltonian, the master equation has the steady-state solution, i.e., ${\rm d}\rho_{ij}/{\rm d}t=0$ with $t\rightarrow\infty$~\cite{ZHANG201412}. The same phenomenon can be found in the blue line in Fig.~\ref{fig:2}, which describes the excitation by THz wave only. In this case, the Hamiltonian (\ref{E12}) reduces to $\hat{H}_{\rm blue}=\hbar\Omega_{120}
[\exp(i\Delta t+i\theta_T)\hat{\sigma}_{12}+
\exp(-i\Delta t-i\theta_T)\hat{\sigma}_{21}]$ with $\xi=0$. This Hamiltonian can be also written in a time-independent form, i.e.,  $\hat{H}'_{\rm blue}=\hbar\Omega_{120}
[\exp(i\theta_T)\hat{\sigma}_{12}+
\exp(-i\theta_T)\hat{\sigma}_{21}]-\hbar\Delta |1\rangle\langle 1|$, by simply employing the unitary transformation of $\exp(i\Delta t|1\rangle\langle 1|)$.
So, the blue line is also the damped one. Readers may notice that the blue line vibrates much faster than the red line. This can be simply explained by the analytic solution of Schr\"{o}dinger equation of $\hat{H}'_{\rm blue}$. The solution says the electron excitation probability $P_e=[\Omega_{120}^2/(2\Delta^2_{\Omega})]\times[1-\cos(2\Delta_{\Omega}t)]$
with the frequency $\Delta_{\Omega}=\sqrt{(\Delta^2/4)+\Omega_{120}^2}$\,.  In the large detuning regime, i.e., $\Delta\gg\Omega_{120}$, the vibrational frequency $\Delta_{\Omega}$ is obviously larger than the standard Rabi frequency $\Omega_{120}$, but the amplitude is small (the rotating-wave effect).

The green line is the numerical solution of Hamiltonian (\ref{E12}) without the rotating-wave approximation. Then, the vibration has not only the low frequency component but also the high frequency component. In this case, the persistent oscillation exists, because Hamiltonian (\ref{E12}) is associated with the rotating-wave terms and can not be written in the time-independent form by any unitary transformation, as well as the original Hamiltonian (\ref{E8}). However, the frequency of persistent oscillation is on the order of the inputting GHz wave and is not the usable signal that a frequency-mixer should produce. Thus we suggest using the low frequency signal predicted by the red line to detect the THz inputting, although the signal is the damped one. This means that a pulsed source of GHz microwave is needed. Furthermore, we note that the above detuning is set at $\Delta=\omega_G$. This condition may not exactly satisfy in the practical experiments, and therefore the effective Hamiltonian (\ref{E17}) becomes
\begin{equation}
\begin{aligned}
\hat{H}_{R}=\hbar\Omega_{\rm eff}
\left[e^{i(\theta_T-\theta_G)}e^{i\delta t}\hat{\sigma}_{12}+
e^{-i\delta t}e^{-i(\theta_T-\theta_G)}\hat{\sigma}_{21}\right]\,,
\end{aligned}
\label{E32}
\end{equation}
with $\delta=\Delta-\omega_G$. The solution of this Hamiltonian is similar to that of $\hat{H}_{\rm blue}$, but can still generate the low-frequency signal with $\sqrt{(\delta^2/4)+\Omega_{\rm eff}^2}$ and $\delta\sim\Omega_{\rm eff}$.

\begin{figure}[tpb]
\centering
\resizebox{0.7\textwidth}{!}{\includegraphics{FIG3.eps}}
\caption{Numerical solutions of $\rho_{22}$ for the three-wave mixing (\ref{E34}), with $\Delta=1$~GHz, $\omega_G=0.9$~GHz, $\omega_{G2}=1.1$~GHz, $\xi=\xi_2=0.5$,  $\theta_G=\theta_{G2}$, $\Omega_{120}=0.1\Delta$, $\gamma_1=\gamma_2=0.5\Gamma_{21}$, and $\Gamma_{21}=10$~MHz. The blue and red lines are obtained from Hamiltonian (\ref{E34}) and its rotating-wave approximated version (\ref{E37}), respectively.}
\label{fig:3}
\end{figure}

Finally, we describe a possibility to generate the persistent signal, i.e., monitor THz incoming online. Based on the above discussion, we apply another GHz wave $\textbf{E}_{G2}\cos(\omega_{G2}t+\theta_{G2})$ to the electrons. Such that the inputting field (\ref{E7}) extends to
\begin{equation}
\begin{aligned}
\textbf{E}'(t)&=\textbf{E}_{T}\cos(\omega_{T}t+\theta_{T})
+\textbf{E}_{G}\cos(\omega_{G}t+\theta_{G})
+\textbf{E}_{G2}\cos(\omega_{G2}t+\theta_{G2})\,,
\end{aligned}
\label{E33}
\end{equation}
and consequently, the Eqs.~(\ref{E12}) and (\ref{E13}) become, respectively,
\begin{equation}
\begin{aligned}
\hat{H}'_{I}=\hbar\Omega_{120}
\left[e^{i\phi'(t)}\hat{\sigma}_{12}+
e^{-i\phi'(t)}\hat{\sigma}_{21}\right]\,,
\end{aligned}
\label{E34}
\end{equation}
and
\begin{equation}
\begin{aligned}
e^{i\phi'(t)}=e^{i\theta_T}
e^{i\Delta t}e^{i\xi\sin(\omega_{G}t+\theta_{G})}
e^{i\xi_2\sin(\omega_{G2}t+\theta_{G2})}\,.
\end{aligned}
\label{E35}
\end{equation}
The derivation for $\xi_2=2[\Omega_{110}(\textbf{E}_{G2})-\Omega_{220}(\textbf{E}_{G2})]/\omega_{G2}$ is similar to that of parameter $\xi$ in Eq.~(\ref{E14}). Considering also $\xi_2\ll1$, we have
\begin{equation}
\begin{aligned}
e^{i\phi'(t)}\approx-\frac{\xi}{2}
e^{i(\Delta-\omega_G)t}e^{i\theta_{TG}}
-\frac{\xi_2}{2}e^{i(\Delta-\omega_{G2})t}e^{i\theta_{TG2}}\,,
\end{aligned}
\label{E36}
\end{equation}
by employing the rotating-wave approximation again, and where $\theta_{TG}=\theta_{T}-\theta_{G}$ and $\theta_{TG2}=\theta_{T}-\theta_{G2}$.
Furthermore, considering the typical case that $\xi\approx\xi_2$ and $\delta=\Delta-\omega_G=\omega_{G2}-\Delta$, such that the effective Hamiltonian (\ref{E17}) becomes,
\begin{equation}
\begin{aligned}
\hat{H}'_{R}\approx\hbar\Omega_{\rm eff}
\left\{e^{i\theta_{TG}}\left[e^{i\delta t}+e^{-i\delta t}e^{i(\theta_{G}-\theta_{G2})}\right]\hat{\sigma}_{12}+
e^{-i\theta_{TG}}\left[e^{-i\delta t}+e^{i\delta t}e^{-i(\theta_{G}-\theta_{G2})}\right]\hat{\sigma}_{21}\right\}\,.
\end{aligned}
\label{E37}
\end{equation}
Due to the small detuning $\delta$, this Hamiltonian can be never written in a time-independent form by any unitary transformations, and thereby the master equation has no steady-state solution. This causes the persistently oscillating charge, as shown in Fig.~\ref{fig:3}, which serves as a source for the low-frequency microwave outputting and could be applied to realize the real-time detection of THz incoming.

\section{Conclusion}
In summary, we have studied the frequency-mixed effects of THz and GHz waves in the cryogenic system of electrons floating on liquid helium. Different from the natural atoms, the transition frequency between the lowest two levels of electrons floating on liquid is in the THz regime. Moreover,  the surface-state of electrons is symmetry breaking due to the barrier at the liquid surface. Therefore, both of THz and GHz waves can effectively drive the electrons via the electric dipole interaction. Specifically, the THz wave near-resonantly excites the transition between the lowest two levels of surface-bound electrons, the GHz wave does
not excite the transition but generates the GHz-varying Stark effect. Using the unitary transformation approach and the rotating-wave approximation, we found an effective Hamiltonian (\ref{E17}) of two-wave mixing with the detuning $\Delta=\omega_{12}-\omega_T=\omega_G$, which could generate the significant ac current of frequency much lower than GHz, as shown in Fig.~\ref{fig:2}. The Hamiltonian is time-independent, so the generated low-frequency signal is the damped one due to the decay of surface-state electrons. To generate the persistent low-frequency signal, the time-dependent Hamiltonian is required, for example, the effective Hamiltonian (\ref{E37}) proposed for the case of three-wave mixing.

\textbf{Acknowledgements}:
This work was supported by the National Natural Science Foundation of China, Grants No. 12047576, and No. 11974290.

\textbf{Data Availability Statement}:
The data generated or analyzed during this study are included in this published article.


\section{Introduction}\label{introduction}

Choosing the right loss function is crucial for getting the expected behavior from deep learning based
models trained for any task. While the
token-level cross-entropy (CE) loss continues to excel in training natural
language generation (NLG) models for various tasks, including dialog-response
generation \citep{roller2020recipes, zhang2019dialogpt}, it is well accepted among
NLP researchers that CE is not the most appropriate loss function to use for
training dialog generation models. Thus, finding the right loss function for training
dialog generation is still an open problem and an active area of research \citep{shen2017conditional,zhao2017learning,saleh2020hierarchical,li2021improving}.

Both unconditional or conditional NLG models,
including dialog generation models, are almost universally trained using
the token-level cross-entropy (CE) loss. The CE loss for language generation models is computed by
comparing the predicted token probabilities to the ground truth target sequence
from the dataset. This computation process of CE is unconditional or
context-free as it does not depend on the available prompt/context in the
case of conditional NLG tasks like dialog generation. Without the context it is not possible to grade the generated response for qualities that human users truly care about, like relevance, coherence, etc. Another major concern with training dialog generation models using the CE loss is the assumption of a fixed target. While training any NLG model using the CE loss,
  the probability of the ground truth response is maximized. Here, we
  make an implicit assumption that the ground truth is the only response
  possible for the given context. But this property does not hold for most dialogs where each context may have a large number of possible responses \citep{dou2021multitalk}\footnote{These problems are
specific to dialog systems only and may or may not apply to other NLG
tasks.}.

Previous attempts in training Seq2Seq dialog generation models using the
CE loss have led to various complications. Mode collapse is one of the
most common issues when training a Seq2Seq model with the CE loss,
mainly at smaller scales \citep{li2016deep,li2021improving}. Here, the model will just assign a
high probability to one or more generic and bland responses e.g. ``I
don't know", ``I have a problem", ``Yes" etc., irrespective of the context.
Previous research
works have also explored various augmentations to the model architecture \citep{serban2016building,zhao2019rethinking}
and/or the loss function \citep{serban2017hierarchical,shen2017conditional,zhao2017learning,li2021improving} to resolve the common problems with CE, as
mentioned above.

In order to facilitate a training process that aims to maximize some user-expected quality in dialog generation models, we propose to directly optimize an estimate of human perceived quality of a context-response pair. \citet{Sinha2020LearningAU,yeh2021comprehensive} have
shown that the output score of retrieval models correlates strongly with human
perception of dialog response quality (given that it was trained on a
dataset of domain similar to the target domain of application).
Further,~\citet{santra2021representation} showed that representations learned by a
response retrieval (using binary cross-entropy or a contrastive loss) capture
important dialog understanding features even better than large-scale dialog generation models trained using CE.

These observations triggered us to propose a novel \textbf{Contextual Response Retrievability} loss function or \textbf{CORAL} that circumvents these
issues with CE loss and relies on a
response retrieval model.
Further, we leverage CORAL to design a learning framework for dialog generation models that treats score calculated by retrieval models as a reward function, \textbf{Response Retrievability Reward} ($R_3$). We then use reinforcement learning (RL) to train dialog models that maximizes above reward between context and generations from the optimized network.
To test the proposed loss function, we train transformer-based Seq2Seq models \citep{vaswani2017attention}
using CORAL for open-domain dialog generation using DailyDialog dataset \citep{li2017dailydialog} and for domain-specific dialog generation using DSTC7-Ubuntu dataset \citep{yoshino2019dialog}. We compared the performance against state-of-the-art pretrained CE-based Seq2Seq models and various other baselines using both automatic metrics and through manual analysis. To summarize, our contributions are as follows.
\begin{enumerate}
  \item We propose CORAL loss function for training dialog generation models by directly optimizing for an estimate of human preference of a $\langle$context,response$\rangle$ pair. To the best of our knowledge, we are the first to propose a loss function for dialog generation model that relies upon the context also.
    \item Further, we propose a recipe to train improved seq2seq dialog models which uses the CORAL loss in a reinforcement learning setup.
  \item We experimentally prove the effectiveness of CORAL against other dialog generation models that use CE and its variants.
\end{enumerate}

\section{Literature Review}\label{literature-review}

The use of the Seq2Seq model for training chitchat (now known as
open-domain) dialog generation model was first proposed by \citet{ritter2011data}. It was realized very soon that, unlike other NLG problems such
as NMT, training dialog generation models using the CE loss leads to
various complications. Researchers found that at, smaller scales, it was
almost impossible to train a dialog generation model \citep{serban2016building,serban2017hierarchical,zhao2017learning,li2016deep}. \citet{serban2016building} argued that standard RNNs fail to encode the context properly and
proposed a hierarchical encoder-based architecture, HRED, that
captures utterance-level and context-level information using separate
RNNs. To capture the notion of multiple possible responses per context,
\citet{serban2017hierarchical} further augmented their HRED model with a VAE to
propose VHRED. Contemporary work also proposed a VAE based grounded
response generation model kg-CVAE \citep{zhao2017learning}.

Several prior works have also tried to formulate reward functions for training dialog generation models using Reinforcement Learning. To generate coherent and engaging responses, \citet{li2016deep} proposed to maximize expected rewards from future turns estimated using a self-play mechanism in a simulated environment.
\citet{sankar2019deep} proposes a space of interpretable, discrete and composed attributes for dialogs as the action space for an RL based solution to the open domain dialog generation problem.
\citet{saleh2020hierarchical} uses a hierarchical reinforcement learning framework to allow the model to learn a proper credit assignment at the utterance level instead of the more common word level assignment. They also used self-play for training the model.
\citet{zhao2019rethinking} propose a learnable latent action space for applying RL algorithms to train conversational agents. This relaxes the researcher from making the choice for action spaces when training RL-based dialog systems. Usual choices are dialog acts or output vocabulary.

Although at smaller scales CE-based training leads to degenerate dialog
generation models, at (extremely) large scales of training data and
model-parameter-size, we do get models that can generate diverse,
coherent and interesting responses. Various large-scale dialog models
have been built by researchers for a wide range of model sizes, from
100s of millions to 10s of billions of parameters. DialoGPT \citep{zhang2019dialogpt} is
based on the GPT-2 \citep{radford2019language} language model (LM) further finetuned on a large
conversational corpus crawled from Reddit. Blenderbot \citep{roller2020recipes}
focuses on generating high-quality responses and is based on a
Transformer-based Seq2Seq architecture \citep{vaswani2017attention}. To improve the quality of
responses, Blenderbot finetuned their model on the Blended-Skill-Talk
dataset (combination of multiple datasets, \citealp{smith2020together}) after pretraining on
Reddit. DialogRPT \citep{gao2020dialogrpt} proposes a sample-and-rank method for
candidates generated from DialoGPT. The ranker was trained using a
dataset of upvotes/downvotes and the number of replies on Reddit
comments. Another family of recently proposed dialog generation models \citep{wu2019response,komeili2021internet} work
based on the idea of retrieve-and-refine for efficient training large language models.

In this paper, we focus on testing the proposed CORAL loss mainly at
smaller scales because 1) the changes are more apparent for smaller scale models, and 2) it becomes more difficult to judge the efficiency and robustness of
  the approach at larger scales where all loss functions seem to work
  upto the same extent. Although we train small-scale models only, our small models outperform several large-sized models across multiple metrics.

\section{Methods}\label{sec:methods}

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{images/coral-compare-losses.drawio2.pdf}

    \caption{Schematics of Cross-Entropy (CE) and CORAL Losses at train time. The main idea of CORAL loss is to optimize a measure of the compatibility between the context and a candidate response. For our implementation, we have used a response retrieval model for measuring this compatibility. Another main difference between CE and CORAL is that, CE is more strict and trains the model only based on true response targets. CORAL also utilizes randomly sampled response targets and increases/decreases its probability of decoding based on $R_3$. The encoder part of the Seq2Seq architecture has been hidden here to focus on the relevant part only.}
    \label{fig:compare-losses}
\end{figure}

The overall approach presented in this section comprises a three-step design process. First, we train a reward function ($R_3$) using a response retrieval model architecture. Next, we apply reinforcement learning for training a Seq2Seq context-to-response generation model that maximizes this response retrievability reward. Applying the REINFORCE~\citep{williams1992simple} algorithm leads to the final differentiable objective function (CORAL Loss). Finally, we design Algorithm \ref{alg:coral} to train a Seq2Seq dialog model using minibatch gradient descent with the CORAL loss.

\subsection{The CORAL Loss Function}\label{a.-the-coral-loss-function}

The main idea of CORAL or Contextual Response Retrievability Loss is to
directly optimize some notion of compatibility between the context and
the response. Such a measure is learned by retrieval models like \citet{chen2019sequential,Henderson2020ConveRTEA}. \citet{santra2021representation} recently showed that such models
actually optimize an estimate of the mutual information between context
and response. Their experiments also prove that these representations
are more feature-rich than large-scale pretrained generative dialog
models (based on CE loss). That is why we consider contextual response retrievability, defined below,
as a suitable target to be optimized for training dialog
generation models.

\paragraph{Contextual Response Retrievability Reward ($R_3$)}\label{sec:response-retrievability-reward} We define the output score of an already
trained response retrieval model for a context-response pair as the Contextual Response Retrievability Reward. This score is usually normalized between $0$ and $1$ and indicates the probability that the response is a valid (coherent and on-topic) continuation to the context.
In general, retrieval models compute this score based on both the input context and the output response from a human annotator or a generative model.
As the reward function is
not differentiable, we first formulate response generation as an RL task. Then, we use the REINFORCE algorithm to
obtain a differentiable objective function as will be described next.

\paragraph{Formulation of the CORAL
Loss}\label{formulation-of-the-coral-loss}

For formulating the final expression for the CORAL loss function, we
first pose the response generation problem as a reinforcement
learning task. Each instance of the context-to-response generation task
is considered as an episode in the RL formulation. The episode consists
of several actions taken by the agent, in our case the decoder. Each action corresponds to
generation of an output token. When the agent generates an EOS
(end-of-sequence) token or has produced a max number of allowed tokens
(T), the episode ends and the environment (the response retrieval model)
generates the $R_3$ reward, for the $\langle$context $c$, generated response $r\rangle$ pair.

The updates to the Seq2Seq model weights are then determined by the Episodic REINFORCE
algorithm\footnote{See Section 5 in \cite{williams1992simple}} as:
\begin{align}
  \Delta W_{\text{CORAL}} = \eta R_3(\langle c,r\rangle) \sum_{t=1}^{T} \frac{\partial\log P(r|c)}{\partial W_{\text{CORAL}}}
  \label{eqn:coral-update}
\end{align}

Equivalently, the loss function to be minimized (for an auto-regressive decoder) can be written as follows.
\begin{align}
    L_{\text{CORAL}}=-\eta R_3(\langle c,r\rangle) \sum_{t=1}^{T} \log P(r_t|r_{<t},c)
    \label{eqn:loss-coral}
\end{align}

Fig.~\ref{fig:compare-losses} gives a sense of how the standard cross-entropy loss and the proposed CORAL loss differs from each other. Decoder using CORAL loss also incorporates quality of response predicted by a response retrieval model as a factor in computing the loss.

\subsection{Training Seq2Seq Models using CORAL Loss}

Next, we explain the algorithm used for training a Seq2Seq model for dialog generation using the CORAL loss and an existing dialog dataset $D$. Fig.~\ref{fig:s2s-coral} shows the architecture of our Seq2Seq model which generates a response given the context using a Transformer encoder-decoder model trained under CORAL loss.

\begin{algorithm}
\caption{Training Algorithm for Seq2Seq Models using CORAL Loss}\label{alg:coral}
\begin{algorithmic}
\State $D \gets \{\langle c,r^+\rangle\}$ \Comment{Positive pairs from training dataset}
\State $R \gets $ Pool of all utterances in dataset (train split)
\For{$\langle c,r^+\rangle \in D$} \Comment{Actual implementation uses batch gradient descent}
    \If{$rand() > p_+$}
        \State $r\leftarrow$ Sample $r^- \sim R$ \Comment{Uniform Negative Sampling}
    \Else
    \State $r\leftarrow$ $r^+$ \Comment{Use Positive Response}
    \EndIf
    \State $R_3(\langle c,r\rangle)=f_R(\langle c,r\rangle) - m$ \Comment{$f_R$: Response Retrieval Model Score Function}
    \State Compute $P_{S2S}(r_t|r_{<t},c) \forall t \in [1,T]$ \Comment{$P_{S2S}$: Decoder output token distribution}
    \State $L_{\text{CORAL}}=-R_3(\langle c,r\rangle)\sum_{t=1}^{T}P_{S2S}(r_t|r_{<t},c)$
    \State Update parameters of $P_{S2S}$ using gradient descent on $L_{\text{CORAL}}$
\EndFor
\end{algorithmic}
\end{algorithm}

\paragraph{Training Stability}
In the true spirit of RL, one would use responses sampled from the
Seq2Seq model for each context and train the model based on the reward
for these $\langle$context, decoded response$\rangle$ pairs.
But, since a high positive reward is obtained only if the generated sequence is an appropriate enough response and only after all the tokens are generated (i.e. the full episode is over), it becomes very difficult to train the model just based on sampling.
Also, because of the combinatorial complexity of the space
of all possible token sequences, the reward will initially be very close to zero for almost all sampled sequences.

Hence, given a dataset $D$ of positive context response pairs, $\langle c, r^+\rangle$, with probability $p_+$, we use the response $r^+$ itself. With the remaining probability ($1-p_+$), we sample utterances randomly from the whole training set as
response candidates $r^-$ for each context $c$. A response retrieval model $R$ is then used to compute the retrieval score $f_R(\langle c,r\rangle)$. To make sure that the probability of good response candidates is increased, we
offset the retrieval score by a fixed margin to get $R_3(\langle c,r\rangle)$. The Seq2Seq decoder computes $P_{S2S}(r_t|r_{<t},c)$ for all tokens $t$ in the response.
The two factors $R_3$ and $P_{S2S}$ are used to compute the CORAL loss. Then based on the CORAL loss, the
model will increase or decrease the probability of generating these candidate responses while decoding. We illustrate the algorithm to train our Seq2Seq dialog model using minibatch gradient descent with CORAL loss in Algorithm \ref{alg:coral}.

\begin{figure}
    \centering
    \includegraphics[width=0.8\textwidth]{images/coral-s2s-coral.drawio.pdf}
    \caption{This figure shows the model architecture used for training the Seq2Seq dialog model using the CORAL loss. The context input to the Transformer-encoder is denoted by a token sequence $[c_1, c_2, ..., c_L]$. The response input sequence $r = [ r_1,  r_2, ..., r_T]$ denotes the positive or a randomly sampled negative response candidate.}
    \label{fig:s2s-coral}
\end{figure}

\subsection{Hyperparameters of CORAL}\label{hyperparameters-of-coral}
One needs to tune the following hyperparameters when using CORAL for training

\begin{itemize}
    \item  \textbf{Probability of positive samples ($p_+$)} During training each context in a batch is paired up with a response candidate from the full utterance pool (from training set). With probability $p_+$, we select the true response, and with probability $(1-p_+)$, we randomly select a utterance as response.
    \item  \textbf{Margin ($m$)} To make sure that the probability of good response candidates is increased and that of negative responses is
    decreased, we offset the retrieval model score by  a margin\footnote{This is more commonly known as \textit{baseline} in RL literature.} $m$. This margin denotes the minimum score value that we expect for a good response. $R_3 = score-m$. This ensures that the $R_3$ value will be negative for candidates unsuitable for the input context. E.g., if score is based on a sigmoid activation at the output of the model, margin $m$ will be some positive number between 0 and 1.
    \item  \textbf{Retrieval model} There is also the choice of the retrieval model that is used for measuring $R_3$. Here we have chosen ESIM \citep{chen2019sequential}, BERT \citep{devlin2018bert} and DMI \citep{santra2021representation} as the base models for training the response retrieval (i.e., $R_3$ reward) models.
\end{itemize}

\subsection{Similarities and Differences between CORAL and CE Loss Functions}

In this section, we explore the similarities and differences between the proposed CORAL loss and the CE loss function. Although CORAL is derived from quite a different viewpoint, under certain hyperparameter settings CORAL approximates a weighted version of the CE loss.

\begin{enumerate}
    \item If we only consider positive samples as candidate responses and set the score range ($score\in[0,1]$) and margin m ($m=0$) such that $R_3$ is always greater than zero, CORAL is equivalent to a weighted version of CE.
    \item Cross-entropy loss has always relied \textit{strictly} on the positive responses in the dataset. CORAL utilizes both positive and negative response candidates.
    \item
      Training of a dialog generation model using CE may over-weigh generic responses more than more informative ones as there is no mechanism for automatically assigning weight to different $\langle$context,response$\rangle$ pairs.
      CORAL has provision for assigning different weight for different $\langle$context, candidate response$\rangle$ pairs.
    \item
      CORAL uses randomly sampled response candidates for training which allows us to utilize more samples of $\langle$context,response$\rangle$ pairs during training.
      This provides a richer training signal from the same dataset.
    \item
    CE loss decomposes to a token level comparison between the predicted and the target token. Its main goal is to increase the probability of the tokens in ground truth response strictly in the given form and order.
    CORAL loss works quite differently as it treats responses as whole units. It will either increase or decrease probability of responses as a whole, based
    on their semantics and compatibility to the context.
\end{enumerate}

\section{Experimental Setup}\label{sec:experimental-setup}

\subsection{Model Setup}\label{sec:model-architecture}

We use a standard transformer-based Seq2Seq architecture \citep{vaswani2017attention} for training a
dialog generation model using the CORAL loss as shown in Fig.~\ref{fig:s2s-coral}. We use PyTorch's inbuilt
Transformer module to implement the model and did not change anything
else in the architecture. For tokenization and vocabulary, we used the
word-piece tokenizer from the BERT \cite{devlin2018bert} model\footnote{We used the implementation by huggingface \citep{wolf2019huggingface} library
(bert-base-uncased).}. It has a vocabulary comprising $30,~522$ subword tokens. We train our models using early-stopping, upto a maximum 50 epochs, based on validation-$R_3$ (average $R_3$ score of generated responses on the validation set). We train the model using Adam optimizer with a peak learning rate of $1e-4$ that is warmed up (first 1000 steps) and decayed linearly.

For the retrieval model, we did our main experiments with the LSTM-based ESIM model
proposed by \citep{chen2016enhanced} and transformer-based BERT \cite{devlin2018bert} and DMI \cite{santra2021representation} models. The ESIM model has two LSTM encoding layers (Encoding and Composition Layers) for individually encoding the context and a candidate response, interleaved by a cross-attention layer. The sigmoid output from the ESIM model is used as
the score for a context-response pair. Since both BERT and DMI are pretrained models, we added an MLP, with two hidden layers
and sigmoid activation at the output, on top of the [CLS] token embedding and finetuned them on DailyDialog or DSTC7-Ubuntu for the response retrieval tasks.
We obtained the reward ($R_3$) subtracting the margin ($m$) (also between 0 and 1) from the output score (sigmoid output).

\begin{itemize}
\item  \textbf{Configuration of ESIM}  For faster learning, we also initialized the ESIM model's token embeddings with Blenderbot-3B embeddings, which has a size of 2560. The encoding and composition layers both used 1 layer LSTMs.

\item \textbf{Configuration of BERT and DMI} For BERT and DMI based retrieval models, we used the ``bert-base-uncased'' and the ``DMI\_Medium'' checkpoints, respectively.

\item \textbf{Configuration of S2S (CORAL)} The Transformer-Seq2Seq model had 4 self-attention layers for both the encoder and the decoder. We used 8 self-attention heads and 1024 as the size of hidden representations.

\item \textbf{Training Setup}
We used a single NVIDIA V100-16GB GPU-based system for training all our
models. We used early stopping based on a validation set for both R3 and S2S
training.
\end{itemize}

\subsection{Data}\label{sec:data-and-code}

We used the \textbf{DailyDialog} (DD) \citep{li2017dailydialog} and \textbf{DSTC7-Ubuntu}\footnote{\url{https://github.com/IBM/dstc-noesis}} \citep{yoshino2019dialog} datasets for all our
experiments. DD is an open-domain dialog dataset in English. For
training the retrieval model, we used randomly sampled utterances as
negative samples. DSTC7-Ubuntu is a domain-specific dataset based on
conversations from the Ubuntu IRC channel. DSTC7-Ubuntu contains majorly English conversations, with a small percentage instances from other languages.
This is available directly as
a dataset for training retrieval models with single positive and
multiple negative responses per context. A summary of the datasets'
stats is provided in Table~\ref{tab:data-stats}. We will make the code and trained model checkpoints publicly available upon publication of the paper.

\begin{table}[!htbp]
\centering
\caption{This table reports some basic stats about the two datasets. These represent the number of context-response pair that was extracted from the various splits of the datasets.}
\label{tab:data-stats}
\begin{tabular}{lrrr}
\toprule
Dataset & Train & Valid & Test \\ \midrule
DailyDialog & 76052 & 7069 & 5740 \\
DSTC7-Ubuntu & 470860 & 23478 & 3247 \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Baselines}\label{baselines}

\begin{itemize}

  \item
  \textbf{Mirror} This is a recently proposed \citep{li2021improving} Seq2Seq model trained with a
  backward-reasoning loss function. For making the model generate more informative responses, authors of Mirror extended the CVAE model for dialog generation \citep{shen2017conditional,zhao2017learning} to optimize for generating final and pre-final utterances in a bidirectional fashion. This
  is also currently state-of-the-art loss function for training small-scale dialog generation models and has been shown to outperform \citep{shen2017conditional,zhao2017learning,saleh2020hierarchical}.

  \item
  \textbf{AdaLabel}
  In the AdaLabel model \cite[]{wang2021diversifying}, the authors applied an adaptive label smoothing to prevent the model from being overconfident over a single choice. The main idea of this paper is to use a soft-target distribution depending on the context, instead of usual one-hot distribution.

\item
  \textbf{Blenderbot} This large-scale model is based on the Transformer-based Seq2Seq architecture-based. Blenderbot \citep{roller2020recipes} was first pretrained on a large dialog corpus
  based on Reddit and then further finetuned on Blended Skill Talk dataset \citep{smith2020together}. This final finetuning helps to improve the quality of responses by reducing unwanted response generation.
\item
  \textbf{DialoGPT} DialoGPT \citep{zhang2019dialogpt} is a GPT-2 \citep{radford2019language} based language model further finetuned on dialogs from Reddit.
\item
    \textbf{DialogRPT} DialogRPT \citep{gao2020dialogrpt} is a response ranking model that was trained based on a dataset of upvote/downvote and number of replies on Reddit comments. For generation purposes it reranks sampled responses from DialoGPT and return the one with highest rank predicted by DialogRPT.
\end{itemize}

\subsection{Evaluation Metrics}\label{evaluation-metrics}

We use a standard set of referenced evaluation metrics and a recently proposed reference-free metric for automatic evaluation. These are described below.

BLEU \citep{papineni2002bleu} and METEOR \citep{banerjee2005meteor} measure lexical match upto $n=4$-grams between the predicted and ground truth response. Since a valid response generated by the model can be different from the ground truth, these two metrics cannot always capture the validity of a generation.

MAUDE \citep{Sinha2020LearningAU} is a recently proposed metric that can capture the suitability between a context and any response without a ground truth reference. MAUDE is also a metric based on a response retrieval model and thus our model effectively optimizes MAUDE through the CORAL loss function. \cite{yeh2021comprehensive} observed that MAUDE performs particularly well when the model has been trained on a dataset from a similar domain to its target application domain. Hence, we report results for two MAUDE variants based on ESIM \citep{chen2016enhanced,chen2019sequential} and BERT \citep{devlin2018bert} finetuned on the target dataset (DSTC7 or DailyDialog).

The Distinct-n \citep{liu2016not} metric measures the diversity of n-grams in the overall set of generated responses. This number should not be too low, and an ideal target for this is the Distinct-n of the ground truth responses.

\section{Results and Discussions}\label{sec:results}

\subsection{Automatic Evaluation}
In Table~\ref{tab:results-dd-new}, we present the results of automatic evaluation
metrics for response generation tasks on DD
dataset. For this experiment, we first trained the model on the DD and DSTC7 datasets and saved
the best checkpoint based on the average $R_3$ reward on the validation set. For our proposed CORAL models, we used a margin is 0.2 for DSTC7 and 0.45 for DailyDialog datasets respectively. Also, we set $p^+$ as 1 for these experiments. Fig.~\ref{fig:hparams} show sensitivity analysis for hyper-parameters $p_+$ and $m$ for both the datasets.

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DailyDialog-ESIM-hparams.pdf}
         \caption{$\text{CORAL}_\text{ESIM}$ (DailyDialog)}
         \label{fig:dd1}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DailyDialog-BERT-hparams.pdf}
         \caption{$\text{CORAL}_\text{BERT}$ (DailyDialog)}
         \label{fig:dd2}
     \end{subfigure}

     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DailyDialog-DMI-hparams.pdf}
         \caption{$\text{CORAL}_\text{DMI}$ (DailyDialog)}
         \label{fig:dd3}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DSTC7-Ubuntu-ESIM-hparams.pdf}
         \caption{$\text{CORAL}_\text{ESIM}$ (DSTC7-Ubuntu)}
         \label{fig:dstc7-1}
     \end{subfigure}

     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DSTC7-Ubuntu-BERT-hparams.pdf}
         \caption{$\text{CORAL}_\text{BERT}$ (DSTC7-Ubuntu)}
         \label{fig:dstc7-2}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
         \includegraphics[width=\textwidth]{plots/DSTC7-Ubuntu-DMI-hparams.pdf}
         \caption{$\text{CORAL}_\text{DMI}$ (DSTC7-Ubuntu)}
         \label{fig:dstc7-3}
     \end{subfigure}
    \caption{Hyperparameter Sensitivity Analysis: These plots showcase the effect of $p^+$ and margin on the final validation-$R_3$ score obtained by the corresponding CORAL model. Each lineplot corresponds to a single $p^+$ value as indicated by the legend.}
    \label{fig:hparams}
\end{figure}

The \textbf{diversity} of the generated responses, in terms of number of unique unigrams and bigrams, is indicated by the dist-1 and dist-2 metrics. All baselines, except AdaLabel, have very low diversity for the generated responses. The RNN-based  Mirror model mostly generates the same bland response (e.g., \textit{``I don't know how to do that''}, \textit{``I don't know what to suggest''}) for most contexts leading to a low diversity score. Although DialogRPT and Blenderbot-3B produce relevant responses (as indicated by a high MAUDE score), the response produced by these models sometimes does not change after the initial few turns, and repeatedly generates the same utterance for all remaining turns which leads to a poor Distinct-n score. The diversity of most baselines (except AdaLabel) is quite low compared to that of CORAL models.
From Table \ref{tab:results-dd-new}, it is quite clear that only the proposed CORAL-based models are able to produce responses which are both diverse (high dist-n) and coherent to the context (high MAUDE), while most baselines have failed to maintain this consistently.

Next, we also report the \textbf{BLEU} and \textbf{METEOR} scores which are two standard metrics used for evaluating NLG models. Though the reader should note that these metrics do not correlate strongly with human judgments on the quality of generated responses, as shown by several prior works \citep{liu2016not,Sinha2020LearningAU,yeh2021comprehensive}. In terms of these word overlap-based metrics, we outperform most of the baselines, except AdaLabel which beats $\text{CORAL}_\text{ESIM}$ and $\text{CORAL}_\text{DMI}$ on DailyDialog.

In terms of \textbf{average length} of the generated responses, the pretrained models outperform other baselines and also the CORAL models. Though it should be noted that high generation length does not necessarily mean higher quality responses, as also indicated by the MAUDE scores.

Finally, in terms of the \textbf{MAUDE} metric, our proposed method, CORAL, outperforms all baselines by a significant margin. Among all the metrics reported here, MAUDE has been shown to correlate well with human preferences for generated responses, along several important quality aspects (e.g., interestingness, specificity, relevance etc. \citealp{Sinha2020LearningAU}). This was the motivation behind the $R_3$ reward. The diverse and high quality responses justify the choice and design of the CORAL loss function.

\begin{table}[]
  \caption{Results for DailyDialog and DSTC7 datasets: From the results, we can see that by optimizing the contextual response retrievability score directly, using REINFORCE, the S2S-CORAL model is able to produce coherent and diverse responses at the same time. The average length is reported to make sure that the model is not resorting to short utterances, such as ``I don't think I know about \textit{[topic\_word]}'', just to be coherent. Among the baselines, Blenderbot-3B, which is a large-scale pretrained model, produces the most human-like responses as per MAUDE but not as diverse as the CORAL-based models. \textbf{$\text{CORAL}_x$} denotes a Seq2Seq model trained with CORAL loss. `x' identifies the retrieval model used for the $R_3$ reward. Note that we used the DialoGPT-medium, DialogRPT-medium, Blenderbot-400M-distill and Blenderbot-3B checkpoints. \textit{We have kept these large-scale baselines as a separate group, in the table.} Each value reported for CORAL models is computed as average of 5 runs.}
  \label{tab:results-dd-new}
  \centering
  \small
  \resizebox{\textwidth}{!}{%

    \begin{tabular}{clcrrrrrr}
    \toprule
    &\multicolumn{8}{c}{\textbf{DSTC7}}\\ \midrule
    &Model & Avg. Len & BLEU & METEOR & Dist-1 & Dist-2 & MAUDE-ESIM & MAUDE-BERT  \\
    \midrule
    &Ground Truth & 13.73 & NA & NA & 0.0922 & 0.5231 & 0.8583 & 0.7363\\
    \midrule
    \multirow{4}{*}{\rotatebox{90}{\makecell{Large\\ Models}}}&DialogRPT & 14.17 & 0.0714 & 0.0424 & 0.0351 & 0.1333 & 0.7328 & 0.6813 \\
    &DialoGPT & 8.59 & 0.0565 & 0.0369 & 0.0504 & 0.1723 & 0.7058 & 0.6557 \\
    &Blenderbot-400M-distill & 17.78 & 0.0869 & 0.0571 & 0.0266 & 0.0893 & 0.5683 & 0.5294  \\
    &Blenderbot-3B & 16.94 & 0.0947 & 0.0591 & 0.0320 & 0.1240 & 0.7300 & 0.7404 \\
    \midrule

    \multirow{5}{*}{\rotatebox{90}{\makecell{Small\\ Models}}}&Mirror & 6.19 & 0.0299 & 0.0387 & 0.0068 & 0.0145 & 0.5121 & 0.4926 \\
    &AdaLabel & 15.70 & 0.2022 & 0.1824 & 0.0539 & 0.3099 & 0.7929 & 0.7046  \\
    &$\text{CORAL}_\text{ESIM}$ & 10.28 & 0.1729 & 0.1581 & \textbf{0.0744} & \textbf{0.4242} & \textbf{0.8590} & 0.7600  \\
    &$\text{CORAL}_\text{BERT}$ & 10.53 & 0.0949 & 0.0724 & 0.0694 & 0.4093 & 0.8534 & \textbf{0.8108} \\
    &$\text{CORAL}_\text{DMI}$ & 10.39 & \textbf{0.2194} & \textbf{0.2111} & 0.0707 & 0.4151 & 0.8274 & 0.7309  \\
    \midrule
    \midrule
    &\multicolumn{8}{c}{\textbf{DailyDialog}}\\ \midrule
    & Model & Avg. Len & BLEU & METEOR & Dist-1 & Dist-2 & MAUDE-ESIM & MAUDE-BERT  \\
    \midrule
    &Ground Truth & 11.97 & NA & NA & 0.0681 & 0.4061 & 0.8180 & 0.8603\\
    \midrule
    \multirow{4}{*}{\rotatebox{90}{\makecell{Large\\ Models}}}&DialoGPT & 8.89 & 0.0667 & 0.0445 & 0.0415 & 0.1682 & 0.6327 & 0.6016  \\
    &DialoRPT & 15.94 & 0.0809 & 0.0530 & 0.0273 & 0.1192 & 0.6523 & 0.6055  \\
    &Blenderbot-400M-distill & 17.27 & 0.1049 & 0.0771 & 0.0223 & 0.0977 & 0.6363 & 0.6322  \\
    &Blenderbot-3B & 16.47 & 0.1063 & 0.0783 & 0.0302 & 0.1284 & 0.7137 & \textbf{0.6812} \\
    \midrule

    \multirow{5}{*}{\rotatebox{90}{\makecell{Small\\ Models}}}&Mirror & 7.93 & 0.0618 & 0.0538 & 0.0371 & 0.1485 & 0.6223 & 0.4995  \\
    &AdaLabel & 11.59 & 0.1208 & 0.0881 & 0.0381 & 0.2292 & 0.5814 & 0.4890 \\
    &$\text{CORAL}_\text{ESIM}$& 10.00 & \textbf{0.1910} & \textbf{0.1727} & 0.0459 & 0.2840 & \textbf{0.7402} & 0.6474  \\
    &$\text{CORAL}_\text{BERT}$& 9.68 & 0.1838 & 0.1656 & \textbf{0.0462} & \textbf{0.2902} & 0.7279 & 0.6526 \\
    &$\text{CORAL}_\text{DMI}$& 9.44 & 0.1727 & 0.1567 & 0.0461 & 0.2844 & 0.7229 & 0.6234  \\
    \bottomrule
    \end{tabular}
  }
\end{table}

\begin{table}[]
  \caption{This table shows sample generations from different baseline models and our $\text{CORAL}_\text{BERT}$ model. Compared to Mirror, responses from our model feels like a more natural continuation to the context. The AdaLabel baseline also has a higher tendency to generate a generic/dull response to the context. A common issue that we notice in all small-scale models is it's random behavior when it encounters a new concept unseen during training.}
  \label{tab:manual-samples}
  \centering
  \scriptsize
  \resizebox{\textwidth}{!}{%
  \begin{tabular}{llll}
  \toprule
  \multicolumn{4}{c}{\textbf{Example Generations from all Models}} \\ \midrule
  \multicolumn{4}{L{14cm}}{\textbf{Context}: I really need to start eating healthier. EOU  I have to start eating better too. EOU  What kind of food do you usually eat? EOU  I try my best to eat only fruits, vegetables, and chicken. EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: What kind of food do you want ?} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: I try to eat a lot of veggies and lean meats.} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel} i have no idea what i'm supposed to eat .} &
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$}: I usually eat a bowl of dumplings.}\\ \midrule

  \multicolumn{4}{L{14cm}}{\textbf{Context}: Room service, is there anything I can do for you? EOU  Yes, I am afraid there are not enough outlets. EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: Have you got any wine ?} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: I'm not sure what you mean by that, but I'm sure there's something you can do.} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel}: ok. thank you .} &
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$}: OK, sir. Can I take your room key to the Room 20 minutes?} \\ \midrule

  \multicolumn{4}{L{14cm}}{\textbf{Context}: Good evening, madam. Can I help you? EOU Yes. I bought this sheet here yesterday. EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: Did it happen ?} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: How much did you pay for it?? I'm a madam} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel}: ok. when will you come back ?} &
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$}: I'd like to buy a new dress.} \\ \midrule

  \multicolumn{4}{L{14cm}}{\textbf{Context}: Could I have my bill, please? EOU Certainly, sir. EOU I'm afraid there's been a mistake. EOU I'm sorry, sir. What seems to be the trouble? EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: Yes, I have been looking forward to it .} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: I don't know what to do with my life. I feel like I have no purpose.} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel} oh, i see. then, let me see. ... oh, this switch down to the shop tonight. i' ve got a lot} &
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$} I'm sorry, sir. I've got you a wrong case. I've been waiting for the wrong medicine. I'm sorry.} \\ \midrule

  \multicolumn{4}{L{14cm}}{\textbf{Context}: Are you busy tomorrow morning? EOU I'm free. What's up? EOU Someone has to pick up the boss at the airport. EOU} \\ \midrule
  \multicolumn{1}{L{3.2cm}|}{\textbf{Mirror}: Oh, I see. Thank you .} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{Blender}: Yes, I have to get up early to go to work. How about you?} &
  \multicolumn{1}{L{3.2cm}|}{\textbf{AdaLabel}: i' m sorry to hear that. do you want to go to the hospital ?} &
  \multicolumn{1}{L{3.2cm}}{\textbf{$\text{CORAL}_\text{BERT}$}: Yes. I'd like to. What time do you want to} \\ \bottomrule
  \end{tabular}
  }
\end{table}

\subsection{Case Study: Generation Quality of Models}\label{human-evaluation}
Aside from automatic evaluation, we also manually went through a random selections of generated outputs from all models and compared them. We wanted to find out the strengths and weaknesses of each of the approaches, both for baselines and CORAL. We summarize these points below. To help the reader to visualize these qualities, we provide samples generated from each of the models in Table \ref{tab:manual-samples}.\\
\textbf{Strengths} Since Blenderbot-3B \citep{roller2020recipes} model was trained with a much larger dialog corpus, it is able to generate responses in many cases where small-scale models failed to understand the context. In such cases, the small-scale models generate some random utterance. Among the small scale models, CORAL generally replies with more engaging and specific responses, whereas Mirror tends to produce generic responses most of the time.\\
\textbf{Weaknesses} Blenderbot fails to realize the switch in the turns sometimes and generates the same response repeatedly, only based on the context topic. This is the reason for low diversity scores of Blenderbot. For all small-scale models, because of limited training data, they lack knowledge of many entities that only appears in test set but not during training. In such cases, the small scale models just produces some unrelated utterance.

\section{Limitations}

While running the experiments, we discovered a few limitations for the specific implementation of the CORAL loss as described in this paper. First problem is regarding the difficulty of exploration in this current setup. The space of all possible sequences is combinatorially large and provides reward very sparsely. Because of this it is really difficult to do on-policy training. The second problem is with randomly sampled negative response candidates which are often too trivial (leading to zero reward) to help the model while training.

\section{Conclusion}
Natural Language Generation models in NLP have usually been
trained using the cross-entropy loss, and it has been reasonably
successful too. But dialog-response generation poses a few
unique challenges for the CE loss.
First, cross-entropy loss assumes
that for any given input, the only possible output is the one available
as the ground truth through the dataset.
Secondly, cross-entropy does not take the context into consideration
while processing the response.
In this paper, we
propose a novel loss function, CORAL, that circumvents these
shortcomings of CE and is more effective for training dialog-generation
models.
Using CORAL, we can train dialog generation models without
assuming a fixed ground-truth response and the value of the loss function is based on both the context and response.
The CORAL loss is based on pretrained response retrieval models that, in prior literature, have been shown to correlate with human preferences.
The proposed loss function will make it possible to train future models focused on maximizing human preference.
We also hope that our work will motivate the NLP
community to look for more suitable loss functions for training dialog
generation models and to rely less on the cross-entropy loss.

We plan to extend this framework for training larger scale models that can capture more patterns from larger training data. We are also looking into the possibility of designing a learning curriculum using the retrieval model. This will help make the training of the dialog generation model more efficient.

Research in loss functions also gives us insights into which attributes for a generative dialog system are most appreciated in the final model as we compare their outcomes.

\section*{Acknowledgments}
This work was partially supported by Microsoft Academic Partnership Grant (MAPG) 2021. The first author was also supported by Prime Minister's Research Fellowship (PMRF), India.

\bibliographystyle{abbrvnat}