\section{I. Introduction}

The nucleus is a quantum manybody system composed of protons and neutrons. The fission limits the amount of nuclear mass \cite{Bo,Ho}. The shell effect increases the fission barrier and enable the survival of superheavy nucleus (SHN). Exploring the limit of element or atomic nucleus is a long-term task, especially close to the 'island of stability' predicted by the shell model \cite{WD,AF,Mo94}. Up to now, there are more than 3000 nuclides existing on the nuclide chart, and most of them are located in the proton-rich region \cite{Th16}. For the past few decades, new transactinide nuclei have been synthesized by the fusion-evaporation reactions in different laboratories. Before 1974, the isotopes of element Z=93-106 were created by the neutron capture reactions and heavy-ion fusion reactions. Combinations with a doubly magic nucleus or nearly magic nucleus are usually chosen owing to the larger reaction $Q$ values. Reactions with $^{208}$Pb or $^{209}$Bi based targets were firstly proposed by Oganessian et al. \cite{Og75}. The SHEs from Bh to Cn were synthesized in the cold fusion reactions at GSI (Darmstadt, Germany) with the heavy-ion accelerator UNILAC and the SHIP separator \cite{Ho00,Mu15}. Experiments on the synthesis of element Nh (Z=113) in the $^{70}$Zn+$^{209}$Bi reaction have been performed successfully at RIKEN (Tokyo, Japan) \cite{Mo04}. However, it is very difficulty to create the superheavy isotopes beyond Nh in the cold fusion reactions because of very low cross section ($\sigma<$0.1 pb). The superheavy elements from Fl (Z=114) to Og (Z=118) have been synthesized at the Flerov Laboratory of Nuclear Reactions (FLNR) in Dubna (Russia) with the double magic nuclide $^{48}$Ca bombarding actinide nuclei \cite{Og99,Og06,Og15}, in which more neutron-rich SHN were produced and identified by the subsequent $\alpha$-decay chain. With constructing the new facilities in the world such as RIBF (RIKEN, Japan), SPIRAL2 (GANIL in Caen, France), FRIB (MSU, USA), HIAF (IMP, China), the SHNs on the 'island of stability' by using the neutron-rich radioactive beams induced fusion reactions or via the multinucleon transfer (MNT) reactions might be possible in experiments.

The MNT reactions and deep inelastic heavy-ion collisions were extensively investigated in experiments since 1970s, in which the new neutron-rich isotopes of light nuclei and proton-rich actinide nuclei were observed \cite{Art71,Art73,Art74,Hil77,Gla79,Moo86, Wel87}. The reaction mechanism and fragment formation were investigated thoroughly, i.e., the energy and angular momentum dissipation, two-body kinematics, shell effect, fission of actinide nuclei etc. Recently, the MNT reactions have been extensively investigated in experiments for new isotope production. The MNT reactions have the advantage of extensive isotope distributions, e.g., more than 100 nuclides with Z=82-100 in the reaction of $^{48}$Ca + $^{248}$Cm and five new neutron-deficient isotopes $^{216}$U, $^{219}$Np, $^{223}$Am, $^{229}$Am and $^{233}$Bk \cite{De15}. The reaction mechanism is concentrated on the neutron-rich nuclei around the shell closure N = 126. The MNT fragments were measured in the reactions of $^{136}$Xe + $^{208}$Pb \cite{JW15,EE12}, $^{136}$Xe + $^{198}$Pt \cite{YX15}, $^{156,160}$Gd + $^{186}$W \cite{EM17}, and $^{238}$U + $^{232}$Th \cite{SK18}. The neutron-rich nuclides around N = 126 have important application in understanding the origin of heavy elements from iron to uranium in the \emph{r}-process of astrophysics. It has been confirmed that the shell closure plays an important role on the production of neutron-rich nuclei and more advantage with the multinucleon transfer (MNT) reactions in comparison to the projectile fragmentation \cite{Ku14}.

Several models have been proposed for describing the MNT reactions, i.e., the dinuclear system (DNS) model \cite{Fe09,GG10,GG81}, the GRAZING model \cite{Win94,Win95}, the dynamical model based on multidimensional Langevin equations \cite{Zag15}. Moreover, the microscopic approaches based on the nucleon degree of freedom, the time-dependent Hartree-Fock (TDHF) approach \cite{CC09,KK13,WN18,Gu19} and the improved quantum molecular dynamics (ImQMD) \cite{TJ08,ZK13,ZK15} are also used to describe the MNT reactions. Some interesting issues have been investigated, e.g., the production cross sections of new isotopes, total kinetic energy spectra and polar angle distribution of transfer fragments, structure effect on the fragment formation etc. There are still some open problems for the MNT reactions, i.e., the mechanism of preequilibrium cluster emission, the stiffness of nuclear surface during the nucleon transfer process, the mass limit of new isotopes with stable heavy target nuclides, interplay of nuclear fission and particle evaporation in the MNT dynamics, etc. The extremely neutron-rich beams are favorable for creating neutron-rich heavy or superheavy nuclei owing to the isospin equilibrium \cite{Das94}.

In this work, the systematics on the production of neutron-rich heavy nuclei around N = 162 has been investigated. The article is organized as follows. In  Section II we give a brief description of the DNS model. In Section III, the transfer cross section and angular distribution of actinide collisions are studied. Summary and perspective on the neutron-rich heavy nuclei are shown in Section IV.

\section{II. Brief description of the model}
\subsection{ The MNT fragment formation }

The DNS concept was assumed to be formal at the touching configuration in nucleon collision and was proposed by Volkov for describing the deep inelastic transfer reaction\cite{Vo20}. The sticking of projectile and target forms DNS. Sticking times are on the order of $(10^{-21}-10^{-20})$s. Once DNS is formed,the exchange of nucleons and energy stars. The motion of nucleons in the interacting potential is governed by the single-particle Hamiltonian \cite{Fe07}. Only the particles within the valence space are active for nucleon excitation and transfer\cite{W24,S25}. The transition probability is related to the local excitation energy and nucleon transfer\cite{Fe07}\cite{Fe06}, which is microscopically derived from the interaction potential in valence space. The local excitation energy is determined by the dissipation energy from the relative motion and the potential energy surface of the DNS. The dissipation of the relative motion and angular momentum of the DNS is described by the Fokker-Planck equation \cite{Fe07a}.  The cross sections of the primary fragments (Z$_{1}$, N$_{1}$) after the DNS reaches the relaxation balance are calculated as follows:
\begin{eqnarray}
	\sigma_{pr}(Z_{1},N_{1},E_{c.m.})=&&\sum^{J_{max}}_{J=0}\sigma_{cap}(E_{c.m.},J)\int f(B) \nonumber\\
	&&  \times P(Z_{1},N_{1},E_{1},J_{1},B)dB
\end{eqnarray}

The cross sections of secondary fragments that is the product of the primary fragment after the evaporation of particles in the MNT reactions are evaluated by
\begin{eqnarray}
	\sigma_{sur}(Z_{1},N_{1},E_{c.m.})=&&\sum^{J_{max}}_{J=0}\sigma_{cap}(E_{c.m.},J)\int f(B) \nonumber\\
	&&  \times P(Z_{1},N_{1},E_{1},J_{1},B) \nonumber \\
	&&  \times W_{sur}(E_{1},J_{1},s)dB
\end{eqnarray}
Here, $E_{1}$ and $J_{1}$ denote the excitation energy and the angular momentum for the fragment$(Z_{1},N_{1})$ with the attempting barrier B, respectively. The maximal angular momentum $J_{max}$ is taken to be the grazing collision of two nuclei. J is the relative angular momentum. The expression of capture cross is $\sigma_{cap}=\pi\hbar^{2}(2J+1)/(2\mu E_{c.m.})T(E_{c.m.},J)$ and the cross is calculated within the Hill-Wheeler formula and the barrier distribution approach. The transferred cross section is smoothed with the barrier distribution and the function is taken to have the Gaussian form $f(B) =\frac{1}{N}exp[-((B-B_{m})/\Delta)^{2}]$, with the normalization constant satisfying the unity relation $\int f(B)dB=1$. The quantities $B_{m}$ and $\Delta$ are evaluated by $B_{m}=(B_{C} + B_{S})/2$ and $\Delta = (B_{C} - B_{S})/2$, respectively. $B_{C}$ and $B_{S}$ are the Coulomb barrier at waist-to-waist orientation and the minimum barrier from varying the quadrupole deformation parameters of the colliding partners.

The nucleon transfer is related to the relative motion by solving a set of microscopically derived master equations by distinguishing protons and neutrons \cite{Fe06,Fe07}. The time evolution of the distribution probability $P(Z_{1},N_{1},E_{1},t)$ for a DNS fragment 1 with proton number $Z_{1}$ and neutron number $N_{1}$ and excitation energy $E_{1}$ is governed by the master equations as follows,
\begin{eqnarray}
	&&\frac{d P(Z_{1},N_{1},E_{1}, t)}{dt}           \nonumber\\
	&=& \sum_{Z^{'}_{1}}W_{Z_{1},N_{1};Z^{'}_{1},N_{1}}(t)[d_{Z_{1},N_{1}}P(Z^{'}_{1},N_{1},E^{'}_{1},t)  \nonumber\\
	&& -d_{Z^{'}_{1},N_{1}}P(Z_{1},N_{1},E_{1},t)]     \nonumber\\
	&& +\sum_{N^{'}_{1}}W_{Z_{1},N_{1};Z_{1},N^{'}_{1}}(t)\times[d_{Z_{1},N_{1}}P(Z_{1},N^{'}_{1},E^{'}_{1},t)  \nonumber\\
	&& -d_{Z_{1},N^{'}_{1}}P(Z_{1},N_{1},E_{1}, t)]    \nonumber\\
\end{eqnarray}
Here the $W_{Z_{1},N_{1};Z_{1}^{\prime},N_{1}}$ ($W_{Z_{1},N_{1};Z_{1},N_{1}^{\prime}}$) is the mean transition probability from the channel $(Z_{1},N_{1},E_{1})$ to $(Z_{1}^{\prime},N_{1},E_{1}^{\prime})$ (or $(Z_{1},N_{1},E_{1})$ to $(Z_{1},N_{1}^{\prime},E_{1}^{\prime})$), and $d_{Z_{1},N_{1}}$
denotes the microscopic dimension corresponding to the macroscopic state $(Z_{1},N_{1},E_{1})$. Sequential one-nucleon transfer is considered in the model with the relation of $Z_{1}^{\prime}=Z_{1}\pm 1$ and $N_{1}^{\prime }=N_{1}\pm 1$.

DNS is affected by potential energy surface (PES) during nuclear transfer. The projectile and target is transferred under the action of the driving potential. The potential energy surface of the DNS is given by
\begin{eqnarray}
	U(\{\alpha\})=&&B(Z_{1},N_{1})+B(Z_{2},N_{2})\nonumber \\&&-\left[B(Z,N)+V^{CN}_{rot}(J)\right]
	+V(\{\alpha\})
\end{eqnarray}
Here Z and N are the proton and neutron number of the composite system respectively with $Z_{1}+Z_{2}=Z$ and $N_{1}+N_{2}=N$ \cite{Fe09}.  The symbol $\{\alpha\}$ denotes the sign of the quantities $Z_{1},N_{1}, Z_{2}, N_{2}; J, R; \beta_{1}, \beta_{2}, \theta_{1}, \theta_{2}$. The $B(Z_{i},N_{i}) (i=1,2)$ and $B(Z,N)$ are the negative binding energies of the fragment $(Z_{i},N_{i})$ and the compound nucleus $(Z,N)$, respectively. The $V^{CS}_{rot}$ is the rotation energy of the compound system. The $\beta_{i}$ represent the quadrupole deformations of the two fragments at ground state. The $\theta_{i}$ denote the angles between the collision orientations and the symmetry axes of deformed nuclei. The interaction potential between fragment $(Z_{1},N_{1})$ and $(Z_{2},N_{2})$ includes the nuclear, Coulomb and centrifugal parts. In the calculation, the distance $R$ between the centers of the two fragments is chosen to be the value at the touching configuration, in which the DNS is assumed to be formed. So the PES depends on the proton and neutron numbers of the fragments.

The survival probability $W_{sur}$ of each fragment is evaluated with a statistical approach based on the Weisskopf evaporation theory\cite{P22}, in which the excited primary fragments are cooled by evaporating $\gamma$ rays, light particles including neutrons, protons, $\alpha$ in competition with binary fission. Shell correction, odd-even effect, Q value, etc. are the structure effects that could be particularly significant in the formation of the primary fragments and in the decay process. the probability in the channel of evaporating the x-th neutron, the y-th proton and the z-alpha is expressed as
\begin{eqnarray}
    W_{sur}(E^{*}_{CN},x,y,z,J)= && P(E^{*}_{CN},x,y,z,J)          \nonumber\\
	&&  \times \prod_{i=1}^{x} \frac{\Gamma_{n} (E^{*}_{i},J)} {\Gamma_{tot}(E^{*}_{i},J)} \prod_{j=1}^{y}\frac{\Gamma_{p}(E^{*}_{j},J)}{\Gamma_{tot}(E^{*}_{j},J)}       \nonumber \\
	&&  \times \prod_{k=1}^{z}\frac{\Gamma_{\alpha}(E^{*}_{k},J)}{\Gamma_{tot}(E^{*}_{k},J)}
\end{eqnarray}

Here the $E^{*}_{CN}$, J are the excitation energy evaluated from the mass table\cite{PM95} and the spin of the excited nucleus, respectively. The total width $\Gamma_{tot}$ is the sum of partial widths of particle evaporation, $\gamma$ -emission and fission. The excitation energy $E^{*}_{s}$ before evaporating the s-th particle is evaluated by
\begin{eqnarray}
E^{*}_{s+1}=E^{*}_{s}-B^{n}_{i}-B^{p}_{j}-B^{\alpha}_{k}-2T_{s}
\end{eqnarray}
with the initial condition  $E^{*}_{1}$ =  $E^{*}_{CN}$ and s = i+j+k. The $B^{n}_{i}$, $B^{p}_{j}$, $B^{\alpha}_{k}$ are the separation energy of the i-th neutron, j-th proton, k-th alpha, respectively. The nuclear temperature $T_{i}$ is given by $E^{\ast}_{i}=a T_{i}^{2} - T_{i}$ and the level density parameter $a$. The evaporation width and realization probability are described in detail in reference \cite{P22}.

\subsection{  Angular distribution }
The detection of emission angle of fragments formed in MNT reaction of primary fragments is beneficial to the placement of detectors in experiments. Different systems and incident energies affect the angular distribution. In this work, we use a deflection function method to evaluate the fragment angle which can be changed by the mass of fragment, angular momentum, and incident energy. The deflection angle is composed of the Coulomb and nuclear interactions as \cite{Ch17,GW78}
\begin{eqnarray}
\Theta(l_{i}) = \Theta(l_{i})_{C}+\Theta(l_{i})_{N}
\end{eqnarray}
with
\begin{eqnarray}
	\Theta(l_{i})_{C}=2\arctan \frac{Z_{1}Z_{2}e^{2}}{2E_{inc}b}.
\end{eqnarray}
The Coulomb deflection function is given by the Rutherford function. The expression of nuclear deflection angle is shown as
\begin{eqnarray}
\Theta(l_{i})_{N} = -\beta\Theta(l_{i})_{C}^{gr}\frac{l_{i}}{l_{gr}}(\frac{\delta}{\beta})^{l_{i}/l_{gr}}.
\end{eqnarray}
Here $\Theta(l_{i})_{C}^{gr}$ is the Coulomb scattering angle at the grazing angular momentum $l_{gr}$ and $l_{gr}=0.22R_{int}[A_{red}(E_{c.m.}-V(R_{int}))]^{1/2}$. The $l_{i}$ is the incident angular momentum.  $A_{red}$, and $V(R_{int})$ are reduced mass of projectile and target and interaction potential, respectively. $R_{int}$ is the distance from the entrance channel. The $\delta$ and $\beta$ are parameterized by fitting the deep inelastic scattering in massive collisions as
\begin{eqnarray}
\label{eq10}
\beta = &&  75 f(\eta) + 15,   \qquad  \eta <375      \nonumber       \\
&& 36 \exp(-2.17\times 10 ^{-3} \eta),   \qquad  \eta \geq 375
\end{eqnarray}

\begin{eqnarray}
\label{eq11}
\delta = && 0.07 f(\eta) + 0.11,     \qquad  \eta <375        \nonumber   \\
&&  0.117 \exp(-1.34\times 10 ^{-4} \eta),  \qquad  \eta \geq 375
\end{eqnarray}
and
\begin{equation}
\label{eq13}
f(\eta) = [1 + \exp{\frac{\eta-235}{32}}]^{-1}
\end{equation}
where the Sommerfeld parameter $\eta = \frac{Z_1Z_2e^2}{\upsilon}$, and the relative velocity $\upsilon = \sqrt{\frac{2}{A_{red}}(E_c.m. - V(R_{int}))}$. For the $i-$th DNS fragment, the emission angle is determined by $\Theta_{i}(l_{i})=\Theta(l_{i})\xi_{i}/(\xi_{1}+\xi_{2})$ with the moment of inertia $xi$.

\section{III. Results and discussion}

The synthesis of superheavy nuclei in the fusion-evaporation reactions has entered the bottleneck stage because of the lack of neutron number of formed compound nucleus. There are several pathways to reach the 'island of stability', i.e., the MNT reactions, incomplete fusion reactions and complete fusion reactions induced with radioactive nuclides etc. The damped collisions of two actinide nuclei were investigated and motivated for producing superheavy nuclei in 1970s at Gesellschaft f\"{u}r Schwerionenforschung (GSI) \cite{Hil77,Gla79}. The experimental data were reanalyzed for investigating the MNT dynamics, in particular for the transactinide production \cite{Kr13}. As a test of the DNS model, the available data in collisions of $^{238}$U+$^{248}$Cm at the beam energy of E$_{lab}$ = 7.0 MeV/nucleon are compared as shown in Fig. 1. It is obvious that the isotopic yields are consistent with the available data. The maximal yield of each isotopic chain is located on the line of $\beta-$stability. The distribution structure of the isotopic yields is related to the PES, nucleon transfer dynamics, nuclear fission, particle evaporation etc. The extension to the neutron-rich side is possible but monotonically decreasing with the mass number of target-like fragment.

\begin{figure*}
\includegraphics[width=15 cm]{fig1}
\caption{Production cross sections of transcurium isotopes in the $^{238}$U+$^{248}$Cm reaction at E$_{lab}$ =7.0 MeV/nucleon and compared with the available experimental data at GSI with error bars \cite{Kr13}. }
\end{figure*}

The properties of neutron-rich nuclei beyond N=126 are particular of importance in understanding the nucleosynthesis in stellar evolution, the nuclear fission dynamics, the new decay modes of isotopes on the 'island of stability' etc. The shape evolution, shell effect, nucleon transfer dynamics, dissipation mechanism of the relative motion and rotation of the dinuclear system influence the MNT fragment formation. Shown in Fig. 2 is a comparison of isotopic distribution in the $^{238}$U induced reactions on $^{252}$Cf, $^{254}$Es and $^{257}$Fm at the beam energy of 7 MeV/nucleon. The open symbols denote the new isotopes in the nuclear chart \cite{Wa21}.  Compared to the neutron-deficient nuclei via the fusion-evaporation reactions, the MNT reactions exhibit a broad isotopic distribution. The production cross sections decrease with increasing the proton number of fragments. The heavier target $^{257}$Fm is favorable for creating the neutron-rich isotopes owing to the larger isospin ratio neutron/proton of colliding system. However, the different systems almost have the same order magnitudes of isotopic yields in the proton-rich domain. The maximal yields of the isotopic spectra in the MNT reactions correspond to the line of $\beta-$stability, in which the shell effect and fission barrier are of significance on the spectrum structure.

\begin{figure*}
\includegraphics[width=15 cm]{fig2}
\caption{Isotopic distribution of Rf, Db, Sg and Bh in the MNT reactions at E$_{lab}$ = 7.0 MeV/nucleon with $^{238}$U bombarding $^{252}$Cf, $^{254}$Es and $^{257}$Fm, respectively. The open symbols denote the new isotopes \cite{Wa21}. }
\end{figure*}

The isotopic dependence of initial systems in the MNT reactions is of significance in the selection of projectile-target combination in experiments and the investigation of isospin dissipation mechanism in theories. Figure 3 shows the isotopic distribution of Rf, Db, Sg and Bh in the reactions of $^{238}$U + $^{252}$Cf (soild line) and $^{238}$U + $^{254}$Cf (dotted line) at E$_{lab}$ =7.0 MeV/nucleon. The isotopic targets are selected for producing the MNT fragments. It is obvious that the overall products move to the neutron-rich side with the neutron-rich target $^{254}$Cf. However, the isotopes in the proton-rich region exhibit an opposite trend. The entrance system with the larger neutron/proton ratio is available for producing the neutron-rich fragments owing to the isospin equilibrium in the dissipation process. The shell effect and isospin relaxation play significant roles on the MNT fragment formation \cite{Fe17,NF17,NF18}.

\begin{figure*}
	\includegraphics[width=15 cm]{fig3}
	\caption{Comparison of Rf, Db, Sg and Bh production in the MNT reactions at E = $_{lab}$7.0 MeV/nucleon with $^{238}$U bombarding $^{252}$Cf and $^{254}$Cf, respectively. The open symbols denote the new isotopes with the recent nuclear mass evaluation \cite{Wa21}.}
\end{figure*}

The beam energy influences the dissipation energy into the DNS and consequently the MNT fragment formation. The more local excitation energy is obtained with increasing the incident energy and leads to the wider domain nucleon transfer \cite{NF17}. A number of rare isotopes might be created with more energetic nuclear collisions. However, the less survival of the primary fragment is obtained by the transfer dynamics. The competition of the diffusion of nucleon transfer and the survival of formed fragment leads the isotopic structure of final MNT fragments, which is associated with the beam energy and colliding system. It is found that the total mass and charge distributions of the secondary fragments around the shell closure N = 126 weakly depend on the bombarding energy in the MNT reaction of $^{136}$Xe + $^{198}$Pt \cite{Fe17}. Systematic investigation of energy dependence on the fragment formation in the MNT reactions would be helpful for selecting the optimal beam energy in experiment. Figure 4 shows isotopic distribution of elements of Rf, Db, Sg and Bh in the reaction of  $^{238}$U + $^{252}$Cf at the collision energies E$_{lab}$ = 6.7, 7.0 and 7.4 MeV/nucleon, respectively. The production cross section decreases with the beam energy because of the reduction of survival probability of formed MNT fragment. We calculated the excitation functions of selected nuclides $^{266}$Rf, $^{267}$Db and $^{268}$Sg produced in the reaction of $^{238}$U + $^{252}$Cf as shown in Fig. 5. There exists an optimal incident energy for producing the MNT fragments with the larger cross sections. More beam energy is not suitable for the heavy fragment formation owing to the reduction of survival probability. Roughly, the selection of beam energy around the Coulomb barrier (interaction potential at the touching position) is enough for the MNT reaction.

\begin{figure*}
	\includegraphics[width=14 cm]{fig4}
	\caption{Incident energy dependance of isotopic distribution of Rf, Db, Sg and Bh in the MNT reaction of  $^{238}$U + $^{252}$Cf. }
\end{figure*}

\begin{figure*}
	\includegraphics[width=10 cm]{fig5}
	\caption{Excitation functions of $^{266}$Rf, $^{267}$Db and $^{268}$Sg production in the MNT reaction of  $^{238}$U + $^{252}$Cf.}
\end{figure*}

\begin{figure*}
	\includegraphics[width=10 cm]{fig6}
	\caption{Isotopic fragments with the neutron shell closure N=162 in the MNT reactions with $^{238}$U bombarding $^{252}$Cf, $^{254}$Es and $^{257}$Fm at E$_{lab}$ = 7.0 MeV/nucleon.}
\end{figure*}

The neutron shell closure around N=126 is favorable for enhancing the cross sections of neutron-rich isotopes and extensive investigation in the MNT reactions with the DNS model \cite{Fe17,Ch20}. The shell effect enhances the fission barrier and enlarges the separation energy of the MNT fragment. On the other hand, the strong shell effect leads to the appearance of pocket in the PES, which also enhances the fragment formation in the nucleon transfer process. A number of neutron-rich isotopes around N = 162 might be created with the damped collisions of two actinide nuclei. More pronounced structure effect is shown in Fig. 6 from the isotonic yields of N=162 in the MNT reactions of $^{238}$U+$^{252}$Cf, $^{238}$U + $^{254}$Es and $^{238}$U+$^{257}$Fm at E$_{lab}$=7.0 MeV/nucleon. The collisions of $^{238}$U+$^{257}$Fm have the larger cross sections because of the neutron/proton ratio and primary fragment yields. The three systems have the same maximal yields around the new isotope $^{263}$Md. The systematic investigation of the new isotope production around N=162 is complicated both in experiments and in theories. Different with the neutron shell closure of N=126, the fission barrier and decay mode is not known in the superheavy region. There are uncertainties of some physical quantities and level spectra in the domain of isotopes around N=162. More experiments and dynamical models in the MNT reactions around N=162 are expected.

\begin{figure*}
	\includegraphics[width=14 cm]{fig7}
	\caption{Cross sections as functions of the atomic numbers and the mass numbers of primary and secondary fragments with shell effect or not in the reaction of  $^{238}$U + $^{252}$Cf at E$_{lab}$ = 7.0 MeV/nucleon.}
\end{figure*}

The collisions of two actinide nuclides provide the possibility to produce the new-neutron isotopes of transfermium elements. The yields of neutron-rich and long-living SHNs in the MNT reactions might be significantly enhanced owing to the shell effect via the 'inverse quasi-fission' process. The shell effect is of significance in the MNT fragment formation, which is manifested via the PES, fission barrier and particle evaporation in the DNS model. Figure 7 shows the production cross sections as functions of atomic number (left panel) and mass number (right panel) in the reaction of  $^{238}$U + $^{252}$Cf at E$_{lab}$ = 7.0 MeV/nucleon. It is obvious that the production cross sections of primary fragments are enhanced around the neutron shell closure N=162, which are caused from the shell effect in the reaction dynamics. The production of secondary fragments is reduced by several orders of magnitude owing to the deexcitation process, in particular in the superheavy region. The decay stage of the MNT fragments is associated with the local excitation energy and angular momentum at the equilibrium time, fission barrier, separation energy of evaporated particles (neutron, proton, deuteron, triton, alpha etc). The strong shell effect can enhance the fission barrier and separation energy, which is favorable the survival of the primary fragment. For the cases without the shell effect, the production cross sections of the primary and secondary fragments monotonically decrease with the mass or charge number, which reduces the formation of heavy rare isotopes in the MNT reactions. The shell effect increases the MNT yields above the two order magnitude. The results are consistent with the model of multidimensional Langevin equations \cite{Zag15}. Inclusion of the structure factors such as the shell effect, odd-even effect, nucleus stiffness etc is particulary significant in the dynamical models for the MNT reactions.

\begin{figure*}
	\includegraphics[width=14 cm]{fig8}
	\caption{The angular distributions of superheavy nucleus of Z = 104-107 in the laboratory frame in the $^{238}$U+$^{252}$Cf reaction at E$_{lab}$ = 7.0 MeV/nucleon (left panels) and the angular distributions of Rf with different incident energy (right panels). }.
\end{figure*}

The emission of the MNT fragments is anisotropic and related to the reaction system and beam energy \cite{EE12}. Accurate estimation of the angular distribution would be helpful for managing the detector system in experiments, which is associated with the dynamical characteristics in MNT reactions, i.e., the deformation of the DNS fragments, dissipation of relative energy and angular momentum, nucleon transfer etc\cite{NF20}. Figure 8 shows the selected nuclides with Z=104-107 produced in the MNT reaction of $^{238}$U+$^{252}$Cf at E$_{lab}$ =7.0 MeV/nucleon (left panel) and the angular distribution of Rf at the incident energies of 7, 7.5, 8 MeV/nucleon, respectively (right panel). Slight movement of the emission polar angles with the maximal yields for the SHEs Rf, Db, Sg and Bh appears from 80$^{o}$ to 95$^{o}$ at the incident energy of 7 MeV/nucleon. The nuclides produced in the MNT reactions move to the forward emission with the maximal cross section from 85$^{o}$ at 7 MeV/nucleon to 50$^{o}$ at 8 MeV/nucleon and manifest the narrow angular distribution with increasing the beam energy.

\section{IV. Conclusions}

In summary, the production of neutron-rich isotopes via the MNT reactions was investigated within the DNS model in the $^{238}$U induced reaction on  $^{252,254}$Cf, $^{254}$Es and $^{257}$Fm at the incident energy around Coulomb barrier. By comparing the available experimental data, the DNS model is favorable for describing the MNT reaction dynamics. The shell effect, fission barrier, neutron separation energy and nucleon transfer dynamics influence the formation of MNT fragments. A broad isotopic spectrum is exhibited in the MNT reactions. A number of neutron-rich isotopes might be created around the Coulomb barrier energy. The energy dependence of the neutron-rich nuclide production is weak. But the proton-rich isotopes are related the incident energy. The heavier target is favorable for the SHN production. The yields of MNT fragments monotonically decrease with the mass number. The shell effect, in particular around N=162, results in the appearance of the bump structure in the MNT mass spectra. The phenomena is reduced by the secondary decay process. The MNT fragments tend to be emitted in the forward direction with increasing the incident energy. The slight difference on the angular distribution exists for different isotopes.

\section{Acknowledgements}

This work was supported by the National Natural Science Foundation of China (Projects No. 11722546 and No. 11675226) and the Talent Program of South China University of Technology.


\section{Introduction}
With the advances in sequence to sequence models,
neural dialogue systems have attracted more and more research attention.
Neural conversation response generation could be formulated as a Seq2Seq task: given a dialogue history, the model is asked to generate a high-quality response.
A large number of end-to-end generative neural conversation models have been applied to open-domain conversation and chatbot, have achieved success in generating fluent responses.
However, the usual Seq2Seq models tend to produce shorter and simpler responses, which are not informative~\cite{DBLP:conf/naacl/LiGBGD16,lian_learning_2019}.

In order to generate informative and meaningful responses, a number of methods have been proposed by leveraging external knowledge.
Besides the dialogue history, knowledge-based dialog system also combines several external knowledge (in this paper we only discuss unstructured textual knowledge) to construct an input sample. Generally, the collection of candidate knowledge is obtained through rough text retrieval in the knowledge base with a certain amount of noise. The noisy knowledge will lead the conversation to meaningless themes~\cite{lin_generating_2020}, so it is essential for a model to identify and select the appropriate knowledge.

However, most existing methods ~\cite{lian_learning_2019,dinan_wizard_2019,kim_sequential_2020,chen_bridging_2020} can only select one knowledge with the highest confidence from the candidate knowledge to participate in dialogue generation, while abandoning other knowledge with low confidence but possibly containing useful information.
In order to make full use of the external knowledge, some approaches like ~\cite{lin_generating_2020} try to use all the knowledge to participate in the response generation. Nevertheless, when the size of candidate knowledge set is too large, this method will bring serious computational overheads and noise knowledge will lead the conversation to irrelevant topics.

Besides, many existing methods ~\cite{kim_sequential_2020,chen_bridging_2020} require labels that indicate the ground-truth knowledge to train the knowledge-grounded dialog models. However, these knowledge labels are difficult to obtain and need to be constructed through a large number of manual annotations, which is labor-intensive.

Motivated by the above issues, we propose Knoformer, a novel knowledge-grounded dialogue response generation model.
Firstly, we present a knowledge-aware dialogue module, which take the concatenation of dialogue history and selected knowledge as inputs and generation responses.
We perform supervised learning to train this module.
Secondly, we propose a knowledge selection module to select all appropriate knowledge according to hidden states from dialogue module.
Due to the lack of knowledge labels, the selection module uses the feedback from the dialogue module as reward and uses the policy gradient algorithm for training. Besides, we also add a weak supervision loss to the selector to further boost the accuracy.
The dialogue module and knowledge selector are optimized jointly in a recurrent way.

We conduct experiments on Wizard-of-Wikipedia~\cite{dinan_wizard_2019} and Holl-E~\cite{DBLP:conf/emnlp/MogheABK18}, and results shows that our model achieves the new state-of-the-art performance.

\section{Related Work}
Knowledge-based conversation have shown promising results in improving response informativeness ~\cite{DBLP:conf/emnlp/ZhouPB18,dinan_wizard_2019}.
PostKS~\cite{lian_learning_2019} uses prior and posterior distributions over knowledge to train a selector which can choice the most suitable candidate (and discard others) to participate in the response generation;
KIC~\cite{lin_generating_2020} uses recurrent knowledge interaction among response decoding steps incorporate  appropriate knowledge and uses pointer-generate networks copy tokens from external knowledge according to knowledge attention distribution;
~\cite{kim_sequential_2020} proposes Sequential Knowledge Transformer (SKT) to model knowledge selection history in multi-turn dialogue;
PIPM/KDBTS~\cite{chen_bridging_2020} improves on PostKS and SKT by using posterior information prediction and knowledge distillation to bridge the gap between prior and posterior knowledge selection.
PostKS, SKT and PIPM/KDBTS only select one knowledge but our Knoformer could select multiple knowledge from knowledge pool; To train SKT and PIPM/KDBTS, ground-truth knowledge labels are needed, but KnoFormer does not need to specify ground-truth knowledge in advance; KIC uses a special structure to realize knowledge attention which has poor portability, and has no ability to filter out noise knowledge, while our model can directly use Transformer to realize knowledge attention and has the ability to select highly relevant knowledge.

One of the most related models to ours may be KnowledGPT~\cite{zhao_knowledge-grounded_2020}, who also focus on the multiple knowledge selection issue in the knowledge-grounded dialogue by reinforcement learning. Our work is novel in that during training, KnowledGPT only calculates one reward for all knowledge selection actions, which is equivalent to only one action in an \emph{episode}, while our model calculates rewards for each action, which can accurately punish or reward each action.

\section{Methodology}
We use capital letters for sequences (e.g., $Y$), lowercase letters for  tokens in sequences (e.g., $u$), bold capital letters for matrices (e.g., $\mathbf{K}$), and bold lowercase letters for vectors (e.g., $\mathbf{v}$).

Given a conversation history $U=\{u_1, u_2, ..., u_m\}$ and a set of external knowledge $\mathcal{K}=\{K_j\}_{j=1}^r$,
where $K_j=\{k_1^j, k_2^j, ..., k_n^j\}$ and $r$ is the size of external knowledge set,
the task of our proposed approach is to learn a knowledge selection module to select a subset of $\mathcal{K}$:
\begin{equation}
    p(\mathcal{K}_{sub}) = \prod_{i=1}^o p\left(K_{a_i} \mid U, K_{a_1},...,K_{a_{i-1}}\right) \quad \mathcal{K}_{sub} = \{K_{a_i}\}_{i=1}^o \quad 1 \leq a_i \leq r
\end{equation}
where $a_i$ represents the index of $i$-th knowledge in $\mathcal{K}_{sub}$. Specifically, we use a recurrent method to select knowledge one by one.
The first knowledge is selected based on the dialogue history $U$, and then the second knowledge is selected based on $U$ and the first knowledge $K_{a_1}$, and so on, to reach the maximum number of knowledge choices.
$\mathcal{K}$ usually has a lot of noise, so the filtered $\mathcal{K}$ is more closely related to the conversation topic.
In addition, we also learn a dialogue generation model generate response using $U$ and $\mathcal{K}_{sub}$: $p(\hat{Y}) = f(U, \mathcal{K}_{sub})$.

Compared with some models~\cite{lian_learning_2019,kim_sequential_2020,chen_bridging_2020} that can only select one knowledge, our model uses a recurrent selection mechanism to pick multiple suitable external knowledge.

Our proposed model consists of three parts, a dialogue module which can fuse the features of knowledge and dialogue history to generate a response ($\S$ \ref{sec:diagmodel}), a knowledge encoding module which can encode all external knowledge separately ($\S$ \ref{sec:knomodel}), and a knowledge selection module which can be trained without ground-truth knowledge based on reinforcement learning ($\S$ \ref{sec:knoselmodel}).
$\S$ \ref{sec:traininfer} introduces the training and inference of our model.
The architecture of our proposed model is show in Fig. \ref{fig:struct}.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=12cm]{imgs/struct_train}
    \caption{The architecture of our proposed model (suppose the model chooses two knowledge). Modules of the same color share parameters. The orange circles represent the loss of the dialogue model, and they will be accumulated and averaged to form the final dialogue loss $\mathcal{L}_u^\prime$.}
    \label{fig:struct}
\end{figure}
\subsection{Dialogue Module}\label{sec:diagmodel}
Our model operates in an iterative manner. Each time a knowledge is selected, the dialogue module will perform feature fusion on the currently selected knowledge. Our dialogue module is based on Transformer~\cite{vaswani_attention_2017}, which takes the dialogue history and several selected external knowledge as input, and generates a response token by token.

Assuming that $c-1$ knowledge has been selected at present, then the input of the dialogue model is $X_c=\left[\texttt{[SOS]};U;K_{a_1};...;K_{a_{c-1}};\texttt{[EOS]}\right]$, where \texttt{[SOS]} and \texttt{[EOS]} are special tokens.
We pass $X_c$ to Transformer encoder to obtain the context-aware representation $\mathbf{X}_c$:
\begin{equation}
    \mathbf{X}_c = \text{TFEncoder}\left(e(X_c), \theta_e \right) \in \mathbb{R}^{m \times d}
    \label{eq:encodeutt}
\end{equation}
where $e(\cdot)$ means to convert a sequence (or token) into word embedding vectors using a lookup table $\mathbf{M} \in \mathbb{R}^{vocab \times d}$ and $\theta_e$ represents the learnable parameters. After encoding, the information of dialogue history and external knowledge is fully integrated.

We take $\mathbf{X}_c$ (as memory) and golden response $Y$ (as input) into Transformer decoder to generate response token by token.
\begin{equation}
    \mathbf{h}_t = \text{TFDecoder}\left(\mathbf{X}_c, e(Y_{1:t-1}),\theta_d \right) \in \mathbb{R}^{d} \quad p(y_t) = \frac{e(y_t)\mathbf{h}_t^\top}{\sum_{y^\prime} e(y^\prime)\mathbf{h}_t^\top}
\end{equation}

\subsection{Knowledge Encoder}\label{sec:knomodel}
For $i$-th ($1 \leq i \leq r$) knowledge $K_i$ in external knowledge set, we wrap it with two special characters \texttt{[SOS]} and \texttt{[EOS]}, and sent to a Transformer encoder:
\begin{equation}
    \mathbf{K}_i = \text{TFEncoder}\left(e(K_i), \theta_k \right) \in \mathbb{R}^{n \times d}
    \label{eq:encodekno}
\end{equation}
We use the representation of \texttt{[SOS]} token (denoted as $\mathbf{k}_i \in \mathbb{R}^{d}$) as the overall representation of $i$-th knowledge.

\subsection{Knowledge Selector}\label{sec:knoselmodel}

Assuming that we will choose the $c$-th knowledge in current step, we formulate the problem of learning-to-select under the framework of reinforcement learning. We define the \textbf{state} $s=\{K_{a_1}, ... , K_{a_{c-1}}\}$ (omit some constants) of the model to be the knowledge that the selector has selected before the current step, e.g., $s=\{3, 2, 1, 4\}$.
The \textbf{action} $a$ is which knowledge to select.
We define the \textbf{action space} of current step as the knowledge index in $\mathcal{K}$ (i.e., \{1,2,...,r\}). During implementation, we use mask technology  to avoid choosing repeated action.
Since $\mathbf{X}_c$ integrates the features from dialogue history and selected knowledge, in order to propagate the information across different steps, we represent the state $s$ with the representation of \texttt{[SOS]} token (denoted by $\mathbf{v}_c \in \mathbb{R}^d$) in $\mathbf{X}_c$.

The selection policy gives the probability $\pi \left(a \mid s\right)$ of taking an action $a$ at the current state $s$, which is modeled by a bilinear matrix:
\begin{equation}
    \pi \left(a \mid s\right) = \mathop{softmax}_{a \in \mathcal{A}_c}\left(\mathbf{v}_c \mathbf{W} \mathbf{k}_a^\top\right)
    \label{eq:act}
\end{equation}
where $\mathbf{W} \in \mathbb{R}^{d \times d}$ is a learnable parameter and $\mathbf{k}_a$ is the knowledge representation ($\S$ \ref{sec:knomodel}). In order to achieve the early stop mechanism, a stop action can be added to action space in implementation.
\subsection{Modules Integration} \label{sec:traininfer}

In the training phase of the recurrent selection mechanism, the actions of selecting next knowledge are sampled according to the probability given by the selection policy. Our model generates a sequence of knowledge for each dialogue history. We train the dialogue model with supervised learning, and we train the selector network via reinforcement learning and weekly supervised learning.

\subsubsection{Supervised Learning for Response Generation} The ground-truth response has been given, so we train the conversation model via supervised learning. Consistent with most Seq2Seq models, the training objective is to minimize the negative log likelihood (NLL):
\begin{equation}
    \mathcal{L}_u = -\frac{1}{z}\sum_{t=1}^z \log p (y_t)
    \label{eq:diagloss}
\end{equation}

\subsubsection{Reinforcement Learning for Selection Policy} Based on the assumption that the ground-truth knowledge labels are not available, supervised learning cannot be used to train the knowledge selection model, it is natural to train it via reinforcement learning.

First of all, the accumulated reward for taking action $a$ at state $s$ is denoted as $\text{R}(s, a)$, which is derived in a recursive manner:
\begin{equation}
    \text{R}(s, a) =  e^{-\text{PPL}_{(s,a)} / \gamma} + \text{R}(s^\prime, a^\prime)
\end{equation}
where $\gamma$ is a constant and we empirically set it to $10$. $\text{PPL}_{(s,a)}$ denotes the perplexity
of ground-truth response $Y$ in current step and $\text{R}(s^\prime, a^\prime)$ denotes the next state-action pair. We use perplexity as reward to make the knowledge selector more inclined to select knowledge that can generate high-quality responses.

The selection policy network can be trained by maximizing the expected accumulated reward through the policy gradient algorithm ~\cite{DBLP:journals/ml/Williams92}:
\begin{equation}
    \mathcal{J} = \mathbb{E}_{a\sim \pi}\left[\tilde{\text{R}}(s, a)\right]
    \label{eq:accreward}
\end{equation}
where $\tilde{\text{R}}(s, a)=\text{R}(s, a)-b$ and $b \approx \mathbb{E}[\text{R}(s, a)]$ is the baseline that is
used to reduce the variance of gradient estimation~\cite{DBLP:conf/emnlp/ClarkM16}.
To be consistent with the notations in response generation, we denote the loss function of selection policy as $\mathcal{L}_k$, which is the negative expected accumulated reward $\mathcal{J}$ in Eq. \ref{eq:accreward}: $\mathcal{L}_k = -\mathcal{J}$. Thus, the gradient of $\mathcal{L}_k$ over a series of action-pair $\mathcal{B}$ is given by:
\begin{equation}
    \nabla \mathcal{L}_k = - \sum_{(s,a)\in \mathcal{B}}\nabla \log \pi \left(a \mid s\right) \text{R}(s, a)
\end{equation}
\subsubsection{Weakly Supervised Learning for Selector}
Valuable knowledge usually has a higher text similarity with the ground-truth response, so we add an additional selection loss to the selector when selecting the first knowledge. We take the response as the query, use the TF-IDF algorithm to score the knowledge in $\mathcal{K}$, mark the index of knowledge with the highest score as $a^+$, and the additional loss is defined as:
\begin{equation}
    \mathcal{L}_s = -\log \pi \left(a^+ \mid s\right)
    \label{eq:knoloss}
\end{equation}

\subsubsection{Training and Inference} The training procedure is show in Algorithm \ref{alg:rlconv} (for ease of understanding, assume batch size is 1 and only optimize one step).
The overall loss is defined as $\mathcal{L} = \mathcal{L}_u^\prime /o+ \mathcal{L}_k + \lambda \mathcal{L}_s$. Weakly supervised labels may not be correct, so we put a smaller weight $\lambda$ on $\mathcal{L}_s$.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=7.5cm]{imgs/struct_eval}
    \caption{Sketch of inference.}
    \label{fig:struct_eval}
\end{figure}
During inference, the knowledge selector take the best action (instead of sampling) at step $c$ according to the selection policy:
\begin{equation}
    a^* = \mathop{\arg\max}_{a \in \mathcal{A}_c} \pi  \left(a \mid s\right)
\end{equation}
After the action $a^*$ is taken, a new knowledge is taken from the knowledge pool and appended to the input sequence, and so on, until reaching the upper bound of knowledge selection $o$. Because there is no need to calculate the reward during inference, in order to save overhead, the decoder only decodes when $c=o$. The sketch of inference is shown in Fig  \ref{fig:struct_eval}.

\begin{algorithm}[tb]
\small
    \caption{Optimization Step}
    \label{alg:rlconv}
    \textbf{Input}: $U$, $Y$,  $\mathcal{K}=\{K_i\}_{i=1}^r$ and number of selections $o$;\\
    \textbf{Output}: Generation loss $\mathcal{L}_u^\prime$, selection loss $\mathcal{L}_k$ and $\mathcal{L}_s$;
    \begin{algorithmic}[1]
        \STATE Encode knowledge independently according to Eq. \ref{eq:encodekno} and get the knowledge representation $\{\mathbf{k}_i\}_{i=1}^r$;
        \STATE Initialize $s=\phi$, $\mathcal{B}=\phi$, $\mathcal{L}_u^\prime=0$;
        \FOR{$c$ \textbf{in} $\{1,...,o\}$}
        \STATE Construct $X_c$ from U and $s$ according to $\S$ \ref{sec:diagmodel};
        \STATE Encode $X_c$ to obtain $\mathbf{X}_c$ and $\mathbf{v}_c$ according to Eq. \ref{eq:encodeutt};
        \STATE Decode and calculate $\mathcal{L}_u$ according to Eq. \ref{eq:diagloss};
        \STATE $\mathcal{L}_u^\prime \gets \mathcal{L}_u^\prime + \mathcal{L}_u$;
        \IF {$c=1$}
        \STATE Calculate $\mathcal{L}_s$ according to Eq. \ref{eq:knoloss};
        \ENDIF
        \IF {$c<o$}

        \STATE Calculate $\pi$ according to Eq. \ref{eq:act};
        \STATE Sample an action $a$ using Gumbel-Max Trick;
        \STATE $\mathcal{B} \gets \mathcal{B} \cap \{(s, a)\}$, $s \gets s \cap \{a\}$;
        \ENDIF
        \ENDFOR
        \STATE Calculate $\mathcal{L}_k$ according to Eq. \ref{eq:accreward};

        \STATE \textbf{return} $\mathcal{L}_u^\prime$, $\mathcal{L}_s$ and $\mathcal{L}_k$;
    \end{algorithmic}
\end{algorithm}

\section{Experiments}
\subsection{Datasets}
We conduct experiments on Wizard-of-Wikipedia~\cite{dinan_wizard_2019} and Holl-E~\cite{DBLP:conf/emnlp/MogheABK18}. Wizard-of-Wikipedia contains 18,430 training dialogues, 1,948 validation dialogues and 1,933 test dialogues on 1365 topics.  And test set is split into two subsets according to topics, which are Test Seen with 965 dialogues and Test Unseen with 968 dialogues whose topics are never seen in training and validation set. There are about 61 sentences on average in the knowledge pool per turn, which are retrieved from Wikipedia based on the context.
Holl-E contains 7,228 training dialogues, 930 validation dialogues and 913 test dialogues. Each dialogue is assigned with about 60 documents on average as the knowledge pool. Here, we use the modified version ~\cite{kim_sequential_2020} which fits for knowledge selection.

It should be noted that in this paper, we focus on the scenarios where ground-truth knowledge is unknown. Thus, we does not use the ground-truth knowledge labels.

\subsection{Models for Comparison}
We compare our Knoformer with the following baselines:
\begin{itemize}
    \item \textbf{KnowledGPT}: Another knowledge-based dialogue system based on BERT and GPT-2 which can select multiple external knowledge via reinforcement learning~\cite{zhao_knowledge-grounded_2020}. KnowledGPT does not provide complete source code and experimental results on other datasets, so we can only compare with it on the Wizard-of-Wikipedia dataset.
    \item \textbf{TF-IDF}: TF-IDF is a commonly used algorithm in information retrieval. We perform TF-IDF to sort all documents in knowledge set (dialogue context as query), and concatenate dialogue context and top-$20$ of them as the input.
    \item \textbf{PIPM/KDBTS}: A latent variable model that uses specially designed Posterior Information Prediction Module (PIPM) to select knowledge and Knowledge Distillation Based Training Strategy (KDBTS) to train the decoder with the knowledge selected from the prior distribution~\cite{chen_bridging_2020}. It is an improvement of PostKS and SKT, but still can only select one knowledge.
\end{itemize}
For fair comparison, we only use the knowledge selector of the above methods, and use BART~\cite{lewis_bart_2020} as the unified dialogue response generator. It should be noted that most pre-trained language models (e.g., BERT, GPT-2, BART) has a limit of max number of input tokens they can handle, so for BART+TF-IDF, text exceeding the maximum length will be truncated, and only keep the first 384 tokens. The source code of KIC and some other methods is not available, so we will not compare with them.

\subsection{Implement Details}
We implement our model over PyTorch framework.
The parameters of dialogue module and knowledge encoder are initialized with BART-base and BERT-base respectively.
We train our model using AdamW \cite{DBLP:journals/corr/abs-1711-05101} optimizer with a batch size of 16 and learning rate 5e-5 at 3 epochs on a NVIDIA QUADRO RTX 8000 machine, and other hyperparameters are detailed in Table \ref{tab:hyper}.
When decoding, the number of beams is between 1 to 5.
We adopt widely used public evaluation toolkit NLTK to evaluate the model performance.
\begin{table}[htbp]
    \caption{Hyperparameters over different datasets.}
    \small
    \centering
    \setlength{\tabcolsep}{2mm}{
        \begin{tabular}{lccc}
            \toprule
                                    & Wizard seen & Wizard Unseen & Holl-E \\ \midrule
            Max Context Length      & 384         & 384           & 256    \\
            Max Knowledge Length    & 64          & 64            & 48     \\
            Max Response Length     & 48          & 48            & 48     \\
            Max Number of Selection & 6           & 3             & 2      \\
            $\lambda$ & 0.3           & 0.7             & 0.7      \\
            \bottomrule
        \end{tabular}}
    \label{tab:hyper}
\end{table}

\subsection{Automatic Evaluation}
We adopt three automatic metrics include BLEU, Div, and corpus-level unigram F1 for evaluation. BLEU is well-known machine translation evaluation metric, and generated response with higher BLEU / F1 is closer to the ground-truth response and has preferable fluency. Div-$n$
reflects the $n$-gram diversity of text \cite{DBLP:conf/naacl/LiGBGD16}, and
 response with higher Div-$n$  could present more information.
\begin{table}[htbp]
    \caption{Automatic evaluation results on Wizard-of-Wikipedia test seen. BART in the first row means no knowledge is used.}
    \centering
    \small
    \setlength{\tabcolsep}{2mm}{
        \begin{tabular}{lcccccc}
            \toprule
            Model           & F1   & BLEU-1 & BLEU-2 & BLEU-3 & Div-1 & Div-2 \\ \midrule
            BART            & 24.0 & 22.4   & 10.2   & 5.0    & 6.0   & 23.8  \\
            BART+KnowledGPT & 26.2 & 25.6   & 13.6   & 8.8    & 7.8   & 24.6  \\
            BART+TF-IDF     & 24.8 & 23.3   & 11.0   & 5.9    & 7.1   & 29.8  \\
            BART+PIPM       & 25.0 & 23.6   & 11.3   & 6.1    & 7.0   & 28.7  \\\midrule
            Knoformer       & \textbf{27.4} & \textbf{26.4}   & \textbf{14.3}   & \textbf{9.2}    & \textbf{7.9}   & \textbf{32.1}  \\ \bottomrule
        \end{tabular}}
    \label{tab:wowseen}
\end{table}

\begin{table}[htbp]
    \caption{Automatic evaluation results on Wizard-of-Wikipedia test unseen.}
    \centering
    \small
    \setlength{\tabcolsep}{2mm}{
        \begin{tabular}{lcccccc}
            \toprule
            Model           & F1   & BLEU-1 & BLEU-2 & BLEU-3 & Div-1 & Div-2 \\ \midrule
            BART            & 23.1 & 21.6   & 9.5    & 4.7    & 4.5   & 19.6  \\
            BART+KnowledGPT & 24.7 & 24.6   & \textbf{12.6}   & 7.8    & 4.9   & \textbf{23.6}  \\
            BART+TF-IDF     & 23.5 & 22.3   & 10.0   & 5.4    & 5.1   & 21.4  \\
            BART+PIPM       & 24.0 & 23.0   & 10.5   & 5.6    & 4.7   & 20.8  \\\midrule
            Knoformer       & \textbf{25.4} & \textbf{24.8}   & \textbf{12.6}   & \textbf{8.0}    & \textbf{5.1}   & 23.1  \\ \bottomrule
        \end{tabular}}
    \label{tab:wowunseen}
\end{table}
Table \ref{tab:wowseen} and Table \ref{tab:wowunseen} shows the automatic evaluation results on Wizard, and Table \ref{tab:holle} shows the results on Holl-E.
We have the following observations:
(1) Our Knoformer significantly surpasses all baselines in most evaluation metrics of all datasets, which means our knowledge selection module is more targeted and can select more valuable knowledge;
(2) External knowledge is of great help to improve performance. If external knowledge is removed, performance will decline;
(3) Only using prior experience (TF-IDF with context as query) to select knowledge is not effective.

\begin{table}[htbp]
    \caption{Automatic evaluation results on Holl-E.}
    \centering
    \small
    \setlength{\tabcolsep}{2mm}{
        \begin{tabular}{lcccccc}
            \toprule
            Model       & F1   & BLEU-1 & BLEU-2 & BLEU-3 & Div-1 & Div-2 \\ \midrule
            BART        & 24.7 & 20.6   & 11.5   & 5.2    & 4.0   & 16.1  \\
            BART+TF-IDF & 27.8 & 25.0   & 11.0   & 5.9    & \textbf{7.1}   & \textbf{28.3}  \\
            BART+PIPM   & 36.3 & 33.4   & 27.5   & 24.6   & 4.6   & 20.8  \\\midrule
            Knoformer   & \textbf{39.5} & \textbf{37.9}   & \textbf{31.0}   & \textbf{28.4}   & 7.0   & 25.8  \\ \bottomrule
        \end{tabular}}
    \label{tab:holle}
\end{table}
\subsection{Human Evaluation}
Besides automatic evaluation, we also recruit three human annotators to do qualitative analysis on response quality. For each corpus, we randomly sample 200 samples, and each sample contains the dialog history, response, and external knowledge set. The annotators then judge the quality of the responses from three aspects, including context coherence, language fluency and response diversity, and assign a score in \{0, 1, 2\} to each response for each aspect. Each response receives 3 scores per aspect, and the agreement among the annotators is measured via Fleiss’ kappa~\cite{fkappa}. The human evaluation result is shown in Table \ref{tab:humaneval}, and we observe that responses from our Knoformer are more fluent and more contextually coherent than those from baselines.
\begin{table}[htbp]
    \caption{Human evaluation results on Wizard-of-Wikipedia. }
    \centering
    \small
    \setlength{\tabcolsep}{1.8mm}{
        \begin{tabular}{lcccccccc}
            \toprule
            \multirow{2}{*}{Models} & \multicolumn{4}{c}{Wizard Test Seen} & \multicolumn{4}{c}{Wizard Test Unseen}                                             \\ \cmidrule(lr){2-5}\cmidrule(lr){6-9}
                                    & CC                                   & LF                                     & RD   & Kappa & CC   & LF   & RD   & Kappa \\ \midrule
            BART+KnowledGPT         & 1.78                                 & 1.80                                   & 1.64 & 0.61  & 1.72 & 1.74 & 1.66 & 0.62  \\
            BART+PIPM               & 1.73                                 & 1.75                                   & 1.65 & 0.59  & 1.69 & 1.70 & 1.62 & 0.61  \\
            Ours                    & 1.80                                 & 1.84                                   & 1.69 & 0.60  & 1.74 & 1.77 & 1.66 & 0.61  \\\bottomrule
        \end{tabular}}
    \label{tab:humaneval}
\end{table}
\subsection{Analysis}

\subsubsection{Ablation study.} In order to explore the importance of components in Knoformer, we conducted ablation experiments on Wizard-of-Wikipedia valid set, and the results are summarized in Table \ref{tab:as}.
First of all, we remove the loss item $\mathcal{L}_k$ and $\mathcal{L}_s$ of knowledge selection, making the selection of knowledge completely random. Results show that meaningless selection will harm the performance of the model.
Secondly, we remove $\mathcal{L}_s$ only, and performance is also degraded, which means that valuable knowledge does have a higher similarity with the ground-truth response, and building the link between knowledge and response directly helps improve model performance.
Besides, we change the joint training to separate training, which means that selector and dialogue module are trained alternately. Results indicate that joint training can make better use of the feedback of dialogue model.
In addition, when replacing our reward function with that of KnowledGPT (only return a reward for all actions in an episode), the performance drops significantly, which means that gives a reward to each action can effectively punish bad actions and enhance the knowledge selection.

\begin{table}[htbp]
    \caption{Results on Wizard valid set.}
    \centering
    \small
    \setlength{\tabcolsep}{2.2mm}{
        \begin{tabular}{lcccccc}
            \toprule
            \multirow{2}{*}{Models} & \multicolumn{3}{c}{Wizard Test Seen} & \multicolumn{3}{c}{Wizard Test Unseen}                                   \\ \cmidrule(lr){2-4}\cmidrule(lr){5-7}
                                    & F1                                   & BLEU-1                                 & BLEU-3 & F1   & BLEU-1 & BLEU-3 \\ \midrule
            Ours                    & 27.1                                 & 26.5                                  & 9.2   & 25.3 & 24.6   & 8.2   \\
            -$\mathcal{L}_k,\mathcal{L}_s$        & 24.6                                 & 23.1                                   & 5.9   & 23.6 & 22.1   & 5.3   \\
            -$\mathcal{L}_s$        & 26.8                                 & 26.3                                   & 8.5   & 24.9 & 24.1   & 7.6   \\
            -joint                  & 26.9                                 & 26.3                                   & 9.1   & 25.1 & 24.3   & 8.0   \\
            $\text{R}^\dagger$       & 26.2                                 & 25.7                                   & 8.2   & 24.1 & 23.4   & 7.3   \\ \bottomrule
        \end{tabular}}
    \label{tab:as}
\end{table}

\subsubsection{Case study.} Table \ref{tab:case1} shows a case in Wizard test unseen. Our model select two pieces of knowledge and incorporate them into response. Compared with baselines and even reference, response generated by Knoformer is more informative.
\begin{table}[htbp]
\scriptsize
\centering
\caption{A case from test unseen of Wizard-of-Wikipedia.}
\begin{tabular}{ll}
\toprule
\begin{tabular}[c]{@{}l@{}}Context\end{tabular}    & \begin{tabular}[c]{@{}l@{}}A: Elvis was such an amazing singer, but he also an incredible musician and actor. \\B: Yes, The King. Did you know he sold more than 1 billion units in 20 years.\end{tabular}
        \\ \midrule
 Reference    & \begin{tabular}[c]{@{}l@{}}I didn't know that! He was born in 1935 and died in 1977 though.\end{tabular}           \\ \midrule
\begin{tabular}[c]{@{}l@{}} Selected\\Knowl.\end{tabular}       & \begin{tabular}[c]{@{}l@{}}1. Elvis Aaron Presley (January 8, 1935 – August 16, 1977) was an American singer... \\ 2. ... he is often referred to as the King of Rock and Roll or simply the King.
\end{tabular}
        \\ \midrule
\multicolumn{2}{l}{\begin{tabular}[c]{@{}l@{}}
\textbf{(Ours)} Yes, he was king of rock and roll. He was born in 1935 and died in 1977.  \\
\textbf{($\text{BART}$+TF-IDF)} Yes, he sold over 100 million records worldwide. \\
\textbf{($\text{BART}$+PIPM)}  I did not know that.  I do know that he founded the Wall Street firm in 1960.
\end{tabular}}                                                                                \\ \bottomrule
\end{tabular}
\label{tab:case1}
\end{table}
\subsubsection{Impact of $o$.} To explore the influence of the number of knowledge choices on model performance, we vary the value of $o$ in $\{1, 2, ..., 12\}$ and report the evaluation results in Fig \ref{fig:io}. The smaller $o$, the smaller the probability that ground-truth knowledge will be captured. When $o$ reaches a certain extent, the performance improvement is very weak or even slightly decreased, implying that the noise in the knowledge pool will interfere with the generation of responses.

\begin{figure}[htbp]
    \center
    \includegraphics[width=12cm]{imgs/impo}
    \caption{The performance of the model with different $o$ in Wizard valid set (seen).}
    \label{fig:io}
\end{figure}
\section{Conclusion}
In this paper, we propose Knoformer, a knowledge-grounded dialogue generation model.
Evaluation results on four benchmarks indicate that our model can significantly outperform state-of-the-art methods.
\section*{Acknowledgments}
This work is supported by the National Key R\&D Program of China (No.2018YFC0830701), the National Natural Science Foundation of China (No.61572120), the Fundamental Research Funds for the Central Universities (No.N181602013 and No.N171602003), Ten Thousand Talent Program (No.ZX20200035), and Liaoning Distinguished Professor (No.XLYC1902057).
\bibliographystyle{splncs04}