\section{Introduction}

Electrides are a novel class of crystals with vacant cavities accommodating intrinsic excess electrons, which are not tightly constrained by certain nuclei, but on the contrary, form ionic bonds with the cationic framework, playing the role of anions \cite{dye1990663, dye2003electrons}. In contrast to random defects or color centers, sites of interstitial electrons in electrides adhere to the translational periodicity. Since the experimental synthesis of the first room-temperature stable inorganic electride [Ca$_{24}$Al$_{28}$O$_{64}$]$^{4+}(4e^-)$ (C12A7:2e$^-$) \cite{matsuishi2003high}, electrides have attracted great attention among the communities of materials science, chemistry and condensed matter physics, owing to the intriguing properties exhibited by non-nucleus-bound electrons. For instance, in magnetic electrides the spin-unpaired anionic electrons are well suited for spin injection in a spintronic device \cite{sui2019prediction, wolf2001spintronics}, on account of their low work function. In addition, the interstitial electronic states are found to favor band inversion near the Fermi level, giving rise to the high tendency for electrides towards topological phases \cite{hirayama2018electrides, zhu2019computational}. One could therefore implement electrides as an excellent testbed to study topological physics \cite{PhysRevLett.120.026401, huang2018topological, PhysRevLett.123.206402, zhang2019topological, nie2020application}.

Hitherto, known electrides especially magnetic electrides, however, are extremely scarce\cite{liu2020electrides}. Consecutive efforts have been devoted by a number of researchers to the computational discovery of new electrides. Approaches of the mainstream involve: structure screening from the materials database \cite{zhu2019computational, PhysRevX.4.031023, Tada2014High-Throughput, Burton2018High-Throughput}, crystal structure prediction (CSP) for unknown prototype structures \cite{tsuji2016structural, ming2016first, zhang2017computer, wang2017exploration}, and functionality-tailored electrides design \cite{PhysRevLett.120.026401, PhysRevMaterials.3.024205}. Here we present an alternative computational strategy to design new electrides. It is widely noticed that every natural electride corresponds to a potential oxidized form, such as Ca$_2$N to Ca$_2$NCl, Y$_2$C to Y$_2$H$_2$C. One is therefore motivated to ask whether the reverse direction holds \cite{liu2020electrides}. In other words, if the anions with large electronegativity is taken away from a selected compound (e.g. remove Cl from Ca$_2$NCl), the remaining structure (Ca$_2$N) is suspected to be a possible electride.
To ascertain this question, we mainly focus on ternary halides containing trivalent rare-earth elements, i.e. scandium, yttrium, and lanthanides. Optimizing the reduced structure using a CSP method, a stable or metastable crystal is expected to be the candidate of a binary electride. Chemical substitution of magnetic lanthanides is followed to induce spin-polarized interstitial electronic states, by which magnetic electrides can be constructed. Starting from 11 halides serving as structure hosts, our method predicts 30 thermodynamically stable and metastable nonmagnetic electrides, 29 of which are newly designed. While retaining the thermodynamic (meta)stability, we replace cationic elements with the lanthanide Gd to induce spin polarization for these crystals. This chemical substitution procedure successfully generates 28 candidate structures of magnetic electrides, with 27 of them proposed for the first time. Electronic properties of two representative systems are analyzed in depth. In addition to electride and magnetism features, topological nodal lines in the electronic bands of these two systems are unveiled as well, further confirming the intimate relationship between electrides and topological materials.

\begin{figure}[htp]
	\centering
	\includegraphics[width=0.95\linewidth]{./figures/flowchart.pdf}
	\caption{Flowchart of designing halide-derived electrides.}
	\label{flowchart}
\end{figure}

\section{Methods}
\subsection{Design scheme}
\label{subsec_design_scheme}

The procedure of our design scheme is presented in Figure \ref{flowchart}, beginning with the selection of halides as electride hosts. For the sake of concreteness, we pick 11 promising ternary halides as host crystals (see Appendix \ref{sec_allcrystals}), which can be classified into 4 stoichiometric families, $A_2$C$X_p$, $A$C$X_p$, $A_2$Ge$X_p$ and $A$Ge$X_p$, where $A$ and $X$ stand for rare-earth and halogen elements, respectively; $p$ be the ratio of mole fraction of the halogen. The halogen atoms in these materials are surrounded by cations, so that the exclusion of halogens gives rise to an intrinsic excess of electrons as well as interstitial sites to accommodate those excess electrons, both of which are necessary for electride formation \cite{zhu2019computational}. We have restricted our scope to rare-earth based halides, considering that rare-earth elements are favorable electron donors in electrides because of their low eletronegativity and standard valence (usually +3). Moreover, close radii among rare-earth atoms form a good condition for chemical substitution (e.g. Y $\rightarrow$ Gd). Owing to intermediate electronegativities (2.55 and 2.01, respectively), C and Ge from group \rom{4}A are chosen as anionic elements, so that electrons could deviate from cationic atoms, and reside in interstitial orbitals at the same time \cite{wang2017exploration}. Si with the electronegativity 1.90 is not considered here, as it is less electronegative compared to C and Ge. In fact, electrons are barely attracted by rare-earth atoms even in the case of Ge, see Sec. \ref{subsec_overview}. Thus, the consideration of Si, which is less likely to attract electrons, is obsolete. We point out that the materials screening here is merely based on rules of thumb, which cannot rule out the possibility of potential electride hosts that disobey these conditions, but would always be an efficient guide of searching electrides \cite{zhang2017computer}.

Given the selected halides, we generate initial structures for the CSP. Taken away the halogens, the corresponding 4 categories of binary compounds are $A_2$C, $A$C, $A_2$Ge and $A$Ge, or collectively designated by $A_nB$ ($n=1,2; B=$ C, Ge). Further, the original cation element $A$ is replaced with Y and La separately to develop 2 groups of compositions for each category. For clarity and consistency, throughout the paper we will insist on a classification in four stoichiometric categories, i.e. the member of one category is independent of the chemical nature of $A$. Structure optimization and comparison would be conducted on the Y and La groups of materials individually for each category of $A_nB$. Since neither of Y and La has partially filled 4\textit{f} orbitals, the CSP is entirely freed from spin-polarized calculations to reduce the computational costs. Another reason for this replacement is related to the chemical substitution of magnetic lanthanides, which will be elaborated below.

CSP then follows to eliminate the structure instability caused by the exclusion of halogen atoms. From the viewpoint of CSP, the reduced structure of a known material acts as a reasonable initial guess, assisting CSP to reach the ground state more efficiently, in contrast to random starting structures \cite{zhang2017computer}. Notice that although the size of interstitial regions might be compressed during the structure optimization, proper vacant spaces adopting anionic electrons are often found among energy-low-lying structures in practice.

Subsequent to the CSP, convex hull diagrams are constructed in order to estimate the stability and potential synthesizability of predicted structures relative to competing phases with different stoichiometries. The most stable structures whose energies above the convex hull (denoted as $E_\text{hull}$) are smaller than 0.20 eV/atom \cite{oganov2019structure, wang2017exploration} are processed for electride identification, in order not to rule out the synthesis possibility of metastable polymorphs \cite{PhysRevLett.116.075503, sun1600225thermodynamic}. Under each category of $A_nB$, the first five promising electrides with the lowest energies are kept as final candidates for both $A=$ Y and La. We note that $E_\text{hull}$ of these candidates mostly turn out to be smaller than 0.10 eV/atom (see Sec. \ref{subsec_overview}).

Next, for every stoichiometric category of potential electrides $A_nB$, the cations of 10 candidate structures (5 from Y-based systems and 5 from the La-based) are substituted by magnetic lanthanides (we focus on Gd in this work) so that magnetism could be induced. As the ionic radius of Gd is between the ones of Y and La, it is expedient to assume stable or metastable Gd-based phases could be found among structures inherited from those of Y and La, especially in the case that Y$_nB$ and La$_nB$ share an identical crystal structure. This is the other reason why Y and La are simultaneously introduced at the stage of generating initial structures. In addition to the close radii of substitution elements, the stability is also assessed by constructing convex hulls. For simplicity, only ferromagnetic (FM) configurations of each Gd-based electride candidates are computed. At the same time, anionic electrons easily become spin-unpaired in the case of FM configurations of Gd. Applied to all other lanthanides with in-between ionic radii, this element substitution strategy could massively explore stable or metastable phases of lanthanide-based compounds without an expensive CSP for every lanthanide. This is a key point of our work.

Finally, having all the predicted electrides, we concentrate on the structures of interest for detailed analysis, including the dynamical stability, further confirmation of electride features, etc. The magnetic ground state of selected Gd systems is computed (see Sec. \ref{subsec_comp_methods}), estimating the relative stability of the FM state compared to other possible magnetic states. Also, we particularly characterize the topological nature encoded in electronic bands of the designed electrides. Notice that these procedures are applied to only a subset of predicted crystals, as given in the last box of Figure \ref{flowchart}.

\subsection{Computational methods}
\label{subsec_comp_methods}

All the electronic structure calculations are performed with the density functional theory (DFT) based on the projector augmented wave scheme \cite{PhysRevB.59.1758} as implemented in the Vienna \textit{ab initio} simulation package ({\sc vasp}) \cite{PhysRevB.54.11169}. The exchange-correlation functional is given by the generalized gradient approximation (GGA) of the Perdew-Burke-Ernzerhof (PBE) type \cite{PhysRevLett.77.3865}. Strong intra-atomic interactions in 4\textit{f} orbitals of Gd are approximated by the GGA+U \cite{PhysRevB.52.R5467}. The nonlocal Heyd-Scuseria-Ernzerhof (HSE06) hybrid functional is supplemented to check the band crossing \cite{heyd2003hybrid}. The energy cutoff of the plane wave is chosen as $30\%$ above of the maximum value recommended in the pseudopoetential library. The Brillouin zone is sampled by a $\Gamma$-centered grid with the resolution of $2\pi \times 0.03 \text{ \AA}^{-1}$. All atoms are fully relaxed with a force criterion of 0.001~eV/{\AA}. Pre/Post-processing tools and utilities for solids computation \cite{wang2019vaspkit, herath2020107080, setyawan2010299, togo2018texttt, momma2008vesta} are employed.

Crystal structure searches for various compositions of electrides are performed with the minima hopping (MH) \cite{goedecker2004minima} algorithm with a generalized version  to periodic systems employing variable cell shape molecular dynamics \cite{amsler2010crystal,flores2020crystal}. This is a well-tested methodology that aims at finding the global minimum of a complex condensed matter system \cite{flores2020perspective}. For crystals with a small number ($<10$) of atoms per primitive cell, the supercells composed of two even three primitive cells are included as well to extend the searching space. As aforementioned, not only the lowest energy configuration, but also neighboring local minima are evaluated for their potential as electrides. In order to build convex hulls for final candidates, stable phases recorded in Materials Project \cite{Jain2013Materials, Ong2008Li} are adopted. The formation energy is estimated with respect to the corresponding stable elemental polymorph, where the {\sc vasp} pseudopotential C\_$\,$h is employed for all carbon-based systems, as some $A_n$C compounds contain $C_2$ dimers; the plane wave energy cutoffs are 875.0 eV and 320.6 eV for C- and Ge- based materials, respectively. To assess the lattice dynamics, we use the supercell approach, on which the total energy and pattern of displacements (within the harmonic approximation) are evaluated with {\sc vasp}. The harmonic interatomic force constants are built and diagonalized using the {\sc phonopy} software \cite{togo2015first}.

We utilize a recently developed scheme termed CMP+SDFT \cite{huebsch2020benchmark}, that is based on a combination of the cluster multipole (CMP) theory \cite{PhysRevB.95.094406, PhysRevB.99.174407} and calculations in the framework of spin-density functional theory (SDFT), in order to explore the possible stable magnetic states for Gd-based candidates of interest. The CMP basis is an orthogonal basis set of magnetic multipole configurations that can be used to expand an arbitrary magnetic structure. By an analysis of experimental data, it has been shown that the most stable magnetic configurations in nature are linear combinations of only few CMPs. Hence, an exhaustive list of candidate magnetic configurations can be generated with the initial magnetic moments generated by CMPs and their combinations. Subsequently, SDFT calculations are exploited to seek the stable configurations. In the case that the FM state is a local minimum instead of a global one, the energy difference higher than the global minimum is employed to evaluate the energetic stability of the FM structure. We point out again that the magnetic ground state is determined only for a subset of designed magnetic electrides.

\begin{figure*}[htp]
	\centering
	\includegraphics[width=0.98\linewidth]{./figures/overview.pdf}
	\caption{A statistical overview of predicted materials. (a)-(f) Convex hull diagrams of binary compounds $A_nB$ ($A=$ Y, La, Gd; $B=$ C, Ge). Electride candidates are colored by red, and distinguished by the attached numbers that represent their energy rankings. Notice that the number 1 means the corresponding phase has the lowest formation energy merely within the predicted electride structures; there could be a non-electride structure that is more stable. Grey dots stand for the other (meta)stable structures predicted by the CSP, which showing no evidence of electrides or have relatively high formation energy. No grey dots in diagrams of the Gd systems, since the Gd-based structures are obtained by chemical substitution instead of the CSP. (g) Stability (measured by $E_\text{hull}$) distribution of 58 designed electrides. (h) New stable polymorphs ($E_\text{hull} = 0$) of each stoichiometric category discovered by the CSP. Non-electride systems are also presented.}
	\label{overvew}
\end{figure*}

Along the high-symmetry line, crystalline symmetries of energy bands are determined via \texttt{irvsp}, a software package that can obtain irreducible representations of the electronic Bloch states \cite{gao2020irvsp}. Tight-binding models in Wannier representation \cite{wu2018wanniertools} are constructed through Wannier functions without the iterative maximal-localization procedure \cite{PhysRevB.56.12847, PhysRevB.65.035109, pizzi2020wannier90}. Considering the non-nucleus-bound electronic states are mostly \textit{s}-like \cite{matsuishi2003high}, we set the \textit{s} orbital as the trial projection at interstitial sites.

\subsection{Electrides identification}
The electron localization function (ELF) \cite{becke1990simple, savin2005electron} is one of the mature theoretical descriptors of electrides \cite{liu2020electrides, zhu2019computational, zhao2016electride, dale2018theoretical, wang2017exploration}. The emergence of ELF local maxima (irregardless of its magnitude \cite{zhu2019computational}) at the vacant region indicates a potential electride. For the further identification of candidates of interest, we do the fatband analysis, namely, projecting eigenstates onto the \textit{s} orbital of empty spheres (pseudoatoms) located at the vacancy positions with ELF maxima. The wavefunction on the interstitial-state dominating bands as well as partial charge density close to the Fermi energy represent additional proof.
\vspace{1.5cm}

\section{Results}

\subsection{Overview of predicted electrides}
\label{subsec_overview}

\begin{figure*}[tbp]
	\centering
	\includegraphics[width=0.75\linewidth]{./figures/LaC_bulk.pdf}
	\caption{Electronic properties from the bulk calculation of LaC. (a) Primitive cell of LaC. Yellow bubbles represent the ELF. For a clear visualization, the ELF around atoms is not displayed. (b) The Brillouin zone of LaC. The high-symmetry points and the path chosen are highlighted in red. The $y$-direction projected Brillouin zone is indicated too. (c) Projected band structure of LaC along the path denoted in (b). The Fermi energy is set 0. As the color changes from red to blue, the contribution of interlayer states diminishes. (d) Nodal line of LaC in the reciprocal space. The left and right panels are viewed from \textit{x} and \textit{y} directions respectively. (e) The Zak phase integrated along the \textit{y} direction. Green and white separately stand for $\pi$ and 0 values. High-symmetry points in this reduced reciprocal space are also shown. (f)(g) Real branches of wavefunctions with the corresponding positions indicated in (c). The yellow and cyan mean positive and negative parts, respectively; red and blue stand for their cross-sections.}
	\label{LaC_bulk}
\end{figure*}

Within the 4 stoichiometric categories of binary compounds, $A_2$C, $A$C, $A_2$Ge and $A$Ge ($A=$ Y, La, Gd), 58 stable or metastable phases of potential electrides are identified. Among the former three categories, 56 of them are identified for the first time and hence, have not been recorded previously in the Inorganic Crystal Structure Database (ICSD) \cite{bergerhoff1983inorganic}. In $A$Ge no electride appears within the range of $E_\text{hull} < 0.20$ eV/atom, implying that Ge is less favorable for electride phases than C in the rare-earth based systems, which might be due to the smaller electronegativity. Figure \ref{overvew} (a)-(f) illustrate the distribution of predicted stable or metastable structures in convex hull diagrams. The potential electrides are highlighted in red, ranked and labeled according to their formation energies from low to high. The (meta)stable phases that are disregarded because of either showing no evidence to be electrides or yielding a high formation energy compared to other structures are marked in grey. As given in Figure \ref{overvew}(g), all the 58 electride structures have the $E_\text{hull}$ smaller than 0.14 eV/atom, and 49 ($84.5\%$) of them are below 0.10 eV/atom, revealing the possible synthesizability of these candidate structures \cite{oganov2019structure}.

Figure \ref{overvew}(h) lists the the newly discovered materials corresponding to the most stable phase ($E_\text{hull} = 0$). Two non-electride systems are attached too for reference. In the category $A_2$C, being the ground state, La$_2$C with the anti-CdCl$_2$ structure is predicted to be a promising electride. Note that the identical structure is shared by the stable phases of the Y$_2$C and the FM Gd$_2$C, while the latter two systems have been experimentally confirmed as stable 2D electrides \cite{zhang2014two, lee2020ferromagnetic}. Additionally, the FM configuration is verified to be the energetically favorable magnetic state of Gd$_2$C according to CMP+SDFT, in agreement with experiments \cite{lee2020ferromagnetic}. Therefore, the stable phases of $A_2$C represent a benchmark of our design scheme, indicating the repeatability of our CSP strategy. Moreover, the presumption of chemical substitution in Sec. \ref{subsec_design_scheme} is demonstrated to hold, i.e. stable or metastable Gd-based structures could be derived from those of Y and La.

We proceed by introducing the other stable structures in Figure \ref{overvew}(h). In the stoichiometric family $A$C, the possible electrides LaC with the space group $I4_1/amd$ (No. 141) and the FM GdC with the space group $C2/m$ (No. 12) are revealed to be the ground states, neither of which was reported previously. For the category $A_2$Ge, we find an unreported global minimum structure La$_2$Ge that belongs to the space group $Amm2$ (No. 38), though it is unlikely to be an electride. Finally, in the category $A$Ge, despite the absence of electride possibility amongst the most stable phases, a new stable polymorph of YGe is found, sharing the same structure with the known orthorhombic LaGe. Crystal structures of these stable materials are presented in Appendix \ref{sec_stable_structures}.

\begin{figure}[tbp]
	\centering
	\includegraphics[width=0.95\linewidth]{./figures/LaC_surf.pdf}
	\caption{Electronic structure of a 15-layer $(010)$ LaC slab. (a) Electronic band structure of the LaC slab. High-symmetry points in the reduced Brillouin zone are given by Figure \ref{LaC_bulk}(b) or \ref{LaC_bulk}(e). Energy is measured from the Fermi level. The red indicates a large proportion of floating electronic states that localize on the top and bottom of the slab. (b) The squared wavefunction of a surface state localized around the top layer. The corresponding point in the band structure is marked by a green circle in (a).}
	\label{LaC_surf}
\end{figure}

Relative to the corresponding stable phases, metastable phases can exhibit superior physical and chemical characteristics \cite{sun1600225thermodynamic}. In the subsequent discussion, we shall focus on two metastable systems to exemplify the crystal structures and electronic properties of the new electride candidates. One system (denoted as the $A$C system) is represented by the monoclinic LaC with $E_\text{hull} = 4.0$ meV/atom, being a promising 2D electride. The other (denoted as the $A_2$Ge system) is based on an orthorhombic structure, found in Y$_2$Ge and La$_2$Ge concurrently. These two electride systems will be thoroughly discussed in Sec. \ref{subsec_AC} and Sec. \ref{subsec_A2Ge}, respectively.

\subsection{The \textit{A}C system}
\label{subsec_AC}

\begin{figure*}[htp]
	\centering
	\includegraphics[width=0.7\linewidth]{./figures/GdC_bulk.pdf}
	\caption{Electronic properties of GdC. (a) Primitive cell of GdC. Partial charge densities around the Fermi level ($-0.05 \sim 0.05$ eV) are shown for spin up (left) and spin down (right). Yellow bubbles represent the partial charge density, the green and blue are the cross sections. (b) Electronic band structure of FM GdC for spin up (top) and spin down (bottom). The high-symmetry points are given in (c). Arrows are indicative of the degeneracies between the highest VB and the lowest CB. (c) Nodal lines of FM GdC for the spin-up (left) and spin-down (right) channels. The Brillouin zone together with high-symmetry points are depicted too. (d) The Zak phase for spin up (left) and spin down (right), integrated along the \textit{y} direction, i.e. $\Gamma$-$X$ direction. Green and white mean $\pi$ and 0 values, respectively.}
	\label{GdC_bulk}
\end{figure*}

Starting from Y$_2$Br$_2$C$_2$ \cite{simon1996supraleitung}, we predict a metastable nonmagnetic crystal LaC , with the formation energy $-0.18$ eV/atom, $4.0$ meV/atom above the convex hull, showing no imaginary phonon modes (see Appendix \ref{sec_phonon}). This is an original material not reported in the database, whose crystal structure is shown in Figure \ref{LaC_bulk}(a). LaC crystallizes in a monoclinic lattice with the space group $P2_1/c$ (No.14), and is composed of four La atoms and four C atoms in a primitive cell. Aside from the time reversal ($\mathcal{T}$) symmetry, the following two symmetries are essential: an inversion $\mathcal{P}$ whose center is at the corner of the unit cell, and an \textit{a}-axis twofold screw operation $\mathcal{S}_{2[100]}$. Indicated by the ELF, LaC is a promising 2D electride, where excess electrons are trapped between cationic layers [Figure \ref{LaC_bulk} (a)].

We calculate the electronic band structure along the high-symmetry path given in Figure \ref{LaC_bulk}(b), in the absence of spin-orbit coupling (SOC). Entangled bands around the Fermi level reveal the metallic behavior of LaC. Placing pseudoatoms with the Wigner-Seitz radius 1.50 \AA$\ $at positions with local ELF maxima, i.e. fractional coordinates (0, 0.5, 0) and (0.5, 0.5, 0.5), yields the contribution of anionic electrons to the Bloch wavefunctions. One can notice that interstitial states are closely related to bands around the Fermi energy $E_\text{F}$ and show conducting behavior. The number of valence electrons in this calculation is 60. Hence, the 30 lower bands are counted as valence bands (VBs), while the other higher bands are regarded as conduction bands (CBs). In Figure \ref{LaC_bulk} (f)(g), the real parts of two eigenstates affiliated to the highest VB and the lowest CB are depicted respectively, of which the corresponding positions are marked in the band structure of Figure \ref{LaC_bulk}(c), close to a gapless point. Non-nuclear orbitals are observed for the VB, further confirming the electride characteristics. Nevertheless, this VB claimed by the projection analysis to have little portion of interstitial states, also contains non-nuclear orbitals according to the wavefunction visualization [Figure \ref{LaC_bulk}(g)]. Such contradiction originates from the fact that those non-nuclear orbitals are accidentally distributed in the uncovered region of pseudoatoms. The projection with empty spheres therefore underestimates the contribution from anionic electrons.

Next we move to the topic of band topology. The gapless point between a VB and a CB, mentioned in the last paragraph, is located along the $A$-$X$ line that is invariant under the $\mathcal{S}_{2[100]}$ operation. This degeneracy is protected by the opposite $\mathcal{S}_{2[100]}$ eigenvalues of the two crossing bands, indicated by \texttt{irvsp}. Additionally, the VB and CB remain gapless under the HSE06 hybrid functional. Owing to the $\mathcal{PT}$ symmetry in this spinless system \cite{chan20163, fang2016topological}, this degenerate point extends to general \textit{k}-points and forms a snake-shape nodal line [Figure \ref{LaC_bulk}(d)]. In the manifold of the Brillouin zone, the disconnected nodal line segments should constitute a whole loop. It is noteworthy that this nodal line partly comes from non-nucleus-bound electronic states (e.g. the band crossing in the $A$-$X$ direction), which are insensitive to the relativistic effect of the nuclei field, therefore might exhibit a negligibly small gap even if SOC is present \cite{zhang2019topological}.

The nodal line here is protected by the $\pi$ Berry phase, defined as
\begin{equation}
	\label{Zak_phase}
	\gamma(\boldsymbol{k}_{\parallel}) = \sum_{n}^{\text{occ.}} \int_{0}^{|\boldsymbol{K}|}\mathrm{d}k_{\perp}\ i\langle u_{n\boldsymbol{k}} | \nabla_{k_{\perp}} | u_{n\boldsymbol{k}} \rangle
\end{equation}
for an arbitrary wave vector $\boldsymbol{k}$, where $\boldsymbol{K}$ stands for the reciprocal lattice vector, $k_{\perp}$ is the magnitude of $\boldsymbol{k}$ component along $\boldsymbol{K}$ and $\boldsymbol{k}_{\parallel}$ is the other component perpendicular to $\boldsymbol{K}$; $|u_{n\boldsymbol{k}}\rangle$ represents the cell-periodic Bloch function, with the gauge $|u_{n,\boldsymbol{k}+\boldsymbol{K}}\rangle = e^{-i\boldsymbol{K}\cdot\boldsymbol{r}}|u_{n\boldsymbol{k}}\rangle$. Note that the Berry phase specified in Eq. (\ref{Zak_phase}), with the integral path along a reciprocal lattice vector, is also known as the Zak phase \cite{PhysRevLett.62.2747}. Making $\boldsymbol{K}$ parallel with the \textit{y} direction of the reciprocal space, yields the Zak phase distribution displayed in Figure \ref{LaC_bulk}(e), which is quantized to be 0 or $\pi$ (mod $2\pi$) as a result of the $\mathcal{P}\mathcal{T}$ symmetry. The nontriviality of this nodal line is demonstrated by the fact that the profile of $\pi$ value region of the Zak phase just matches the nodal line projection along the \textit{y} direction [Figure \ref{LaC_bulk}(e) and the right panel of Figure \ref{LaC_bulk}(d)]. LaC is therefore a TNLE \cite{hirayama2018electrides}.

While underlying the fingerprint of nodal lines in the bulk, the Zak phase also discloses information on the surface. To elaborate this point, we construct a 15-layer (010) slab with the electron layer terminated, and compute the electronic band structure using DFT, as presented in Figure \ref{LaC_surf}(a), whose high-symmetry points in the surface Brillouin zone are given by Figure \ref{LaC_bulk}(b) or \ref{LaC_bulk}(e). Contribution from excess electrons localized on the surfaces is highlighted by projecting eigenstates onto pseudoatoms on the top and bottom ends of the slab. At the wave vector $\boldsymbol{k}_{\parallel}$ with $\pi$ Zak phase [e.g., $\bar{\Lambda}_1$ in Figure \ref{LaC_bulk}(b)], the bulk polarization is nonzero and quantized, and the topological charge $e/2$ (mod $e$) would be induced at the slab surface by this bulk polarization \cite{PhysRevB.48.4442, hirayama2017topological}. As the two surfaces of the slab are equivalent on account of the $\mathcal{P}$ symmetry, the total charge at $\bar{\Lambda}_1$ is different from that in the region with 0 Zak phase [e.g., $\bar{Z}$ in Figure \ref{LaC_bulk}(b)] by \textit{e}. The charge difference can be evidenced by the slab band structure. Along the $\bar{Z}\rightarrow\bar{\Gamma}$ direction, one unoccupied state traverses the bulk line node and merges into the valence spectrum in the end, as marked by the blue circle in Figure \ref{LaC_surf}(a). As a result, in comparison with $\bar{Z}$, there exists one more VB at $\bar{\Lambda}_1$, being responsible for the charge difference \textit{e}. In addition to the surface charge, midgap states emerging from the line node are observed. On the left side of the line node, the midgap states are dominated by interstitial orbitals. In Figure \ref{LaC_surf}(b), visualization of a midgap state in real space unveils the fact that this boundary state is not tightly attached to any atoms, but floats on the surface instead. Such floating surface states are also predicted in other TNLEs like Y$_2$C \cite{hirayama2018electrides}.

\begin{figure*}[htbp]
	\centering
	\includegraphics[width=0.75\linewidth]{./figures/Y2Ge_bulk.pdf}
	\caption{Electronic properties from the bulk calculation of Y$_2$Ge. (a) Primitive cell of Y$_2$Ge. Yellow bubbles are ELF. The isosurface value in the bottom panel is smaller than those in the top and middle panels. Connected interstitial states are highlighted by a dashed red circle in the bottom panel. (b) Projected band structure of Y$_2$Ge with the high-symmetry points denoted in (c). The Fermi energy is set 0. As the color changes from red to blue, the contribution of interstitial states decays. Some gapless points are indicated by arrows. (c) Nodal lines of Y$_2$Ge accompanied by the Brillouin zone. Two inequivalent groups of nodal lines are labels as NL1 and NL2, respectively. Spinless Weyl points are also displayed, with the orange and magenta representing the monopole charge $+1$ and $-1$ separately. (d)(e) show the real branches of the wavefunctions at the positions indicated in (b); the positive part is in yellow and the corresponding cross-sections are colored by red; the cyan and blue are for the negative part.}
	\label{Y2Ge_bulk}
\end{figure*}

We now discuss the crystal after chemical substitution Y $\rightarrow$ Gd, namely, the isostructural GdC. As described in Sec. \ref{subsec_design_scheme}, we have restricted our discussion to the FM configuration of Gd atoms, in which case the interstitial electrons can be easily magnetized. For the GdC here, although an antiferromagnetic (AFM) configuration is predicted by CMP+SDFT to be the magnetic ground state, it is not highly energetic-favorable compared to the FM state (see Appendix \ref{sec_ground_mag}). This makes our 'FM restriction' more plausible. GdC possesses a formation energy equal to $-0.22$ eV/atom and $E_\text{hull}= 0.032$ eV/atom. To look into the electronic structure, the Hubbard $U$ for Gd 4\textit{f} orbitals is included, with $U - J = 6$ eV and $J = 0.7$ eV \cite{PhysRevX.4.031023}; other values ($U - J = 3$ eV and 9 eV) are also checked, showing that only the dense flat $4f$ bands shift while other bands near the Fermi level remain unchanged. Figure \ref{GdC_bulk}(a) presents the partial charge density of the FM GdC, in the vicinity of $E_{\text{F}}$ ($-0.05 \sim 0.05$ eV) for two spin channels. While only intralayer electrons can be seen for spin up, excess electrons rest in the interlayer gap for spin down. Therefore, spin-polarized interstitial states are achieved in this FM system, although the macroscopic magnetism is dominated by Gd atoms. Figure \ref{GdC_bulk}(b) gives the electronic band structure of the FM GdC without SOC. One can notice that bands of two spin channels resemble those in LaC, as a result of the close electronic configuration between Y and Gd, and the nearly identical crystal structure. Such a similarity has been also observed in the electride systems Y$_2$C and Gd$_2$C \cite{PhysRevX.4.031023, liu2020ferromagnetic}. Hence, we expect GdC exhibits a similar electronic behavior to LaC, especially regarding the band topology. Band degeneracies around the Fermi level are displayed in Figure \ref{GdC_bulk}(b), all of which are preserved at the level of HSE06 functionals. In parallel with LaC, these degenerate points turn out to extend in the reciprocal space, forming nodal rings for both spin up and down [Figure \ref{GdC_bulk}(c)]. All the degeneracies still exist in the HSE06 band structure. The nontriviality is again verified by the Zak phase [Figure \ref{GdC_bulk}(d)]. In spite of the broken $\mathcal{T}$ symmetry, the remaining $\mathcal{P}$ symmetry still protects the $\pi$ Berry phase nodal lines. This time reversal breaking TNLE state was also reported in Gd$_2$C \cite{liu2020ferromagnetic}.

\subsection{The \textit{A}$_2$Ge system}
\label{subsec_A2Ge}

\begin{figure*}[htbp]
	\centering
	\includegraphics[width=0.7\linewidth]{./figures/Gd2Ge_bulk.pdf}
	\caption{Electronic properties of Gd$_2$Ge. (a) Primitive cell of Gd$_2$Ge. Partial charge densities around the Fermi level ($-0.05 \sim 0.05$ eV) are presented for spin up (left) and spin down (right). The yellow bubbles represent the partial charge density, and the green means the cross-section. (b) Electronic band structure of FM GdC for the spin-up (top) and spin-down (bottom) parts. The high-symmetry points are indicated in (c). Degenerate points between the highest VB and lowest CB are marked by arrows. (c) Nodal lines of FM GdC for spin up (left) and spin down (right). The Brillouin zone, together with high-symmetry points, are depicted too. The blue or green distinguishes nodal lines that are distributed on the vertical or horizontal mirror plane. Weyl points are also displayed, with the orange and magenta representing the monopole and antimonopole, respectively.}
	\label{Gd2Ge_bulk}
\end{figure*}

Y$_2$Ge is another new inorganic electride, derived from the halide La$_2$I$_2$Ge\cite{mattausch2005reduced}, with the structure metastability supported by the negative formation energy $-0.62$ eV/atom ($0.13$ eV/atom higher than the hull) and the absence of imaginary phonon frequencies (see Appendix \ref{sec_phonon}). The base-centered orthorhombic lattice of Y$_2$Ge belongs to the space group $Amm2$ (No. 38). The same structure emerges as a metastable phase of La$_2$Ge too. Figure \ref{Y2Ge_bulk}(a) depicts the primitive cell of Y$_2$Ge that contains nine Y atoms and three Ge atoms. The symmetry operations are: a twofold rotation $\mathcal{C}_{2[0\bar{1}1]}$, two mirror reflections $\mathcal{M}_{[011]}$ and $\mathcal{M}_{[100]}$, whose axes are perpendicular to each other. Notice that the inversion symmetry $\mathcal{P}$ is absent. The spatial distribution of the ELF is given in Figure \ref{Y2Ge_bulk}(a). Two ELF maxima related by the $\mathcal{C}_{2[0\bar{1}1]}$ or  $\mathcal{M}_{[011]}$ symmetry exist in the cationic cavities (the top and middle panels). These two interstitial sites are not isolated but connected with each other, which is uncovered in the bottom panel with a smaller isosurface value of ELF. Now that these interstitial states are connected through the one-dimensional (1D) channel, we classify this candidate as a 1D electride. Set pseudoatoms with the Wigner-Seitz radius 0.5 \AA$\ $at the interstitial locations with local maxima of the ELF, yielding the projected band structure of Y$_2$Ge (without SOC) given by Figure \ref{Y2Ge_bulk}(b), in which the high-symmetry points are denoted in Figure \ref{Y2Ge_bulk}(c). Eigenstates around the Fermi level mostly emanate from Y $4d$ and interstitial \textit{s} orbitals. Again the number of VBs is selected to match the half number of valence electrons in the primitive cell.

Along the $T$-$Z$-$\Gamma$-$Y$ path, the gap between the highest VB and the lowest CB disappears at four positions, which have been labeled by arrows NL1 or NL2 in Figure \ref{Y2Ge_bulk}(b). While the crossings marked by NL1 on the $T$-$Z$ and $Z$-$\Gamma$ paths are confirmed by the band structure computed with the HSE06 hybrid functional, the other two crossings NL2 on the $\Gamma$-$Y$ path disappear when employing HSE06 (see Appendix \ref{sec_hse_bands}). For completeness, we take the latter two degenerate points into account too. Since the $T$-$Z$-$\Gamma$-$Y$ line is located on the $\mathcal{M}_{[011]}$-invariant plane $(011)$, each state on this line has a well-defined eigenvalue $\pm1$ of $\mathcal{M}_{[011]}$, as verified by \texttt{irvsp}. It turns out that every pair of crossing bands have opposite mirror eigenvalues, protecting the degeneracy. In Figure \ref{Y2Ge_bulk}(d)(e), two eigenstates---one is reflection antisymmetric and the other is symmetric with respect to the (011) plane---are depicted. Their corresponding positions have been indicated in Figure \ref{Y2Ge_bulk}(b), belonging to a CB and a VB, respectively. Such opposite crystalline symmetries guarantee the band crossing in the $T$-$Z$ direction. Moreover, one of the crossing bands is dominated by anionic electrons [Figure \ref{Y2Ge_bulk}(e)], again suggesting the significant role of interstitial states for band inversion in electrides. Figure \ref{Y2Ge_bulk}(c) reveals the nodal lines stemming from those gapless points are confined within the mirror plane $(011)$. The big ring (labeled as NL1) crosses the boundary of the Brillouin zone; two small rings (labeled as NL2) are the inversion of each other, associated by the $\mathcal{T}$ symmetry. They are related to the band crossings NL1 and NL2 in Figure \ref{Y2Ge_bulk}(b), respectively. As the degenaracies on the $\Gamma$-$Y$ line disappear at the HSE06 level, the small nodal rings NL2 might be absent under an experimental observation. We then implement the $\mathbb{Z}$-type reflection invariant to characterize the mirror-symmetry protected nodal lines \cite{PhysRevB.90.205136, fang2016topological}. Within the mirror plane, we pick two points $p_a$ and $p_b$ on the two sides of the nodal line, at which the VBs and CBs are separated. Then count the number of VBs with the mirror eigenvalue $+1$, denoted by $N_{a,b}$. The reflection invariant reads
\begin{equation}
	\label{Z-invariant}
	\zeta = N_a - N_b.
\end{equation}
A nonzero $\zeta$ confirms the nontrivial nodal line, which means that the band inversion cannot be adiabatically eliminated without breaking the mirror symmetry. In our case, we obtain $\zeta_1 = \zeta_2 = 1$ for the big and small nodal rings. As a result, the stability of the line nodes in Y$_2$Ge is protected by the reflection symmetry $\mathcal{M}_{[011]}$.

Aside from the line nodes, there appear six pairs of point nodes (spinless Weyl points) owing to the absence of $\mathcal{P}$ symmetry. We compute the monopole charge (also known as the chirality) of each node using the Berry flux across an enclosed surface accommodating the node \cite{doi:10.7566/JPSJ.87.041002}. Figure \ref{Y2Ge_bulk}(c) depicts the positions and monopole charges of the Weyl points. Each pair of Weyl points are linked by the $\mathcal{T}$ symmetry, possessing the same monopole charge.

Next, we replace the Y with Gd while retaining the crystal structure, obtaining a metastable magnetic electride whose formation energy is $-0.58$ eV/atom and $E_\text{hull}=0.12$ eV/atom. Figure \ref{Gd2Ge_bulk}(a) displays the partial charge density distribution in the primitive cell, within the energy range $-0.5\text{ eV} < E-E_{\text{F}} < 0.5\text{ eV}$. Similar to GdC, interstitial states prefer the minority spin (spin down) in Gd$_2$Ge. That is, the magnetization opposite to Gd is induced among anionic electrons. Utilizing the GGA+U method, we show the electronic band structure in Figure \ref{Gd2Ge_bulk}(b) without SOC. Compared to the bands of Y$_2$Ge, the spin-up bands are lowered and the spin-down bands are lifted as expected. Degenerate points along high-symmetry lines are labeled by arrows. All the degenaracies are confirmed by the computation using the HSE06 functional, except for the right two band crossings on the $T$-$Z$ path of the spin-down band structure, which are replaced by one crossing (see Appendix \ref{sec_hse_bands}). These gapless points are guaranteed by the mirror symmetries $\mathcal{M}_{[011]}$ and $\mathcal{M}_{[100]}$. Figure \ref{Gd2Ge_bulk}(c) presents the nodal lines generated from the band crossings within the mirror planes. In contrast to Y$_2$Ge, the $k_x=0$ (spin up)  and $k_x=\pi$ (spin down) planes also hold nodal lines (highlighted in green), allowed by the $\mathcal{M}_{[100]}$ symmetry. Additionally, the perpendicular mirror planes give rise to two sets of nodal chains in the reciprocal space of spin down. In parallel with Y$_2$Ge, Weyl points are observed for both spin channels [Figure \ref{Gd2Ge_bulk}(c)]. The intriguing properties arose out of the coexistence of Weyl points and nodal chains remain to be explored.

\section{Conclusion}
In summary, we put forward a new electride design scheme, predicting electrides as the reduced form of halides. To demonstrate the success of this design strategy, three stoichiometric categories of binary compounds, $A_2$C, $A$C, $A_2$Ge ($A$ = Y, La, Gd), with 56 newly discovered stable or metastable phases derived from ternary halides, are identified as promising electrides based on first-principles crystal structure prediction and chemical substitution. Of all the candidate structures, the energies above the convex hull are smaller than $0.14$ eV/atom, indicating potential synthesizability. Benefiting from the similarities of atomic and chemical properties of rare-earth elements, stable or metastable Gd-based systems are generated by element substitution from the Y- and La-based structures so that magnetism can manifest itself. Subsequently, we study the ferromagnetic case, a situation where the anionic electrons can easily be magnetized. Two electride systems, the monoclinic $A$C and the orthorhombic $A_2$Ge, have been discussed in detail. According to our results for $A=$ Gd, their magnetic states are suitable for spin injection. The topological phases investigated show that both of the two designed electride systems manifest topological nodal lines in the reciprocal space, which cements the close relation between topological materials and electrides because of the interstitial states around the Fermi level.

Our strategy to design magnetic and topological electrides can be easily generalized to other sets of materials.  For example, one could expand the searching space to include hydrides, oxides, nitrides, etc. accessible through material databases in combination with high-throughput techniques \cite{setyawan2010high-throughput}. The replacement of other lanthanides in host systems is likely to open new avenues to synthesize magnetic electrides. One future task would be collaborating with experimentalists to synthesize the designed electrides, and devising incorporation into spintronic devices.

\begin{acknowledgments}
M. H. acknowledges financial support from JSPS KAKENHI Grants No. 20K14390. J. A. F.-L. acknowledges computational resources provided by the Swiss National Supercomputing Center withing the project s970. T. N. acknowledges financial support from JST PRESTO (Grant No. JPMJPR20L7). M.-T. H. and R. A. were supported by a Grant-in-Aid for Scientific Research (No. 19H05825, and No. 16H06345) from the Ministry of Education, Culture, Sports, Science and Technology.
\end{acknowledgments}

\newpage


\section*{Acknowledgements}
We thank  Hou Pong Chan, Jian Li, and our anonymous reviewers for their insightful feedback on our paper.
This work was partially supported by the Research Grants Council of the
Hong Kong Special Administrative Region, China (No. CUHK 14210717, General Research Fund; No. CUHK 2410021, Research Impact Fund, R5034-18). Jing Li is supported by the Hong Kong Polytechnic University internal funds (1-BE2W and 1-ZVRH) and NSFC Young Scientists Fund 62006203.

\section{Conclusion}

This paper studies cross-media keyphrase prediction on social media and presents a unified framework to couple the advantages of generation and classification models for this task.
Moreover, we propose a novel \emph{Multi-Modality Multi-Head Attention}
to capture the dense interactions between texts and images, where image wordings explicit in optical characters and implicit in image attributes are further exploited to bridge their semantic gap. Experimental results on a large-scale newly-collected Twitter corpus show that our model significantly outperforms SOTA  generation and classification models with  traditional attention mechanisms.

\section{Multi-modal Tweet Dataset}\label{sec:data}
\paragraph{Data Collection and Statistics.}
Since there are no publicly available datasets for multi-modal keyphrase annotation, we contribute a new dataset with social media posts from \textbf{Twitter}.
Specifically, we employ the Twitter advanced search API\footnote{\url{https://twitter.com/search-advanced}} to query English tweets that contain both images and hashtags from January to June 2019.
For keyphrases, we consider to use user-generated hashtags  following common practice~\cite{DBLP:conf/emnlp/ZhangWGH16,DBLP:conf/naacl/ZhangLSZ18}.
We further clean the raw data in the following ways:
1) we only retain tweets with one color image in JPG form;
2) we remove tweets with less than $4$ tokens or more than $5$ hashtags to filter out noise data;
3) rare hashtags (occurring less than $10$ times) and their corresponding tweets are removed to alleviate sparsity issue;
4) we remove the duplicate tweets (e.g., retweets) and images and obtain 53,701 tweets  with each containing a distinct tweet text-image pair.
We randomly split the data into $80\%$, $10\%$, $10\%$ corresponding to training, validation, and test set.
The data split statistics of tweet texts are displayed in Table~\ref{tables:dataset_stat}.

\input{tables/data_statistics}
\vspace{-0.3em}
\paragraph{Preprocessing.} We employ an open-source Twitter preprocessing tool~\cite{baziotis-pelekis-doulkeridis:2017:SemEval2} to tokenize the tweets, segment the hashtags, and apply common spelling corrections.
To  reduce the errors introduced by the automatic hashtag segmentation, we manually check them and construct a complete mapping list.
Following~\citet{wang-etal-2019-topic-aware}, we retain tokens in hashtags (without \# prefix) for those occurring in the middle of the posts due to their inseparable semantic roles.
We further remove all the non-alphabetic tokens and replace links, mentions (@username), digits into special tokens as \textit{$\langle$url$\rangle$}, \textit{$\langle$mention$\rangle$}, and \textit{$\langle$number$\rangle$} respectively.

\vspace{-0.3em}
\paragraph{Tweet Image Analysis.}
To further analyze the Twitter image characteristics,
we sample $200$ text-image tweets and analyze their  distributions over varying types  in Figure~\ref{fig:image_types}.
We observe a diverse set of categories and only around half of the images ($54\%$) are natural photos, which is rather different from other standard image data such as MS-COCO.
Moreover, we  conduct a pilot study to categorize the text-image relations following~\citet{DBLP:conf/acl/VempalaP19} and find $52\%$ of them have either texts or images useless to represent semantics (see Figure~\ref{tables:four_association_examples} for some examples).
Such diverse category and complex text-image relationship pose unique challenges compared to traditional vision-language tasks like image captioning and visual question answering, where they focus on more natural images, and more importantly, their two modalities have most semantics shared.
To deal with this, we propose M$^3$H-Att and image wordings to better capture essential information from noisy cross-media data.

\input{figure_inputs/image_types}
\vspace{-0.3em}
\paragraph{Image Wording Analysis.}
Here we shed light on some interesting statistics on image wordings.
We first analyze the top $5$ attributes predicted from the images in our dataset: \textit{\{man, shirt, woman, sign, white\}}, which shows that \textit{most of the images on Twitter are about people and daily life}.
For OCR texts, we employ a widely used OCR engine Tesserocr\footnote{\url{https://pypi.org/project/tesserocr/}} to extract optical characters.
From all matching images, there are around \textbf{$35\%$} of them contain characters, significantly larger than the corresponding number in COCO images ($4\%$), indicating \textit{social media users' preference to post images containing optical characters}.
To mitigate the effects of OCR errors,
we only consider tokens present in the vocabulary of tweet texts and
find about $17\%$ left with a median length of  $16$ tokens.
Besides, $32\%$ of the remaining data have words appearing in their corresponding keyphrases and $13\%$ contain the entire keyphrases, suggesting its potential help in keyphrase prediction.

\section{Experiments and Analyses}

\subsection{Experimental Setup}\label{ssec:setup}
\vspace{-0.3em}
\paragraph{Evaluation Metrics.}
We mainly evaluate our model with popular information retrieval metrics macro-average F1@K, where K is $1$ or $3$ as there are $1.33$ keyphrases on average per tweet (Table~\ref{tables:dataset_stat}).
To further measure the keyphrase orders (as we can generate a keyphrase ranking list with beam search), we employ mean average precision (MAP) for the top five predictions following~\citet{DBLP:conf/naacl/ChenCLBK19}.
The higher scores from all the metrics indicate better performance.
For word matchings in evaluation, we consider the results after processed with Porter Stemmer following~\citet{DBLP:conf/acl/MengZHHBC17}.

\vspace{-0.3em}
\paragraph{Comparison Models.}
We first consider the upper-bound performance of  extractive methods, denoted as \textsc{EXT-ORACLE}.
Then, the following baselines are compared. (1) \textbf{Image-only} models: we apply max/average pooling on the grid-level VGG features or object-level BUTD ~\cite{DBLP:conf/cvpr/00010BT0GZ18} and aggregate them for classification.
(2) \textbf{Text-only} models: we consider  classification-based (\textsc{CLS}) or sequence generation-based (\textsc{GEN}) methods. For \textsc{CLS} models, we
consider simple max/average pooling on the text features learned from Bi-GRU encoder and the  Topic Memory Network (\textsc{TMN})~\cite{DBLP:conf/emnlp/ZengLSGLK18} (a SOTA short text classification model).
For \textsc{GEN} models, we employ the seq2seq with attention~\cite{DBLP:journals/corr/BahdanauCB14}, copy mechanism~\cite{DBLP:conf/acl/SeeLM17}, and latent topics~\cite{wang-etal-2019-topic-aware}  (the SOTA topic-aware model for social media keyphrase generation).
(3) \textbf{Text-image} models: we consider the SOTA \textsc{CLS} model  for multi-modal hashtag recommendation~\cite{DBLP:conf/ijcai/ZhangWHHG17} using co-attention and its variant with image-attention~\cite{DBLP:conf/cvpr/YangHGDS16}, as well as  Bilinear Attention Networks (\textsc{BAN})~\cite{DBLP:conf/nips/KimJZ18} (a SOTA variant for Visual Question Answering~\cite{DBLP:conf/iccv/AntolALMBZP15}).
For our models, we first adopt the basic variants with M$^3$H-Att separately applying to either \textsc{CLS} or \textsc{GEN}. Then we additionally combine image wordings and the joint training strategy (Eq.~(\ref{eq:overall_objective})). Our full model is obtained by further aggregating the \textsc{CLS} and \textsc{GEN} outputs (Eq. (\ref{eq:aggregate})).

\vspace{-0.3em}
\paragraph{Parameter Settings.}
We maintain a  generation  vocabulary $V_{gen}$ of $45K$ tokens and the keyphrase classification vocabulary $V_{cls}$ with 4,262 labels.
We apply $200$-d Twitter GloVe embedding~\cite{DBLP:conf/emnlp/PenningtonSM14} for encoding inputs.
We employ two layers of Bi-GRU  for the encoder and a single layer GRU for the decoder with hidden size set to $300$.
For visual signals, we extract either $49$ grid-level VGG  $512$-d features or $36$ object-level BUTD $2048$-d features.
For the M$^3$H-Att, we employ $4$ heads with $64$-d subspace, where $4$ layers are stacked for attention to text modality, and $1$ layer for vision or attribute modality.
In training, we set the loss coefficient $\gamma=1$ and employ Adam optimizer~\cite{DBLP:journals/corr/KingmaB14} with a learning rate  as $0.001$.
We decay it by $0.5$ if validation loss does not drop and apply gradient clipping with the max gradient norm as $5$.
Early stop~\cite{caruana2001overfitting} is adopted via monitoring the change of validation loss.
For inference, we employ beam search with beam size set to $10$ to generate a ranking list of keyphrases.
For the baselines, we re-implement \textsc{CLS-IMG-ATT} and \textsc{CLS-CO-ATT}, and employ the released codes to produce results for \textsc{CLS-TMN}\footnote{\url{https://github.com/zengjichuan/TMN}}, \textsc{GEN-TOPIC}\footnote{\url{https://github.com/yuewang-cuhk/TAKG}}, and \textsc{CLS-BAN}\footnote{\url{https://github.com/jnhwkim/ban-vqa}}.

\input{tables/main_exp_new}
\subsection{Main Comparison Results}\label{ssec:main-results}

We first report the main comparison results in Table~\ref{tables:main_exp_new} and draw the following observations:

$\bullet$~\textit{Textual features are more important than visual signals}. It is seen from the text-only models' better performance compared with their counterparts relying solely on images.
For image-only models, we find that object-level BUTD outperforms grid-level VGG, while for pooling methods, average pooling works better for visual signals while max pooling is more suitable for texts.\footnote{In experiments, we find that VGG works better than BUTD features for M$^3$H-Att in our variants. We show results with the better setting without otherwise specified.}

$\bullet$ \textit{Vision modality can provide complementary information to the text.}
Most models considering cross-media signals perform better than text-only and image-only baselines.
An exception is observed on the classification models with traditional attention, where the best F1@1 score $42.12$ from \textsc{CLS-CO-ATT} is still less than the  text-only model \textsc{GEN-TOPIC}'s  $43.17$.
This indicates the limitation of traditional co-attention to well exploit multi-modal features from social media.

$\bullet$ \textit{Both M$^3$H-Att and image wordings are helpful to encode social media features.} We find that both M$^3$H-Att and image wordings contribute to the performance boost of keyphrase classification or generation or their joint training results, which showcase their ability to handle multi-modality data from social media.
We will discuss more in \cref{ssec:m3h_iw}.

$\bullet$ \textit{Our output aggregation strategy is effective.} Seq2seq-based keyphrase generation models (especially armed with the copy mechanism to enable better extraction capability) perform better than most classification models and even upper bound results of extraction models.
It is probably because of the high absent keyphrase rate and the large size of keyphrases (Table~\ref{tables:dataset_stat}) exhibited in the noisy social media data. Nevertheless, \textsc{GEN-CLS-M$^3$H-ATT}, coupling advantages of classification and generation, obtains the best results ($47.06$ F1@1), dramatically outperforms the SOTA text-only model ($43.17$)  and text-image one ($42.12$).

\input{figure_inputs/four_subfigures}

\subsection{Quantitative Analyses}\label{ssec:freq-postlen}
We examine how our models  perform in diverse scenarios: present vs. absent keyphrases and varying keyphrase frequency and post length in Figure~\ref{fig:four_sub}.
\vspace{-0.2em}
\paragraph{Present vs. Absent Keyphrases.}
We report the F1@1 for evaluating  present keyphrases and recall@5 for absent keyphrases.
As shown in  Figure~\ref{fig:four_sub} (a-b), generation models with copy mechanism consistently outperform classification models for present keyphrases, while the latter works better for absent keyphrases.
Nonetheless, our output aggregation strategy is able to cover generation models' inferiority for absent keyphrases and exhibits much better results ($41.19$ vs. $35.83$ recall@5 score) from \textsc{GEN-CLS-M$^3$H-Att} than \textsc{GEN-M$^3$H-Att}.
Besides, visual signals are helpful for  generation and classification models in predicting both present and absent keyphrases, where a larger boost is observed for the latter, probably owing to the inadequate clues available from texts.

\vspace{-0.2em}
\paragraph{Keyphrase Frequency.}
From Figure~\ref{fig:four_sub} (c), we observe better F1@1 from all models to produce more frequent keyphrases, because common keyphrases allow better representation learning from more training instances.
For extremely rare keyphrases (occur $<10$ times in training), generation models with copy mechanisms exhibit better capability to handle them than classification ones.

\vspace{-0.2em}
\paragraph{Post Length.}
From Figure~\ref{fig:four_sub} (d), we observe that longer post length does not guarantee better performance and the best results are obtained for posts with $15\sim35$ tokens.
It might be attributed to the noisy nature of social media data --- longer posts provide both richer contents and more noise.
For the posts with $<15$ tokens, all multi-modal methods perform better than the text-only ones, indicating that the image modality plays a more important when texts contain limited features.

\subsection{Analyses of M$^3$H-Att and Image Wording}\label{ssec:m3h_iw}
We proceed to quantify the effects of different settings in M$^3$H-Att and image wording.

\input{tables/m3h_layer_head_dim}

\input{tables/ocr_attr_comp}

\vspace{-0.2em}
\paragraph{M$^3$H-Att Analysis.}
We investigate how various configurations ($L_{vis}\in\{1,2,3,4\}$, $H\in\{2,4,8,12\}$,  $d_H\in \{64,128,256\}$ ) of our M$^3$H-Att affect the prediction results in Table~\ref{tables:exp-m3h}.
Here we only show the classification results (and similar trends are observed from generation).
We notice that more complex models do not always present better results and even render performance deteriorate in some cases due to the overfitting issue.
The best performance is attained by $4$ stacked layers of $4$ heads with  a $64$-d subspace.

\vspace{-0.2em}
\paragraph{Image Wording Analysis.}
To examine image wording effects, we compare four models in three settings: no image wording, OCR (only), and image attributes (only) in Table~\ref{tables:ocr_attr}.
The results are shown in three test sets:
the entire test set (Full), the $889$ subset instances with OCR tokens (OCR), and the $266$ ones containing keyphrases from ImageNet labels (Attr)~\cite{DBLP:journals/ijcv/RussakovskyDSKS15}.
For the \textsc{CLS-MAX} and \textsc{GEN-COPY}, we add attributes by using its max-pooled features to attend the text memory, which is later used for prediction.

We observe that either OCR texts or image attributes contribute to better F1@1 on the entire test set for all chosen models, while much more performance gain can be observed on their subsets with OCR texts or ImageNet keyphrases, indicating that images with optical characters and natural styles can benefit more from image wordings.\footnote{Here we assume that multimedia posts with ImageNet keyphrases have a higher probability to contain natural photos.}

\input{figure_inputs/attn_vis}

\subsection{Qualitative Analysis}\label{ssec:case_study}
To explore whether M$^3$H-Att is able to attend different aspects from the image, we probe into its attention weights via heatmap visualization in Figure~\ref{fig:attn_vis}.
Here \textsc{CLS-M\textsuperscript{3}H-ATT} is employed with a single layer of $12$ heads, whose image-to-text and text-to-image attention are examined.
The top figure shows that all its heads attend to the text based on the visual cues, where some attend to ``turtle'' while others attend to ``world'' and ``globe'' with various emphasis.
Interestingly, Head 11 highlights the ``happy'' token, which also appears in the image.
For the text-to-image attentions (bottom), we find some heads tend to highlight the specific local objects, such as the two players by Head 0 and 5 and the textual regions by Head 9, while some capture a more global view of the image like Head 11.
More examples are shown in Figure~\ref{fig:more_attn_cases}.

\input{tables/new_case_study}
We further illustrate how images (visual signals, image attributes, and  OCR tokens) help cross-media keyphrase prediction by analyzing their predictions in Figure~\ref{fig:case_study}.
In post (a),  visual features help both \textsc{CLS-CO-ATT} and our model correctly predict its keyphrase, where our model precisely attends the cat's face (key region reflecting the image's semantics).
Without such context, \textsc{GEN-COPY} wrongly predicts \textit{``star wars''}, which might be caused by the misleading token \textit{``mysterious''} in the texts.
Besides, the cat keyphrase  is also revealed in the top  predicted attribute.
In post (b-c), only our model with image wordings makes  correct predictions, where we observe that the ground-truth keyphrases directly appear in the attributes or  OCR texts. See  Figure~\ref{tables:OCR_cases} and \ref{tables:attr_cases} for more  examples.

\section{Introduction}

The prominent use of social media platforms (such as Twitter)
exposes individuals with an abundance of fresh information in a wide variety of forms such as texts, images, videos, etc.
Meanwhile, the explosive growth of multimedia data has far outpaced individuals' capability to understand them.
This presents a concrete challenge to digest the massive amount of data, distill the salient contents therein, and provide users with a quicker access to the information they need when navigating a large multitude of noisy social media data.

\input{tables/intro_example.tex}

To that end, extensive efforts have been made to \textbf{social media keyphrase prediction}\footnote{We consider a hashtag as a post's keyphrase annotation following the common practice~\cite{DBLP:conf/emnlp/ZhangWGH16,DBLP:conf/naacl/ZhangLSZ18}. } --- aiming to produce a sequence of words that reflect a post's key concern.
Nevertheless, previous work mostly focuses on the use of textual signals~\cite{DBLP:conf/naacl/ZhangLSZ18,wang-etal-2019-topic-aware,wang-etal-2019-microblog}, which sometimes provide limited features as social media language is essentially informal and fragmented.
To enrich the contexts, here we resort to exploiting the matching images, which are widely used in social media posts to deliver auxiliary information from authors (e.g., opinions, feelings, topics, etc.), primarily due to the flourish of mobile Internet.

To  illustrate our motivation,
Figure~\ref{tables:intro_example} shows the texts and images of two Twitter posts (tweets).
The left is tagged with a keyphrase ``cat'', which can be clearly signaled with its image while the paired text is an anthropomorphic description and hardly unveils its real semantics.
For the right, the image depicts a basketball game scene with optical characters ``2019 NBA FINALS'', directly indicating its keyphrase, which is difficult to identify from the texts. In both examples, images play a more vital role than texts in reflecting the key information.
These points motivate our cross-media keyphrase prediction study that examines how the salient contents can be indicated by the coupled effects of post texts and their matching images.

Previous work~\cite{DBLP:conf/ijcai/ZhangWHHG17,DBLP:conf/aaai/Zhang00TY019} employs co-attention networks~\cite{DBLP:conf/nips/LuYBP16,DBLP:conf/eccv/XuS16} to encode multimedia posts, where a single attention function is concurrently performed to infer either visual or textual distributions.
We argue that they might be suboptimal to model intricate text-image associations, as a recent finding~\cite{DBLP:conf/acl/VempalaP19} points out there can be four diverse semantic relations held by images and texts on Twitter.
To allow for better modeling, in this work,
we take advantage of the recent advances in multi-head attention~\cite{DBLP:conf/nips/VaswaniSPUJGKP17}, which is capable of learning representation from different subspaces.
We extend it to capture diverse cross-media interactions,
named as \emph{Multi-Modality Multi-Head Attention} (M$^3$H-Att).
Moreover, to well align the images' semantics to texts', we adopt \emph{image wordings} and define two forms for that
--- explicit \textit{optical characters}  (such as ``NBA Finals'' in post (b)) detected from the optical character reader (OCR) and implicit \textit{image attributes}~\cite{wu-etal-2006-boosting}, high-level text labels predicted to summarize the image's semantic concepts (such as a ``cat'' label for post (a)).

Furthermore, unlike prior work employing either classification ~\cite{DBLP:conf/ijcai/GongZ16} or generation models~\cite{wang-etal-2019-topic-aware},  we propose a \emph{unified} framework to couple the advantages of keyphrase classification and generation.
Specifically, in addition to the joint training of both modules,
we further extend the copy mechanism~\cite{DBLP:conf/acl/SeeLM17} to adaptive aggregate classification outputs together with  source input tokens.
Empirical results  show that our proposed unified model  not only  keeps classification's superiority to predict common keyphrases (Figure~\ref{fig:four_sub}~(c)) while enabling keyphrase creation beyond a predefined candidate list, but also largely benefits the keyphrase prediction especially for  absent keyphrases (Figure~\ref{fig:four_sub} (b)).

For experiments, we collect a large-scale tweet dataset with texts and images, which is presented as part of our work.
Extensive results show that our model significantly outperforms the state-of-the-art (SOTA) methods using traditional attention mechanisms.
For example, we obtain $47.06\%$ F1@1 compared with $43.17\%$ by \citet{wang-etal-2019-topic-aware} (keyphrase generation from texts only) and $42.12\%$ by~\citet{DBLP:conf/ijcai/ZhangWHHG17} (multi-modal keyphrase classification).
We then examine how we perform to handle absent and present keyphrases, and varying keyphrase frequency and post length.
The results indicate the consistent performance boost brought by our M$^3$H-Att design and unified framework in diverse scenarios~(\cref{ssec:freq-postlen}).
We further quantify the effects of different settings of multi-head attention and image wordings
to see when and how they work the best~(\cref{ssec:m3h_iw}).
Lastly, we provide qualitative analysis
to interpret why our model results in superior multimedia understanding~(\cref{ssec:case_study}).

\section{Our Unified Cross-Media Keyphrase Prediction Framework}

\input{figure_inputs/model.tex}

Given a collection $\mathcal{C} $ with $|C|$ text-image post pairs $\{(\mathbf{x}^n, I^n)\}_{n=1}^{|C|}$ as input, we aim to predict a  keyphrase set
$\mathcal{Y} =\{{\bf y}^i\}_{i=1}^{|\mathcal{Y}|}$ for each of them.
Following~\citet{DBLP:conf/acl/MengZHHBC17}, we copy the source input pair multiple times to allow each paired to have one keyphrase.
We represent each input as a triplet $(\mathbf{x}, I, \mathbf{y})$, where  $\mathbf{x}$ and  $\mathbf{y}$ are formulated as word sequences $\mathbf{x}=\langle x_1,..., x_{l_x}\rangle$ and $\mathbf{y}=\langle y_1,..., y_{l_y}\rangle$ ($l_x$ and $l_y$   denote the  number of words).

We show the  overview of our proposed cross-media keyphrase prediction model in Figure~\ref{fig:framework}.
We first encode a text-image tweet into three modalities: \textit{text}, \textit{attribute}, and \textit{vision}  (\cref{ssec:encoder}), and propose a Multi-Modality Multi-Head
Attention (M$^3$H-Att) to capture their intricate interactions
(\cref{ssec:model-m3h}).
Then, we feed the learned multi-modality representations for either keyphrase classification or generation, followed with  a tailored aggregator to combine their outputs (\cref{ssec:decoder}). Lastly, the entire framework can be jointly trained via multi-task learning (\cref{ssec:training}).

\subsection{Multi-modality Encoder}\label{ssec:encoder}

\paragraph{Learning Text Representation.}
We first embed each token $x_i$ from the input sequence into a high-dimensional vector via a pre-trained lookup table, and then employ bidirectional gated recurrent unit (Bi-GRU)~\cite{DBLP:conf/emnlp/ChoMGBBSB14} to encode the embedded input token $e(x_i)$:
\begin{align}
    \overrightarrow{\mathbf{h}_i}&=GRU(e(x_i),~~\overrightarrow{\mathbf{h}_{i-1}}),\\ \overleftarrow{\mathbf{h}_i}&=GRU(e(x_i)
    ,~~\overleftarrow{\mathbf{h}_{i+1}}).
\end{align}
Forward hidden state $\overrightarrow{\mathbf{h}_i}$ and backward one $\overleftarrow{\mathbf{h}_i}$ are later concatenated into $\mathbf{h}_i=[\overrightarrow{\mathbf{h}_i};\overleftarrow{\mathbf{h}_{i}}]$.
We employ it as the context-aware representation of $x_i$ and pack all of them
in the input sequence into a textual memory bank $\mathbf{M}_{text} =\{\mathbf{h}_i,...,\mathbf{h}_{l_x}\} \in \mathbb{R}^{l_x \times d}$, where $d$ denotes the hidden state dimension.

\paragraph{Encoding OCR Text.}
To detect optical characters from images, we use an open-source toolkit~\cite{DBLP:conf/icdar/Smith07} to extract OCR texts in  form of a word sequence.
It is then appended into the post text  with a delimited token \textit{$\langle$sep$\rangle$} to notify the change of text genres, which is shown to be a simple yet effective design to combine OCR features.

\paragraph{Learning Image Representation.}
We consider two types of  image representations: \textit{grid-level} or \textit{object-level} visual features.
For the former, we apply a pre-trained VGG-16 Net~\cite{DBLP:journals/corr/SimonyanZ14a} to extract $7 \times 7$ convolutional feature maps for each image $I$.
For the latter, inspired by bottom-up attention~\cite{DBLP:conf/cvpr/00010BT0GZ18}, we use the Faster-RCNN~\cite{DBLP:conf/nips/RenHGS15} pre-trained on Visual Genome~\cite{DBLP:journals/ijcv/KrishnaZGJHKCKL17} to detect the objects and extract their features.
Each feature map is further transformed into a new vector $\mathbf{v}_i$ through a linear projection layer.
As such, we construct a visual memory bank  as $\mathbf{M}_{vis}=\{\mathbf{v}_1,...,\mathbf{v}_{l_v}\}\in \mathbb{R}^{l_v \times d}$, where $l_v$ denotes the number of image regions or objects.

\paragraph{Encoding Image Attribute.}
Following~\citet{DBLP:conf/acl/CaiCW19}, we first train an attribute predictor based on the Resnet-152~\cite{DBLP:conf/eccv/HeZRS16} features on MS-COCO 2014 caption dataset~\cite{DBLP:conf/eccv/LinMBHPRDZ14}.
Specifically, we extract noun and adjective tokens from the image captions as the attribute labels.
Afterward, the top five predicted attributes of each image are transformed with another linear layer to an attribute memory bank  $\mathbf{M}_{attr}=\{\mathbf{a}_1,...,\mathbf{a}_5\}\in \mathbb{R}^{5 \times d}$, which aims to  capture images' high-level semantic concepts.

\subsection{Multi-modality Multi-Head Attention}\label{ssec:model-m3h}

Our design of multi-head attention is inspired by its prototype in Transformer~\cite{DBLP:conf/nips/VaswaniSPUJGKP17}.
We extend it to capture multiple forms of cross-modality interactions for a multimedia post, which is therefore named as M$^3$H-Att, short for Multi-Modality Multi-Head Attention.
Compared to its original use as a self-attention over texts only, we instead operate on three modalities (text, attribute, and vision) in a \textit{pairwise} co-attention manner.

For each  co-attention, we perform  scaled dot attention  $\mathcal{A}$ on a set of \{\textit{Query}, \textit{Key}, \textit{Value}\}:
\begin{align}
    \mathcal{A}(Q, K, V)&=\texttt{softmax}(\frac{QK^T}{\sqrt{d_K}})V,\\
    \mathcal{A}^M(Q,K,V)&=[head_1; ...; head_H]W^O,\\
 \text{where}~~~ head_h&=\mathcal{A}(QW^Q_h, KW^K_h, VW^V_h),
\end{align}
where $W^Q_h, W^K_h, W^V_h\in\mathbb{R}^{d\times d_H}$ are learnable weights to project the query, key, and value from dimension $d$ to a lower space of $d_H$-dimension and  $H$ is the head number.
Outputs from all the heads are  concatenated (in $\mathcal{A}^M$)  and  passed to a feedforward network  with  residual connections~\cite{DBLP:conf/eccv/HeZRS16} and layer normalization~\cite{DBLP:journals/corr/BaKH16}.

Specifically, we employ the text features as a query to attend to the vision/attribute modality and vice versa.\footnote{We also try other combinations, e.g., M$^3$H-Att between the vision and attribute, but the improvements are negligible.}
Here max/average-pooling is adopted to obtain one holistic query vector for each modality instead of token-level queries considering the noisy nature of social media data.
Moreover, we stack  multiple co-attention layers to  empower its modeling capability, where $L_{text}, L_{attr}, L_{vis}$ denote the number of stacked layers for text, attribute, and vision queries, respectively.
After that, the outputs from all co-attention layers are summed up with a linear multi-modal fusion layer to produce a
context vector $\mathbf{c}_{fuse} \in \mathbb{R}^d$.
It will be  fed into a keyphrase classifier and generator for the unified prediction.
Notably, this  indicates that our  M$^3$H-Att's great potential to  serve as a generic module  for  benefiting other cross-media applications.

\input{figure_inputs/multi_head_att}

\subsection{Unified Keyphrase Prediction}\label{ssec:decoder}
We describe how we  combine the keyphrase classification and generation into a unified prediction for coupling their advantages below.

\paragraph{Keyphrase Classification.}
As each keyphrase $\mathbf{y}$ usually consists of only several tokens, it can be considered as a discrete integral label and predicted it with a keyphrase classifier.
Here we directly pass the multi-modal context vector $\mathbf{c}_{fuse}$ into a two-layer of multi-layer perceptron (MLP) and map it to the distribution over the label vocabulary $V_{cls}$:
\begin{align}
    P_{cls}(\mathbf{y}) = \texttt{softmax}(\texttt{MLP}_{cls}(\mathbf{c}_{fuse})).
\end{align}

\paragraph{Keyphrase Generation with Pointer.}
For keyphrase generation, we base on a sequence-to-sequence  framework to predict the keyphrase word sequence $\mathbf{y}=\langle y_1,..., y_{l_y}\rangle$, where the generation probability is defined as
$\prod_{t=1}^{l_y}P(y_t\,|\,\mathbf{y}_{<t})$.

Concretely, we use an unidirectional GRU decoder to model the generation process, which emits the hidden state $\mathbf{s}_t = GRU( \mathbf{s}_{t-1}, \mathbf{u}_t) \in \mathbb{R}^d$ based on the previous hidden state $\mathbf{s}_{t-1}$ and the embedded decoder input $\mathbf{u}_t$.
The decoder state is initialized by the last hidden state $\mathbf{h}_{l_x}$ of the text encoder.
Here  attention mechanism~\cite{DBLP:journals/corr/BahdanauCB14} is adopted to obtain a textual context $\mathbf{c}_{text}$:
\begin{align}
\mathbf{c}_{text}&=\sum_{i=1}^{l_x}\alpha_{t,i}\mathbf{h}_i,\\
\alpha_{t,i}&=\texttt{softmax} (S(\mathbf{s}_t,\mathbf{h}_i)),\label{eq:alpha} \\
S(\mathbf{s}_t,\mathbf{h}_i) &= \mathbf{v}^T_{ \alpha}~ \tanh(\mathbf{W}_{\alpha}[\mathbf{s}_t;\mathbf{h}_i] + \mathbf{b}_{\alpha}),
\end{align}
where $S(\mathbf{s}_t,\mathbf{h}_i)$ is a score function to measure the compatibility between the $t$-th word to be decoded and the $i$-th word from the text encoder.  $\mathbf{W}_{\alpha}\in \mathbb{R}^{d \times 2d},\mathbf{b}_{\alpha}, \mathbf{v}\in \mathbb{R}^{d}$ are trainable weights.

Next, we incorporate the  static multi-modal vector  $\mathbf{c}_{fuse}$ (produced by M$^3$H-Att and independent of the decoding step $t$)
to construct a context-rich representation $\mathbf{c}_t = [\mathbf{u}_t;\mathbf{s}_t;\mathbf{c}_{text} + \mathbf{c}_{fuse}]$. Based on it, we apply another MLP with softmax to produce a word distribution over  vocabulary $V_{gen}$:
\begin{equation} \label{eq:copy_score}
  P_{gen}(y_t)= \texttt{softmax}(\texttt{MLP}_{gen}(\mathbf{c}_t)).
\end{equation}

To further allow the decoder to explicitly extract words from the source post, we apply the copy mechanism~\cite{DBLP:conf/acl/SeeLM17} by
calculating a soft switch $\lambda_t \in [0,1]$ with a sigmoid-activated MLP on $\mathbf{c}_t$.
It indicates whether to generate the word from the vocabulary $V_{gen}$ or copy it from the input sequence, where the extractive distribution is decided by the text attention weights $\alpha_{t,i}$ in Eq.~(\ref{eq:alpha}).

\paragraph{Classification Output Aggregation.}
We further extend the copy mechanism to aggregate the classification's outputs to benefit keyphrase generation.
First, we retrieve the top-K predictions from the classifier and convert each into the word sequence $\mathbf{w}=\langle w_1,..., w_{l_w}\rangle$, where $l_{w}$ is the sequence length of the combined predictions.
Then, we normalize their classification logits using softmax into a word-level distribution $\mathbf{\beta} \in \mathbb{R}^{l_{w}}$, which represents the extractive probability from the classification output.
Finally, we obtain the unified prediction via:
\begin{align}\label{eq:aggregate}
    P_{unf}&(y_t) = \lambda_t\cdot P_{gen}(y_t)~~+  \\ &(1-\lambda_t)\cdot(a\cdot\sum_{i:x_i=y_t}^{l_x}\alpha_{t,i} + b\cdot\sum_{j:w_j=y_t}^{l_{w}}\beta_j), \nonumber
\end{align}
where $a,b~(a+b=1)$ are  hyper-parameters to decide whether to copy from the input sequence or the classification outputs.
To stabilize the aggregation of classification outputs, we warm up the classifier for several epochs first by setting $a$ to $1$ and $b$ to $0$ and then both to $0.5$ for further training.

\subsection{Joint Training Objective}\label{ssec:training}
We employ the standard negative log-likelihood loss and define the entire framework's training objective with the linear combination of the label classification loss and the token-level sequence generation loss for multitask learning:
\begin{equation}\label{eq:overall_objective}
    \mathcal{L}(\theta) = -\sum_{n=1}^N [\underbrace{\log P_{cls}(\mathbf{y}^n)}_{\text{Classification}} + \gamma \cdot \sum_{t=1}^{l_y^n}\underbrace{\log P_{unf}(y^n_t)}_{\text{Unified}}],
\end{equation}
where $N$ is size of the training text-image  pairs and $\gamma$ is a hyper-parameter to balance the two losses (empirically set to $1$) and $\theta$ denotes the   trainable parameters shared for the whole framework. Intuitively, jointly training  keyphrase classification would benefit the unified prediction by not only implicitly better parameter learning, but also explicitly providing more precise outputs to be copied to the keyphrase generator by the aggregation module.

\section{Related Work}
\paragraph{Social Media Keyphrase Prediction.}
Traditional keyphrase prediction studies focus on using two-step pipeline methods:
candidates are first extracted with handcrafted features (e.g. part-of-speech tags~\cite{DBLP:conf/dl/WittenPFGN99}) and then ranked by unsupervised~\cite{DBLP:conf/aaai/WanX08} or supervised algorithms~\cite{DBLP:conf/emnlp/MedelyanFW09}.
These methods undergo labor-intensive feature engineering and hence lead to the growing popularity of adopting data-driven neural networks.
Specifically for social media keyphrase prediction,
most efforts are based on sequence tagging style extraction~\cite{DBLP:conf/emnlp/ZhangWGH16,DBLP:conf/naacl/ZhangLSZ18} or classification from  a predefined candidate list~\cite{DBLP:conf/ijcai/GongZ16,DBLP:conf/ijcai/ZhangWHHG17}, which cannot  produce keyphrases absent in the post or  the fixed list.

Inspired by the recent success of keyphrase generation~\cite{DBLP:conf/acl/MengZHHBC17,conf/acl/chan19keyphraseRL} for scientific articles,
\citet{wang-etal-2019-topic-aware,wang-etal-2019-microblog} employ sequence-to-sequence (seq2seq) models to allow unseen keyphrases to be flexibly created for social media posts.
Unlike them, we propose a novel unified framework to combine the benefits of keyphrase classification and generation.
Similar to this, \citet{DBLP:conf/naacl/ChenCLBK19} also exploit the power of  classification for keyphrase generation but in a separated retrieval manner, where we elegantly integrate them with a tailored copy mechanism and allow for the end-to-end joint training.
While most of prior work focuses on the modeling of texts, we additionally exploit the matching  images and study their coupled effects for indicating keyphrases.

\paragraph{Cross-media Research.}
We are also related to cross-media research, where texts and images are jointly exploited for a variety of applications, such as  personalized image captioning~\cite{DBLP:journals/pami/ParkKK19},  event extraction~\cite{DBLP:conf/acl/LiZZWLJC20}, sarcasm detection~\cite{DBLP:conf/acl/CaiCW19}, and text-image relation classification~\cite{DBLP:conf/acl/VempalaP19}.
Some of them have pointed out the usefulness of OCR texts~\cite{DBLP:conf/mm/ChenHK16} and image attributes~\cite{DBLP:conf/cvpr/WuSLDH16} to endow images with higher-level semantics beyond visual features, where we are the first to study how OCR texts and image attributes work together to indicate keyphrases.
Closest to our work, \citet{DBLP:conf/ijcai/ZhangWHHG17,DBLP:conf/aaai/Zhang00TY019} study multimedia hashtag classification and employ  co-attention networks~\cite{DBLP:conf/nips/LuYBP16,DBLP:conf/eccv/XuS16} to model the text-image associations, while we extend the multi-head attention~\cite{DBLP:conf/nips/VaswaniSPUJGKP17} to better capture such diverse styles of cross-media interactions.

While multi-head attention has been widely exploited in many vision-language (VL) tasks, such as image captioning~\cite{DBLP:conf/aaai/ZhouPZHCG20}, visual question answering~\cite{tan2019lxmert,DBLP:conf/nips/LuBPL19}, and visual dialog~\cite{DBLP:conf/emnlp/KangLZ19,DBLP:conf/emnlp/vdbert20}, its potential benefit to model flexible cross-media posts has been previously ignored.
Due to the informal style in social media, cross-media keyphrase prediction brings unique difficulties mainly in two aspects: first, its text-image relationship is rather complicated~\cite{DBLP:conf/acl/VempalaP19} while in conventional VL tasks the two modalities have  most semantics shared; second, social media images usually exhibit a more diverse distribution and a much higher probability of containing OCR tokens~(\cref{sec:data}), thereby posing a hurdle for effectively processing.