text:
\section*{Introduction}
\label{introduction}

Electrons injected into chiral molecules like DNA become spin polarized after being transmitted through the molecule.~\cite{Goehler2011,Xie2011,Mishra2020,Naaman2019,Waldeck2021}.
Such a spin-filtering effect has been termed ``chirality-induced spin selectivity" (CISS)~\cite{Naaman2012,Michaeli2016,Michaeli2017}.
This is a remarkable effect, since organic molecules do not contain magnetic atoms, which would be apparent candidates for spin-dependent phenomena.

Although CISS is established experimentally, its theoretical understanding is still debated~\cite{Evers2022}.
In many theories, the spin-orbit interaction (SOI) is considered to be the origin of the spin asymmetry~\cite{Guo2012,Gutierrez2013,Guo2014,Matityahu2016,Michaeli2019,Geyer2020,YU2020,SierraBioMol2020,Liu2021,Wolf2022,Michaeli2022}.
However, since the SOI does not break time-reversal symmetry, the appearance of  SOI-induced spin-filtering is a non-trivial effect:
Bardarson's theorem \cite{Bardarson2008} imposes a constraint stating that \textit{in time-reversal-symmetric systems with half integer spins, the transmission eigenvalues of the scattering matrix come in degenerate pairs}.
Assuming that this Kramers-type degeneracy carries up and down spins in the same direction, the theorem prohibits spin filtering in systems coupled to two terminals.
However, the theorem forbids spin selectivity through two-terminal time-reversal symmetric systems only when there is only one orbital-channel.
Therefore, several previous theories broke time-reversal symmetry by introducing spin dissipation~\cite{Guo2012,Guo2014,Matityahu2016}, which effectively introduces many terminals.

Another option to overcome Bardarson's theorem, recently formulated explicitly~\cite{YU2020}, is to introduce two-orbital conducting channels.
Bardarson's theorem does not specify which spin states are associated with the doubly-degenerate transmission eigenvalues.
Therefore, spin-filtering is possible if there exist two pairs of doubly-degenerate transmission eigenvalues, in which one pair carries two up spins in one direction and the other pair carries two down spins in the opposite direction.
The origin of this idea is spin filtering brought about by the Rashba SOI in two-terminal quantum point contacts~\cite{Eto2005}, in tubular two-dimensional gases~\cite{Entin-WohlmanPRB2010}, and in quasi-one dimensional quantum wires~\cite{NagaevPRB2014}.
In the context of CISS, the idea appeared implicitly in the models of a particle traveling on~\cite{Michaeli2019,Geyer2020} or along~\cite{Gutierrez2013} the surface of a helical tube, and in the double-helix model with two orbitals residing on different helices~\cite{SierraBioMol2020}.

In a previous paper~\cite{YU2020}, we demonstrated that a $p$-orbital helical atomic chain, a toy model of a single strand of the DNA molecule, can be reduced to an effective two-orbital tight-binding model realizing a two-terminal spin filter  for specific parameters, without breaking time-reversal symmetry.
In that paper we mainly focused on an ideal configuration: two pairs of up and down spins propagating in opposite directions.
In the present paper we extend our previous work to a broader range of parameters.
Especially, we analyze the helical symmetry of our model for all $p$-orbital states.
The consequences of including the orbital degrees of freedom has been discussed before:
(i) Orbital polarization emerges in the $p$-orbital helical atomic chain~\cite{Otsuto2021};  (ii) The sign of the  hopping matrix elements in a neighboring $p$-$p$ block is related to the chirality and to the direction of spin-polarization~\cite{Zoellner2020}.
In particular we discuss the consequences of  the helical symmetry of the hopping matrix elements connecting neighboring $p$-orbitals.
Since the DNA molecule is complex and it is difficult to derive systematically its effective $p$-orbital tight-binding model, we only account for the constraint imposed by the helical symmetry.
In the following, we use $\hbar=1$.

\section*{Spin filtering in a two-terminal two-orbital time-reversal symmetric conductor}
\label{sec:ttsf}

We begin by briefly explaining the reason why the SOI, which does not break time-reversal symmetry, cannot realize two-terminal spin filtering in a single-orbital conducting channel.
In such a system, there are 2 channels when the spin degree of freedom is accounted for [Fig.~\ref{fig:even_helical} (a)]:
For each spin $\sigma$, there are right- and left-going modes, $|k; \sigma \rangle$ and $|-k; \sigma \rangle$.
Spin filtering occurs if, e.g., the right-going $\downarrow$-spin and left-going $\uparrow$-spin states, $|k;\downarrow \rangle$ and $|-k;\uparrow \rangle$, dominate the transport.
For this purpose, the other two states, $|k;\uparrow \rangle$ and $|-k;\downarrow \rangle$, have to be gapped away.
Note that they are time-reversed of one another, $\hat{\Theta}|k;\sigma \rangle = \sigma |-k; \bar{\sigma} \rangle$, where $\hat{\Theta}=-i \hat{\sigma}_y \hat{K}$  is the time-reversal operator ($\hat{\sigma}_y$  is the Pauli matrix and $\hat{K}$ is the complex conjugate operator)~\cite{JJSakurai1985}.
Here $\bar{\sigma}=\downarrow (\uparrow)$ for $\sigma=\uparrow (\downarrow)$
and the coefficient $\sigma$ has the values $\sigma = +1 (-1)$ for $\sigma=\uparrow (\downarrow)$.
The Hamiltonian which hybridizes time-reversed states is given by,
\begin{align}
\hat{V} = a \hat{c}_{k;\uparrow}^\dagger \hat{c}^{}_{-k;\downarrow} + a^* \hat{c}_{-k;\downarrow}^\dagger \hat{c}^{}_{k;\uparrow} \, , \label{eqn:V}
\end{align}
where $a$ is a complex number.
Since the annihilation operator transforms as $\hat{\Theta} \hat{c}^{}_{k;\sigma} \hat{\Theta}^{-1} = \sigma \hat{c}^{}_{-k; \bar{\sigma}}$ ~\cite{Bernevig2013},
the mixing Hamiltonian is odd under the time reversal operation as $\hat{\Theta} \hat{V} \hat{\Theta}^{-1} = - \hat{V}$.
Consequently, it is not possible to realize spin filtering without breaking time-reversal symmetry in a single-orbital conducting channel.

The situation is different when there are two orbital channels, which we denote as orbital $1$ and orbital $2$ [Fig.~\ref{fig:even_helical} (b)].
In this case one may consider hybridizing the right-going $\uparrow$-spin and the left-going $\downarrow$-spin residing on different orbitals.
Such a Hamiltonian,
\begin{align}
\hat{V}^\prime =& a \, \hat{c}_{k;2, \uparrow}^\dagger \hat{c}^{}_{-k;1, \downarrow} - a \, \hat{c}_{k;1, \uparrow}^\dagger \hat{c}^{}_{-k;2, \downarrow} \nonumber \\
&+ a^* \hat{c}_{-k;1, \downarrow}^\dagger \hat{c}^{}_{k;2, \uparrow} - a^* \hat{c}_{-k;2, \downarrow}^\dagger \hat{c}^{}_{k;1, \uparrow} \, , \label{eqn:Vp}
\end{align}
 where $a$ is a complex number, is even under the time-reversal operation, $\hat{\Theta} \hat{V}^\prime \hat{\Theta}^{-1} = \hat{V}^\prime$.
Therefore, the SOI can in principle lead to spin filtering when there are two-orbital channels.

In the next section we provide a different argument, based on the scattering matrix of the molecule, and demonstrate that spin filtering by a two-terminal two-orbital  setup  does not contradict Bardarson's theorem~\cite{Bardarson2008}.

\begin{figure}
\begin{center}
\includegraphics[width=1.0 \columnwidth]{orbital_channel.pdf}
\caption{
(a) Single-orbital case.
For each spin, there are left- and right-going states.
Time-reversed states (TRSs) are indicated by dashed arrows.
The Hamiltonian, which mixes up and down spins propagating in the opposite directions is odd under the time-reversal operation.
(b) Two-orbital case.
The Hamiltonian, which mixes states connected by solid arrows can be even under the time-reversal operation. }
\label{fig:even_helical}
\end{center}
\end{figure}

\subsection*{Scattering matrix}
\label{sec:Bardarson_T}

Let us consider a chiral molecule attached to left and right leads.
The scattering states in the left ($s=L$) and right ($s=R$) leads are,
\begin{align}
|\psi \rangle^{}_s = |{k} \rangle^{}_s c_s^{\mathrm  in}  + |-{k} \rangle^{}_s c_s^{\mathrm  out}  \, .
\label{eqn:scattering_state_}
\end{align}
The scattering matrix ${S}$, which connects the amplitude of right-going $|{k} \rangle^{}_s$ and left-going $|-{k} \rangle^{}_s$ states,
\begin{align}
{c}^{\mathrm  out} = {S} c^{\mathrm  in} \, ,
\;\;\;\;
{c}^{\mathrm  out} = \left[ \begin{array}{c} {c}_L^{\mathrm  out} \\ {c}_R^{\mathrm  out} \end{array} \right] \, ,
\;\;\;\;
c^{\mathrm  in}= \left[\begin{array}{c} c_L^{\mathrm  in} \\ c_R^{\mathrm  in} \end{array} \right]\, ,
\label{eqn:s_def_1}
\end{align}
reads
\begin{align}
{S} = \left[\begin{array}{cc} {r} & {t}' \\ {t} & {r}' \end{array} \right] \, .
\label{eqn:s_matrix}
\end{align}

When there are $N^{}_s$ orbital channels in terminal $s$, the amplitudes $c^{\mathrm in}_{s}$ and ${c}^{\mathrm out}_{s}$ form 2$N^{}_s-$component vectors,
\begin{align}
c_s^{{\mathrm in}} = \left[ \begin{array}{c} c_{1 \uparrow s}^{{\mathrm in} } \\ c_{1 \downarrow s}^{{\mathrm in} } \\ \vdots \\ c_{N_s \uparrow s}^{{\mathrm in} } \\ c_{N_s \downarrow s}^{{\mathrm in} }
\end{array} \right]\ ,
\;\;\;\;
{c}_s^{{\mathrm out}} = \left[ \begin{array}{c} {c}_{1 \downarrow s}^{{\mathrm out} } \\ {c}_{1 \uparrow s}^{{\mathrm out}} \\ \vdots \\ {c}_{N_s \downarrow s}^{{\mathrm out} } \\ {c}_{N_s \uparrow s}^{{\mathrm out}} \end{array} \right] \, .
\label{amp}
\end{align}
For a time-reversal symmetric system, the scattering matrix is self-dual~\cite{Bardarson2008},
\begin{align}
S=({\bm 1}_{2 N^{}_{s}} \otimes \sigma^{}_y) S^T ({\bm 1}_{2 N^{}_{s}} \otimes \sigma^{}_y) \ ,
\label{eqn:BT}
\end{align}
where $S^{T}_{}$ is the transposed scattering matrix (${\bm 1}_{ n}$ is the $n \times n$ unit matrix).
The block-diagonal component of the scattering matrix is the reflection matrix, $r=({\bm 1}_{N^{}_{s}} \otimes \sigma^{}_y) r^T ({\bm 1}_{N^{}_{s}} \otimes \sigma^{}_y)$.
Hence, the reflection amplitude from the state with orbital $\alpha'$ and spin $\sigma'$ into the state with orbital $\alpha$ and spin $\sigma$, $r^{}_{\alpha \sigma,\alpha' \sigma'}$, satisfies,
\begin{align}
r^{}_{\alpha \sigma , \alpha' \sigma'} = \sigma \sigma' \, r^{}_{\alpha' \bar{\sigma}' , \alpha \bar{\sigma}} \, . \label{eqn:sym_ref}
\end{align}
The transmission eigenvalues mentioned above are the eigenvalues of the matrix of transmission probabilities,
\begin{align}
{t}^\dagger {t} = {\bm 1}_{2 N_s}- r^\dagger_{} r  \, .
\end{align}
For the single-orbital channel, $N_{s}^{}=1$, the reflection matrix is a $2 \times2$ matrix.
It is diagonal \cite{Bardarson2008}, as by Eq. (\ref{eqn:sym_ref})
\begin{align}
r^{}_{\uparrow , \uparrow} =& r^{}_{\downarrow , \downarrow}=r^{}_0 , \\
r^{}_{\sigma , \bar{\sigma}} =& -r^{}_{\sigma , \bar{\sigma}}=0.
\end{align}
The matrix of transmission probabilities is also diagonal, ${t}^\dagger {t} = (1-|r^{}_0|^2) \sigma^{}_0$.
Therefore, the transmission eigenvalues are degenerate.
Since spin asymmetry is absent, spin filtering is forbidden.

For the two-orbital channel case, $N_{s}^{}=2$, the reflection matrix is a $4 \times 4$ matrix.
A simple example, which satisfies (\ref{eqn:sym_ref}) and is capable of producing spin filtering, is~\cite{YU2020}:
\begin{align}
r &= \left[ \begin{array}{cccc}
0 & 0^{}_{} & 0^{}_{} & r^{}_{1 \uparrow , 2 \downarrow} \\
0 & 0 & r^{}_{1 \downarrow , 2 \uparrow} & 0 \\
0 & r^{}_{2 \uparrow, 1 \downarrow} & 0 & 0 \\
r^{}_{2 \downarrow , 1 \uparrow} & 0 & 0 & 0
\end{array} \right]\nonumber \\
&= \left[ \begin{array}{cccc}
0 & 0^{}_{} & 0^{}_{} & r^{}_{1 \uparrow , 2 \downarrow} \\
0 & 0 & -r^{}_{2 \downarrow , 1 \uparrow} & 0 \\
0 & -r^{}_{1\uparrow,2\downarrow} & 0 & 0 \\
r^{}_{2 \downarrow , 1 \uparrow} & 0 & 0 & 0
\end{array} \right] \ .
\label{eqn:rm_2c}
\end{align}
The matrix (\ref{eqn:rm_2c}) can be rearranged in a block-diagonal form,
$r=\mathrm{diag}(r^{}_+,r^{}_-)$, where
\begin{align}
r^{}_+ = \left[ \begin{array}{cc} 0 & r^{}_{1 \uparrow , 2 \downarrow} \\ r^{}_{2 \downarrow , 1 \uparrow} & 0 \end{array} \right] ,\
r^{}_- = \left[ \begin{array}{cc} 0 & -r^{}_{1 \uparrow , 2 \downarrow} \\ -r^{}_{2 \downarrow , 1 \uparrow} & 0 \end{array} \right] \ . \label{eqn:rp_rm}
\end{align}
The two matrices  $r^{}_{+}$ and $r^{}_{-}$ are time-reversed of one another,  $r^{}_- = \sigma^{}_y r_+^T \sigma^{}_y$.
The four transmission eigenvalues are the solutions of the characteristic polynomial equation
\begin{align}
\mathrm{det} \{ \Lambda {\bm 1}_{4}- t^\dagger t \} &= ( \mathrm{det} \{ (\Lambda-1) {\bm 1}^{}_{2} + r_\pm^\dagger r^{}_\pm \})^2=0 \ .
\end{align}
They come in pairs of degenerate eigenvalues~\cite{Bardarson2008},
\begin{align}
1+|r_{1 \uparrow, 2 \downarrow}|^2, 1+|r_{1 \uparrow, 2 \downarrow}|^2, 1+|r_{2 \downarrow, 1 \uparrow}|^2, 1+|r_{2 \downarrow, 1 \uparrow}|^2 \, .
\end{align}

Let us examine the spin filtering associated with the reflection matrix (\ref{eqn:rm_2c}).
The spin conductance at the left lead is given by~\cite{YU2020},
\begin{align}
G^{}_{j;LL} =  {\mathrm Tr} [ {\bm 1}^{}_2 \otimes \sigma^{}_j ( {\bm 1}^{}_4 - r r^\dagger _{}) ]/(2\pi)\ , \label{eqn:Gj}
\end{align}
where $\sigma^{}_{j}$ ($j=x,y,z$) is the $j$th Pauli matrix.
Inserting Eq. (\ref{eqn:rm_2c}) into Eqs.~(\ref{eqn:Gj}) yields a finite spin conductance for the $z$ component of the spin
\begin{align}
G^{}_{z;LL} =( |r^{}_{2 \downarrow, 1 \uparrow}|^2 - |r^{}_{1 \uparrow, 2 \downarrow}|^2 )/\pi \ .
\end{align}
The spin conductances for the other components vanish, $G^{}_{x;LL} =G^{}_{y;LL}=0$.
The charge conductance is obtained from Eq.~(\ref{eqn:Gj}) by replacing the Pauli matrix with the unit matrix $\sigma_0={\bm 1}_2$,
\begin{align}
G^{}_{0;LL} =(2 -  |r^{}_{2 \downarrow, 1 \uparrow}|^2 - |r^{}_{1 \uparrow, 2 \downarrow}|^2)/\pi\ ,
\end{align}
It follows that the (normalized) spin polarization is
\begin{align}
P^{}_{z;L} = \frac{G^{}_{z;LL}}{G^{}_{0;LL}} = \frac{ |r_{2 \downarrow, 1 \uparrow}|^2 - |r_{1 \uparrow, 2 \downarrow}|^2 }{2 -  |r^{}_{2 \downarrow, 1 \uparrow}|^2 - |r^{}_{1 \uparrow, 2 \downarrow}|^2} \ .
\label{eqn:pzL}
\end{align}
Perfect spin-filtering $P^{}_{z;L} =1$ (or $P^{}_{z;L} =-1$) is achieved for
$|r_{2 \downarrow, 1 \uparrow}|^2=1$ (or $|r^{}_{1 \uparrow, 2 \downarrow}|^2=1$). The condition $|r_{2 \downarrow, 1 \uparrow}|^2=1$ corresponds to the Hamiltonian (\ref{eqn:Vp}).

The specific reflection matrix in the example (\ref{eqn:rm_2c}) demonstrates the possibility of spin filtering in a two-terminal molecule.
However, it is still a non-trivial task to construct a microscopic model realizing the two-orbital spin-filter.
In the next section, following our previous work~\cite{YU2020}, we describe a $p$-orbital helical atomic chain realizing the spin-filtering when it is attached to two terminals.

\section*{A $p$-orbital helical tight-binding model}
\label{sec:effective_hamiltonian}

\begin{figure}[ht]
\begin{center}
\includegraphics[width=1 \columnwidth]{setupDNA_.pdf}
\caption{
(a) Schematic picture of a $p$-orbital helical atomic chain, a toy model for a single strand of a double-stranded DNA.
${\bm R}(\phi^{}_{n})$ is the radius-vector to the $n$th site, within the Frenet-Serret frame [Eq. (\ref{Rphi})],
$\Delta h$ is the pitch, $\Delta\phi=2\pi/N$, and $\phi^{}_{n}=n p \Delta\phi$.
(b) Ladders threaded by a fractional flux $2\pi/N$.
The vertical lines represent the tunneling amplitudes
$\pm p \Delta_{so} \exp(ip \phi_n)$ connecting $\uparrow$- and $\downarrow$-spins on different orbitals at the $n$th rung.
The site index $n$ increases from left to right.
}
\label{fig:setupDNA}
\end{center}
\end{figure}

Here we summarize the construction of the tight-binding Hamiltonian describing the $p$-orbital helical atomic chain shown in Fig.~\ref{fig:setupDNA} (a) (see Appendix B of Ref.~\cite{YU2020}).
The vector from the origin to a point on a continuous helix of radius $R$ and pitch $\Delta h$ is,
\begin{align}
{\bm R}(\phi) = \left [  R \cos ( \phi),R \sin (p \phi), \Delta h \, \phi/(2 \pi) \right] \ ,
\label{Rphi}
\end{align}
where $p=1$ ($p=-1$) for a helix twisted in the right-handed (left-handed) sense.
In the Frenet-Serret frame, the tangent ${\bm t}$ (along  the helix), normal ${\bm n}$, and bi-normal ${\bm b}$ unit vectors at the point on the helix are
\begin{align}
&{\bm t}(\phi) = \left [ - \kappa \sin (\phi), p \kappa \cos (\phi), |\tau| \right]\ ,\nonumber\\
&{\bm n}(\phi) = \left [- \cos (\phi), - p \sin (\phi), 0 \right] \ ,\nonumber\\
&{\bm b}(\phi) = {\bm t}(\phi) \times {\bm n}(\phi) = \left [ p |\tau| \sin (\phi), - |\tau| \cos (\phi), p \kappa \right] \ ,
\label{tnb}
\end{align}
where the `normalized' curvature and torsion, $\kappa$ and $\tau$, are
\begin{align}
\kappa = \frac{R}{\sqrt{R^2+[\Delta h/(2 \pi)]^2}} , \;\; \tau = \frac{p \Delta h/(2 \pi)}{\sqrt{R^2+[\Delta h/(2 \pi)]^2}}  .  \label{eqn:tau}
\end{align}

The position of the $n$th site in the tight-binding scheme is specified by ${\bm R}(\phi_n)$, where the increment of $\phi$ between neighboring sites is $\Delta\phi=2\pi/N$, and $\phi^{}_n = p 2 \pi n/N$.
The tight-binding Hamiltonian of the helical atomic chain is,

\begin{align}
\hat{\mathcal H}^{}_{\mathrm mol} =&\Big (\sum_{n}^{} - \hat{c}_{n+1}^\dagger {\bm J} \otimes \sigma^{}_0 \tilde{c}^{}_n  + {\mathrm H.c.} \Big ) \nonumber \\ &+ \sum_{n}^{} \epsilon^{}_0 \, \hat{c}_{n}^\dagger \hat{c}^{}_{n} - 2 \Delta^{}_{so} \, \hat{c}_n^\dagger {\bm L} \cdot {\bm S} \hat{c}^{}_n  \nonumber \\ &+ \sum_{n}^{} K_{ {\bm t} } \, \hat{c}_n^\dagger [ ( {\bm t}(\phi_n) \cdot {\bm L})^2 - {\bm 1}_3 ] \hat{c}^{}_n  \nonumber \\ &+ \Delta \epsilon \, \hat{c}_n^\dagger  [ ( {\bm b}(\phi_n) \cdot {\bm L})^2 - ( {\bm n}(\phi_n) \cdot {\bm L})^2 ] \hat{c}^{}_n \ ,
\label{eqn:original_hamiltonian}
\end{align}
 where
\begin{align}
\hat{c}^{\dagger}_{n} = \left[ \begin{array}{cccccc} \hat{c}^{\dagger}_{n;p_x \uparrow} & \hat{c}^{\dagger}_{n;p_x \downarrow} & \hat{c}^{\dagger}_{n;p_y \uparrow} & \hat{c}^{\dagger}_{n;p_y \downarrow} & \hat{c}^{\dagger}_{n;p_z \uparrow} & \hat{c}^{\dagger}_{n;p_z \downarrow} \end{array} \right] \ , \label{eqn:vec_c}
\end{align}
is the vector of creation operators: $\hat{c}_{n;o \sigma}^\dagger$ is a creation operator of an electron residing at site $n$ with orbital $o$ and spin $\sigma$.
The first term on the right-hand side of Eq.~(\ref{eqn:original_hamiltonian}) describes the hopping between nearest-neighbor sites, with the hopping amplitude ${\bm J}$ being a $3 \times 3$ matrix.
Since the system is time-reversal symmetric, all the components  of the matrix are real
\begin{align}
\hat{\Theta} {\bm J} \hat{\Theta}^{-1}  = \hat{K} {\bm J} \hat{K}^{-1} = {\bm J} ,
\end{align}
where we have used $\Theta \hat{c}^{}_{n;\alpha} \Theta^{-1} = -i  {\bm 1}_3 \otimes \sigma^{}_y \hat{c}^{}_{n;\alpha}$.
In the second term, $\epsilon_0$ is the on-site energy.
The third term represents the intra-atomic spin-orbit interaction whose strength is denoted $\Delta^{}_{so}$.
Here ${\bm L}= (L_x,L_y,L_z)$ is the vector of the orbital angular-momentum operators,
\begin{align}
L^{}_x =& \left[ \begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & -i \\ 0 & i & 0 \end{array} \right], \\
L^{}_y =& \left[ \begin{array}{ccc} 0 & 0 & i \\ 0 & 0 & 0 \\ -i & 0 & 0 \end{array} \right], \\
L^{}_z =& \left[ \begin{array}{ccc} 0 & -i & 0 \\ i & 0 & 0 \\ 0 & 0 & 0 \end{array} \right]
,
\end{align}
and ${\bm S}={\bm \sigma}/2$ is the  vector of the spin angular-momentum, with ${\bm \sigma}$ being the vector of the Pauli matrices.

The other terms in the Hamiltonian describe the crystalline  fields created by neighboring atoms:
$K_{\bm t}$ is the orbital anisotropy energy along the tangential direction ${\bm t}(\phi_n)$.
$\Delta \epsilon$ is the difference between the orbital anisotropy energies along the normal direction ${\bm n}(\phi_n)$ and the bi-normal direction ${\bm b}(\phi_n)$.
We assume the leading orbital anisotropy energy is the one along the tangential direction, $K_{\bm t}$.
 This condition would effectively mimic the situation discussed in the helical tube models~\cite{Michaeli2019,Geyer2020,Gutierrez2013} in a simple manner.

\subsection*{Helical symmetry}
\label{Helical_symmetry}

In the following, we analyze the helical symmetry for the infinite chain.
Although the system is finite in the transport experiment, for a sufficiently long molecule, the transport properties would be dominated by the bulk electric states.
Then the system we consider possesses helical symmetry, i.e., the Hamiltonian is invariant under the screw operation~\cite{Otsuto2021}: a translation by one site and a rotation  by $p \Delta \phi = p 2 \pi/N$ along the $z$ axis,
\begin{equation}
\hat{D}_z(p \Delta \phi) \hat{T} \hat{\mathcal H}^{}_{\mathrm mol} \hat{T}^{-1} \hat{D}_z^{-1}(p \Delta \phi) =  \hat{\mathcal H}^{}_{\mathrm mol} .
\end{equation}
The translation operator $\hat{T}$ shifts the site index of the operators by one,
\begin{align}
\hat{T} \hat{c}_{n;o,\sigma} \hat{T}^{-1} = \hat{c}_{n+1;o,\sigma} \, .
\end{align}
The rotation operator around the $z$ axis is,
\begin{align}
\hat{D}_z(p \Delta \phi) = e^{-i (\hat{L}_z + \hat{S}_z) p \Delta \phi} ,
\end{align}
where
$\hat{L}_z=\sum_n \hat{c}_n^\dagger \left( L_z \otimes {\bm 1}_2 \right) \hat{c}_n$ and
$\hat{S}_z=\sum_n \hat{c}_n^\dagger \left( {\bm 1}_3 \otimes S_z \right) \hat{c}_n$.
Thus, the rotation changes Eq.~(\ref{eqn:vec_c}) to be
\begin{align}
\hat{D}_z(p \Delta \phi) \hat{c}_n \hat{D}_z^{-1}(p \Delta \phi) = e^{i L_z p \Delta \phi} \otimes  e^{i S_z p \Delta \phi} \hat{c}_n \, .
\end{align}

This operation does not change the on-site energy term in the Hamiltonian.
The intra-atomic SOI does not change either since $[L_z+S_z, {\bm L} \cdot {\bm S}]=0$.
The crystalline field terms in the third and forth lines of (\ref{eqn:original_hamiltonian}) do not change as well.
This can be verified by exploiting the following relations:
\begin{align}
{\bm t}(\phi_{n+1}) \cdot {\bm L} =&  e^{-i L_z p\Delta \phi} [ {\bm t}(\phi_{n}) \cdot {\bm L} ] e^{i L_z p\Delta \phi} \, , \\
{\bm n}(\phi_{n+1}) \cdot {\bm L} =&  e^{-i L_z p\Delta \phi} [ {\bm n}(\phi_{n}) \cdot {\bm L} ] e^{i L_z p\Delta \phi} \, , \\
{\bm b}(\phi_{n+1}) \cdot {\bm L} =&  e^{-i L_z p\Delta \phi} [ {\bm b}(\phi_{n}) \cdot {\bm L} ] e^{i L_z p\Delta \phi} \, .
\end{align}
The first (hopping)  term is transformed as,
\begin{align*}
\sum_{n}^{} \hat{c}_{n+1}^\dagger {\bm J} \otimes \sigma^{}_0 \hat{c}^{}_n
\to
\sum_{n}^{} \hat{c}_{n+2}^\dagger e^{-i L_z p \Delta \phi} {\bm J} e^{i L_z p \Delta \phi} \otimes \sigma^{}_0 \hat{c}^{}_{n+1} \, .
\end{align*}
Therefore, the hopping matrix ${\bm J}$ satisfies,
\begin{align}
\bm J =e^{-i L_z p \Delta \phi} {\bm J} e^{i L_z p \Delta \phi} \, . \label{eqn:constraint}
\end{align}
Consequently, the elements of the hopping matrix are parameterized by three parameters, $J$, $\alpha$ and $\varphi$ as,
\begin{align}
{\bm J} = J \left[ \begin{array}{ccc} \alpha \cos \varphi & - \alpha \sin \varphi & 0 \\ \alpha \sin \varphi  & \alpha \cos \varphi & 0 \\ 0 & 0 & 1 \end{array} \right] .
\end{align}
The three parameters are real numbers, since the Hamiltonian is time-reversal symmetric.

In our previous work~\cite{YU2020}, we demonstrated the two-terminal two-orbital spin filtering for a specific condition: $K_t \to \infty$, $\varphi=-p \Delta \phi$ and $\alpha=1$.
In the next section, we analyze the filtering for other various parameters.

\section{Band structure}
\label{sec:band_structure}

\begin{figure*}[ht]
\begin{center}
\includegraphics[width=16.4cm]{bandSOI_IJC.pdf}
\caption{Band structures for various parameters
:
(a) $\alpha=1$, $\varphi=-\Delta \phi$, $\tau=0$ and $N=4$,
(b) $\alpha=1.2$, $\varphi=-\Delta \phi$, $\tau=0$ and $N=4$
(c) $\alpha=\sqrt{2}$, $\varphi=\pi/4$, $\tau=0$ and $N=4$,
and
(d) $\alpha=\sqrt{2}$, $\varphi=\pi/4$, $\tau=0.48$ and $N=10$.
Other parameters are fixed as $p=1$, $\Delta_{so}=0.4 J$, $\epsilon_0=\Delta \epsilon=0$ and $K_t=7 J$.
The color scheme indicates the $z$ component of the average spin (red for $\uparrow$ spin and blue for $\downarrow$  spin, see the color bar).
}
\label{fig:band}
\end{center}
\end{figure*}

Figure \ref{fig:band} shows the band structure in the reduced zone scheme for various parameters, obtained by imposing periodic boundary condition $\hat{c}^{}_{MN+n}=\hat{c}^{}_n$ with $M \to \infty$.
The color scheme indicates the $z$ component of the average spin (red for $\uparrow$ spin and blue for $\downarrow$  spin, see the color bar).
In the following, we fix $p=1$, and $\epsilon_0=\Delta \epsilon=0$.
The strength of the SOI is taken as $\Delta^{}_{so}=0.4 J$.
This estimation is based on a band of width $4J \sim 120 \mathrm{meV}$~\cite{Gutierrez2012} and the intra-atomic SOI energy in carbon nanotubes $\Delta_{so} \sim 12 \mathrm{meV}$~\cite{HuertasHernandoPRB2006}.
The crystalline field is taken to be sufficiently large, as $K_t=7J$.
Due to  this strong crystalline field along the tangential direction of the helix, $K_t$, there are two energetically split bands.
The lower band is the $\sigma$-band and the upper band is the $\pi$-band.

Panel (a) in Fig. \ref{fig:band} shows the band structure for $\alpha=1$, $\varphi=-\Delta \phi$ and $\tau=0$, parameters for which the spin filtering is almost ideal.
It almost recovers our previous result in Ref.~\cite{YU2020}.
As seen, the lower band is spin degenerate.
The upper band can be effectively described by two decoupled ladders threaded by a fractional flux in each rung [Fig.~\ref{fig:setupDNA} (b)]~\cite{YU2020}.
The flux is induced by the intra-atomic SOI and the helical structure.
The Hamiltonians of two decoupled ladders, $\hat{\mathcal H}^{}_+$ and $\hat{\mathcal H}^{}_-$, are~\cite{YU2020},
\begin{align}
\hat{\mathcal H}^{}_\pm =&\sum_{n}^{} ( - J \hat{a}_{n+1;\pm}^\dagger \hat{a}^{}_{n;\pm} +{\mathrm H.c.}  ) \nonumber \\ & \pm  p \Delta^{}_{so} \, \hat{a}_{n;\pm}^\dagger \left[\begin{array}{cccc} 0 & e^{-i p \phi^{}_n} \\ e^{i p \phi^{}_n} & 0 \end{array} \right] \hat{a}^{}_{n;\pm} \ ,  \label{eqn:H_pm}
\end{align}
where
\begin{align}
\hat{a}^{\dagger}_{n;+} = \left[ \begin{array}{cc} \hat{a}^{\dagger}_{n;p_x \uparrow} & \hat{a}^{\dagger}_{n;p_z \downarrow} \end{array} \right],
\ \ \hat{a}^{\dagger}_{n;-} = \left[ \begin{array}{cc} \hat{a}^{\dagger}_{n;p_z \uparrow} & \hat{a}^{\dagger}_{n;p_x \downarrow} \end{array} \right] \, ,
\label{eqn:ann_FB}
\end{align}
and $\hat{a}^{}_n = e^{i L_z p \phi_n} \hat{c}^{}_n$.
The two ladders are time-reversed of one another $\hat{\Theta} \hat{\mathcal H}^{}_+ \hat{\Theta} ^{-1} = \hat{\mathcal H}^{}_-$~\cite{YU2020}, which is reminiscent of the quantum spin Hall system~\cite{Bernevig2013}.
At the boundary of the Brillouin zone,  $k=\pm \pi$, i.e., around $E=\pm 2 J \cos(\pi/N)$ (indicated by dotted lines), there are left-going $\uparrow$($\downarrow$)-spin states and right-going $\downarrow$($\uparrow$)-spin states.
The width of the energy window in which the helical states reside is compatible with the intra-atomic SOI, $\Delta^{}_{so}$.
These states are responsible for the spin filtering~\cite{YU2020}.
They are degenerate and, away from the prescribed condition, the degeneracy is lifted:
Panel (b) is drawn for $\alpha=1.2$, which also broadens the width of the bands.
Pairs of up and down spin states propagating at opposite directions are clearly observed around  $E=\pm 2 J \cos(\pi/N)$ (dotted lines).
Since we take $\tau=0$, the pitch is zero, $\Delta h=0$, and this ideal situation is realized only hypothetically.

In panel (c), we chose the parameter $\varphi \neq -\Delta \phi$.
In this case the Hamiltonian cannot be separated into two time-reversed ones as in Eqs.~(\ref{eqn:H_pm}).
The deviation from $\varphi = -\Delta \phi$ induces the mixing between $\sigma$  and $\pi$ orbitals.
As seen in the figure, there appears spin splitting in the lower band, induced by the inter-atomic SOI:
It results from the intra-atomic SOI combined with the mixing of the $\sigma$ and $\pi$ orbitals on neighboring atoms due to the curved geometry~\cite{HuertasHernandoPRB2006,VarelaPRB2016}.
The inter-atomic Rashba-like SOI induced in this way is reduced by $\sim J \Delta_{so}/K_t$ as compared with the bare intra-atomic SOI.

Panel (d) shows the result for parameters taken to mimic a DNA molecule.
The number of sites in each turn is $N=10$ corresponds to the number of base pairs.
The dimensionless torsion is taken to be $\tau=0.48$, as estimated for a B-form DNA: $R=1 \mathrm{nm}$ and $\Delta h=3.4 \mathrm{nm}$~\cite{Sasao2019}.
The finite torsion approximately reduces the energy window of the helical states by ${\kappa} \Delta^{}_{so}$~\cite{YU2020}.
In panel (d), one still finds helical states close to the top and bottom of the upper band.

Although our model realizes the spin current without breaking time-reversal symmetry, it is not sufficient to explain the experimentally observed magneto conductance~\cite{Xie2011,Mishra2020}.
Earlier papers~\cite{Yang2019,Naaman2020,Yang2020,Yang2020a} argued that the Onsager relations forbid linear magneto-conductance in chiral molecules which connect a ferromagnet with a normal metal, and attributed the observed non-linear magneto-conductance to electro-electron~\cite{Fransson2019} or electron-phonon~\cite{Fransson2020,Michaeli2022} interactions.
Here we showed that a linear spin conductance can be generated even when time-reversal symmetry is not broken.
However, the full explanation of the experimental observations probably also need these additional interactions.

\section*{Conclusion}
\label{conclusion}

Chirality-induced spin selectivity, invoked by the spin-orbit interaction, has been discussed within the single-particle picture.
The appearance of spin current in a time-reversal symmetric system when two orbital channels participate in the transport is demonstrated.

We analyze the helical symmetry of the infinite $p$-orbital helical atomic chain with intra-atomic spin-orbit interaction and a strong crystalline field along the helix introduced in Ref.~\cite{YU2020}.
The helical symmetry imposes constraints on the nearest-neighbor $p$-orbital hopping matrix elements:
They are parameterized by 3 independent real numbers.
We explore parameters away from the condition analyzed in Ref.~\cite{YU2020}, in which the ideal spin-filtering is realized.
We demonstrate that for a wide range of parameters, pairs of up and down spins propagating along opposite directions survive around the top and the bottom of the band.
These helical states in the infinite atomic chain would be responsible for spin filtering in the two terminal transport experiments.
The deviation from the ideal spin-filtering condition would not spoil our previous findings~\cite{YU2020}.

As pointed out in Ref.~\cite{Gutierrez2013}, the two orbitals can be on the same helix and thus the intra-atomic SOI is sufficient for the spin-filtering.
In our simple $p$-orbital helical atomic chain, the typical energy scale of the helical states is approximately the intra-atomic SOI times the curvature of the helix.
The intra-atomic SOI is typically larger than the inter-atomic SOI induced by the mixing between $\pi$- and $\sigma$-bands and thus would be a likely candidate for explaining the CISS effect.

\section*{Acknowledgments}

This work was supported by JSPS KAKENHI Grants No. 18KK0385, No. 20H01827 and No. 20H02562.

\begin{shaded}
\noindent\textsf{\textbf{Keywords:} \keywords}
\end{shaded}

\setlength{\bibsep}{0.0cm}
\bibliographystyle{Wiley-chemistry}


text:
\section{Introduction}\label{sec1}
The task of generating new data from samples has always exerted a particular fascination in machine learning, both because of the potential for almost endless streams of new and original data, as well as for the implications on the knowledge extracted by
a model about the data manifold. It is clear that the effectiveness of generative techniques
crucially depends on data representation, and different encodings may result in
more or less entangled combinations of the different explanatory factors of variation
behind the data \cite{BengioCV13,KimMnih18}. The key idea behind unsupervised
learning of disentangled representations is that real-world data depends on a relatively
small number of explanatory factors of variation which can be compressed and recovered by unsupervised learning techniques \cite{Locatello19,helpful19,Locatello20}. Strictly related to
representation learning, the task of {\em exploration} of the latent space of
generative models aims to understand the ``arithmetic" of the variational factors \cite{Radford15,Closed-Form-fact}, and
the effect that particular trajectories inside the latent space could produce in the visible
domain \cite{Shen22, learning_latent18, li2021discovering}.

In spite of the huge amount of work devoted to the exploration of latent spaces, relatively
little attention has been so far devoted to the problem of {\em comparing} the latent space
of different generative techniques, i.e. to the problem of locating the internal
representation $z_X$ of $X$ in a given space starting from its representation in the latent space of a different model (see Figure \ref{fig:mapping}).

\begin{figure}[h]
\begin{center}
\includegraphics[width=\columnwidth]{mapping.png}
\caption{
\label{fig:mapping} Given a generative model, it is usually possible to have an encoder-decoder pair mapping the visible space to the latent one (even GANs can be inverted, see Section \ref{sec:inversion}).
From this assumption, it is always possible to map an internal representation in a space $Z_1$ to the corresponding internal representation
in a different space $Z_2$ by passing through the visible domain. This provides a supervised set of input/output pairs: we can try to learn a direct map, as simple as possible.
The astonishing fact is that a simple linear map gives excellent results, in many situations. This is quite surprising, given that
both encoder and decoder functions are modeled by deep, non-linear
transformations.}
\end{center}
\end{figure}

\begin{figure*}
     \centering
     \begin{subfigure}[b]{\textwidth}
         \centering
         \includegraphics[width=\textwidth]{space1.png}\vspace{-.2cm}
         \caption{Relocation of Type 1, between latent spaces relative to different training instances of the same generative model, in this case a particular Variational Autoencoder \cite{SVAE}. The two reconstructions are almost identical.}
         \label{fig:reloc_type_1}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{\textwidth}
         \centering
         \includegraphics[width=\textwidth]{space_type2_1.png}
         \caption{Relocation of Type 2, between a Vanilla VAE and a state-of-the-art
         Split-VAE \cite{SVAE}. The SVAE produces better quality images, even if not necessarily in the direction of the original: the information lost by the VAE during encoding cannot be recovered by the SVAE, which instead makes a reasonable guess.}
         \label{fig:reloc_type_2}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{\textwidth}
         \centering
         \includegraphics[width=\textwidth]{space_type3_1.png}
         \caption{Relocation of Type 3, between a vanilla
          GAN and a SVAE. Additional examples involving StyleGAN are given in Section\ref{sec:StyleGANspace}. To map the original image
(first row) into the latent space of the GAN we use an inversion network.
Details of reconstructions may slightly differ, but colors pose and the overall
appearance is surprisingly similar. In
some cases (e.g. the first picture) the reconstruction
re-generated by the VAE (from the GAN encoding!) is closer to
the original than that of the GAN itself.}
         \label{fig:reloc_type_3}
     \end{subfigure}
        \caption{Examples of relocations of different Types. In the first row we have the original, in the second row the image reconstructed by the first generative model, and in the third row the image obtained by the second model after linear relocation in its spcae.}
        \label{fig:three}
\end{figure*}

The key questions we are interested in are the following:
\begin{enumerate}
    \item Do different trainings of the same generative model induce the extraction of similar
    features from data, and hence substantially isomorphic spaces up to, say permutations or linear
    transformations? We refer to this type of transformations as being of Type 1.
    \item Do different architectural models driven by common learning objectives (e.g. maximizing log-likelihood) learn similar
    features? How much do the extracted features depend on the neural network structure?
    We refer to this type of transformations, between spaces of variants of models in
    the same class, as being of Type 2.
    \item Finally, what is the influence of the learning objective on the internal representation?
    Is e.g. a Generative Adversarial Network learning the same features of a Variational AutoEncoder? We refer to these transformations as
    being of Type 3.
\end{enumerate}
Any answer, whether positive or negative, could substantially improve our knowledge of generative techniques.

Our surprising preliminary results, reported in this article, seem to suggest that (provided models have not been taught or explicitly conceived to act differently)
it seems to be possible to pass from a latent space to another by means of a {\em simple linear mapping} preserving most of the
information.

This linear transformation may be computed directly through linear regression, but we advocate a learning-based technique based on a suitable small ``support set" of data samples enucleating, in the visible space, the key variational factors of the data manifold. When we say ``small", we mean that the set has a cardinality comparable with the number
of variables in the latent space (so, {\em really small}): for instance, in the case of CelebA, we experimented with a support set of 150 images. Locating these 150 samples in the two spaces is enough to allow the definition of a relocation map for all data.

The main results of our investigation are summarized in Figures~\ref{fig:three}. Figure~\ref{fig:reloc_type_1} describes an example of relocation between different trainings of a same network (relocation of Type 1); Figure~\ref{fig:reloc_type_2} is relative to the relocation between
different models of a same class---two different VAEs, in this case (relocation of Type 2); Figure~\ref{fig:reloc_type_3} is an example of relocation from a VAE to a GAN, that is between different models with different learning objectives (relocation of Type 3). While details may slightly differ, especially
for transformations between different generative models, the
overall appearance (pose, colors and background) is substantially
preserved. Considering the non-linearity of these generative processes, the result is, at a first glance, quite surprising:
pairs of points related by a simple linear mapping in the latent spaces of two different generative models are decoded by the respective decoders in closely related---in some cases almost identical---images!

\subsection*{Structure of the article}
The structure of the article is the following. We start by providing, in Section \ref{sec:generative}, a quick introduction to generative modeling, and in particular
to latent variables models, comprising the popular Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs); in this section, we also discuss the problem of inverting GANs. Section \ref{sec:semantic_interpretation} cover the domain of
semantic exploration of latent spaces, representation learning and disentanglement.
In Section \ref{sec:datasets} we start introducing the datasets, the models and the methodology that we used for our experiments. Since we focus on linear transformations,
they can be defined by a small set of points, that we call Support Set: locating the
points in the Support Set in the two latent spaces is enough to define the transformation. Our approach to get a good Support Set is discussed in
Section \ref{sec:support_set}. In Section \ref{sec:results} we give numerical results about the mappings (visual examples, more readily interpretable, are spread over the article).
Section \ref{sec:StyleGANspace} is devoted to the discussion of the latent space of StyleGAN,
that seems to present some pathological issues: many faces in the CelebA dataset lie
outside of its generative range. Even in this case, however, provided we confine the transformation to
the StyleGAN subspace, we discover interesting linear mapping to other spaces.
Conclusion and future works are discussed in Section \ref{sec:Conclusions}.
Additional material is given in appendix:
a detailed description of the models used in this work
(Section \ref{appendix:models}), a full list of all images in the CelebA Support Set
(Section \ref{appendix:labels}).

\section{Generative Modeling}
\label{sec:generative}
Generative modeling is the task of learning the high-dimensional probability distribution of a data manifold starting from a representative set of
samples. When successfully trained, generative models can be used to create new samples from the underlying distribution, possibly providing estimations of their likelihood.
The learning process provides an essential and valuable insight of the kind of features
used to encode the distribution, and the way the model ``interpreted'' and ``understood'' data.

At the heart of generative techniques there is a relatively small set of
techniques \cite{generative_introduction, generative_survey_2018}: Auto-Regressive models \cite{pixelrnn, pixelsnail}, Flow models \cite{kingma2018glow, dinh2016density, voleti2021multi, flow_methods}, Energy-based models \cite{ho2020denoising, du2019implicit, energy_models_survey} and Latent-Variable models, particularly GANs \cite{large_gan, InvidiaGAN18, Radford15} and VAEs \cite{Kingma13, vq_vae_2, IWAE}.

In this article, we shall mostly focus on the popular and effective Latent-Variable models, that is models
where the actual distribution $p(x)$ of a data point $x$ is expressed through marginalization
over a vector $z$ of {\em latent variables}:
\begin{equation*}
    p(x) = \int_{z} p(x\rvert z)p(z) dz = \EX_{p(z)} [p(x\rvert z)]
\end{equation*}
where $z$ is the latent encoding of $x$ distributed with a known distribution $p(z)$ named {\em prior distribution}.
The distribution $p(x\rvert z)$ is usually learned by a deep neural network; after training
it can be used to generate new samples via ancestral sampling:

\begin{enumerate}
    \item sample $z \sim p(z)$;
    \item generate $x \sim p(x\rvert z)$.
\end{enumerate}

\subsection{Variational Autoencoders}
\label{sec:vae}

A Variational AutoEncoder (VAE) \cite{VAEKingma} has a structure similar to a classical auto-encoder \cite{DeepBook,autoencoders}, being composed of an {\em encoder} producing a latent vector $z$ from an input $x$ and of a {\em decoder} which reconstructs the input $\hat x$ from a latent code; the two components are simultaneously trained using, e.g., a mean squared error loss $\mid x - \hat x \mid_2$. However, in order to regularize the latent space, which is a precondition to support semantically meaningful generation \cite{generative_introduction}, latent variables are interpreted as parameters of a local distribution $q(z\rvert x)$ and a Kullback-Leibler component $KL(q(z\rvert x)\; \| \;\mathcal{N}(0, 1))$ is added to the reconstruction loss,  with the purpose of pushing the marginal distribution $q(z)$ towards a standard Gaussian $\mathcal{N}(0, 1)$.
Balancing of these two loss components, usually via a $\gamma$ or $\beta$ parameter, is crucial for better generation and learning of disentagled features \cite{vae_loss_balancing, beta-vae17, understanding-beta-vae18}.

Several issues affect the performance of VAEs, most importantly blurriness of generated images \cite{VAEGreen}. As such, many variants
have been proposed over the years to improve results by addressing the mismatch between the aggregate inference distribution $q(z)$ and the prior $p(z))$. These comprise: quantization of the latent code (VQ-VAE \cite{VQVAE}), use of normalizing flows (Hybrid VAE \cite{autoregressive16}), two-Stage architectures \cite{2_stage_vae}, and hierarchical models \cite{DRAW,kingma2018glow}.

\subsection{Generative Adversarial Networks}
\label{sec:gan}
In a Generative Adversarial Network (GAN) \cite{GANs, tutorial-GAN, Radford15} a {\em generator}, acting as a sampler for the desired distribution, is jointly trained with a {\em discriminator}, evaluating  the output of the generator by attempting to distinguish real from generated  (``fake'') data. This can be formalized in the form of a zero-sum game, where one agent's gain is another agent's loss; the generator and the discriminator must be trained alternately, freezing the respective adversarial component; at the end of the process the generator is supposed to win, producing samples that the discriminator is unable to distinguish from real.

GANs are known to have unstable training and several issues among which the well known mode collapse phenomenon \cite{tutorial-GAN}. Indeed, multiple variations for the loss function have been studied over time \cite{GAN_losses_survey}, including the Wasserstein loss \cite{Wasserstein_GAN}, least squares loss \cite{LSL_GAN} or the introduction of a penalty term for the discriminator \cite{DRAGAN}. Furthermore, a myriad of variations on the structure itself have been proposed, among which: maximizing the mutual information between specific latent variables \cite{InfoGAN}; exploiting pairs of GANs to perform style transfer between images in distinct datasets \cite{cycleGAN}; GANs with attention layers \cite{attention_GAN}.

A particularly interesting series of works come from the application of style transfer concepts to GANs (StyleGAN and its successors \cite{karras2019style, karras2020analyzing, karras2021alias}). StyleGAN builds on Progressive GANs \cite{Progressive_GAN}, whose structure is unchanged from that of a baseline GAN but is implemented progressively: the architecture is trained starting from down-sampled images at very low resolution, and at each progression step the input size is increased while additional layers are introduced to both generator and discriminator.

StyleGAN further builds on this structure by adding to the generator (\textit{Synthesis} network) a fully connected \textit{Mapping} network which takes the usual seed $z \in Z$ and produces a ``style'' vector $w \in W$. This vector is then specialized per-layer through Adaptive Instance Normalization (AdaIN), which according to the authors produce a behavior similar to style-transfer. Furthermore, a small amount of noise is added to all blocks of the Synthesis network to better fill in the output details. The full structure of StyleGAN can be seen in Figure \ref{fig:styleGAN}.

\begin{figure}[ht]
\begin{center}
\includegraphics[width=\columnwidth]{Summary-of-the-StyleGAN-Generator-Model-Architecture.png}
\caption{Structure of the StyleGAN generative network (picture from \cite{karras2019style}).  Observe: (1) the two distinct latent spaces $Z$ and $W$;
(2) the {\em mapping network} taking a randomly sampled point $z\in Z$ as input and generating a {\em style vector} $w$; (3) the use of Adaptive Instance Normalizationation, or AdaIN (Blocks A), to apply style vectors after each convolution layer of the Synthesis network;
(4) the exploitation of noise as an additional source of randomness passed through learned scaling layers (Blocks B).
\label{fig:styleGAN}}
\end{center}
\end{figure}

\subsubsection{GAN Inversion}
\label{sec:inversion}
The generator of a GAN usually takes as input a seed $z \sim \mathcal{N}(0, 1)$, and has a role directly comparable to that of a VAE decoder. However, GANs lack a direct encoding process of the original input sample, unlike a VAE encoder. If, as is the case for our study, both generative and encoding processes are needed, a third neural network has to be added to a pre-trained GAN as a sort of plug-in encoder. This re-coder component is known as an inverse GAN, and building an accurate re-coder is a known problem in the literature \cite{gan_inversion_survey}.

\begin{figure*}[ht]
\begin{center}
\includegraphics[width=\textwidth]{gan_inv1.png}
\caption{Results of our own network for StyleGAN inversion. Images in the first row have been generated by StyleGAN; they are re-coded into the W space and regenerated (second row).
The two images are hardly distinguishable. However, as we shall see in Section~\ref{sec:StyleGANspace}, inversion can be more problematic for images outside the generative range of the model; in principle,
a good generative model should be able to produce any sample, provided it is
not too atypical.\label{fig:gan_inv1}}
\end{center}
\end{figure*}

Several approaches to inversion have been explored \cite{perarnau2016invertible, bau2020semantic, daras2020your, anirudh2020mimicgan}, mostly for editing applications, the simplest being SGD optimization \cite{gan_inversion_optimization} or a learning-based approach such as using a neural network trained on generated images to reconstruct the original latent vector using a mean squared error loss $\mid z - \hat z \mid_2$, with the advantage that over-fitting is never an issue since training is not constrained to samples of the original data. Hybrid methods combining both efforts have also been explored \cite{gan_inversion_first_hybrid, gan_inversion_hybrid_2}.

Recent works have focused mostly on the inversion of the popular StyleGAN, building on previous work with a variety of inversion structures and minimization objectives \cite{stylegan_inversion, collins2020editing, abdal2020image2stylegan, stylegan_overparameterization, alaluf2022hyperstyle} with the aim of generalization to any dataset. However, we used a simpler and narrow approach by developing our own StyleGAN inverter for the $W$ space using a naive recoding network. It works surprisingly well for commonly generated samples, with a final mean square error close to $0.0040$. We show some examples of recoding in Figure~\ref{fig:gan_inv1}.

\section{Semantic Interpretation of Latent Spaces}
\label{sec:semantic_interpretation}
The latent space of a generative model efficiently synthesizes information from data, however the resulting compressed vectors cannot be easily mapped onto understandable features such as labels or attributes. Therefore, it is also unknown how exactly a model learns from data, in terms of how well it encodes its features, biases and human-meaningful characteristics. At the same time, this knowledge could fundamentally influence the quality of models and provide a foundation on which to improve their performances without relying solely on empirical and qualitative analyses.

Conditional architectures \cite{cVAE, InfoGAN} can indeed mitigate this issue by explicitly feeding features alongside samples during training, but in doing so they remodel the task as a supervised problem with respect to the classes on which conditioning is done, with all other data features remaining non-explainable.
These approaches do not provide interesting information about the way the neural
network understand data, and for this reason, they will not be discussed in this work.

\subsection{Exploration and Disentaglement}
Many works attempt to understand the latent space of GANs by performing exploration on the latent space, that is, they introduce small nudges in a direction based on the empirical principle that they will correspond to a small change in the corresponding generated data. The approach can be particularly useful for image editing, as once a semantically meaningful direction is found (eg. color, pose, shape), it can be traveled to tweak an image, introducing a desired feature without the need for a conditional generation model. InterFaceGAN \cite{Shen22} supposes that for a given feature taking values in $(-\infty; \infty)$ there exists an hyperplane in the latent space whose normal vector allows for a gradual modification of the feature, which can be found e.g. via an SVM \cite{SVM}. Further work based on this idea searches for these directions as an iterative or an optimization problem \cite{li2021interpreting} and also extend it to controllable walks in the latent space \cite{li2021discovering}.

A different, more systemic approach to the problem is by \cite{Closed-Form-fact}, which use a closed-form equation to find the editing direction $n_i$ applied per-layer $i$ of a generator, which is then composed to find the overall direction $n$. Another approach of the same ``arithmetic" flavor comes from \cite{Exploration_PCA}, where a generative application of PCA with a non-linear kernel is used to determine the hidden features of a small-scale dataset, without any reliance on a particular generative model.

Much less work on exploration has been devoted to VAEs. An example is given by \cite{learning_latent18}, which however works on a conditional architecture, in order to produce lower-dimensionality subspaces that are easier to analyze.

\section{Datasets, Models, Methodology}
\label{sec:datasets}
\subsection{Datasets}
As stated in the abstract, we confined our analysis to the familiar and largely investigated data manifold of human faces.
Our dataset of reference is CelebA \cite{celeba}, including its higher-quality version CelebAHQ \cite{InvidiaGAN18}. Images taken from CelebA have been aligned as per their paper \cite{celeba} and then cropped to size $128\times 128$ with a $y$ offset of $45$ and an $x$ offset of $25$ in order to remove as much background information as possible. The crop is then downsampled to size $64\times 64$ with bilinear interpolation).

CelebaHQ is a dataset of 30K images at resolution
$1024\times 1024$, obtained from a subset of CelebA with a
complex methodology explained in appendix C of \cite{Progressive_GAN},
comprising a sophisticated preprocessing phase, super-resolution
techniques, and selection of best quality samples.

\subsection{Generative models}
\label{subsec:models}
For our experiments we took into considerations 4 different models, two GANs and
two VAEs; in each class, we investigated a basic, average quality "vanilla" version and a more sophisticated, state-of-the-art model. A summarizing Table \ref{table:models_basic} for these models is provided. More in-detail, we have investigated the following architectures:
\begin{enumerate}
    \item Vanilla VAE \cite{VAEKingma} using $\gamma$ balancing \cite{vae_loss_balancing} with a latent dimension $Z$ = 64 trained on the cropped CelebA;
    \item Vanilla GAN \cite{GANs} with a latent dimension $Z$ = 64 trained on the cropped CelebA;
    \item SVAE \cite{SVAE} with a latent dimension $Z$ = 150 trained on the cropped CelebA;
    \item StyleGAN \cite{karras2019style} pre-trained on CelebA-HQ, which has a latent dimension $Z$ of size $512$ and a style-vector latent dimension $W$ of the same size.
\end{enumerate}
The structure of the StyleGAN has been already briefly discussed in Section \ref{sec:gan}. The in-depth architecture of the other models, not central to the topic of this article, is given in Appendix \ref{appendix:models}.

The dimension of the latent space and the resolution of
the different models is summarized in Table \ref{tab:modelinfo}.

\begin{center}
\begin{table}
\centering
\begin{tabular}{ |c|c c|}
 \hline
 \textbf{Model} & \textbf{Latent dim} & \textbf{Resolution} \\
 \hline
 GAN & 64 & $64 \times 64$ \\
 VAE & 64 & $64 \times 64$  \\
 SVAE & 150 & $64 \times 64$ \\\hline
 StyleGAN & 512 & $1024 \times 1024$ \\
 \hline
\end{tabular}
\caption{\label{table:models_basic} Dimension of the Latent Space and Resolution for
the different models.
\label{tab:modelinfo}}
\end{table}
\end{center}

\subsection{Methodology}
For each one of the previous models, apart StyleGAN where we only had at our disposal a single set of pre-trained parameters, we trained and tested five different instances.
When reporting values in the results, if not differently stated, they have to be
understood as an average over the different trainings.

Mapping between different models (transformations of Type 2 and 3) can have a lot of additional issues. Firstly, the two latent spaces may have sensibly different dimensions, for instance $512$ for StyleGAN versus $150$ for the SVAE and  for the other models, and may work at different resolutions, for instance $1024\times 1024$ for StyleGAN versus $64\times 64$ for the other models. Furthermore, the two generative models may have been trained on the two different datasets which, albeit similar, have different data
and different crops. To this aim, when
passing from CelebA-HQ to CelebA we take a simplified crop of dimension $880\times 880$ with an
height offset of 20 and a width offset of 60, which is then downsampled to size $64\times 64$ with bilinear interpolation.

Since we are interested in linear mappings, the transformations may be defined by
a small set of "corresponding" points common to both spaces: this is what we call a Support Set.
Our methodology to build it is defined in Section \ref{sec:support_set}.
The support Set is defined in the visible domain; we trace their respective encodings
in the different spaces, and define the map by linear regression with
mean squared error as a loss. When we cannot use a Support Set, we may directly work
with the whole visible domain (or the subset of the visible domain common to the two
spaces), sampling minibatches in it.

\section{Support Set}
\label{sec:support_set}
In this Section we explain the technique used to build a
small support set of examples driving the linear transformation.
This is based on the following steps, each one detailed in a respective subsection:
\begin{description}
\item[features ordering] \hspace{1mm} we order latent
variables according to their relevance for reconstruction, using a suitable metric discussed below;
\item[features selection] \hspace{1mm} we select  a small number $n$ of particularly significant latent variables; $2^n$ must be lower
than the cardinality of the support set;
\item[sample selection] \hspace{1mm} we select points in the space belonging
to extremal regions with respect to the selected features.
\end{description}

\subsection{Features ordering}
\label{sec:features_ordering}
Feature importance---the task of associating
a score to input features based on how useful they are for
solving a specific problem---is a major subfield of Machine
Learning. In the case of generative modeling, the goal is
to maximize the (log)likelihood of data, and it is natural
to associate a score to features according to their
contribution to this objective. It is worth observing that
different techniques, like e.g. PCA, would not be beneficial
to this aid, due to the shape of the prior latent
distribution which is, typically, a spherical Gaussian
distribution\footnote{Even the potential mismatch between the prior and
the aggregate inference distribution in the case of VAEs
cannot be exploited by PCA, since this technique only takes
into consideration the first two moments of the distribution.}.

Our feature importance technique requires an encoder in addition to a decoder: it fits particularly well with VAEs, but
it can be generalized to GANs by exploiting a re-coder network (see Section~\ref{sec:inversion}).
Specifically, in order to evaluate the contribution of the variable to the loss function, we compute over a large number of data the average difference between the reconstruction error when the latent
variable is zero-ed out with respect to the case when
it is normally taken into account.
We call this information the {\em reconstruction gain}
associated with the latent variable. It was introduced in
\cite{sparsity} where it was used to compare the reconstruction error and
the Kullback-Leibler divergence on a per-variable base, in order to clarify the variable collapse phenomenon  \cite{BurdaGS15,RobustPCA,PosteriorCollapse}.

We did the experiment on the SVAE, which in our experiments has a latent space of 150 variables. In Figure~\ref{fig:info_gain} we show the information gain relative to all its latent variables, ordered by relevance.

\begin{figure}[ht]
\begin{center}
\includegraphics[width=\columnwidth]{info_gain.png}
\caption{Information gain for all variables, in
decreasing order. Only a bunch of variables are in charge
of the macroscopic factors of variations.
\label{fig:info_gain}}
\end{center}
\end{figure}
Eleven variables have a score higher than 10, although the distribution has a relatively long tail: the first 20 variables are responsible for about 75\% of the information.

\begin{figure*}[ht]
\begin{center}
\includegraphics[width=\textwidth]{latent_vars.png}\vspace{-0.8cm}
\caption{Effect of the seven most informative latent variables
in the visible domain. Each image is obtained by varying a
specific variable in the range [-2.25; +2.25].
Considering these are the variables with the largest information
gain, it may be argued that their impact is less pronounced than expected.
Most of the variables are associated with a change in luminosity
of all or part of the image, possibly associated with modifications in hair color, source of illumination
and tiny variations in the pose. In the case of variable 21,
there seems to be progressive Female-Male transition (and vice-versa for variable 114).
\label{fig:latent_vars}}
\end{center}
\end{figure*}

\subsection{Feature Selection}
We keep a small number of the most informative variables.
For the way we shall use it, this number must be smaller
than the logarithm of the cardinality of the support set.
In our case, we aim to a support set of dimension 150,
so we focus on the 7 most relevant variables.

In Figure~\ref{fig:latent_vars} we show examples of the effect of some of these variables on generated images: we take a random point and progressively modify the
given variable in the range between -2.25 and 2.25 (remember that
the latent space standard deviation is 1).

\subsection{Sample selection}\label{ss:sample_selection}
Finally, we divide the latent space in sectors corresponding
to extreme values for the previously selected variables, and
pick up samples in these sectors.
\begin{figure}[ht]
\begin{center}
\vspace{-.5cm}\includegraphics[width=\columnwidth]{sectors.png}\vspace{-0.8cm}
\caption{Example of sectors in 3 dimensions (cropped to distance 2
from the origin). The distance between sectors is equal to twice a
configurable threshold. We work with the 7 most informative latent
variables, obtaining a total of $2^7=128$ sectors.
\label{fig:sectors}}
\end{center}
\end{figure}

More precisely, having defined a threshold {\em th} and a "direction" {\em dir} given by a $+/-$ sign for each selected variable,
a sector defined by the pair $(th,dir)$ is the set of points with
direction compatible with {\em dir} and at a distance from the origin
larger than {\em th}. Since we consider all possible directions,
this gives a total of $2^n$ sectors where $n$ is the number of selected variables (for a fixed {\em th}).
In each sector, we pick up a sample at random (enlarged {\em th} sectors become progressively less inhabitated).

It is interesting to observe that the number of latent points
in the dataset within different sectors at a given
threshold is far from uniform. This seems to be a confirmation
that the actual image distribution is far from the desired Gaussian
normal prior and, in a VAE, a symptom of the potential mismatch between the generative prior and the aggregate inference distribution computed by the encoder, which is a well known
and problematic aspect of VAEs
 \cite{ELBOsurgery,rosca2018distribution,aboutVAE}.
Attempts to solve this issue have been
made both by acting on the loss function \cite{WAE} or by exploiting more complex priors \cite{autoregressive16,Vamp,resampledPriors}; the actual effects on the latent space of these techniques is an interesting
research direction for future investigations.

In Figure we show typical inhabitants for a few given sectors. As expected,
they share macroscopic features like background color, pose, hairs, and illumination.

\begin{figure}[ht]
\begin{tabular}{c}
\includegraphics[width=\columnwidth]{sector_102.png}\vspace{-.4cm}\\
\tiny{sector 102}\vspace{-.2cm}\\
\includegraphics[width=\columnwidth]{sector_109.png}\vspace{-.4cm}\\
\tiny{sector 109}\vspace{-.2cm}\\
\includegraphics[width=\columnwidth]{sector_126.png}\vspace{-.4cm}\\
\tiny{sector 126}\vspace{-.2cm}\\
\includegraphics[width=\columnwidth]{sector_60.png}\vspace{-.4cm}\\
\tiny{sector 60}
\end{tabular}
\caption{Examples of data in different sectors. For each sector,
images are different,
but share macroscopic features: background color, pose, hairs, illumination,
etc.
\label{fig:samples_in_sectors}}
\end{figure}

Part of the 128 images resulting from our selection process are depicted in
Figure~\ref{fig:support_set1}. The complete list of labels for the support set
is reported in the appendix.
\begin{figure}[ht]
\includegraphics[width=\columnwidth]{set1.png}
\caption{Part of the images in the support set resulting from our
selection process. The samples are supposedly representative of the
principal factors of variations in the dataset. Additional examples are given in
the appendix.
\label{fig:support_set1}}
\end{figure}
The samples in the support set occupy ``extreme"
positions in the latent space with respect to the most informative
directions: for this reason, they as supposed to be representative of the
principal factors of variations in the dataset.

As a partial confirmation of the previous hypothesis, we expect the distance
between elements in the support set to be sensibly higher than the average
distance between points in the full dataset. This is actually the case:
the mean squared error between random CelebA images is 0.116, versus 0.183 for
samples in the support set.

\section{Results}
\label{sec:results}
This Section contains numerical results relative to the transformation between latent spaces.
The discussion of StyleGAN, for its relevance and some
interesting pathological issues, will be postponed to the next Section.

Here, with we shall use the names VAE, GAN and SVAE to refer to our specific implementations
of these models, discussed in Section \ref{subsec:models} and detailed in appendix \ref{appendix:models}.

We build a set of correspondent input-output pairs by encoding the Support Set (or the full
set of visible data) into the two latent spaces. Then, we directly build a linear map by linear regression, minimizing the mean squared error between target and computed latent vectors.

For each transformation, we provide three values:
\begin{description}
\item[L-MSE] Latent Mean Squared Error. This is the loss of the model, namely the mean squared error between the target vectors and those computed by the model;
\item[R-MSE] Reconstruction Error. This is the mean squared error between the original image in the visible domain and its reconstruction via the source generative model;
\item[M-MSE] Mapped Error. This is the mean squared error, in the visible domain, between original images and images reconstructed by the target generative model after linear mapping.
\end{description}
The three errors are graphically described in Figure \ref{fig:errors}.

\begin{figure}[ht]
\begin{center}
\includegraphics[width=.9\columnwidth]{errors.pdf}
\end{center}
\caption{Relocations Errors. An original point $o$ in the visible domain is mapped into internal representations $z_1$ and $z_2$ in the latent spaces $Z_1$ and $Z_2$. The map $M$ is trained to reconstruct $z_2$ from $z_1$: L-MSE is the mean squared error between $z_2$ and $M(z_1)$. R-MSE is the mean squared error, in the visible domain, between $o$ and its reconstruction according to the first generative model. M-MSE is the mean squared error, in the visible domain, between $o$ and $D_2(M(z_1))$.
\label{fig:errors}}
\end{figure}
The latent error L-MSE is not easily deciphered; the comparison between R-MSE and M-MSE provides a more intelligible information about the quality of the translation.

Results are given in Table \ref{tab:results}.

\begin{table}
\centering
\begin{tabular}{ |c c|c c c| }
 \hline
 \textbf{From} & \textbf{To} & \textbf{L-MSE} & \textbf{R-MSE} & \textbf{M-MSE} \\\hline
 VAE & VAE & 0.03 & 0.0073 & 0.0103\\
 VAE & SVAE & 0.72 & 0.0073 & 0.0105\\
 VAE & GAN & 0.49 & 0.0073 & 0.0339\\
 GAN & VAE & 0.50 & 0.0284 & 0.0254\\
 GAN & SVAE & 0.86 & 0.0284 & 0.0275\\
 GAN & GAN & 0.43 & 0.0284 & 0.0335\\
 SVAE & VAE & 0.195 & 0.0035 & 0.0125 \\
 SVAE & GAN & 0.63 & 0.0035 & 0.0388 \\
 SVAE & SVAE & 0.20 & 0.0035 & 0.0067 \\
 \hline
 \hline
\end{tabular}
\caption{\label{table:mapping_results}Mapping results for different model pairs: (L-MSE) MSE between the target and Mapped Latent vectors; (R-MSE) MSE between the original and Reconstructed (encoded-decoded) images; (M-MSE) MSE between the original and mapped images via the learned linear mapping. When source and target coincide, we mean different trainings of the same model (Type 1 transformations).}
\label{tab:results}
\end{table}
For the sake of comparison, it it worth to recall that the mean squared error between CelebA images
is $0.116$; in all models the M-MSE is always below $0.039$.

\section{The StyleGAN space}
\label{sec:StyleGANspace}
The ``extreme'' nature of the images in the Support Set makes them a very natural benchmark of the expressiveness of generative models: is it possible to reconstruct these images by passing them through an encoding-decoding process?

\begin{figure*}[ht]
\includegraphics[width=\textwidth]{support_gan4.png}
\caption{StyleGAN inversion on images in the Support Set.
The macro structure (background, pose, illumination, etc.)
is preserved, but all other features are lost: images
in the Support Set seem to lie outside of the generative
range of StyleGAN. Note also the more ``conventional'' nature
of the images obtained by the inversion.
\label{fig:support_gan}}
\end{figure*}

For StyleGAN trained on CelebA-HQ, results are disappointing (see Figure \ref{fig:support_gan}, and compare them with the inversion of generated images in Figure~\ref{fig:gan_inv1}). Although the macrostructure is preserved (background, pose, illumination), details are sensibly different. Numerically,
while the average mean squared error on generated images is $0.026$, the corresponding value for the Support Set is  $0.251$, almost ten times higher.

Our conjecture is that StyleGAN is simply unable to generate data in the support set: they do not belong to its latent space, specifically due to its training dataset. To check this claim we implemented a gradient ascent technique to generate latent representations corresponding to a desired output. Once again, the gradient ascent technique provides almost
perfect results on generated images but substantially fails
on images in the CelebA support set, as shown in Figure \ref{fig:gradient_ascent}.

\begin{figure}[ht]
\begin{center}
\includegraphics[width=.3\columnwidth]{ga1.png}
\includegraphics[width=.3\columnwidth]{ga2.png}
\includegraphics[width=.3\columnwidth]{ga.png}
\caption{Gradient ascent technique for StyleGAN on data
in the Support Set. The original is in the first row, and the
image generated through gradient ascent, in the second. The technique confirms that these images cannot be generated by StyleGAN.
\label{fig:gradient_ascent}}
\end{center}
\end{figure}
We believe that the latent space of StyleGAN, trained on CelebA-HQ, only faithfully reflects a subspace of the latent space of our other models, trained on the full CelebA
dataset. In particular, points in our extreme {\em sectors} seem to lie
outside of the generative range of StyleGAN, or to be
severely underrepresented (Figure~\ref{space_vs_sectors}).
\begin{figure}[ht]
\begin{center}
\includegraphics[width=\columnwidth]{Space_vs_sectors.png}
\caption{CelebA Sectors seem to be external to the latent space of StyleGAN
\label{space_vs_sectors}}
\end{center}
\end{figure}
The problem is possibly also related to the well-known fact that faces generated by StyleGAN (and other generative networks) can be easily distinguished from reals \cite{Distinguishing2022,deep_fake_recognition20,attributing19}.

\subsection{Comparison with different spaces}
Since exploiting the Support Set is not a viable solution, we need to define a direct
mapping by regression on all data. Furthermore, we choose to work with the $W$ StyleGAN space, since the $Z$ space is passed through a series of fully connected layers (the {\em Mapping} network) which we suppose, by construction, cannot be inverted linearly.
Here we try to map the $W$ space of StyleGAN, trained
over CelebA-HQ, to the latent space of SVAE trained over CelebA.
The input to the transformation map is the vector $w$, obtained by ancestral sampling from the
$Z$ space. The expected output $z$ is obtained by synthesizing with StyleGAN the image
corresponding to $w$, cropping and resizing it to dimension $64\times 64$ and encoding it
in the SVAE latent space. The result of the linear map will be called $\hat{z}$; let
$\mathit{SVAE}(z)$, and $\mathit{SVAE}(\hat{z})$ the corresponding decodings to the visible domain.
As usual, input vectors $w$ may be generated {\em ad libitum}, with no risk of overfitting.

After training, the mean squared error between $z$ and $\hat{z}$ is around $0.45$
with a standard deviation of $0.05$.
The mean squared error between $\mathit{SVAE}(z)$, and $\mathit{SVAE}(\hat{z})$ is $0.014$
with standard deviation of $0.002$.
All results have been repeated over 5 different parameters configurations of SVAE, relative to
5 different trainings (obviously, each experiment results in a different linear transformation).

Result are shown in Figure~\ref{fig:StyleGAN2SVAE}. They are not perfect, but definitely interesting.

\begin{figure*}[ht]
\includegraphics[width=\textwidth]{StyleGAN2VAE1.png}
\caption{Mapping from the W space of StyleGAN to the latent space of SVAE.
In the first row we have sources, sampled by StyleGAN from $w\in W$.
In the second row we have the
SVAE reconstruction, starting from a suitably cropped and rescaled images
(SVAE work at resolution $64\time 64$): these images are the best possible approximation
of the source images obtainable by SVAE. In the third row we show the output produced
by the SVAE decoder after mapping each $w$ in its latent space: results are very similar
to those of the second row.
\label{fig:StyleGAN2SVAE}}
\end{figure*}

\begin{figure*}[ht]
\includegraphics[width=\textwidth]{VAE_to_styleGAN5.png}
\caption{Mapping from the latent space of SVAE to the W space of StyleGAN.
In the first row we have images generated by StyleGAN: $StyleGAN(w)$, for $w\in W$.
In the second row we have their
SVAE reconstructions, starting from suitably cropped and rescaled versions.
Images in the third row are obtained by first encoding $StyleGAN(w)$ in the latent space
of the SVAE, obtaining a latent representation $z$. This $z$ is then linearly transformed
to a vector $\hat{w} \in W$; the final image is $StyleGAN(\hat{w})$.
\label{fig:VAE_to_styleGAN}}
\end{figure*}

We also tested a few variants weighting the distance between latent variables according to
their ``information relevance", but we did not observe significant improvements.

Let us come to the mapping from the latent space of VAE to that of the StyleGAN. To train
the transformation model (as usual, a single dense layer with no bias), we simply invert input and
output of the previous network. After training, the mean squared error between $w$ and $\hat w$
is around $0.029$ with a standard deviation of $0.004$.
The mean squared error between $StyleGAN(w)$, and $StyleGAN(\hat{w})$ is $0.076$
with standard deviation of $0.014$.
Results are visually really good, as can be visually checked in Figure~\ref{fig:VAE_to_styleGAN}.

\section{Conclusions}\label{sec:Conclusions}
In this article we addressed the problem of comparing the latent space of different generative models, defining transformations
between them. Specifically, we proved that we can pass from a latent space to another by means of a simple {\em linear map} preserving most of the information. Hence, the organization of the
latent space seems to be largely independent from
  \begin{itemize}
  \item the training process
  \item the network architecture
  \item the learning objective: GANs and VAEs share the same space
  \end{itemize}
The result is original, surprising and largely unexpected; apparently, the latent space, if not
artificially constrained with different objectives, seems
to naturally organize itself in a way that is merely dependent
from the data manifold. Of course, we expect that this ``natural" structure
can be altered in many different ways, e.g. through conditioning,
which strongly impacts the latent structure, or via transformations
like normalizing flows, explicitly aiming towards a strong regularization
of the space. We also do not expect the two spaces $Z$ and $W$ of StyleGAN to be linearly related, since otherwise
the long chain of 8 dense layers between them would have no purpose.

Our result is full of implications from the point of view of representation learning and disentanglement. The fact that the latent space has a sort
of implicit and native structure raises promising expectations about the
possibility of learning features in a completely unsupervised
way. Moreover, the recent observation \cite{Shen22, li2021interpreting} that variations over a single semantical feature is a quasi-linear manifold in the latent space of
generative models fits well with our empirical observations,
opening interesting perspectives about the possibility of
``porting'' disentanglement between different spaces, and
more generally, to better understand the issue in a more general
framework.

The fact that the transformation between spaces is linear
obviously permits its definition in terms of a small set of
independent points of the same cardinality of the dimension
of the latent space; this is what we call a {\em Support Set}.
Locating these points in the two latent spaces is enough to
define the map.
In principle, any set of independent points could serve as a
Support Set, but for robustness reasons, it seems preferable
to chose points as apart as possible between each other.
We described a possible approach for defining such a set,
based on "sectors" in the space. This set is of interest in
its own, as it is representative of the
principal factors of variations in the dataset. Due to
this fact, it also provides a natural benchmark
to test the expressiveness of generative models.

This leads to an additional side contribution of our work:
in contrast with the usual belief, StyleGAN trained on CelebA-HQ seems to have serious generative deficiencies: many images, in particular most of the images in our Support Set from CelebA, seem to lie outside the generative range of StyleGAN. In particular, as it is also evident in inversion results, The StyleGAN generative process is privileging standardization, strongly penalizing defects, oddities and eccentricities: the StyleGAN space is not a space for minorities.

This could be a cause for concern about CelebA-HQ. Not only
it is computationally demanding, but one could also wander if it has statistical relevance: an assortment of 30K images in a space of dimension $3\times2^{20}$ looks more like a collection of scattered points than a data manifold.

Our results also raise serious worries  about the increasing
use of generative techniques for data augmentation purposes.
All generative techniques seem to have serious biases,
privileging likelihood over diversity: using them for data
augmentation may have no statististical significance. It is a bad practice that should be discouraged and deprecated.

As for future developments, most of the work just lies ahead. Here is a short, not-exhaustive list of possible topics:
\begin{itemize}
\item test and hopefully confirm our mapping results on different datasets;
\item deepen the relationship between the field of disentanglement through suitable linear manipulations of the latent space;
\item define and test a Support Set for StyleGAN and Celeba-HQ;
\item investigate the possibility to improve the transformation with residual non-linearities, and in that case study them;
\item better investigate and possibly find a remedy to the generative deficiencies of StyleGAN.
\end{itemize}

\subsection*{Data Availability}\label{sec5}
The training datasets can be found at \href{CelebA-dataset}{https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html} and \href{CelebAHQ-dataset}{https://www.kaggle.com/datasets/lamsimon/celebahq}.

The code relative to this work is available on Github in the following repository:
 \href{https://github.com/asperti/We_love_latent_space}{https://github.com/asperti/We_love_latent_space}. We also provide pretrained weights that can be downloaded using suitable facilities.
\bigskip

\subsection*{Acknowledgements} We would like to thank Fabio Merizzi for many interesting discussions on the subject of this article.\bigskip

\subsection*{Conflict of interest} On behalf of all authors, the corresponding author states that there is no conflict of interest.

redpajama_set_name: RedPajamaArXiv