text:
\section{Introduction}

If $A$ is a Boolean algebra, we denote by $K_A$ the Stone space of $A$,
that is a  compact Hausdorff totally disconnected
space such that $A$ is isomorphic to the algebra of all clopen subsets of $K_A$ (see \cite{koppelberg}).
 $C(K)$ denotes the Banach space of real-valued continuous functions on a compact Hausdorff space $K$
with the supremum norm.

Given an infinite cardinal $\kappa$, let $\mathcal B_\kappa$ denote the class of Banach spaces of density at most
 $\kappa$, $\mathcal C_\kappa$ denote
the class of compact spaces of weight at most $\kappa$ and
$\mathcal A_\kappa$ denote the class of Boolean algebras of cardinality at most $\kappa$.
We say that $X\in \mathcal B_\kappa$ is isometrically universal for $\mathcal B_\kappa$
if for every $Y\in \mathcal B_\kappa$  there is a linear isometry $T: Y\rightarrow X$ onto its range. We say that $K\in \mathcal C_\kappa$ is universal for $\mathcal C_\kappa$
if for every $L\in \mathcal C_\kappa$ there is a continuous surjection $\phi: K\rightarrow L$ and we say that $ A\in \mathcal A_\kappa$ is universal for $\mathcal A_\kappa$ if for every $B\in \mathcal A_\kappa$
there is a Boolean isomorphism $h: B\rightarrow A$ onto its range.

Classical functorial arguments involving the dual ball of a Banach space imply (see e.g. the introduction to
\cite{universal} or \cite{unisur}) that the first two statements below
are equivalent (by the Stone duality) and that they imply the third one:
\begin{itemize}
\item There is a universal Boolean algebra for $\mathcal A_{2^\omega}$.
\item There is a universal compact Hausdorff space  for $\mathcal C_{2^\omega}$.
\item There is an isometrically universal Banach space for $\mathcal B_{2^\omega}$.
\end{itemize}
 By classical results (e.g. the Banach-Mazur Theorem) the three statements are true if we replace $2^\omega$ by $\omega$ (for a survey on related topics see \cite{unisur}).
Moreover by the results of
I.I. Parovichenko \cite{parovicenko} and A. S. Esenin-Volpin \cite{volpin},
GCH implies that all the above statements are true.
The consistency of the failure has been proved
by A. Dow and K. P. Hart for Boolean algebras and compact spaces (see \cite{dowhart})
and by S. Shelah and A. Usvyatsov for isometric embeddings (see \cite{shelahusvyatsov}).
Actually one can even prove the nonexistence
of a universal Banach space in $\mathcal B_{2^\omega}$
for  isomorphic embeddings (see \cite{universal, ug}).
 Whether the equivalence of all the three statements above  can be proved
without  additional assumptions
  is the main question
that motivates our research in this paper. Not knowing how to attack it, an
intermediate question is whether an isometrically universal
Banach space of the form $C(K_A)$ must be induced by a universal Boolean algebra $A$,
as is known that the existence of an isometrically universal Banach space
implies the existence of one of the form $C(K_A)$ (Fact 1.1. of \cite{universal}).
This is the case on the countable level: one can see that
if a $C(K_A)$ is separable and isometrically universal\footnote{Actually, $C(K_A)$ being isomorphically universal suffices.}
for separable Banach spaces, then $K_A$ cannot be scattered and so
$A$ must contain an infinite free algebra, meaning that $A$ is a universal algebra
among countable algebras and $K_A$ is a universal compact space
among metrizable compact spaces. The main purpose of this paper is to prove that
the situation is different on the uncountable level:

\begin{theorem} For any uncountable $\kappa$, if there exists an isometrically universal
Banach space for $\mathcal B_\kappa$, then there is also
such a space of the form $C(K_A)$ for some Boolean algebra $A$
where $A$ does not contain well-ordered uncountable chains. In particular,
$A$ is not a universal Boolean algebra for $\mathcal A_\kappa$ nor $K_A$ is universal for $\mathcal C_\kappa$.
\end{theorem}
\begin{proof}
Suppose  that $X$ is an isometrically universal Banach space for $\mathcal B_\kappa$. By the above discussion,
by isometrically embedding $X$ into $C(B_{X^*})$ and taking a totally disconnected continuous preimage of $B_{X^*}$, we may assume that $X$ is of the form $C(K_A)$
for some Boolean algebra $A$. Then, Proposition \ref{mainprop} produces a Boolean algebra
$B$ of the same cardinality $\kappa$ such that $C(K_A)$ isometrically embeds into $C(K_B)$
and hence $C(K_B)$ is also isometrically universal for the same class of Banach spaces $\mathcal B_\kappa$.
However, still by Proposition \ref{mainprop}, the Boolean algebra generated by a well-ordered chain of type $\omega_1$
cannot be embedded into $B$. By the Stone duality, $K_B$ cannot be continuously mapped onto
$[0,\omega_1]$ with the order topology. Therefore, $K_B$ is not universal for $\mathcal A_\kappa$ and $K_B$ is not universal for $\mathcal C_\kappa$.
\end{proof}

In the literature there are some results assuming the axiom OCA and concerning similar behaviour
of the Boolean algebra $\wp(\mathbb{N})/Fin$ and
 the Banach space $\ell_\infty/c_0\equiv C(K_{\wp(\mathbb{N})/Fin})$.
 In fact,
there is at present no known example showing that $\ell_\infty/c_0$
is not isometrically universal under OCA while there are
many examples showing that $\wp(\mathbb{N})/Fin$ is not universal (\cite{almostno, dowhartmeasure,
quotients}).
For example, A. Dow and K. P. Hart have proved in \cite{dowhartmeasure} that
OCA implies that the measure algebra $M$ does not embed into $\wp(\mathbb{N})/Fin$,
while it is well known that $C(K_{M})\equiv L_\infty$ isometrically embeds into
$C(K_{\wp(\mathbb{N})/Fin})\equiv\ell_\infty/c_0$ (see e.g.,  p. 304 of \cite{amirarbel}).
At the same time, while well-known that
the algebra of clopen subsets of the cozero set in $\mathbb{N}^*$ does
not embed into $\wp(\mathbb{N})/Fin$ under OCA (see \cite{quotients}),
its is still unknown if the corresponding
Banach space $\ell_\infty(\ell_\infty/c_0)$
 can be embedded under OCA into $\ell_\infty/c_0$ (cf. \cite{sums, dowpfa}).

Our paper contains basically one construction: given a Boolean algebra
$A$, we construct another Boolean algebra $B$ such that $C(K_A)$ embeds isometrically into
$C(K_B)$ but $B$ does not contain well-ordered uncountable chains (Proposition \ref{mainprop}).
Thus, when $A$ does contain uncountable well-ordered chains as, for example, in the cases
of Boolean algebras of the clopen subsets of $[0,\omega_1]$ or $\mathbb{N}^*$, we cannot have
a Boolean embedding of $A$ into $B$. $B$ is the algebra of clopen sets of some (standard)
totally disconnected preimage of the dual ball $B_{C(K)^*}$ of $C(K_A)$ with the
weak$^*$ topology. In Section 2 we show that if it had an uncountable well-ordered chain
of clopen sets we would have a chain of some sets in the ball $B_{C(K)^*}$ and this
cannot occur as is shown in Section 3.

It would be interesting to know if objects other than uncountable well-ordered chains can be used in the
above argument.
Or conversely, for which Boolean algebras $A$, the existence of an isometric embedding
$C(K_A)$ into $C(K_B)$ implies the existence of a Boolean embedding of $A$ into $B$.
By Theorem 12.30 (ii) of \cite{fabian} these cannot be uncountable antichains,
i.e., the algebra $A=FinCofin(\kappa)$ for any $\kappa$ has the above property for any algebra $B$.
Also, if $A$ has an independent family of cardinality $\kappa$ and $C(K_A)$ isometrically
embeds into $C(K_B)$, by the Holszty\'nski Theorem (\cite{holsztynski}) we have a closed set
$F\subseteq K_B$ which maps onto $\{0,1\}^\kappa$. This map can be always extended
to a totally disconnected superspace and in particular $K_B$ maps onto $\{0,1\}^\kappa$,
and so $B$ contains an independent family of cardinality $\kappa$.
But we do not know if an isometric embedding of $C(K_{\wp(\mathbb{N})})\equiv\ell_\infty$
into $C(K_B)$ implies the existence of an isomorphic copy of $\wp(\mathbb{N})$ in $B$.

Terminology should be standard, concerning the Banach spaces we follow
\cite{fabian} and for Boolean algebras we follow \cite{koppelberg}. $f[X]$, $f^{-1}[X]$ denote the image  and the preimage of $X$ under $f$, respectively. $f|X$ denotes the restriction of $f$ to $X$, $\{0,1\}^{<\mathbb{N}}=\bigcup_{n\in \mathbb{N}}\{0,1\}^n$.

\section{Chains in  totally disconnected preimages}\label{secchains}

\begin{lemma}\label{interior} Let $K$ and $L$ be compact spaces and $\psi: K\rightarrow L$ be a surjective
continuous mapping. Suppose $U\subseteq K$ is clopen. For every $x \in U$, if $\psi^{-1}[\{\psi(x)\}] \subseteq U$, then $\psi(x) \in int(\psi[U])$. Hence,
\[\psi[\{x\in U: \psi^{-1}[\{\psi(x)\}]\subseteq U\}]\subseteq int(\psi[U]).\]
\end{lemma}
\begin{proof} Note that given $x \in U$, if $\psi^{-1}[\{\psi(x)\}]\subseteq U$,
then $\psi(x)\not\in \psi[K\setminus U]$. But $L = \psi[U]\cup\psi[K\setminus U]$,
 and so
$L\setminus \psi[K\setminus U]$ is an open set included in $\psi[U]$,  so $\psi(x)\in int(\psi[U])$.
\end{proof}

The previous lemma will be applied to a restriction of the function $\phi_I$ defined as follows. Let $\phi: \{0,1\}^\mathbb{N} \rightarrow [0,1]$ be given by
$$\phi(x)=\sum_{n=1}^\infty \frac{x(n)}{2^n}.$$
Given any set of indices $I$, we  consider
$\phi_I: ( \{0,1\}^\mathbb{N})^I \rightarrow [0,1]^I$ which is
 defined  coordinatewise by
\[\phi_I(x)(i)
= \phi(x(i))\]
 for $x \in ( \{0,1\}^\mathbb{N})^I$ and $i \in I$.
Since $\phi_I$ is defined
coordinatewise we immediately obtain the following:

\begin{lemma}\label{combination} Suppose $x, x', x''\in  ( \{0,1\}^\mathbb{N})^I$ are such that $\phi_I(x)=\phi_I(x')$
and that  that for each $i\in I$ either $x''(i)=x(i)$ or $x''(i)=x'(i)$. Then
$\phi_I(x'')=\phi_I(x)=\phi_I(x')$.
\end{lemma}

We will consider the standard basis of clopen sets of $ \{0,1\}^\mathbb{N}$, i.e., the sets of the form
$$[s]=\{x\in  \{0,1\}^\mathbb{N}: s\subseteq x\}$$
for  $s\in  \{0,1\}^{<\mathbb{N}}$.  By the definition of the product topology, the sets of the form
$$U(i, [s]) = \{x \in (\{0,1\}^\mathbb{N})^I: x(i) \in [s]\},$$
for $i \in I$ and $s\in  \{0,1\}^{<\mathbb{N}}$ form a topological subbasis
 for $(\{0,1\}^\mathbb{N})^{I}$ which consists of clopen sets.
Note that
$$\phi[[s]]=\Big[\sum_{1\leq n\leq |s|}\frac{x(n)}{2^n},\ \
\ \sum_{1\leq n\leq |s|}\frac{x(n)}{2^n}+\frac{1}{2^{|s|}}\Big]
\leqno (*)$$
for any $s\in \{0,1\}^{<\mathbb{N}}$. Hence, $\phi$ sends the standard basic clopen sets onto
closed subintervals of $[0,1]$, in particular onto convex sets.

\begin{definition}
For a subspace $X$ of $(\{0,1\}^\mathbb{N})^I$, $n\in \mathbb{N}$ and a subset $J\subseteq I$
 we will say that $Y\subseteq X$ $n$-depends on $J\subseteq I$
in $X$ if and only if whenever $x, y\in X$ and $x(i)|n=y(i)|n$
for  each $i\in J$, then
\[x\in Y \ \Leftrightarrow\  y\in Y.\]
\end{definition}

It is immediate that $U(i,[s])$ $n$-depends on $\{i\}$ in $(\{0,1\}^\mathbb{N})^I$ where $n=|s|$
and so, $U(i,[s]) \cap L$ $n$-depends on $\{i\}$ in any $L\subseteq (\{0,1\}^\mathbb{N})^I$. Since clopen
sets of compact Hausdorff spaces are Boolean combinations of finitely many sets from any clopen subbasis, it follows that
any clopen subset of any closed $L\subseteq (\{0,1\}^\mathbb{N})^I$
$n$-depends on $J\subseteq I$ in $L$, for $n$ being any natural number larger than $|s|$ for all $s$ which appear in the finite Boolean combination of sets of the form $U(i,[s])$ and $J$ is the set of all $i$ which appear in such a Boolean combination.
We are now ready
for the main result of this section.

\begin{proposition}\label{chain} Let $I$ be a set and
let $K$ be a closed convex subspace of $[0,1]^I$
 such that no closed convex subspace $F$ of $K$ has an uncountable well-ordered chain of open (in $F$) sets $(V_\alpha)_{\alpha<\omega_1}$ satisfying
 $\overline{V}_\alpha \subseteq V_\beta$ for $\alpha<\beta<\omega_1$. Then $L = {\phi_I}^{-1}[K]$ has no uncountable well-ordered chain of clopen sets $(U_\alpha)_{\alpha<\omega_1}$.
\end{proposition}
\begin{proof} Suppose that
 $(U_\alpha\cap L)_{\alpha \in \omega_1}$ is a sequence
of clopen subsets of $L$ where $U_\alpha$'s
are clopen subsets of $(\{0,1\}^\mathbb{N})^I$. As we noticed, it follows that each $U_\alpha \cap L$
$n_\alpha$-depends on some finite set $J_\alpha\subseteq I$ in $(\{0,1\}^\mathbb{N})^I$,
for some $n_\alpha$.
Using the $\Delta$-system lemma (see \cite{jech}) and the fact that $\{0,1\}^{<\mathbb{N}}$ and $\mathbb{N}$ are countable,
we may assume that $(J_\alpha)_{\alpha < \omega_1}$ is a $\Delta$-system with root $\Delta$
and each $U_\alpha$ $n$-depends on $J_\alpha$ in $(\{0,1\}^\mathbb{N})^I$ for a fixed $n\in \mathbb{N}$ and all $\alpha<\omega_1$.

For each $f\in (\{0,1\}^n)^\Delta$, consider
\[U(f)=\bigcap_{i\in \Delta} U(i, [f(i)]).\]
 $(\{0,1\}^{\mathbb{N}})^I$ is the disjoint union of the family of clopen sets $\{U(f): f\in (\{0,1\}^n)^\Delta\}$.
It follows that there is $f_0\in (\{0,1\}^n)^\Delta$ such that
$(U_\alpha\cap L\cap U(f_0))_{\alpha<\omega_1}$ forms an uncountable sequence. By
going to a subsequence we may assume that all elements $U_\alpha\cap L\cap U(f_0)$ are distinct.
Consider $L'=L\cap U(f_0)$, $F=\phi_I[L']$ and $\psi=\phi_I| L'$ from $L'$ onto $F$ and put
\[V_\alpha=int_{F}(\psi [U_\alpha\cap L']).\]
Note that $F=\phi_I[\phi_I^{-1}[K] \cap U(f_0)] = \phi_I[U(f_0)] \cap K$, which is
convex as the intersection of two convex sets.
Secondly, note that $U_\alpha\cap L'$ $n$-depends on $J_\alpha\setminus\Delta$ in $L'$, for each $\alpha<\omega_1$.
Indeed, whenever $x, y\in L'$
we have that $x(i)|n=f_0(i)
=y(i)|n$ for all $i\in\Delta$ and so, whenever we have additionally that
 $x(i)|n=y(i)|n$ for all $i\in J_\alpha\setminus\Delta$,
we may use the fact that $U_\alpha$ $n$-depends on $J_\alpha$ in $(\{0,1\}^\mathbb{N})^{I}$.

By the hypothesis on convex sets in $K$ applied to $F$, there are $\alpha<\beta<\omega_1$
such that
$\overline{V_\alpha}\not\subseteq V_\beta$. Aiming at a
contradiction, let us assume that $(U_\alpha \cap L)_{\alpha < \omega_1}$ is a well-ordered chain and hence, $U_\alpha\cap L'\subseteq U_\beta\cap L'$. Then, since $\psi[U_\alpha\cap L']$ is a closed set,
$\overline{V_\alpha}\subseteq \psi[U_\alpha\cap L']$ and
we conclude that there is
\[y\in \psi [U_\alpha\cap L']\setminus int_F(\psi [U_\beta\cap L'])
\subseteq \psi [U_\beta\cap L']\setminus int_F(\psi [U_\beta\cap L']).\]
Let $x\in U_\alpha\cap L'$ be such that $\psi(x)=y$.
 Lemma \ref{interior} gives that $\psi^{-1}[\{y\}]\not\subseteq U_\beta\cap L'$ and so
there is $x'\in L'$ such that  $x'\not \in U_\beta\cap L'$ but $\psi(x')=y$.
Now we will combine $x$ and $x'$ following  Lemma \ref{combination} and will obtain
a contradiction with the hypothesis that $U_\alpha\cap L'\subseteq U_\beta\cap L'$.
Define
$$x''(\xi) = \left\{ \begin{array}{ll}
     x'(\xi) & \text{if } \xi \not \in J_\beta \setminus \Delta\\
     x(\xi) & \text{otherwise.}
         \end{array}
                                         \right.$$
By Lemma \ref{combination} the point $x''$ is in $L$.
Note that $x''\in U(f_0)$ as $x''|\Delta=x|\Delta$ i.e., $x''\in L\cap U(f_0)=L'$. Also,
$x''|J_\alpha=x|J_\alpha$ and so $x''\in U_\alpha$
since $U_\alpha\cap L'$ $n$-depends on $J_\alpha$ in $L'$ and $x \in U_\alpha$. On the other
hand, $x''|(J_\beta\setminus\Delta)=x'|(J_\beta\setminus\Delta)$, so
$x''\not \in U_\beta$ as $U_\beta\cap L'$ $n$-depends on $J_\beta\setminus\Delta$ in $L'$ and $x' \notin U_\beta$.
This shows that $U_\alpha \cap L' \not\subseteq U_\beta \cap L'$, contradicting our hypothesis. Hence, $(U_\alpha \cap L)_{\alpha<\omega_1}$
is not a well-ordered chain and this completes the proof of the proposition.
\end{proof}

\section{Well-ordered chains in the dual ball}

\begin{proposition}\label{ballBanach}
If $F$ is any closed convex subspace of the dual unit ball of a Banach space endowed with the weak$^*$ topology, then $F$ does not have an uncountable well-ordered chain of open sets $(V_\alpha)_{\alpha < \omega_1}$ such that
$\overline{V_\alpha} \subseteq V_\beta$ for any $\alpha<\beta<\omega_1$.
\end{proposition}
\begin{proof}
Let $X$ be a Banach space, $B_{X^*}$ its dual unit ball endowed with the weak$^*$
topology and $F$ a closed convex subspace of $B_{X^*}$. Suppose $(V_\alpha)_{\alpha \in \omega_1}$ is a well-ordered chain of
open sets of $F$ such that
$\overline{V_\alpha} \subseteq V_\beta$ for any $\alpha<\beta<\omega_1$. Put $V = \bigcup_{\alpha \in \omega_1} V_\alpha$. Since $F$ is a closed subspace of $B_{X^*}$ and $B_{X^*}$ is weakly$^*$ compact, then $F \setminus V$ is a nonempty closed set of $F$. It is as well
a weakly closed set, since the weak topology is finer than the weak$^*$ topology.
It cannot be a weakly open set in $F$, because $F$ is convex, and since $X$ endowed with the weak topology is a topological vector space, $F$ is weakly connected. Then, there is
$x \in \overline{B_{X^*} \setminus V}^{w} \cap \overline{V}^{w}$.

Now, by Kaplansky's theorem (e.g., 4.49 of \cite{fabian}), every Banach
space has countable tightness in its weak topology. So, there is a countable set
$D \subseteq V$ such that $x \in \overline{D}^w$. Since $D$ is countable, there
is $\gamma\in \omega_1$
such that $D \subseteq V_\gamma$, which implies that
$x \in \overline{V}^w_\gamma \subseteq \overline{V}_\gamma^{w^*} \subseteq V_{\gamma+1} \subseteq V$,
contradicting the fact that $x \in  \overline{F \setminus V}^{w}$.
\end{proof}

The hypothesis of $F$ being convex is crucial, as any compact Hausdorff space $K$
can be homeomorphically embedded in the dual ball of $C(K)$ with the weak$^*$
topology by associating
the Dirac $\delta_x$ to each $x\in K$.
In the context of the above result it is also worthy to mention the following:

\begin{proposition}
Note that if $X$ is a Banach space of density $\kappa$, then there is a well-ordered
increasing chain $(U_\xi)_{\xi<\kappa}$ of open sets  in the dual unit ball $B_{X^*}$ of $X$
with the weak$^*$ topology.
\end{proposition}
\begin{proof}
Using the Hahn-Banach theorem,
construct by transfinite induction a sequence $(x_\xi, x^*_\xi)_{\xi<\kappa} \subseteq X \times B_{X^*}$
such that $x^*_\xi(x_\eta)=0$ for $\eta<\xi<\kappa$ and $x^*_\xi(x_\xi)=1$ for
$\xi<\kappa$. Then
$$U_\xi=\bigcup_{\eta<\xi}\{x^*\in B_{X^*}: x^*(x_\eta)\not=0\}$$
is as required.
\end{proof}

\begin{lemma}\label{lema-convex}
Given a Banach space $X$ and a dense
subset $D$ of its unit ball, the natural restriction mapping $f: B_{X^*} \rightarrow [-1,1]^{D}$ defined
by $f(x^*)=x^*|_{D}$ is a homeomorphism onto its image with respect to
the weak$^*$ and the product topologies, with the property
 that $F \subseteq B_{X^*}$ is convex if and only if $f(F)$ is convex.
\end{lemma}
\begin{proof} The preimages of standard basic open sets in the product are
weakly$^*$ open, so that $f$ is continuous. As two distinct functionals must
differ on an element of the unit ball we see that
 $f$ is a homeomorphism onto its image.
$\Phi: X^*\rightarrow \mathbb{R}^D$ given by $\Phi(x^*)(d)=x^*(d)$ is linear and one-to-one
and hence its inverse is linear and one-to-one. Both mappings preserve convexity.
The lemma follows as $f=\Phi|B_{X^*}$.
\end{proof}

\begin{proposition}\label{pre-image-Banach}
The dual unit ball of every Banach space endowed with the weak$^*$ topology has a continuous preimage which is compact, totally disconnected  of same weight and with no uncountable well-ordered chain of clopen sets.
\end{proposition}
\begin{proof} Let $X$ be a Banach space of density $\kappa$.
Let $D\subseteq X$ be a dense subset of the unit ball of cardinality $\kappa$.
Let $f: B_{X^*} \rightarrow [-1,1]^{D}$ be defined by $f(x^*)=x^*|_{B_{X}}$
as in Lemma \ref{lema-convex}, let $g$ be the linear
order-preserving homeomorphism from $[-1,1]$ onto $[0,1]$
 and let ${g}^D: [-1,1]^{D} \rightarrow [0,1]^{D}$ be defined coordinatewise by
 ${g}^D(x)(d) = g(x(d))$ for any $d\in D$ and $x\in [-1,1]^D$. Clearly
 ${g}^D$ is a homeomorphism such that $F \subseteq [-1,1]^{D}$
 is convex if and only if $g^D[F]$ is convex.
 Then, $h=g \circ f: B_{X^*} \rightarrow [0,1]^{D}$
 is a homeomorphism onto its image such that $F \subseteq B_{X^*}$
is convex if and only if $h[F]$ is convex.

Let $K = h[B_{X^*}] \subseteq [0,1]^{D}$. If $F$ is a convex subset of $K$, then $h^{-1}[F]$ is a convex subset of $B_{X^*}$ and Proposition \ref{ballBanach} guarantees that it
contains no uncountable well-ordered chain of open sets $(V_\alpha)_{\alpha<\omega_1}$ such that $\overline{V}_\alpha^F \subseteq V_\beta$ for $\alpha < \beta <\omega_1$. Since $h$ is a homeomorphism onto $K$, we get that $F$ has no such  chain of open sets either.

Finally, since $K$ satisfies the hypotheses of Proposition \ref{chain},
 we get that $L = {(\phi_D)}^{-1}[K]$ has no uncountable well-ordered chain of clopen sets.
Since the weight of $K$ cannot be bigger than $D$, this concludes the proof.
\end{proof}

\begin{proposition}\label{mainprop} Suppose that
 $A$ is a Boolean algebra.  There is a Boolean algebra ${B}$
of same cardinality as ${A}$ but without uncountable
well-ordered
chains   such that the Banach space $C(K_{B})$ contains an isometric copy of $C(K_{A})$.
\end{proposition}
\begin{proof}
Let $X= C(K_{A})$ and by Proposition \ref{pre-image-Banach}, the dual unit ball $B_{X^*}$ has a continuous preimage $L$ which is compact, totally disconnected, of the same weight as $B_{C(K_{A})^*}$
and with no uncountable well-ordered chain of clopen sets. Hence, ${B}= Clop (L)$ is a Boolean algebra of same cardinality as ${A}$ which has no uncountable well-ordered chains and, therefore, has no isomorphic copy of ${A}$. But $K_{B}$ is homeomorphic to $L$, so that $C(K_{B})$ is isometric to $C(L)$, which contains an isometric copy of $C(B_{X^*})$, which in turn contains an isometric copy of $X = C(K_{A})$.
\end{proof}

\bibliographystyle{amsplain}


text:
\section{Concluding Discussions}
In this paper, we proposed a dimensionality reduction method applied to tensor-structured data in a convolutional neural network,
and also derived the gradients of the loss function with respect to the parameters associated with the TensorProjection Layer.
In addition, we implemented the TensorProjection Layer using TensorFlow, and tested it on two different data sets as a preliminary experimental study.

The first experimental result implies that the TensorProjection Layer can reduce the dimensionality of the data more efficiently than the AveragePooling Layer.
The second experimental result implies that the TensorProjection Layer fits to the data faster than the fully connected layer, and the training result is also more stable.
However, these results do not immediately imply that TensorProjection Layer can completely take the place of the existing methods.
For example, in our preliminary experiment done before these two experiments,
we found that AveragePooling Layers performed better than the TensorProjection Layer if they were embedded after the first convolutional layer.
We will look into the performance of TensorProjection Layer in more details and make refinement in the future.

\section{Introduction}

In convolutional neural networks, pooling layers are widely used to summarize the output image from the previous convolutional layer.
We may consider that pooling layers are used to reduce the dimensionality of the output images from convolutional layers.
The purpose of our research is to study an alternative dimensionality reduction layer which takes the place of pooling layers.

\subsection{Main Idea}\label{sec:introduction:main_idea}

We propose our method as a layer in a neural network.
We call our proposed layer TensorProjection Layer.
We assume that TensorProjection Layer is embedded after a convolutional layer like Figure \ref{fig:use-of-TensorProjectionLayer}.
We will explain the details of the method again in Section \ref{sec:TensorProjectionLayer}, however, we briefly describe the basic idea of thinking here.

Let $\{X_i\}_{i=1}^n$ be the output tensors with the size of $p_1 \times p_2 \times p_3$ from a convolutional layer,
and they will be the input tensors of the TensorProjection Layer.
We reduce the dimensionality of each tensor through our dimensionality reduction layer.
Let $\{Z_i\}_{i=1}^n$ be the output  tensors of our dimensionality reduction layer with the size of $q_1 \times q_2 \times q_3$ where $q_i \leq p_i$.
Let $\{U_k\}_{k=1}^3$ be orthogonal matrices with the size of $p_k \times q_k$.
We transform $X_i$ into $Z_i$ by the following formula.
\begin{equation}\label{equation:forward_path}
Z_i \eqdef X_i \times_1 U_1^T \times_2 U_2^T \times_3 U_3^T.
\end{equation}
In the formula above, $\times_k$ represents $k$-mode product.
Please refer to the Appendix (\ref{appendix:k-mode-product}) if you are unfamiliar with this operation.
We view these orthogonal matrices $\{U_k\}_{k=1}^3$ as layer parameters, and train them via a supervised learning criterion such as minimization of cross-entropy loss function.

\begin{figure}[h!]
    \centering
    \input{figure/model}
    \caption[test]{Use of TensorProjection Layer}
    \label{fig:use-of-TensorProjectionLayer}
\end{figure}

\subsection{Related Methods}

We introduce some related dimensionality reduction methods to describe the characteristics of our proposed method.

\subsubsection{PCA}

Principal Component Analysis (PCA) is a classical dimensionality reduction method applied to vector data.
Let $X \in \mathbb{R}^p$ be a random variable from a population with a finite mean $\mu$ and a finite covariance matrix $\Sigma$.
The purpose of PCA is to reduce the dimensionality of $X$ via linear transformation and explain the data with a fewer number of new variables.

PCA is formulated in the following way.
Let $B \eqdef [b_1, \cdots, b_q]$ where $b_1, \cdots, b_q \in \mathbb{R}^p$ are orthonormal vectors and let $\widetilde{X} \eqdef BB^T (X-m)+m$.
We find
\begin{equation}
\argmin_{B,m} E[(X-\widetilde{X})^T(X-\widetilde{X})]. \label{formula:pca}
\end{equation}
It is well-known that $m = \mu$ and $B=\widetilde{U} \eqdef [u_1, \cdots, u_q]$ minimize the above formula where $u_1, \cdots, u_q$ are the $q$-leading eigenvectors of $\Sigma$.
In this way, we can compress information of $X$ into a new variable,
\begin{equation}
Z \eqdef \widetilde{U}^T(X-\mu) \ (\in \mathbb{R}^q)
\end{equation}
whose dimension is smaller than $X$.

In a general situation, $\mu$ and $\Sigma$ are unknown, so we estimate $\mu$ and $\Sigma$ from observed random samples $\{X_i\}_{i=1}^n$.
In formula (\ref{formula:pca}), we evaluate $E[\cdot]$ by the empirical distribution.
It follows that we have to find
\begin{eqnarray}
\argmin_{B,m}  \frac{1}{n}\sum_{i=1}^n (X_i-m)^T(I-BB^T)(X_i-m).
\end{eqnarray}
Equivalently, we find $\argmax_{B,m}  \sum_{i=1}^n (X_i-m)^TB B^T(X_i-m)$.
Furthermore, we can easily see that $m = \bar{X}$ is the minimizer by an easy rearrangement.
Therefore, it follows that PCA is formulated by
\begin{equation}
\argmax_{B} \sum_{i=1}^n (X_i-\bar{X})^TB B^T(X_i-\bar{X}) \label{formula:pca_general}
\end{equation}
where $B$ is a $p \times q$ orthogonal matrix.
It is well-known that $B = [u_1, \cdots, u_q]$ is a solution where $u_1, \cdots, u_q$ are the leading $q$-eigenvectors of $S \eqdef \frac{1}{n} \sum
_{i=1}^n (X_i - \bar{X})(X_i - \bar{X})^T$.

\subsubsection{MPCA}
Multilinear Principal Component Analysis (MPCA) is an extension of PCA to tensor-structured data.
In PCA, our goal is to find a good approximation of $p$-dimensional data with a smaller dimension $(q<p)$.
Let $\{X_i\}_{i=1}^n$ be tensor-structured samples with the size of $p_1 \times p_2 \times \cdots \times p_m$.
In MPCA, out goal is to find a good approximation of $\{X_i\}_{i=1}^n$ with a smaller dimension $q_1 \times q_2 \times \cdots \times q_m$ ($q_i \leq p_i$).
Let $U_k (k=1,\cdots,m)$ be an orthogonal matrix with the size of $p_k \times q_k$ ($q_i < p_i$).
Let
\begin{equation}
Z_i \eqdef (X_i - \bar{X}) \times_1 U_1^T \times_2 U_2^T \cdots \times_m U_m^T. \label{formula:mpca_Zi}
\end{equation}
In MPCA, we can naturally extend the formula (\ref{formula:pca_general}) as
\begin{equation}
\argmax_{U_1, \cdots, U_m} \sum_{i=1}^n \|  Z_i \|^2_F, \label{formula:mpca}
\end{equation}
where $\| \cdot \|_F^2$ is the square of Frobenius Norm.
Let
\begin{equation}
\widetilde{X}_i \eqdef Z_i \times_1 U_1 \times_2 U_2 \cdots \times_m U_m + \bar{X}.
\end{equation}
We can also interpret the formula (\ref{formula:mpca}) as minimization of reconstruction error,
\begin{equation}
\sum_{i=1}^n \|X_i - \widetilde{X}_i\|_F^2.
\end{equation}
However, it is hard to obtain an analytic solution of (\ref{formula:mpca}).
Some iterative algorithms to obtain (\ref{formula:mpca}) have been proposed such as \cite{10.1109/TNN.2007.901277} (Lu).

\begin{comment}
\subsubsection{Probabilistic PCA}

In PCA, we did not assume any statistical distribution of $X$ except for the assumption that $X$ has a finite mean and a finite covariance matrix.
In Probabilistic Probabilistic Principal Component Analysis \cite{Tipping99probabilisticprincipal},  we assume the observed sample $X \in \mathbb{R}^p$ can be decomposed as
\begin{equation}
X = W_{p \times q} Z_{q \times 1} + \mu_{p \times 1}+ \epsilon_{p \times 1}, \ \ \ Z \sim N(0, I_q),  \ \epsilon \sim N(0,\sigma^2 I_p),
\end{equation}
where $\mu, W$ and $\sigma^2$ are unknown parameters.
We may regard Probabilistic PCA as a special case of the Factor Analysis.
We would like to explain $X$ with another unobservable variable $Z$ with a lower dimensionality.
(We often call $Z$ a latent variable.)
We can estimate $mu, \sigma^2, W$ by the Maximum Likelihood Estimation.
Suppose that we have i.i.d random samples $\{X_i\}_{i=1}^n$.
Let $\Sigma \eqdef WW^T + \sigma^2 I_p$.
Note that $X_i \iid N(\mu,\Sigma)$.
\begin{eqnarray}
L(\mu, \sigma^2, W) &\eqdef& \prod_{i=1}^n f(X_i \mid W, \mu, \sigma^2), \\
f(x \mid \mu, \sigma^2, W) &\eqdef& \left(\frac{1}{2\pi}\right)^{p/2} \det(\Sigma)^{-1/2} \exp\left(-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right)
\end{eqnarray}
We find
\begin{equation}
\argmax_{W, \mu, \sigma^2} L(W, \mu, \sigma^2).
\end{equation}
The Maximum Likelihood Estimators of $W, \mu, \sigma^2$ are given as below.
\begin{equation}
\widehat{\mu} = \bar{X}, \ \ \ \widehat{\sigma^2} = \frac{1}{p-q} \sum_{i=q+1}^p \lambda_i, \ \ \ \widehat{W} = U_q(\Lambda_q-\widehat{\sigma^2} I_q)^{1/2} R,
\end{equation}
where
\begin{itemize}
   \item $\lambda_1 \geq \cdots \geq \lambda_p$ are the eigenvalues of $S \eqdef \frac{1}{n} \sum_{i=1}^n (X_n-\bar{X})(X_n-\bar{X})^T$,
   \item $u_j$ ($j=1,2,\cdots,p$) is the eigenvector of $S$ corresponding to the $j$ th eigenvalue $\lambda_j$,
   \item $\Lambda_q \eqdef \diag(\lambda_1, \cdots, \lambda_q)$, $U_q = [u_1 u_2 \cdots u_q]$,  $R$ is an arbitrary rotation matrix. (In practice, $R$ is often chosen to be $I$.)
\end{itemize}
\end{comment}

\subsubsection{HOPE and its application to CNN}

Zhang and Jiang \cite{DBLP:journals/corr/Zhang015} proposed HOPE which is one of the linear latent variable models and its parameter estimation method.
In HOPE, suppose that each observed sample $X_i \in \mathbb{R}^p$ is decomposed into the latent variable $Z_i \in \mathbb{R}^q$ and the noise part $N_i \in \mathbb{R}^{p-q}$ through an orthogonal transformation
\begin{equation}
\begin{pmatrix} Z_i \\ N_i \end{pmatrix} = \begin{pmatrix} U \\ V  \end{pmatrix} X_i, \ \ \ \ \ \ (U: q \times p, V: (p-q) \times p),
\end{equation}
where $UU^T = I_{q}, VV^T = I_{p-q}$ and $UV^T = O$.
Suppose that $Z_i$ and $N_i$ are stochastically independent, $Z_i$ follows a mixture Gaussian distribution (or a mixture Von Mises–Fisher distribution) and $N_i$ follows a normal distribution $N(0,\sigma^2 I_{p-q})$.
The unknown parameters including $U$ are estimated by Maximum Likelihood Estimation.
In order to satisfy the orthogonal constraint $UU^T = I_q$, the penalty term $-\beta D(U)$ ($\beta > 0$) is added to the likelihood function where
\begin{equation}
D(U) = \sum_{i=1}^{q} \sum_{j=i+1}^q \frac{|u_i^Tu_j|}{|u_i| \cdot |u_j|}, \ \ \ \ \ \ (U^T = [u_1 \ u_2 \ \cdots \ u_q]).
\end{equation}
By adding the penalty term, the obtained vectors $u_1, \cdots, u_q$ are expected to be less correlated.

Pan and Jiang \cite{pmlr-v77-pan17a} applied the techniques of HOPE into the training of convolutional neural networks.
They vectorize the output of a convolutional layer, and then project it onto a lower dimensional space via the orthogonal matrix $U$.
The orthogonal matrix $U$ is viewed as a training parameter in a neural network, and is obtained by minimizing the cross-entropy loss function.
To satisfy the orthogonal constraint, the above penalty term is added to the loss function.

\subsection{Characteristics of Our Proposed Method}
In this part, we describe the characteristics of our proposal by comparing with the above related methods.
Our TensorProjection Layer is different from MPCA and HOPE in the following ways.
We implement tensor-structured dimensionality reduction similar to MPCA without flattening an original tensor into a vector, and the method incorporates the label information into the loss function.

\subsubsection{Supervised dimensionality reduction}
In a TensorProjection Layer, we adopt an idea which is similar to MPCA that transforms the input tensors into smaller ones by multilinear projection.
However, MPCA is an unsupervised dimensionality reduction method based on minimization of reconstruction error.
Our TensorProjection Layer introduced in Section \ref{sec:TensorProjectionLayer} is a supervised one, and the orthogonal matrices $\{U_k\}_{k=1}^3$ will be obtained via minimizing the cross-entropy loss.

\subsubsection{Preserving the tensor-structure which is natural to tensor data}
Our proposed method also aims to reduce the dimensionality of the output tensor from a convolutional layer via projection as well as \cite{pmlr-v77-pan17a}.
However, we implement dimensionality reduction without destroying the tensor-structure of the data.
Furthermore, we directly find the orthogonal matrices $U_1, U_2, U_3$ under the constraint $U_i^TU_i = I_{q_i}$ ($i=1,2,3$).
Please refer to Section \ref{sec:TensorProjectionLayer:ForwardPropagation}.

\section{Examples}

To evaluate the performance of TensorProjection Layer, we solve the classification problem using a convolutional neural network with a TensorProjection Layer.
In this experiment, we use the Fashion MNIST dataset \cite{DBLP:journals/corr/abs-1708-07747} and the Chest X-Ray Images \cite{ChestXRa89:online}.

As we have stated before, we implement the TensorProjection Layer as a custom layer of TensorFlow.
We can directly embed a custom layer defined in TensorFlow to the model defined in Keras.
So we define the model using Keras and embed the TensorProjection Layer into the model.

\subsection{Fashion MNIST Data Set}\label{sec:example:mnist}

We use the Fashion MNIST dataset \cite{DBLP:journals/corr/abs-1708-07747} in this experiment.
The Fashion MNIST dataset consists of 70,000 grayscale 28x28 small images (60,000 training images and 10,000 testing images).
Each image is labeled with one of 10 classes.
The image classes include T-Shirt/Top, Trouser, Pullover, Dress and so on.

We classify the images by a convolutional neural network.
We insert a TensorProjection Layer into the convolutional neural network and evaluate if the proposed method works well.

\subsubsection{Model Structure}

We adopt the model structure shown in Table \ref{table:fashionTP}.
The model contains two convolutional layers.
After the first convolutional layer, we insert an average pooling layer.
After the second convolutional layer, we insert a TensorProjection Layer.

We explain why we do not put a TensorProjection Layer after the first convolutional layer.
In our preliminary experiment, we found that putting an average pooling layer performed better than a TensorProjection Layer after the first convolutional layer.
We guess the reason in the following way.
Two adjacent pixels in an original image are likely to have the close values.
The same can be said to the output of the first convolutional layer.
So summarizing adjacent pixels into one value like pooling layers has a good dimensionality effect.

To compare the dimensionality reduction effect between a TensorProjection Layer and an AveragePooling Layer,
we replace the TensorProjection Layer with an AveragePooling Layer.
We design these two models so that they have the same output size in each layer.
Please refer to the Table \ref{table:fashionAP} for the details of the alternative model structure.
\begin{itemize}
   \item (*1), (*3) The kernel size is $(3,3)$ and the stride size is $(1,1)$. We do zero-padding so that the output has the same shape with the input.
   \item (*2), (*4) The kernel size is $(2,2)$ and the stride size is $(2,2)$.
\end{itemize}

\begin{table}
	\centering
	\begin{tabular}{lllll}
		\toprule
		Layer     & Output Shape  & Parameters & Activation & Remarks \\
		\midrule
		Conv2D & ($n$, 28, 28, 32)  & 320  & ReLU & (*1) \\
		Average Pooling & ($n$, 14, 14, 32) & - & - &(*2)    \\
		Conv2D     & ($n$, 14, 14, 64)  & 18496  & ReLU & (*3) \\
		TensorProjection Layer & ($n$, 7, 7, 64) & 196 & - & \\
		Flatten & ($n$, 3136) & - & \\
		Fully Connected & ($n$, 640) & 2007680 & ReLU & \\
		Dropout (0.5) & ($n$, 640) & - & \\
		Fully Connected & ($n$, 10) & 6410 & SoftMax & \\
		\bottomrule
	\end{tabular}
	\vspace{11pt}
	\caption{Summary of the model with a TensorProjection Layer (Model (I))}
	\label{table:fashionTP}
\end{table}

\begin{table}
	\centering
	\begin{tabular}{lllll}
		\toprule
		Layer     & Output Shape  & Parameters & Activation & Remarks \\
		\midrule
		Conv2D & ($n$, 28, 28, 32)  & 320  & ReLU & (*1)   \\
		Average Pooling & ($n$, 14, 14, 32) & - & - & (*2)  \\
		Conv2D     & ($n$, 14, 14, 64)  & 18496  & ReLU & (*3) \\
		AveragePooling & ($n$, 7, 7, 64) & - &-& (*4) \\
		Flatten & ($n$, 3136) & - & \\
		Fully Connected & ($n$, 640) & 2007680 & ReLU & \\
		Dropout (0.5) & ($n$, 640) & - & \\
		Fully Connected & ($n$, 10) & 6410 & SoftMax & \\
		\bottomrule
	\end{tabular}
	\vspace{11pt}
	\caption{Summary of the alternative model (Model (II))}
	\label{table:fashionAP}
\end{table}

\subsubsection{Training Method}
We use RMSProp algorithm to train the parameters.
(We use the default learning rate.)
We set the batch size to $100$.
We train $15$ epochs.
After finishing each epoch, we test the model with the validation data which is separated from the training data.
We repeat the experiment $20$ times because each experiment result depends on the randomly generated initial values of the training parameters.

\subsubsection{Result}

We present the test result in Table \ref{table:fashionResult_epoch5}, \ref{table:fashionResult_epoch10}, \ref{table:fashionResult_epoch15}.
The tables present the validation accuracy and the validation loss at the $5$th, the $10$th and the $15$th epoch respectively.
Since we implement the experiment 20 times, we summarize them with the best score, worst score and the median value.
Model (I) is the model embedded a TensorProjection Layer and Model (II) is the alternative model without TensorProjection Layer.

We also visualize the change of the validation accuracy and the validation loss in Figure \ref{fig:fashion-val-acc}, \ref{fig:fashion-val-loss}.
In each Figure, we plot the median value of the validation accuracy or validation loss out of the all 20 experiments.

Figure \ref{fig:fashion-val-acc} indicates that the Model (I) with a TensorProjection Layer shows a higher validation accuracy during the training process than the Model (II).
Figure \ref{fig:fashion-val-loss} indicates that the validation loss of the Model (I) decreases faster than the Model (II).
However, the Model (I) starts over-fitting earlier than the Model (II).

\begin{table}[h!]
\centering
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{} & \multicolumn{3}{c}{Validation Accuracy} & \multicolumn{3}{c}{Validation Loss} \\ \midrule
                       & Best     & Worst    & Median    & Best   & Worst   & Median   \\ \midrule
Model (I)   &     0.9224 & 0.9027 & 0.9170 & 0.2205 & 0.2683 & 0.2364    \\
Model (II)    &   0.9154 & 0.9055 & 0.9102 & 0.2420 & 0.2648 & 0.2533   \\ \bottomrule
\end{tabular}
\vspace{11pt}
	\caption{Fashion MNIST: result at the 5th epoch}
	\label{table:fashionResult_epoch5}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{} & \multicolumn{3}{c}{Validation Accuracy} & \multicolumn{3}{c}{Validation Loss} \\ \midrule
                       & Best     & Worst    & Median    & Best   & Worst   & Median   \\ \midrule
Model (I)   &    0.9269 & 0.9193 & 0.9249 & 0.2301 & 0.2620 & 0.2430  \\
Model (II)    &   0.9232 & 0.9159 & 0.9189 & 0.2223 & 0.2891 & 0.2389  \\ \bottomrule
\end{tabular}
\vspace{11pt}
	\caption{Fashion MNIST: result at the 10th epoch}
	\label{table:fashionResult_epoch10}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{} & \multicolumn{3}{c}{Validation Accuracy} & \multicolumn{3}{c}{Validation Loss} \\ \midrule
                       & Best     & Worst    & Median    & Best   & Worst   & Median   \\ \midrule
Model (I)   &   0.9287 & 0.9210 & 0.9249 & 0.2717 & 0.3140 & 0.2920       \\
Model (II)    &    0.9261 & 0.9058 & 0.9198 & 0.2247 & 0.2974 & 0.2512   \\ \bottomrule
\end{tabular}
\vspace{11pt}
	\caption{Fashion MNIST: result at the 15th epoch}
	\label{table:fashionResult_epoch15}
\end{table}

\begin{figure}[h!]
 \begin{center}
  \includegraphics[width=100mm]{figure/fashion-val-acc}
 \end{center}
 \caption{Fashion MNIST: Validation Accuracy (median of 20 trials)}
 \label{fig:fashion-val-acc}
\end{figure}

\begin{figure}[h!]
 \begin{center}
  \includegraphics[width=100mm]{figure/fashion-val-loss}
 \end{center}
 \caption{Fashion MNIST: Validation Loss (median of 20 trials)}
 \label{fig:fashion-val-loss}
\end{figure}

\subsection{Chest X-Ray Image Data Set}\label{sec:example:chest-x-ray}

Next we use the Chest X-Ray Image dataset \cite{ChestXRa89:online}.
The Chest X-Ray Image dataset consists of 5,856 grayscale images (5216 training images, 16 validation images, 624 testing images).
Each image is labeled with 'Normal' or 'Pneumonia'.
We classify healthy people and pneumonia patient from the chest X-ray image using a convolutional neural network.

\subsubsection{Model Structure}

There are only 5,856 training images, and it may not be enough to train the model.
So we take advantage of transfer learning by using the pre-trained model.
We use the VGG16 \cite{DBLP:journals/corr/SimonyanZ14a} pre-trained model as a feature extractor.
The VGG16 was trained using the ImageNet dataset \cite{imagenet_cvpr09}.
The input data of VGG16 is a $224 \times 224 \times 3$ color image.
However, the chest X-ray images are grayscale and the size of each image is not fixed.
Therefore, we use ImageDataGenerator, a component of Keras, and load the training images as color images.
Furthermore, we resize each training image to $224 \times 224$.

The VGG16 contains 14 convolutional layers and 2 fully-connected layers.
We ignore the fully-connected layers in the VGG16 model, and connect a TensorProjection Layer with the last Max-Pooling layer in the VGG16 model.

We present the model structure in Table \ref{table:chest-x-ray-TP} (Model (A)).
We fix the parameters in the VGG16 pre-trained model, in other words, the parameters in VGG16 are not updated anymore during the training process.
So we only train the parameters in the TensorProjection Layer and the output fully-connected layer.

To do a comparison, we also train an alternative model.
We present the alternative model structure in Table \ref{table:chest-x-ray-Dense} (Model (B)).
In the alternative model (Model (B)), we replace the TensorProjection Layer with a fully-connected layer.
Both of the models (Model(A,B)) reduce the dimensionality of the input from $7 \times 7 \times 512 = 25088$ to $256$.

\begin{table}[h!]
	\centering
	\begin{tabular}{lllll}
		\toprule
		Layer     & Output Shape  & Parameters & Activation & Remarks \\
		\midrule
		VGG16 & ($n$, 7, 7, 512)  & -  & - & (*5)   \\
		TensorProjection Layer & ($n$, 14, 14, 32) & 8248 & - &  \\
		Flatten & ($n$, 256) & - &  - \\
		Dropout (0.5) & ($n$, 256) & - & - \\
		Fully Connected & ($n$, 1) & 257 & sigmoid & \\
		\bottomrule
	\end{tabular}
	\vspace{11pt}
	\caption{Summary of the model with a TensorProjection Layer (Model (A))}
	\label{table:chest-x-ray-TP}
\end{table}

\begin{table}[h!]
	\centering
	\begin{tabular}{lllll}
		\toprule
		Layer     & Output Shape  & Parameters & Activation & Remarks \\
		\midrule
		VGG16 & ($n$, 7, 7, 512)  & -  & - & (*5)   \\
		Flatten & ($n$, 25088) & - &  - \\
		Fully Connected & ($n$, 256) & 6422784  & - &  \\
		Dropout (0.5) & ($n$, 256) & - & - \\
		Fully Connected & ($n$, 1) & 257 & sigmoid & \\
		\bottomrule
	\end{tabular}
	\vspace{11pt}
	\caption{Summary of the alternative model (Model (B))}
	\label{table:chest-x-ray-Dense}
\end{table}

\begin{itemize}
   \item $(*5)$ In this experiment, the VGG16 is used as a feature extractor and the parameters in the VGG 16 model are not updated during the training process.
\end{itemize}

\subsubsection{Training Method}
In addition to transfer learning, we also take advantage of data augmentation techniques to prevent over-fitting.
ImageDataGenerator in Keras enables us to randomly shift or zoom the original images.
We implement data augmentation as Code \ref{src:data_augmentation}.

\begin{lstlisting}[language=Python, caption={Data Augmentation with ImageDataGenerator}, label=src:data_augmentation]
train_datagen = tf.keras.preprocessing.image.ImageDataGenerator(
    rescale=1./255,
    width_shift_range=0.05,
    height_shift_range=0.15,
    rotation_range=10,
    zoom_range=0.1,
)
\end{lstlisting}

We use RMSProp algorithm to train the parameters.
We use the default learning rate.
We set the batch size to 100.
We train 15 epochs.
Since the original validation data only contains 16 images, we discard them.
We validate the model with the 624 testing image instead.
After finishing each epoch, we test the model with the 624 testing images.
We repeat the experiment 20 times because each experiment result depends on the randomly generated initial values of the training parameters.

\subsubsection{Result}

We present the test results in Table \ref{table:ChestXRayResult_epoch5}, \ref{table:ChestXRayResult_epoch10}, \ref{table:ChestXRayResult_epoch15}.
The tables present the validation accuracy and the validation loss at the 5th, the 10th and the 15th epoch respectively.
Since we implement the experiment 20 times, we summarize them into the best score, worst score and the median value.
Model (A) is the model embedded a TensorProjection Layer and Model (B) is the alternative model which does not contain a TensorProjection Layer.

We also visualize the change of the validation accuracy and the validation loss in Figure \ref{fig:chest-xray-val-acc}, \ref{fig:chest-xray-val-loss} during the training process.
In each Figure, we plot the median value of the validation accuracy or validation loss out of the all 20 experiments.
From Figure \ref{fig:chest-xray-val-acc}, it is hard to see which one has a higher validation accuracy between Model(A) and Model(B).
However, Model(B)'s accuracy is slightly higher than that of Model(A) at the 15th epoch (Table \ref{table:ChestXRayResult_epoch15}) as a result.
From Figure \ref{fig:chest-xray-val-loss}, we can see that Model(A) fits to the data faster than Model(B).
We consider that this is because Model(B) contains much more parameters than Model(A).

Furthermore, we also plot the worst value of the validation accuracy at each epoch out of the all 20 experiments (See Figure \ref{fig:chest-xray-val-acc-worst}).
We can see that Model(A) performs better than Model(B) in the worst case.
This result implies that Model(A) is more stable than Model (B).

\begin{table}[h!]
\centering
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{} & \multicolumn{3}{c}{Validation Accuracy} & \multicolumn{3}{c}{Validation Loss} \\ \midrule
                       & Best     & Worst    & Median    & Best   & Worst   & Median   \\ \midrule
Model (A)   &   0.9087 & 0.7756 & 0.8854 & 0.2440 & 0.5533 & 0.3027       \\
Model (B)    &  0.9054 & 0.6827 & 0.8758 & 0.8273 & 4.728 & 1.312    \\ \bottomrule
\end{tabular}
\vspace{11pt}
	\caption{Chest-X-Ray Image: result at the 5th epoch}
	\label{table:ChestXRayResult_epoch5}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{} & \multicolumn{3}{c}{Validation Accuracy} & \multicolumn{3}{c}{Validation Loss} \\ \midrule
                       & Best     & Worst    & Median    & Best   & Worst   & Median   \\ \midrule
Model (A)   &   0.9135 & 0.8606 & 0.8950 & 0.2231 & 0.3965 & 0.2855       \\
Model (B)    &  0.9215 & 0.5481 & 0.8942 & 0.2463 & 2.353 & 0.3752    \\ \bottomrule
\end{tabular}
\vspace{11pt}
	\caption{Chest-X-Ray Image: result at the 10th epoch}
	\label{table:ChestXRayResult_epoch10}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{} & \multicolumn{3}{c}{Validation Accuracy} & \multicolumn{3}{c}{Validation Loss} \\ \midrule
                       & Best     & Worst    & Median    & Best   & Worst   & Median   \\ \midrule
Model (A)   &   0.9231 & 0.8718 & 0.8998 & 0.2187 & 0.3844 & 0.2871       \\
Model (B)    &  0.9231 & 0.7788 & 0.9095 & 0.2388 & 0.8579 & 0.3107    \\ \bottomrule
\end{tabular}
\vspace{11pt}
	\caption{Chest-X-Ray Image: result at the 15th epoch}
	\label{table:ChestXRayResult_epoch15}
\end{table}

\begin{figure}[h!]
 \begin{center}
  \includegraphics[width=100mm]{figure/chest-xray-val-acc}
 \end{center}
 \caption{Chest-X-Ray Image: Validation Accuracy (median of 20 trials)}
 \label{fig:chest-xray-val-acc}
\end{figure}

\begin{figure}[h!]
 \begin{center}
  \includegraphics[width=100mm]{figure/chest-xray-val-loss}
 \end{center}
 \caption{Chest-X-Ray Image: Validation Loss (median of 20 trials)}
 \label{fig:chest-xray-val-loss}
\end{figure}

\begin{figure}[h!]
 \begin{center}
  \includegraphics[width=100mm]{figure/chest-xray-val-acc-worst}
 \end{center}
 \caption{Chest-X-Ray Image: Validation Accuracy (worst of 20 trials)}
 \label{fig:chest-xray-val-acc-worst}
\end{figure}

\section{Our Proposal: TensorProjection Layer}\label{sec:TensorProjectionLayer}

\subsection{Forward Propagation}\label{sec:TensorProjectionLayer:ForwardPropagation}
A TensorProjection Layer transforms a 3D tensor into a smaller one by employing $k$-mode product.
Let $\{X_i\}_{i=1}^n$ be the input tensors to the TensorProjection Layer with the size of $p_1 \times p_2 \times p_3$.
We define the output of the TensorProjection Layer as below.
\begin{equation}\label{equation:forward_propagation}
Z_i \eqdef X_i \times_1 U_1^T \times_2 U_2^T \times_3 U_3^T.
\end{equation}
In equation $(\ref{equation:forward_propagation})$, $\{U_k\}_{k=1}^3$ are the training parameters associated with the TensorProjection Layer.
We assume that $\{U_k\}_{k=1}^3$ are the orthogonal matrices with the shape of $p_k \times q_k \ (k=1,2,3)$.
It follows that the size of each output tensor $Z_i \ (i=1,\cdots,n)$ is $q_1 \times q_2 \times q_3$.

Parameters in neural networks are usually trained by gradient descent.
However, if we simply apply the algorithm to the training of the parameters $\{U_k\}_{k=1}^3$ in (\ref{equation:forward_propagation}),
then the trained matrices $\{U_k\}_{k=1}^3$ will not satisfy the constraint about orthogonality.
In order to cope with this constraint, we let $\{W_k\}_{k=1}^3$ be matrices with the size of $p_k \times q_k$ and let
\begin{equation}\label{equation:Uk_}
U_k \eqdef W_k (W_k^T W_k)^{-1/2} \ \ \ \ (k=1,2,3).
\end{equation}
By doing this, we can assure that $\{U_k\}_{k=1}^3$ are orthogonal matrices.
As a consequence, we view $\{W_k\}_{k=1}^3$ as the training parameters associated with the TensorProjection Layer instead of $\{U_k\}_{k=1}^3$.
However, $\{W_k^T W_k\}_{k=1}^3$ in (\ref{equation:Uk_}) are not necessarily full-rank (or invertible).
Therefore we do a small modification to (\ref{equation:Uk_}) as below.
\begin{equation}\label{equation:Uk}
U_k \eqdef W_k (W_k^T W_k + \epsilon_k^2 I_{q_k})^{-1/2} \ \ \ \ (k=1,2,3),
\end{equation}
where $\{\epsilon_k^2\}_{k=1}^3$ are small fixed positive constants.

In conclusion, let $\{W_k\}_{k=1}^3$ be matrices with the size of $p_k \times q_k$ and these are the parameters that we train by gradient descent.
Next we define $\{U_k\}_{k=1}^3$ according to the formula in (\ref{equation:Uk}) instead of (\ref{equation:Uk_}).
For each observation $\{X_i\}_{i=1}^n$, we compute the output tensors $\{Z_i\}_{i=1}^n$ according to the formula (\ref{equation:forward_propagation}).

\subsection{Backpropagation}

Neural network models have complicated structures and  involve a huge number of parameters to train, thus the models are usually unidentifiable, and it is hard to find the analytic solution that minimizes the loss function.
Therefore, in most cases, we find the parameters by gradient descent.
Let $\theta$ be a parameter in a neural network model.
For each $\theta$, we need to compute $\frac{\partial L}{\partial \theta}$ where $L$ is the loss function.
In short, backpropagation is an algorithm to compute $\frac{\partial L}{\partial \theta}$ for each $\theta$.

\subsubsection{Quick Review of Backpropagation}

Firstly, we review the basic idea of backpropagation.
Please refer to Appendix \ref{appendix:notations} for definitions of the notations.
Let us consider a neural network model with $\ell_0$ layers.
Let $n$ be the number of observation (or the minibatch size).
Let $\{X_i^{(\ell)}\}_{i=1}^n$ be the input to the $\ell$ th layer ($\ell=1,\dots,\ell_0$) and let $\{Z_i^{(\ell)} \}_{i=1}^n$ be the output of the $\ell$ th layer corresponding to the $i$ th observation (i.e $X_i^{(\ell)}$).
Let $\{\theta^{(\ell)}_k\}_{k=1}^{K_\ell}$ be the training parameters associated with the $\ell$ th layer.

Let $L$ be the loss function of the neural network.
We find the parameters $\{\theta_k^{(\ell)}\}_{k,\ell}$ that minimize the loss function $L$ by gradient descent.
In the gradient descent method, we compute $\frac{\partial L}{\partial \theta^{(\ell)}_k}$ for each $\ell, k$ to find the minimizer of the loss function $L$.
Backpropagation is merely a simple procedure to compute the gradients $\{\frac{\partial K}{\partial \theta_k^{(\ell)}}\}_{\ell, k}$ using the Chain Rule.

Let us fix $\ell \in \{1,\cdots,\ell_0\}$.
For the time being, we suppose that $\frac{\partial L}{\partial Z_i^{(\ell)}} \ (i=1,\cdots,n)$ are already given.
By the Chain Rule, we obtain $\frac{\partial L}{\partial \theta^{(\ell)}_k} \ (k=1,\cdots, K_\ell)$ in the following way.
\begin{equation}\label{equation:dLdtheta}
\frac{\partial L}{\partial \vect(\theta^{(\ell)}_k)} = \sum_{i=1}^n \frac{\partial L}{\partial \vect(Z^{(\ell)}_i)}  \frac{\partial \vect(Z^{(\ell)}_i)}{\partial \vect(\theta^{(\ell)}_k)}.
\end{equation}
From the above equation $(\ref{equation:dLdtheta})$, we can see that it is sufficient to find $\frac{\partial \vect(Z^{(\ell)}_i)}{\partial \vect(\theta^{(\ell)}_k)}$,
which is relatively easy to compute because both $Z^{(\ell)}_i, \theta^{(\ell)}_k$ are components in the $\ell$ th layer, hence we just need to focus on the design (i.e the definition of output) of $\ell$ th layer.

Next, we discuss how to compute $\{\frac{\partial L}{\partial Z_i^{(\ell)}}\}_{i=1}^n $ for each $i$ and $\ell$, which we assumed to be given in the previous part.
Note that $Z^{(\ell-1)}_i =X^{(\ell)}_i$ because the output of the $\ell-1$ th layer is the input to the $\ell$ th layer.
By the Chain Rule,
\begin{equation}
\frac{\partial L}{\partial \vect(Z^{(\ell-1)}_i)}  = \frac{\partial L}{\partial \vect(X^{(\ell)}_i)} = \frac{\partial L}{\partial \vect(Z^{(\ell)}_i)} \frac{\partial \vect(Z^{(\ell)}_i)}{\partial \vect(X^{(\ell)}_i)}.
\end{equation}
In this way, we compute $\{ \frac{\partial L}{\partial \vect(Z^{(\ell-1)}_i)}  \}_{\ell}$ from the last layer $(\ell = \ell_0)$ toward the first layer $(\ell = 1)$ in order.
This is why the algorithm is called {\it backpropagation}.

\subsubsection{Backpropagation in TensorProjection Layer}

All we have to do is to compute $\{\frac{\partial L}{\partial X_i}\}_{i=1}^n$ and $\{\frac{\partial L}{\partial W_k } \}_{k=1}^3$.
$\{\frac{\partial L}{\partial W_k } \}_{k=1}^3$ is necessary to implement gradient descent.
$\{\frac{\partial L}{\partial X_i} \}_{i=1}^n$  is necessary to find the gradients associated with the previous layer in backpropagation.

For simplicity, we define
\begin{eqnarray}
M_k &\eqdef& W_k^T W_k + \epsilon_k^2 I_{q_k}, \\
G_k &\eqdef& \left(W_k^T W_k + \epsilon_k^2 I_{q_k}\right)^{-1/2} = M_k^{-1/2}.
\end{eqnarray}
Note that we can express $U_k$ as
\begin{equation}
U_k = W_k G_k
\end{equation}
by using $G_k$.
We present the result as below.
We will discuss its derivation in Section \ref{sec:TensorProjectionLayer:Derivation}.
\begin{prop}[Gradients with respect to the TensorProjection Layer]\label{prop:gradients_of_TensorProjectionLayer}
\begin{eqnarray}
\frac{\partial L}{\partial {\rm vec}(X_i)} &=& \frac{\partial L}{\partial {\rm vec}(Z_i)}  \frac{\partial {\rm vec}(Z_i)}{\partial {\rm vec}(X_i)} \label{equation:dLdvecXi} \\
&=&  \frac{\partial L}{\partial {\rm vec}(Z_i)} \left(U_3^T \otimes U_2^T \otimes U_1^T\right) \label{equation:dLdvecXi2} \\
\frac{\partial L}{\partial {\rm vec}(W_k)} &=& \sum_{i=1}^n \frac{\partial L}{\partial {\rm vec}(Z_i)} \frac{\partial {\rm vec}(Z_i)}{\partial {\rm vec}(U_k)} \frac{\partial {\rm vec}(U_k)}{\partial {\rm vec}(W_k)} \label{equation:dLdvecWk}\\
\frac{\partial \vect(Z_i)} {\partial {\rm vec}(U_1)} &=& P^{(1)}_{q_1,q_2,q_3} \left[ \left \{ \left (U_3^T \otimes U_2^T \right ) X_{i_{(1)}}^T \right \} \otimes I_{q_1} \right] K_{p_1, q_1}\label{equation:dvecZidvecU1} \\
\frac{\partial \vect(Z_i)} {\partial {\rm vec}(U_2)} &=& P^{(2)}_{q_1,q_2,q_3} \left[ \left \{\left(U_3^T \otimes U_1^T\right ) X_{i_{(2)}}^T \right \} \otimes I_{q_2} \right ] K_{p_2, q_2} \label{equation:dvecZidvecU2} \\
\frac{\partial \vect(Z_i)} {\partial {\rm vec}(U_3)} &=& P^{(3)}_{q_1,q_2,q_3} \left[ \left \{\left(U_2^T \otimes U_1^T\right ) X_{i_{(3)}}^T \right \} \otimes I_{q_3} \right] K_{p_3, q_3} \label{equation:dvecZidvecU3} \\
\frac{\partial \vect(U_k)}{\partial {\rm vec}(W_k)} &=& G_k^T \otimes I_{p_k} + \left(I_{q_k} \otimes W_k\right) \frac{\partial {\rm vec}(G_k)}{\partial {\rm vec}(W_k)} \label{equation:vecUkvecWk} \\
\frac{\partial \vect(G_k)}{\partial \vect(W_k)} &=& \frac{\partial \vect(G_k)}{\partial \vect(M_k)} \frac{\partial \vect(M_k)}{\partial \vect(W_k)} \label{equation:vecGkvecWk} \\
\frac{\partial \vect(G_k)}{\partial \vect(M_k)} &=& -\frac{1}{2}M_k^{-3/4} \otimes M_k^{-3/4} \label{equation:vecGkvecMk} \\
\frac{\partial \vect(M_k)}{\partial \vect(W_k)} &=& K_{q_k, q_k} \left(I_{q_k} \otimes W_k^T\right) + I_{q_k} \otimes W_k^T \label{equation:vecMkvecWk}.
\end{eqnarray}
\end{prop}

\subsubsection{Derivation}\label{sec:TensorProjectionLayer:Derivation}

(\ref{equation:dLdvecXi}) is immediately obtained by the Chain Rule (Please refer to \ref{appendix:notations:chainrule}).
(\ref{equation:dLdvecXi2}) is obtained by taking $\vect(\cdot)$ to the both sides in (\ref{equation:forward_propagation}).
Please note that
\begin{equation}
\vect(X \times_1 A_1 \times_2 A_2 \cdots \times_k A_k) = \left(A_k \otimes \cdots \otimes A_1\right) \vect(X)
\end{equation}
holds according to the well-known fact.
(\ref{equation:dLdvecWk}) is immediately obtained by the Chain Rule.
In this way, we decompose $\frac{\partial L}{\partial {\rm vec}(W_k)}$ into the product of three components.
Since $\frac{\partial L}{\partial {\rm vec}(Z_i)}$ is given, we discuss how to compute $ \frac{\partial {\rm vec}(Z_i)}{\partial {\rm vec}(U_k)}$ and $\frac{\partial {\rm vec}(U_k)}{\partial {\rm vec}(W_k)}$ in the following part.

The derivations of (\ref{equation:dvecZidvecU1}), (\ref{equation:dvecZidvecU2}) and (\ref{equation:dvecZidvecU3}) are similar.
So it is sufficient for us to discuss (\ref{equation:dvecZidvecU1}).
By unfolding $Z_i$ along the 1st mode (axis or dimension), we obtain
\begin{equation}
Z_{i_{(1)}} = U_1^T X_{i_{(1)}} \left(U_3 \otimes U_2\right),
\end{equation}
where $Z_{i_{(1)}}$ and $X_{i_{(1)}}$ represent the tensor unfoldings of  $Z_{i}$ and $X_{i}$ along the 1st mode respectively.
Note that there are different definitions of $k$-mode tensor unfolding.
In this paper, we follow Kolda and Bader's definition.
Please refer to \cite{TensorUn29:online},\cite{10.1137/07070111X}.
Since $Z_{i_{(1)}} = U_1^T X_{i_{(1)}} (U_3 \otimes U_2) = I_{q_1} U_1^T X_{i_{(1)}} (U_3 \otimes U_2)$,
we obtain
\begin{eqnarray}
\vect\left(Z_{i_{(1)}}\right) &=& \vect\left(I_{q_1} U_1^T X_{i_{(1)}} \left(U_3 \otimes U_2\right)\right) \\
&=&  \left [ \left \{ \left (U_3^T \otimes U_2^T \right) X_{i_{(1)}}^T \right \} \otimes I_{q_1} \right ]  \vect( U_1^T) \\
&=&  \left [ \left \{ \left (U_3^T \otimes U_2^T \right) X_{i_{(1)}}^T \right \}\otimes I_{q_1}  \right ] K_{p_1, q_1} \vect(U_1)
\end{eqnarray}
by vectorizing the both sides, where $K_{p_1, q_1}$ is a commutation matrix.
In the formula above, we applied the fact that
\begin{equation}
\vect(AXB) = (B^T \otimes A) \vect(X).
\end{equation}
A commutation matrix $K_{m,n}$ is a matrix satisfying $\vect(A^T) = K_{m,n} \vect(A)$ for all $m \times n$ matrix $A$.
Please refer to \cite{Xu_2018} for the properties of commutation matrices.
From the result above, it follows that
\begin{eqnarray}
\frac{\partial \vect(Z_{i_{(1)}})}{\partial \vect(U_1)} = \left [\left \{ \left (U_3^T \otimes U_2^T\right) X_{i_{(1)}}^T  \right \} \otimes I_{q_1} \right] K_{p_1, q_1}.
\end{eqnarray}
Note that there exists a permutation matrix $P^{(1)}_{q_1,q_2,q_3}$ s.t
\begin{equation}
\vect(Z_i) = P^{(1)}_{q_1,q_2,q_3} \vect(Z_{i_{(1)}}),
\end{equation}
which only depends on $(q_1, q_2, q_3)$.
By the Chain Rule, we obtain
\begin{eqnarray}
\frac{\partial \vect(Z_i)}{\partial \vect(U_1)} &=& \frac{\partial \vect(Z_i)}{\partial \vect(Z_{i_{(1)}})} \frac{\partial \vect(Z_{i_{(1)}})}{\partial \vect(U_1)} \\
&=& P^{(1)}_{q_1,q_2,q_3} \left [ \left \{ \left (U_3^T \otimes U_2^T\right ) X_{i_{(1)}}^T \right \} \otimes I_{q_1} \right ]K_{p_1, q_1} .
\end{eqnarray}
Now we have (\ref{equation:dvecZidvecU1}).
We can prove (\ref{equation:dvecZidvecU2}), (\ref{equation:dvecZidvecU3}) in a similar way as (\ref{equation:dvecZidvecU1}).

Next we verify (\ref{equation:vecUkvecWk}).
Note that $U_k = W_k G_k$.
Let us consider a small variation $W_k \mapsto W_k + \Delta W_k$.
Since both $G_k$ and $U_k$ are functions of $W_k$, we have
\begin{equation}
U_k + \Delta U_k = (W_k + \Delta W_k) (G_k + \Delta G_k).
\end{equation}
By ignoring $\Delta W_k \Delta G_k$, we have an equation
\begin{eqnarray}
\Delta U_k &=& \Delta W_k G_k + W_k \Delta G_k \\
&=& I_{p_k} \Delta W_k G_k + W_k \Delta G_k I_{q_k}.
\end{eqnarray}
By vectorizing the both sides, we have
\begin{eqnarray}
\vect(\Delta U_k) &=& \vect(I_{p_k} \Delta W_k G_k) + \vect(W_k \Delta G_k I_{q_k}) \\
&=& \left(G_k^T \otimes I_{p_k}\right) \vect(\Delta W_k) + \left(I_{q_k} \otimes W_k \right) \vect(\Delta G_k).
\end{eqnarray}
Note that $\Delta$ and $\vect$ are exchangeable.
\begin{equation}
\Delta \vect(U_k) = \left (G_k^T \otimes I_{p_k} \right) \Delta \vect( W_k) + \left (I_{q_k} \otimes W_k \right) \Delta \vect(G_k).
\end{equation}
The above equation immediately implies (\ref{equation:vecUkvecWk}).
(\ref{equation:vecUkvecWk}) still contains an unsolved part (\ref{equation:vecGkvecWk}).
By the Chain Rule we separate (\ref{equation:vecGkvecWk}) into two parts: (\ref{equation:vecGkvecMk}) and (\ref{equation:vecMkvecWk}).

First we discuss (\ref{equation:vecGkvecMk}).
For simplicity let $M$ be a $q \times q$ symmetric and positive definite matrix.
Let $G \eqdef M^{-1/2}$.
We consider $\frac{\partial \vect(G)}{\partial \vect(M)}$.
Let $\Delta M$ be a variation in $M$ which is also a symmetric and positive definite matrix.
Note that
\begin{equation}
\Delta G = \lim_{\epsilon \rightarrow 0} \frac{(M+\epsilon \Delta M)^{-1/2} - M^{-1/2}}{\epsilon} \label{equation:dG}.
\end{equation}
By simple deformation, we have
\begin{eqnarray}
(M+\epsilon \Delta M)^{-1/2} &=& \Bigl \{ M^{1/2} \left (I_q + \epsilon M^{-1/2} \Delta M M^{-1/2} \right ) M^{1/2} \Bigr \}^{-1/2} \\
&=& M^{-1/4} \left(I_q + \epsilon M^{-1/2} \Delta M M^{-1/2}\right)^{-1/2} M^{-1/4} \\
\label{equation:M_add_epsilonDeltaM_invsqrt_note1} &=& M^{-1/4} \left(I_q + \epsilon V \Lambda V^T \right)^{-1/2} M^{-1/4}   \\
\label{equation:M_add_epsilonDeltaM_invsqrt_note2} &=& M^{-1/4} \left(V (I_q + \epsilon \Lambda) V^T \right)^{-1/2} M^{-1/4}  \\
\label{equation:M_add_epsilonDeltaM_invsqrt_note3} &=& M^{-1/4} V \left (I_q + \epsilon \Lambda \right)^{-1/2} V^T  M^{-1/4}
\end{eqnarray}
\begin{itemize}
  \item (\ref{equation:M_add_epsilonDeltaM_invsqrt_note1}) We consider eigenvalue decomposition of the symmetric matrix $M^{-1/2} \Delta M M^{-1/2} = V \Lambda V^T$.
  \item (\ref{equation:M_add_epsilonDeltaM_invsqrt_note2}) $V V^T = I_q$ because $V$ is an orthogonal matrix.
  \item (\ref{equation:M_add_epsilonDeltaM_invsqrt_note3}) $(V \left (I_q + \epsilon \Lambda \right)^{-1/2} V^T)^2 = V \left (I_q + \epsilon \Lambda \right)^{-1}V^T = (V \left (I_q + \epsilon \Lambda \right)V^T)^{-1}$.
\end{itemize}
Also note that
\begin{equation}
M^{-1/2} = M^{-1/4} V I_q V^T M^{-1/4}.
\end{equation}
By using the facts above, we can deform (\ref{equation:dG}) as below.
\begin{equation}
M^{-1/4} V \Bigl \{ \lim_{\epsilon \rightarrow 0} \frac{ \left(I_q + \epsilon \Lambda \right)^{-1/2} - I_q }{\epsilon} \Bigr \} V^T  M^{-1/4}
\end{equation}
It is not difficult to verify that
\begin{equation}
\lim_{\epsilon \rightarrow 0} \frac{ \left(I_q + \epsilon \Lambda \right)^{-1/2} - I_q }{\epsilon} = -\frac{1}{2} \Lambda.
\end{equation}
This is because the left hand side is a diagonal matrix and all we have to verify is
\begin{equation}
\lim_{\epsilon \rightarrow 0}\frac{1}{\epsilon} \left(\frac{1}{\sqrt{1+\epsilon \lambda_i}} - 1\right) = -\frac{1}{2} \lambda_i.
\end{equation}

As a result, it follows that
\begin{eqnarray}
\Delta G &=& -\frac{1}{2} M^{-1/4} V \Lambda V^T  M^{-1/4} \\
&=&  -\frac{1}{2} M^{-1/4} M^{-1/2} \Delta M M^{-1/2}  M^{-1/4} \\
&=& -\frac{1}{2} M^{-3/4} \Delta M M^{-3/4}
\end{eqnarray}
By vectoring the both sides, we have
\begin{eqnarray}
\vect (\Delta G) = -\frac{1}{2} \left (M^{-3/4} \otimes M^{-3/4}\right) \vect(\Delta M),
\end{eqnarray}
from which we immediately obtain  (\ref{equation:vecGkvecMk}).

Finally we verify  (\ref{equation:vecMkvecWk}).
Similarly, we consider a small variation $W_k \mapsto W_k + \Delta W_k$.
We have
\begin{equation}
M_k + \Delta M_k = (W_k + \Delta W_k)^T(W_k + \Delta W_k).
\end{equation}
By ignoring $\Delta W_k^T \Delta W_k$, we have
\begin{eqnarray}
\Delta M_k &=&  W_k^T \Delta W_k + (\Delta W_k)^T W_k \\
&=& W_k^T \Delta W_k I_{q_k} + I_{q_k} (\Delta W_k)^T W_k.
\end{eqnarray}
By vectorizing the both sides, we have
\begin{eqnarray}
\vect (\Delta M_k) &=& \vect\left(W_k^T \Delta W_k I_{q_k}\right) + \vect\left(I_{q_k} (\Delta W_k)^T W_k\right) \\
&=& \left(I_{q_k} \otimes W_k^T\right) \vect\left(\Delta W_k\right) + \left(W_k^T \otimes I_{q_k}\right) \vect\left(\left(\Delta W_k\right)^T\right) \\
&=& \left(I_{q_k} \otimes W_k^T\right) \vect\left(\Delta W_k\right) + \left(W_k^T \otimes I_{q_k}\right) K_{p_k, q_k} \vect\left(\Delta W_k\right)
\end{eqnarray}
Now the proof is complete.

\subsection{Implementation}
Nowadays, deep learning libraries are capable of automatic differentiation.
We just need to design the forward propagation and it is not always necessary for us to compute the gradients.
We can implement the TensorProjection Layer as a custom layer using TensorFlow \cite{tensorflow2015-whitepaper} without deriving the gradients in Proposition \ref{prop:gradients_of_TensorProjectionLayer}.
We provide the source code of the TensorProjection Layer on GitHub.
Please access to \href{https://github.com/senyuan-juncheng/TensorProjectionLayer}{https://github.com/senyuan-juncheng/TensorProjectionLayer}.

\section{Notations and Matrix Calculus}\label{appendix:notations}

\subsection{gradient}

Let $x \in \mathbb{R}^p$ and let $L \eqdef L(x): \mathbb{R}^p \mapsto \mathbb{R}$ be a function of $x = \left(x_1, x_2, \cdots, x_p \right)^T$.
In this paper, we define the gradients of $L$ with respect to $x$ as below.
\begin{equation}
\frac{\partial L}{\partial x} \eqdef \left(\frac{\partial L}{\partial x_1}, \frac{\partial L}{\partial x_2}, \cdots, \frac{\partial L}{\partial x_p}\right).
\end{equation}
In this paper, the gradient of a scholar function with respect to a (column) vector becomes a row vector.
Now suppose that $L(x)$ is also a multidimensional function.
Let $L \eqdef \left(L_1(x), L_2(x), \cdots, L_d(x) \right)^T: \mathbb{R}^p \mapsto \mathbb{R}^d$ where $x \eqdef (x_1, x_2, \cdots, x_p)^T \in \mathbb{R}^p$.
We define the gradient of $L$ with respect to $x$ as below.
\begin{equation}
\frac{\partial L}{\partial x} \eqdef
\begin{pmatrix}
\frac{\partial L_1(x)}{\partial x_1} & \frac{\partial L_1(x)}{\partial x_2} & \cdots & \frac{\partial L_1(x)}{\partial x_p}  \\
\frac{\partial L_2(x)}{\partial x_1} & \frac{\partial L_2(x)}{\partial x_2} & \cdots & \frac{\partial L_2(x)}{\partial x_p}  \\
\cdots & \cdots & \cdots & \cdots  \\
\frac{\partial L_d(x)}{\partial x_1} & \frac{\partial L_d(x)}{\partial x_2} & \cdots & \frac{\partial L_d(x)}{\partial x_p}
\end{pmatrix}
\end{equation}

\subsection{vectorize operation}

A vectorize operator transforms a tensor (or a matrix) into a vector.
For example, suppose that
\begin{equation}
A \eqdef \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix}.
\end{equation}
Then
\begin{eqnarray}
\vect(A) &\eqdef& \begin{pmatrix} A_{11} & A_{21} & A_{12} &A_{22} & A_{13} & A_{23} \end{pmatrix}^T \\
&=&\begin{pmatrix} 1 & 4 & 2 & 5 & 3 & 6\end{pmatrix}^T.
\end{eqnarray}
Similarly, if $A$ is a tensor with the size of $p \times q \times r$,
then
\begin{equation}
A \eqdef \left(A_{111}, A_{211}, \cdots, A_{p11}, A_{121}, A_{221}, \cdots, A_{p21}, \cdots ,A_{pq1},  A_{112} \cdots \right).
\end{equation}
In Python (NumPy), the reshape action into $pqr \times 1$ with the option order='F' is the same as vectorize operation.
Please refer to \cite{IMM2012-03274} for properties of the vec operator.

\subsection{Chain Rule}\label{appendix:notations:chainrule}

Let $L \eqdef L(u)$ be a scholar function with respect to a vector $u$.
Suppose that $u$ is a function of another vector $x$.
Then the following equation holds.
\begin{equation}
\frac{\partial L}{\partial x} = \frac{\partial L}{\partial u} \frac{\partial u}{\partial x}.
\end{equation}
For example, let $L(u) \eqdef u_1 u_2$ and let $u_1 \eqdef (x_1+x_2), u_2 \eqdef (x_2+x_3)$.
Since $L = (x_1+x_2)(x_2+x_3)$, we can directly obtain
\begin{equation}
\frac{\partial L}{\partial x} = \left(x_2+x_3, x_1+2x_2+x_3, x_1+x_2\right).
\end{equation}
We can also obtain the same result via the Chain Rule above.
\begin{eqnarray}
\frac{\partial L}{\partial u} \frac{\partial u}{\partial x} &=& \left(u_2 , u_1\right) \begin{pmatrix} 1 & 1 & 0 \\ 0 & 1 & 1\end{pmatrix} \\
&=& (u_2, u_1 + u_2, u_1) \\
&=& (x_2+x_3, x_1+2x_2+x_3, x_1+x_2).
\end{eqnarray}

\section{$k$-mode product}\label{appendix:k-mode-product}

We explain the definition of $k$-mode product between a 3-order tensor and a matrix.
The following definition follows Kolda and Bader's definition.
Please refer to \cite{TensorUn29:online},\cite{10.1137/07070111X}.
Although, Kolda-Bader and Tensorly propose different definitions of unfolding and folding,
the results of $k$-mode product are consistent.

\subsection{unfolding}

Let $X$ be a tensor with the size of $p_1 \times p_2 \times p_3$.
Let $X_{(k)}$ be the unfolding along the $k$-th mode where $k=1,2,3$.
$X_{(k)}$ is a matrix defined in the following way.
In the definition below, $\mathrm{permute}(X,[\cdots])$ is an operation to swap the axes according to the order written in $[\cdots]$,
and $\mathrm{reshape}$ is an operation to transform the tensor into a designated shape.
In the reshape operation, we suppose that the index of the 1st mode (axis) changes faster than the 3rd (or the last) mode as well as the vectorize operation which we defined above.
In NumPy's reshape function, we need to add the option order='F' to follow our definition.
\begin{itemize}
  \item $X_{(1)} \eqdef \mathrm{reshape}(\mathrm{permute}(X, [1,2,3]),[p_1, p_2p_3])$
  \item $X_{(2)} \eqdef \mathrm{reshape}(\mathrm{permute}(X, [2,1,3]),[p_2, p_3p_1])$
  \item $X_{(3)} \eqdef \mathrm{reshape}(\mathrm{permute}(X, [3,1,2]),[p_3, p_1p_2])$
\end{itemize}

We also provide a sample code in Python.
Please see the Code \ref{src:unfolding}. (Please import numpy as np.)
In NumPy, the default reshape order option is 'C', which is opposite from our definition.
Please also refer to \cite{cohen2015notations}.
In \cite{cohen2015notations}, the reshape order is compatible with NumPy default order.
\begin{lstlisting}[language=Python, caption={unfolding of a 3-order tensor}, label=src:unfolding]
def unfold(X, k):
    p1 = X.shape[0];
    p2 = X.shape[1];
    p3 = X.shape[2];

    if k == 0:
        n_shape = np.array([p1,p2*p3]);
        A = X.transpose(0,1,2);
    elif k==1:
        n_shape = np.array([p2,p3*p1]);
        A = X.transpose(1,0,2);
    elif k==2:
        n_shape = np.array([p3,p1*p2]);
        A = X.transpose(2,0,1);

    X_k = np.reshape(A,n_shape,order='F');
    return X_k;
\end{lstlisting}

\subsection{folding}

Folding is the inverse transformation of unfolding defined above.
Please see Code \ref{src:folding}. (Please import numpy as np.)
\begin{lstlisting}[language=Python, caption={folding of a 3-order tensor}, label=src:folding]
def fold(X_k, k, new_shape):
    if k==0:
        reshape_order = np.array([0,1,2]);
        permute_order = np.array([0,1,2]);
    elif k==1:
        reshape_order = np.array([1,0,2]);
        permute_order = np.array([1,0,2]);
    elif k==2:
        reshape_order = np.array([2,0,1]);
        permute_order = np.array([1,2,0]);

    new_shape = np.array([new_shape[reshape_order[0]],new_shape[reshape_order[1]],new_shape[reshape_order[2]]]);
    X = np.reshape(X_k,new_shape,order='F');
    X = X.transpose(permute_order);
    return X;
\end{lstlisting}

\subsection{$k$-mode product}

Let $X$ be a tensor with the size of $p_1 \times p_2 \times p_3$.
Let $M_k$ be a matrix with the size of $q_k \times p_k$.
Then we can define the $k$-mode product $X \times_k M_k$.
$X \times_k M_k$ is a tensor with the size of $q_1 \times p_2 \times p_3$ (if $k=1$) or $p_1 \times q_2 \times p_3$ (if $k=2$) or $p_1 \times p_2 \times q_3$ (if $k=3$).
The $k$-mode product is defined in the following way.
\begin{itemize}
  \item [Step 1.] Compute $X_{(k)}$.
  \item [Step 2.] Compute $M_k X_{(k)}$. Note that $X_{(k)}$ is a matrix.
  \item [Step 3.] Fold the matrix $M_k X_{(k)}$ along the $k$-th mode. (The shape is $q_1 \times p_2 \times p_3$ or $p_1 \times q_2 \times p_3$ or $p_1 \times p_2 \times q_3$.)
\end{itemize}
Please see Code \ref{src:k-mode-product}. (Please import numpy as np.)
\begin{lstlisting}[language=Python, caption={k-mode product of a 3-order tensor}, label=src:k-mode-product]
def k_mode_product(X, k, M):
    X_k = unfold(X,k);
    qk = M.shape[0];
    p = np.array(X.shape);
    p[k] = qk;
    MX_k = np.matmul(M,X_k);
    MX_k = fold(MX_k,k,p);
    return MX_k;
\end{lstlisting}

redpajama_set_name: RedPajamaArXiv