summaries: This paper presents a method to reduce the dimensions of resting-state functional connectivity for psychiatry disorders such as autism spectrum disorder, major depressive disorder, and schizophrenia. Their method is based on a conditional variational auto-encoder that utilized the diagnostic label. They evaluated their method on two neuroimaging datasets and obtained clustered 2-dimensional representations of multi psychiatry disorder.

summaries: The authors present a new conditional VAE, which uses diagnostic information as a supervision signal in two different ways, with the aim to identify small latent embedding spaces for functional brain connectivity that preserve diagnostic information. The authors have potential applications in computational psychiatry in mind, which are indeed extensively discussed in the field (transitioning to a continuous nosological approach to mental illness). They compare their approach on synthetic data to various related approaches and ultimately apply it to two functional connectivity datasets from patients with different psychiatric diagnoses, which appears to give good results.  Their main contribution is technical and lies in defining a new cost function to learn the embedding space. Their work incrementally builds on existing work, but appears to be sufficiently novel. There is also a moderate empirical contribution as they analyze the relationship between MDD, ASD and SCZ.

summaries: This paper presents a new conditional variational autoencoder (VAE) approach to learn a low dimensional embedding of neuropsychiatric disorders from resting-state functional connectivity data. The proposed approach uses diagnostic information in 2 ways: 1) to cluster samples from the same disorder together via the conditional VAE model, and 2) to separate the clusters using contrastive learning.  The method is compared against other dimension reduction approaches on both a synthetic dataset and 2 real datasets which showed a consistent nosological relation among 3 disorders.

summaries: The authors propose a novel variational autoencoder to utilize functional connectivity (FC) features from resting state fMRI (rs-fMRI) scans in order to uncover latent nosological relationships between diverse yet related neuropsychiatric disorders. The autoencoder  seeks to learn a mapping  from high-dimensional FC space to a low-dimensional embedding space that is constrained to preserve pairwise relationships between the diagnostic attributes of the disorders. From a clinical standpoint, this work studies the nosology of complex disorders using a continuous dimensional characterization rather than using categorical and discrete diagnostic labels for supervision.    The authors validate their framework on both synthetic data and two separate clinical rs-fMRI datasets consisting of patients diagnosed with Autism Spectrum Disorder (ASD), Major Depressive Disorder (MDD) and Schizophrenia (SCZ). Their experiments evaluate the consistency of uncovered pairwise nosological relationships inferred from their latent representations.  They demonstrate that their model is capable of reliably inferring a dimensional characterization of multiple brain disorders beyond diagnosis labels.

abstractText: The unravelled nosological relation among diverse types of neuropsychiatric disorders serves as an important precursor in advocating the dimensional approach to psychiatric classification. Leveraging high-dimensional abnormal resting-state functional connectivity, the crux of mining corresponded nosological relations is to derive a low-dimensional embedding space that preserves the diagnostic attributes of represented disorders. To accomplish this goal, we seek to exploit the available diagnostic information in learning the optimal embedding space by proposing a novel type of conditional variational auto-encoder that incorporates dual utilisation of diagnostic information. Encouraged by the achieved promising results in challenging the conventional approaches in low dimensional density estimation of synthetic functional connectivity features, we further implement our approach on two empirical neuropsychiatric neuroimaging datasets and discover a reliable nosological relation among autism spectrum disorder, major depressive disorder, and schizophrenia.

title: Mental health: On the spectrum

venue: Nature News Feature,

authors:
Verneri Anttila
Brendan Bulik-Sullivan
Hilary K Finucane
Raymond K Walters
Jose Bras
Laramie Duncan
Valentina Escott-Price
Guido J Falcone
Padhraig Gormley
Rainer Malik

title: Analysis of shared heritability in common disorders

venue: of the brain. Science,

authors:
Wenjun Bai
Changqin Quan
Zhi-Wei Luo

title: Learning flexible latent representations via encapsulated variational encoders

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Simon Baron-Cohen
Sally Wheelwright
Richard Skinner
Joanne Martin
Emma Clubley

title: The autism-spectrum quotient (aq): Evidence from asperger syndrome/high-functioning autism, malesand females, scientists and mathematicians

venue: Journal of autism and developmental disorders,

authors:
Aaron T Beck
Robert A Steer
Roberta Ball
William F Ranieri

title: Comparison of beck depression inventories-ia and-ii in psychiatric outpatients

venue: Journal of personality assessment,

authors:
Christopher P Burgess
Irina Higgins
Arka Pal
Loic Matthey
Nick Watters
Guillaume Desjardins
Alexander Lerchner

title: Understanding disentangling in beta-vae

venue: arXiv preprint arXiv:1804.03599,

authors:
BJ Casey
Nick Craddock
Bruce N Cuthbert
Steven E Hyman
Francis S Lee
Kerry J Ressler

title: Dsm-5 and rdoc: progress in psychiatry research

venue: Nature Reviews Neuroscience,

authors:
Avshalom Caspi
Terrie E Moffitt

title: All for one and one for all: Mental disorders in one dimension

venue: American Journal of Psychiatry,

authors:
Nick Craddock
Michael J Owen

title: The kraepelinian dichotomy–going, going but still not gone

venue: The British Journal of Psychiatry,

authors:
Nat Dilokthanakul
Pedro AM Mediano
Marta Garnelo
Matthew CH Lee
Hugh Salimbeni
Kai Arulkumaran
Murray Shanahan

title: Deep unsupervised clustering with gaussian mixture variational autoencoders

venue: arXiv preprint arXiv:1611.02648,

authors:
Richard O Duda
Peter E Hart

title: Pattern classification

title: Whither dsm–v? The British

venue: Journal of Psychiatry,

authors:
Jacob Goldberger
Geoffrey E Hinton
Sam Roweis
Russ R Salakhutdinov

title: Neighbourhood components analysis

authors:
Xinyu Guo
Kelli C Dominick
Ali A Minai
Hailong Li
Craig A Erickson
Long J Lu

title: Diagnosing autism spectrum disorder from brain resting-state functional connectivity patterns using a deep neural network with a novel feature selection method

venue: Frontiers in neuroscience,

authors:
Raia Hadsell
Sumit Chopra
Yann LeCun

title: Dimensionality reduction by learning an invariant mapping

venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06),

authors:
Anibal Sólon Heinsfeld
Alexandre Rosa Franco
R Cameron Craddock
Augusto Buchweitz
Felipe Meneguzzi

title: Identification of autism spectrum disorder using deep learning and the abide dataset

venue: NeuroImage: Clinical,

authors:
John E Helzer
Helena C Kraemer
Robert F Krueger
Hans-Ulrich Wittchen
Paul J Sirovatka
Darrel A Regier

title: Dimensional approaches in diagnostic classification: Refining the research agenda for DSM-V

venue: American Psychiatric Pub,

authors:
Martin Heusel
Hubert Ramsauer
Thomas Unterthiner
Bernhard Nessler
Sepp Hochreiter

title: Gans trained by a two time-scale update rule converge to a local nash equilibrium

venue: arXiv preprint arXiv:1706.08500,

authors:
Geoffrey Hinton
Nitish Srivastava
Kevin Swersky

title: Neural networks for machine learning lecture 6a overview of mini-batch gradient descent

venue: Cited on,

authors:
Quentin JM Huys
Tiago V Maia
Michael J Frank

title: Computational psychiatry as a bridge from neuroscience to clinical applications

venue: Nature neuroscience,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Diederik P Kingma
Danilo J Rezende
Shakir Mohamed
Max Welling

title: Semi-supervised learning with deep generative models

authors:
Shinsuke Koike
Saori C Tanaka
Tomohisa Okada
Toshihiko Aso
Ayumu Yamashita
Okito Yamashita
Michiko Asano
Norihide Maikusa
Kentaro Morita
Naohiro Okada

title: Brain/minds beyond human brain mri project: a protocol for multi-level harmonization across brain disorders throughout the lifespan

authors:
Leland McInnes
John Healy
James Melville

title: Umap: Uniform manifold approximation and projection for dimension reduction

venue: arXiv preprint arXiv:1802.03426,

authors:
Grégoire Montavon
Alexander Binder
Sebastian Lapuschkin
Wojciech Samek
Klaus-Robert Müller

title: Layer-wise relevance propagation: an overview

venue: Explainable AI: interpreting, explaining and visualizing deep learning,

authors:
Charlie Nash
SM Ali Eslami
Chris Burgess
Irina Higgins
Daniel Zoran
Theophane Weber
Peter Battaglia

title: The multi-entity variational autoencoder

venue: In NIPS Workshops,

authors:
Aaron van den Oord
Oriol Vinyals
Koray Kavukcuoglu

title: Neural discrete representation learning

venue: arXiv preprint arXiv:1711.00937,

authors:
Matthieu Perrot
Denis Rivière
Jean-François Mangin

title: Cortical sulci recognition and spatial normalization

venue: Medical image analysis,

authors:
Kihyuk Sohn
Honglak Lee
Xinchen Yan

title: Learning structured output representation using deep conditional generative models

authors:
Saori C Tanaka
Ayumu Yamashita
Noriaki Yahata
Takashi Itahashi
Giuseppe Lisi
Takashi Yamada
Naho Ichikawa
Masahiro Takamura
Yujiro Yoshihara
Akira Kunimatsu

title: A multisite, multi-disorder resting-state magnetic resonance image database

venue: Scientific data,

authors:
Tomoki Tokuda
Junichiro Yoshimoto
Yu Shimizu
Go Okada
Masahiro Takamura
Yasumasa Okamoto
Shigeto Yamawaki
Kenji Doya

title: Identification of depression subtypes and relevant brain regions using a data-driven approach

venue: Scientific reports,

authors:
Jakub Tomczak
Max Welling

title: Vae with a vampprior

venue: In International Conference on Artificial Intelligence and Statistics,

authors:
Jack Tsai
Robert A Rosenheck

title: Psychiatric comorbidity among adults with schizophrenia: A latent class analysis

venue: Psychiatry research,

authors:
Arash Vahdat
Jan Kautz

title: Nvae: A deep hierarchical variational autoencoder

venue: arXiv preprint arXiv:2007.03898,

authors:
Laurens Van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research,

title: Autism and schizophrenia: one, two or many disorders

authors:
Neil D Woodward
Carissa J Cascio

title: Resting-state functional connectivity in psychiatric disorders

venue: JAMA psychiatry,

authors:
Cedric Huchuan Xia
Zongming Ma
Rastko Ciric
Shi Gu
Richard F Betzel
Antonia N Kaczkurkin
Monica E Calkins
Philip A Cook
Angel Garcia de la Garza
Simon N Vandekar

title: Linked dimensions of psychopathology and connectivity in functional brain networks

venue: Nature communications,

authors:
Meichen Yu
Kristin A Linn
Philip A Cook
Mary L Phillips
Melvin McInnis
Maurizio Fava
Madhukar H Trivedi
Myrna M Weissman
Russell T Shinohara
Yvette I Sheline

title: Statistical harmonization corrects site effects in functional connectivity measurements from multi-site fmri data

venue: Human brain mapping,

authors:
Shengjia Zhao
Jiaming Song
Stefano Ermon

title: Infovae: Information maximizing variational autoencoders

venue: arXiv preprint arXiv:1706.02262,

authors:
Yizhe Zhu
Martin Renqiang Min
Asim Kadav
Hans Peter Graf

title: S3vae: Self-supervised sequential vae for representation disentanglement and data generation

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

title: SRPBS DATABASE & DATA PRE-PROCESSING STEPS The employed UTO and HuShoWa datasets are derived from the SRPBS database, which is consisted of resting-state fMRI scans from 8 different sites. The detailed curation and pre-precessing steps of SRPBS database can be consulted in the work

heading: 1 INTRODUCTION

heading: 1.1 NOSOLOGICAL RELATION AMONG NEUROPSYCHIATRIC DISORDERS

text:
Instead of the traditional discrete, categorical view on nosology of multiple neuropsychological disorders (Frances, 2009), an alternative dimensional, continuous view, which suggests all mental illnesses lie along a single low dimensional spectrum (Adam, 2013; Casey et al., 2013; Helzer et al., 2009), uprises to be a more promising perspective to view the relation among diverse disorders. Despite its commonplace in the symptom-based clinical domain (Caspi & Moffitt, 2018), it was not until recently that the finding of shared genetic components across various neuropsychiatric disorders (Anttila et al., 2018) offered the first computational study to reveal the nosological relation among various disorders.
Aside from the discovered commonalities on the genetic makeup of various disorders, as the restingstate functional connectivity (FC) has proved itself as a valuable biomarker in distinguishing disorder patients from healthy controls in a plethora of computational studies (Woodward & Cascio, 2015), they may also hold great promise for uncovering the complex, continuous relation among different neuropsychiatric disorders. Unfortunately, due to their inherent high dimensionality, it poses a formidable data-analytic challenge in discovering the targeted nosological relation on the highdimensional neuropsychiatric FC feature space. Thence, the crux of unveiling the FC informed nosological relation rests on finding an optimal low-dimensional embedding space to be informative of high-dimensional FC feature space.

heading: 1.2 THE OBJECTIVE OF THIS RESEARCH

text:
One straightforward approach is to apply dimensionality reduction techniques to these highdimensional FC features. However, as stated in Huys et al. (2016), without the supplied supervision signals, over-reliance on data-driven methods may not be sufficient to derive the optimal
low-dimensional embeddings of FC features. Hence, the objective of this research is to seek the utilisation of supervision signals, e.g., diagnostic information, in learning a neural correlate informed low-dimensional embedding space, which allows us to attain the nosological relation among diverse disorders.
To harness diagnostic information in learning a targeted low-dimensional embedding space, in the following sections, we chiefly propose a novel type of conditional variational auto-encoder (VAE) to incorporate dual utilisation of this information, which involves regularising the learning of embeddings through the introduced implicit clustering effect, and encoding the diagnostic difference among disorders. Based on the demonstrated empirical superiority of our approach in a simulation study, we implemented it in mining a consistent nosological relation among three common disorders across two curated neuropsychiatric FC datasets.
In summarisation, main contributions of our work include:
• putting forward a novel type of conditional VAE, which involves two utilisations of diagnostic information in learning an optimal embedding space for high dimensional FC features;
• mining a consistent nosological relation among autism spectrum disorder, major depressive disorder, and schizophrenia across two curated neuropsychiatric FC datasets.

heading: 2 PREVIOUS WORKS

text:
Harnessing the extracted brain features, recent efforts in computational psychiatry are largely acknowledged on developing increasingly complicated classification models to mine discriminative features to aid clinical diagnosis (Guo et al., 2017; Heinsfeld et al., 2018) and discovering the subtypes of a single mental illness via advanced data-driven approaches, e.g., the multi-view clustering method (Tokuda et al., 2018).
Unfortunately, little attention was paid to unveiling complex nosological relations among diverse types of neuropsychiatric disorders based on brain correlates. Among few attempts, a worth mentioned study that shares a similar motivation with ours is the work of Xia et al. (2018), where dimensions of psychopathology were closely tied to certain common brain features at the macro level, e.g., resting-state networks. However, this finding – in support of the differentiation of disorders on the brain feature level – reveals little on the nosological relationship among disorders, let alone their symptom-based methodology, which does not fully harness the essence of brain features.
Narrowing our focus on prevailing methodological modifications on VAE, a plethora of variants ranging from the improved generalisation, e.g., beta-VAE (Burgess et al., 2018), Vamp-VAE (Tomczak & Welling, 2018), the diversified generative process, e.g., EVAE (Bai et al., 2019), Multi-entity VAE (Nash et al., 2017), the optimised inference (encoding) process, e.g., Info-VAE (Zhao et al., 2017), the discretised latent space, e.g., vq-VAE (Oord et al., 2017), to the conditional variant and its recent advents, e.g., a semi-supervised VAE (Kingma et al., 2014), NVAE (Vahdat & Kautz, 2020), CVAE (Sohn et al., 2015), and the Gaussian mixture VAE (Dilokthanakul et al., 2016). Falling into the last category, our proposed method can be regarded as a novel type of conditional VAE that embeds a clustering concept. To interested readers, we attach a detailed methodological comparison between our approach and mentioned alternative approaches in A.1. Additionally, rather than the conventional usage in the work of Zhu et al. (2020), our implementation of the contrastive learning is directly applied in the embedding space instead of the original feature space.

heading: 3 DUAL UTILISATION OF DIAGNOSTIC INFORMATION

text:
To exploit diagnostic information, i.e., the discrete diagnostic labels c ∈ C, in learning low dimensional latent variables in the embedding space z ∈ Z that are aware of both projected high dimensional FC input features x ∈ X , and their diagnostic attributes, as shown in Figure 1, we delineate dual utilisation of diagnostic information in our proposed conditional VAE.
1. The provided diagnostic information first serves as observable variables to regularise the learning of z in a novel type of conditional VAE, ensuring the learned (lowdimensional) z to be clustered around close-to-empirical cluster centers.
2. The z associated diagnostic difference is further encoded in the embedding space via contrastive learning.
3.1 MANNER-1: REGULARISE THE LEARNING OF z IN A NOVEL CONDITIONAL VAE
Under the coinage of VAE, an amortised joint inference distribution qφ(z, x) ≡ qφ(z|x)p(x) is commonly employed to approximate the intractable joint generative distribution pθ(z, x) ≡ pθ(x|z)p(z). Aside from the empirical distribution p(x), we further assume p(z) as a Normal distribution, and both qφ(z|x) and pθ(x|z) are conditional Gaussian distributions in which location and scale parameters can be parameterised by complex neural networks. The optimisation objective is acknowledged as the minimisation of the K-L divergence between two joint densities of latent variable and input, i.e., p(z, x) and q(z, x), as: DKL{qφ(z, x)||pθ(z, x)}. To harness provided diagnostic information, we explicitly add discrete variables c ∈ C to represent available diagnostic information; it leads to a conditional VAE (Figure 1 (b)), which the general optimisation objective can be derived as DKL{qφ(z, x, c)||pθ(z, x, c)}. Crucially, to further specify the inner components of two joint densities qφ(z, x, c), and pθ(z, x, c), we adopt a hierarchical encoding process that admits the following derivation: q(φz,φc)(c, z|x) ∝ qφc(c|z)qφz (z|x). In a
similar vein, a hierarchical decoding process permits: p(θz,θc)(x, z|c) ∝ pθz (x|z)pθc(z|c). As a result, the foregoing joint inference and generative distributions can be derived as: q(φz,φc)(x, z, c) ∝ qφc(c|z)qφz (z|x)p(x) and p(θz,θx)(x, z, c) ∝ pθx(x|z)pθz (z|c)p(c), respectively. From now on, we ignore subscripts for brevity. The general optimisation objective of this novel type of conditional VAE Lcvae with specified hierarchical inference and generative processes can be rewritten into a novel format:
DKL{qφ(z, x, c)||pθ(z, x, c)} = ∑ c ∫∫ q(c|z)q(z|x)p(x) log q(c|z)q(z|x)p(x) p(x|z)p(z|c)p(c) dzdx
= Ex∼p(x) [ Ez∼q(z|x)[− log p(x|z)] ] + Ex∼p(x) [ DKL(q(c|z)||p(c)) + ∑ c q(c|z) log q(z|x) p(z|c) ] ,
(1)
where we ignore the optimisation irrelevant term log p(x) (an empirical distribution) for brevity. With N number of classes, we define a categorical prior distribution Cat(c|π), π ∈ <N+ for p(c). Three different parameterised Gaussian distributions are further defined as p(x|z) = N (x;µx, I), q(z|x) = N (z;µx, σ2xI), and p(z|c) = N (z;µc, I), respectively, where I is the identity matrix. q(c|z) can be modelled by a multi-layer feedforward soft-max network to produce a categorical representation (logits) for each encoded latent representation.
The first component in equation 1, i.e., Ez∼q(z|x)[− log p(x|z)], can be realised as the conventional reconstruction loss in a vanilla VAE to ensure the low-dimensional latent variables encoded from high-dimensional FC features. The entire second component, i.e., DKL(q(c|z)||p(c)) +∑ c q(c|z) log q(z|x) p(z|c) , can be understood as the imposed (implicit) clustering effect in the lowdimensional embedding space.
The first term in the above-mentioned second component, i.e., DKL(q(c|z)||p(c)), serves as a preventative measure to prevent creating a dominant cluster in the face of a class-imbalanced dataset. Intuitively, it encourages the produced categorical representation (logits) c from q(c|z) to be close to its categorical prior distribution p(c). However, since the K-L divergence is an intractable divergence measure, we compute an easy-to-measure alternative quantity, i.e., the cross-entropy of two distributions H(q(c|z), p(c)) in practice. Letting N to indicate the number of input classes, the second term in the second component, i.e., ∑ c q(c|z) log q(z|x) p(z|c) , encourages the learned embedding space to be divided into N different subspaces to ensure each encoded latent representation can be ascribed to 1 of N clusters. Since the loss term that is on the clustering effect can be pinned down to 12 ||z − µc||
2 (see the derivation in attached A.2), where the cluster centre µc is provided directly from the precomputed empirical cluster mean, the first utilisation of diagnostic information can be regarded as enforcing each encoded embedding z to be closer to its belonged cluster centre µc within each of N clusters.

heading: 3.2 MANNER-2: ENCODING DIAGNOSTIC DIFFERENCE IN THE EMBEDDING SPACE

text:
Corresponding to the second manner of utilising diagnostic information, i.e., embedding the diagnostic difference between disorder phenotypes, we resort to contrastive learning (Hadsell et al., 2006) to explicitly model this diagnostic difference. As shown in Figure 1(c), we define a pair of latent vectors−→z1 ,−→z2 ∈ Z to represent two encoded high dimensional brain features, a distance function D to measure the dissimilarity of two latent vectors, and a binary class indicator C = 0, 1 to verify whether latent vectors come from the same disorder phenotype. The contrastive loss function Lclr (Hadsell et al., 2006) in our case can be defined as:
(1− C)1 2 (D(−→z1,−→z2)) 2 + C 1 2 {max(0,m−D(−→z1,−→z2))} 2, (2)
where m is a pre-definable margin to weigh the contribution of dissimilar pairs, and we adopt the simple euclidean distance to captureD(−→z1,−→z2), i.e.,D(−→z1,−→z2) = ||
−→z1−−→z2 ||2. To preserve the authentic diagnostic differences among encoded latent variables, the m is set to 0, i.e., we weigh the equal importance of dissimilar and similar pairs.
Importantly, bridging the two loss terms, i.e., Lclr and Lcvae together, an overall loss function Ltotal through a convex combination of two terms can be derived as:
Ltotal = (1− λ)Lcvae + λLclr, (3) where we enforce the equal contribution of two loss terms, i.e., λ = 0.5, to stress the equal importance of two manners in utilising the diagnostic information to derive our targeted embedding space. In attached A.3, we show the effects of differentiated λ in shaping the embedding space for interested readers.

heading: 4 ASSESSING THE IMPORTANCE OF DIAGNOSTIC INFORMATION

text:
To assess the importance of diagnostic information in deriving the optimal low-dimensional embedding of neuropsychiatric FC features, we implemented our proposed method in a simulation study, along with the ground truth embeddings. The purpose of this simulation study is twofold. (1) We seek to assess the importance of diagnostic information in recovering ground truth embeddings. (2) We also aim to demonstrate the empirical superiority of our proposed approach over several existing supervised representation learning approaches.
In preparing for the synthetic high-dimensional FC features, we strictly follow the following threestep generation procedure.
1. We firstly pre-train a U-map projection function F on a real FC dataset, e.g., the SRPBS multi-disorder brain connectivity database (Tanaka et al., 2021).
2. 300 low dimensional embeddings are then sampled from three 2D multivariate Gaussians N2(0.8, 1);N2(0.4, 1);N2(0.2, 1). The pair-wise distance relations among three 2D Gaussians are served as the ground-truth relation (see Figure 2).
3. The inverted projection function F−1 is further utilised to map the foregoing 300 embeddings from the low-dimensional (2D) ground truth embedding space into the highdimensional FC space. In the case of the SRPBS dataset, this produces a [300, 9730] synthetic FC feature matrix.

heading: 4.1 MODELS AND THEIR IMPLEMENTATIONS

heading: 4.1.1 WITHIN OUR APPROACH

text:
The crux of implementing our proposed approach is to compute the final loss function equation 3 with its involved loss terms Lcvae in equation 1, and Lclr in equation 2. Aside from the easy computed reconstruction loss, in the remaining components of equation 1, i.e., ∑ c q(c|z) log q(z|x) p(z|c) +
DKL(q(c|z)||p(c)), the parameters that are pertaining to the optimisation of ∑ c q(c|z) log q(z|x) p(z|c) can be reduced to σ2x, µx, z, and µc. As σx and µx can be parameterised by a feed-forward neural network, it allows the production of z via the re-parameteristion trick Kingma & Welling (2013), i.e., z = ⊗ σx+µx, ∼ N (0, 1). The cluster-wise mean µc can be produced by a simple one layer Gaussian network with supplied supervision signals, e.g., cluster labels in this simulation study. A simple softmax network is sufficient to measure the KL-divergence between q(c|z) and p(c). The second loss term Lclr in equation 2 can be easily captured via adding a margin (distance) based feed-forward neural network on top of encoded latent representations.
Essentially, to demonstrate the importance of dual utilisation of diagnostic information in our proposed approach, corresponding to above-mentioned two utilisations, three instantiations of our approach are established ranging from the full implementation of our model (Full model) to the mere implementation of Lcvae (Manner-1 model), and Lclr (Manner-2 model). Their layerwise configurations and respective parameter estimation processes are concisely summarised in attached B.1.

heading: 4.1.2 ALTERNATIVE APPROACHES

text:
Three categories of alternative methods are also included in this simulation study, ranging from the conventional semi-supervised VAE (S-VAE) (Kingma et al., 2014), the mainstream manifold learning based dimensionality reduction approach, e.g., supervised Umap (S-Umap) (McInnes et al.,
2018), to the metric learning based neighbourhood component analysis (NCA) (Goldberger et al., 2004). The implementation details of these alternative approaches are also demonstrated in attached B.1.

heading: 4.2 PERFORMANCE EVALUATION

text:
As shown in Figure 2, provided with ground truth embeddings and their pairwise distance relations, we target on evaluating the performance of various models regarding the capability of recovering the ground truth embeddings from synthetic high-dimensional FC features. The evaluation metrics include the qualitative visual inspection of yielded embedding spaces, and a quantitative assessment to verdict whether the distance relations among estimated clusters are preserved.
To quantify the between-cluster distance on the derived low dimensional embedding space, we rely on the pair-wise Wasserstein distance-based Frechet inception distance (FID) based metric (Heusel et al., 2017), i.e., FID(µ1,Σ1,µ1,Σ2), over the conventional L− 1 and L− 2 distance metrics for its consideration on both first and second degree statistics in measuring the discrepancy between two multivariate Gaussians. Its computation can be defined as:
FID(µ1,Σ1,µ1,Σ2) = |µ1 − µ2| 2 + tr(Σ1 + Σ2 − 2(Σ1Σ2)1/2),
where µ; Σ are the first and second degree statistics of inferred low-dimensional multivariate Gaussians, i.e., z ∼ N (µ,Σ). On the basis of suggested FID metric to assess the between-cluster relation, the provided ground-truth relation among three clusters can be expressed into the three-way inequality: FID(Cluster1−Cluster3) > FID(Cluster1−Cluster2) < FID(Cluster2−Cluster3).
Juxtaposed in demonstrated Figure 2, among three instantiations of our approach, the maximum utilisation of diagnostic information, i.e., the Full model, achieved the most promising result on recovering the correct distance relation among three instantiations of our approach, along with the assessed significant pair-wist t-tests among three between-cluster distances in our Full model (see attached C).
This observation, in conjunction with the consistent result in the second simulation study (see attached D), greatly substantiates the importance of diagnostic information in deriving an optimal low-dimensional space for high-dimensional FC like features. The superiority of our approach over included alternative ones are presented less-distorted low-dimensional embedding space and recovered accurate distance matrix. This observed empirical superiority of our approach assures the further implementation on real neuropsychiatric FC datasets.

heading: 5 MINING THE NOSOLOGICAL RELATION AMONG MULTIPLE DISORDERS

text: After demonstrating the importance of diagnostic information in learning low-dimensional embeddings of FC features, we seek to implement our approach, i.e., the Full model, on curated neuropsychiatric FC datasets to explore nosological relations among disorder phenotypes. In short, high-dimensional FC features of included neuropsychiatric disorders are firstly projected into a common low-dimensional (2D) embedding space, where their pairwise distances are served as the nosological relation in this embedding space.

heading: 5.1 DATASETS & METRICS

text:
To acquire a high-quality neuropsychiatric FC dataset that contains multiple types of disorders is a time-consuming task, which demands expertise in diagnosis, and consistency in neuroimaging scanning protocols. The SRPBS multi-disorder brain connectivity database (Tanaka et al., 2021) 1 that pools the fMRI scans of diverse neuropsychiatric patients from 8 different scanning sites with consistent diagnosis guidelines, perfectly serves as our research database.
UTO dataset From the opted SRPBS database, two datasets were formed to suit our need in studying the nosological relation among multiple neuropsychiatric disorders. The primary dataset comes from the data that were acquired at the University of Tokyo Hospital (AKA., UTO dataset), which is
1The URL link for this database can be referred to https://bicr-resource.atr.jp/srpbsfc/.
comprised of fMRI scans from 3 chosen types of clearly diagnosed disorders: autism spectrum disorder (ASD; 10 subjects), major depressive disorder (MDD; 62 subjects), and schizophrenia (SCZ; 35 subjects). These fMRI scans were acquired in the same facility under the identical scanning protocol, which is free from the common threat of site difference in most fMRI studies. The brief introduction of the SRPBS database can be found in attached E.
The subject-wise brain features, i.e., functional connectivities, were obtained from 10-min restingstate fMRI BOLD signals that underwent the identical preprocessing steps. To compute the subjectwise functional connectivities from these preprocessed BOLD signals, the Brainvisa Sulci Atlas parcellation scheme, which each individual image was divided into 140 regions (Perrot et al., 2011) and the standard Pearson R correlation method were adopted for time series extraction, and the computation of connectivity matrix 2, respectively. Hence, for each subject, 9730 connectivity features were crafted to serve as high dimensional brain features, forming a [107, 9730] brain feature matrix.
HuShoWa dataset Aside from the main UTO dataset, a replication dataset from the SRPBS database were further curated from the Hiroshima University and ShoWa University (HuShoWa dataset). A total of 307 subjects were included in our HuShoWa dataset, occupying categories of MDD (173 subjects), ASD (115 subjects), and SCZ (19 subjects). Underwent the identical preprocessing steps, and the feature extraction process, we harvested a [307, 9730] FC feature matrix. To minimise the effect of site-difference on acquired fMRI scans in HuShoWa dataset, we resorted to the harmonisation technique, e.g., the combat approach (Yu et al., 2018) to correct site-difference in our input features.
2To ease the computation, the lower diagonal of connectivity matrix is reserved for connectivity features.
Validation scheme & performance assessment Since empirical neuropsychiatric FC features are seldom accompanied with ground truth low-dimensional embeddings, it is essential to obtain a strict validation scheme to ensure the reliability of our attained nosological relations. For this, we opt for 10-fold cross validation scheme that demands 10 times running of our model on two empirical datasets.
In consist with the previous harnessed FID metric in assessing the between-cluster relation, in both UTO and HuShoWa datasets, we applied the FID metric to compute the cluster-wise FID distance on the basis of yielded low-dimensional embeddings of three disorders, e.g., FID(ASD−MDD);FID(ASD−SCZ);FID(MDD−SCZ), their inequality may inform us the nosological relation among ASD, MDD, and SCZ. Over the implemented 10-fold cross validation, multiple paired t-tests in testing the statistical significance of these pair-wise FID distances were also conducted to assess the reliability of attained nosological relations.

heading: 5.2 DISCOVERED NOSOLOGICAL RELATIONSHIPS AMONG ASD, MDD AND SCZ

text:
As shown in Figure 3 (left panel), on low-dimensional embedding spaces that are derived from the UTO and HuShoWa datasets, with maximum utilisation of diagnostic information, the learned embeddings are compact, clear-cut clustered in accordance with their represented disorder phenotypes 3.
In respect of the attained highly consistent embedding spaces (Figure 3 left panel), two similar FID matrices (Figure 3 middle panel), and high statistical power in conducted paired t-tests (Figure 3 right panel) across two curated neuropsychiatric datasets, it unveils us an interestingly between-phenotype relation among the considered ASD, MDD and SCZ, i.e., FID(ASD−MDD) >
3To interested readers, in attached F, we demonstrate the empirical comparison of the derived embeddings from our approach and several competing alternatives on HuShoWa dataset.
FID(ASD−SCZ) and FID(ASD−MDD) > FID(MDD−SCZ). This suggests that ASD and MDD may belong to two independent nosology entities, which are loosely connected regarding their large differences in corresponding neural correlates (functional connectivities), whereas the closely linked ASD-SCZ, MDD-SCZ nexuses indicate the potential ’nosological closeness’ between SCZ and the other two disorder phenotypes, which is in line with the previous findings on high rates of comorbidity on SCZ-ASD (Wood, 2017) and SCZ-MDD (Tsai & Rosenheck, 2013).
Based on this finding, we are able to align these disorder phenotypes on an FC informed dimensional coordinate, allowing ASD and MDD to situate at two ends of this coordinate, in correspondence with two domains of psychopathology, i.e., the neurodevelopmental and affective pathology, respectively (see attached G) (Craddock & Owen, 2010).

heading: 5.3 NEUROSCIENTIFIC INSIGHT

text:
Furthermore, since the final projected FCs were represented as two-dimensional features, it is of increasing research interest to see whether these two-dimensional features are in agreement with existing disorder related connectivities in the brain space. For this, harnessing the layer-wise relevance propagation method (Montavon et al., 2019), we have projected each ’cluster centre’ (the most representative feature in each of three clusters) back into the high dimensional FC space, and tentatively identified a significant connection between Caudate and Cuneus that obtains the highest relevance in discriminating three disorders (MDD, SCZ, ASD) among the overall 9730 FCs. However, the validity of this preliminary neuroscientific insight on involved functional connectivity is under the threat of used nonlinear projections (multiple ReLU connections in both inference and generative networks), where the isomorphism between the low-dimensional embedding and high-dimensional FC space is not guaranteed.
Lastly, to assess the interpretability of learned low-dimensional embedding space in our context, we focus on probing the potential association between learned embeddings and measured clinical or psychological assessments. On the basis of provided ASD related autism spectrum quotient (AQ score) (Baron-Cohen et al., 2001), and MDD related Beck Depression Inventory (BDI score) (Beck et al., 1996) in the HuShoWa dataset, the associations between our derived embeddings and the clinical factors are captured, i.e., AQ score: ρ = 0.533, p < 0.001 and BDI score: ρ = 0.540, p < 0.001, suggesting the potential clinical awareness of derived low-dimensional embeddings.

heading: 6 LIMITATIONS

text: Two limitations of this study are the confined capability of our approach in discerning a novel subtype of a disorder or the nosological overlap between two disorders, and the scarcity of high-quality independent FC datasets that are curated other than the harnessed SRPBS database. The former limitation may be ameliorated through investigating the finer (possibly hierarchical) structure within low-dimensional embeddings of known phenotypes, whereas the latter one may be tackled by the future release of the high-quality imaging datasets from the Brain/Minds beyond MRI project (Koike et al., 2021).

heading: 7 CONCLUSION

text:
Unveiling the complex nosological relation among diverse neuropsychiatric disorders with the help of diagnostic information, we advocate a neural correlate based approach, which targets on deriving a neural correlate informed low-dimensional embedding space. To learn low-dimensional embeddings of neural correlates that preserves the diagnostic attributes, dual utilisation of diagnostic information that is incorporated in a novel type of conditional VAE, are proposed in this research. Relying on this method, we discovered a reliable and consistent nosological relation among ASD, MDD, and SCZ across two curated neuropsychiatric FC datasets.
Undoubtedly, the current approach – at its embryonic stage -- faces several mentioned limitations, a future approach with enhanced interpretability on the obtained low dimensional embedding space, and the inclusion of an independent validation dataset should enable us to attain more reliable nosological relations on various neuropsychiatric disorders to encourage the age-old, discrete categorical view on mental illness to make an exit at a fast pace.
ETHICS STATEMENT
Concerning the potential ethnic issues of our propose approach in serving as a clinical application, we raise four concerns on future use of our approach.
1. Undoubtably, the sole reliance on the discovered nosological relation on basis of our approach is neither optimal nor responsible in the diagnosis procedure. Given the early stage of our approach, harnessing the discovered nosological relation among disorders is still immature to replace established symptom-based diagnosis in clinical psychiatry.
2. Regarding discovered nosological relations among disorders, given the heterogeneous, complex, and developmental nature of most neuropsychiatric disorders, further validation of such relations is demanded.
3. Meanwhile, since our probed nosological relation among disorders on 2D embedding space is induced from the computed distance relation, it does not imply any potential signs of comorbidity.
4. To yield a wide range of social acceptance on using neuroimaging modality in clinical diagnosis, it demands more future validation studies that come from other aspects, e.g., the genetic aspect, and symptom relevance. Solely relying on the neuroimaging modality in clinical diagnosis can be misleading and unethical.
Unless these raised ethical concerns are addressed, along with additional validation studies and evidence, clinical use of our approach should not be considered and recommended at this moment.
REPRODUCIBILITY STATEMENT
To boost the reproducibility of our study, the executable Python code snippet that allows the implementation of our model is contained within the uploaded Zip file. Within the uploaded Zip file, we also include a code snippet that contains the helper function we used in the computation of FID scores between embeddings. The used SRPBS database can be referred to https://bicr-resource.atr.jp/srpbsfc/. The public available version of this referred database can be requested upon the site.

heading: A METHODOLOGICAL DISTINCTIVENESS & DERIVATIONS

text:
A.1 METHODOLOGICAL COMPARISON BETWEEN OUR APPROACH AND ALTERNATIVES
As summarised in following Table 1, the main difference between our approach and NVAE (Vahdat & Kautz, 2020) lies on the derived hierarchical process in defining the decoding and generative process. In NVAE, the hierarchical process can be seen in the sequential production of z, i.e., z = z1, ..., zl; q(z2|x) = q(z2|z1, x). This sequential encoding and decoding procedure allows NVAE to implement feature combinations from z1 to zl. Differ to NVAE, our hierarchical process focuses on the conditional generation of the latent variable z from both discrete variable c, and the input x, i.e., q(c, z|x) ∝ q(z|x)q(c|z). More importantly, with a defined categorical prior distribution for the discrete variable Cat(c|π), π ∈ <N+ for p(c), the N groups of z are encouraged to be mutually independent, i.e., the implicit clustering effect. Such independence is neither the objective nor the outcome of NVAE approach.
The hierarchical process in our inference model entails the conditional chain q(c|z)q(z|x)p(x), whereas in CVAE (Sohn et al., 2015), such conditional chain is replaced by the conditional density q(z|x, y). In a similar vein, as illustrated in Table 1, the major distinction of our approach to a conventional semi-supervised VAE (Kingma et al., 2014) also lies on their differentiated derivations on inference networks.
In regards to different instantiations on graphical models, our approach also differs from the Gaussian mixture VAE on both inference and generative networks. In Gaussian mixture VAE (Dilokthanakul et al., 2016), the inference network is defined as q(x,w, z|c), and its corresponded generation network is p(c, x, w, z) ∝ p(x|w, z)p(c|x). Ignoring the Gaussian mixture variable w, the foregoing networks can be seen as q(x, z|c) ∝ q(z|x)q(x|c) and p(x|z)p(c|x). In our networks, these two processes are defined as q(c|z)q(z|x) and p(x|z)p(z|c), respectively. Aside from these differences, we acknowledge the explicit modelling of different Gaussian mixtures w, which cannot be achieved by the current version of our approach.
In Section 3.1, we put forward a novel type of conditional VAE that allows implicit clustered embeddings. The core component of our loss function is pertaining to log q(z|x)p(z|c) , where we embed the implicit clustering effect. Given two conditional Gaussians to parameterise q(z|x) and p(z|c), respectively, i.e., q(z|x) ∼ N (µx, σ2x), and p(z|c) ∼ N (µc, I), the general optimisation loss term
for this component can be derived into:
log q(z|x) p(z|c) = log N (µx, σ2x) N (µc, 1) where N (µx, σ2x) = 1√
2πσ2x exp
{ −1
2 ∥∥∥∥z − µxσx ∥∥∥∥2 } , and N (µc, 1) = 1 (2π) exp { −1 2 ‖z − µc‖2 } ,
= 1
2 ‖z − µc‖2 −
1 2 log σ2x − 1 2 ∥∥∥∥z − µ(x)σ(x) ∥∥∥∥2 , as z = µx + ⊗ σx, ∼ N (0, 1)
= 1
2 ‖z − µc‖2 −
1 2 log σ2x + 1 2 || ||2.
Ignoring the last constant term 1
2 || ||2, we arrive at
= 1
2 ‖z − µc‖2 −
1 2 log σ2x.
(4)
From the following general derivation, it clearly indicates the contribution of µc in producing our targeted low dimensional embeddings z. I.e., from the regularisation viewpoint, µc can be viewed as one type of regulariser to regularise the learning of z.
A.3 THE IMPORTANCE OF λ IN DETERMINING THE SHAPE OF z
In the main text, to balance the contribution of two manners, we pre-define the λ = 0.5 to enforce the equal weighting between two manners. However, this λ value be tuned flexibly in determining the shape of each embedded cluster. In the attached Figure 4, through the demonstration on MNIST dataset, we show the effect of differentiated λ in controlling the overlapping degree of encoded clusters. I.e., λ→ 0, clusters of embeddings tend to be overlapped with each other, whereas λ→ 1, encoded clusters are pulling away from each other, in the contribution of the independent, nonoverlapping clustering effect.

heading: B TRAINING CONFIGURATIONS

text:
B.1 CONFIGURATIONS IN THE SIMULATION STUDY (SECTION 4.)
The following two subsections are presented here to demonstrate the default configurations for implementing three instantiations of our proposed approach, i.e., Full model, Manner-1 model, and Manner-2 model, and three alternative models, i.e., S-VAE model, S-Umap model, and NCA model, which were utilised in Section 3 of the main text.
B.2 CONFIGURATIONS IN THE EMPIRICAL STUDY (SECTION 5.)
The following table summarises the training configuration of our approach in exploring the nosological relations among various disorder phenotypes in two empirical datasets (Section 4 of the main text). To improve the reliability of the attained nosological relation, we adopt the nearly identical model configurations to implement our model in UTO and HuShoWa datasets, respectively. Regarding the scarce sample sizes of two datasets, the complex cross validation procedure may not be appropriate here. Therefore, in Figure 4 of the main text, the results were yielded from two full datasets.

heading: C STATISTICAL TESTS

text: To assess the statistical significance of attained embedding relation in the above-mentioned simulation study (see Main text Section 4.), we conduct a series of paired t-tests to assess the significance of pair-wise between-cluster FID across six different approaches. As shown in the following Figure 5, our proposed Full model achieved the most superior t-test results among the harnessed 6 approaches, albeit the comparison between C2-C3 and C1-C2 does not achieve the statistical significance. Note here, the observed strong statistical significance between C2-C3 and C1-C2 in NCA, i.e., C2− C3 > C1− C2, contradicts with the ground-truth, i.e., C2− C3 < C1− C2.

heading: D THE SECOND SIMULATION STUDY

text: We attach a supplementary (2nd) simulation study (see the following Figure 6) to further solidify the observed empirical superiority of our approach, i.e., the full model, in recovering low-dimensional embedding of FC-like high-dimensional features. In this attached 2nd simulation study, all set-ups are kept the same with the 1st simulation study (Section 4. main text) except for the non-linear scattering of simulated ground-truth embeddings. Inconsistent with the rendered 1st simulation study, the only model that recovers the ground-truth between-cluster relation is our proposed Full model.

heading: E SRPBS DATABASE & DATA PRE-PROCESSING STEPS

text:
The employed UTO and HuShoWa datasets are derived from the SRPBS database, which is consisted of resting-state fMRI scans from 8 different sites. The detailed curation and pre-precessing steps of SRPBS database can be consulted in the work of Tanaka et al. (2021). Here, we highlight some of the key aspects of scanning and data pre-processing. At each site, the subject-wise restingstate fMRI data were obtained from 10-min resting-state fMRI scanning session with the eye-open condition. For data curated in HuShoWa dataset, the imaging data were acquired in a Siemens Spec-
tra and a Siemens Verio scanner, whereas for the data in UTO dataset, data were acquired from a GE MR750W scanner.
All resting-state fMRI data underwent the identical pre-processing step include slice-time correlation, realignment, co-registeration, segmentation of T1-weighted structural images, normalisation to Montreal Neurological Institute (MNI) space, and spatial smoothing with an isotropic Gaussian kernel of 6 mm full-width at half-maximum.

heading: F EMPIRICAL COMPARISON OF DERIVED EMBEDDINGS FROM OUR APPROACH AND ALTERNATIVE MODELS

text: Implemented on the same HuShoWa dataset, we empirically evaluate the derived low dimensional embeddings from our approach, and several alternative ones, e.g., neighbourhood component analysis (NCA) (Goldberger et al., 2004), T-SNE (Van der Maaten & Hinton, 2008), Supervised Umap (S-Umap) (McInnes et al., 2018), and a conventional semi-supervised VAE (S-VAE) (Kingma et al., 2014).

heading: G DIMENSIONAL VIEW ON ASD, MDD, & SCZ

text: Demonstrated in the following Figure 8, in accordance with the envisioned dimensional view on diverse neuropsychiatric disorders (Craddock & Owen, 2010), we recast the attained the nosological relation among ASD, MDD, SCZ on the following continua, where ASD and MDD reside at two poles of this continua.

title: DISCOVERING THE NEURAL CORRELATE INFORMED NOSOLOGICAL RELATION AMONG MULTIPLE NEU- ROPSYCHIATRIC DISORDERS THROUGH DUAL UTILISA- TION OF DIAGNOSTIC INFORMATION


summaries: Quantum neural networks have two parts: a quantum embedding circuit that takes in classical data and embeds it into a quantum state, and a variational quantum circuit that learns a quantum circuit for evolving the quantum state before measurement.  The authors propose to encode the classical input data into the parameterized angle in the quantum embedding circuit using a tensor-train network (instead of a dense neural network). This results in a 3-4% improvement in prediction accuracy for MNIST (87.12% accuracy).

summaries: This work proposes an end-to-end learning framework TTN-VQC for quantum neural networks. The main problem is that current quantum computer cannot handle large number of qubits.  The idea of this paper is applying QTT to perform dimensional reduction followed by quantum encoding TPE, which can be combined with existing quantum neural network (VQC) as an end-to-end learning model.

summaries: As it was already studied in the literature, one way to implement a machine learning classifier in quantum computers is to convert classical input data like images into quantum states that are fed to a Variational Quantum Circuit (VQC). However, current quantum computers have a small number of qbits, which requires to perform a dimensionally reduction of the input datasets as a preprocessing step. In this paper, the authors propose to use data compression based on the Tensor Train Network (TTN) model, which is a very well-known compression technique. They compare TTN compression against a straightforward dense layer and the PCA-based dimension reduction techniques. The paper includes experimental results on MNIST dataset by simulating quantum circuits with up to 8qbits for noisy and noiseless scenarios.

summaries: Summary:  This paper is dedicated to designing an end-to-end learning framework for quantum neural networks. The authors design a novel quantum tensor network for dimension reduction and quantum embeddings generation, since it is quite related to the practical usage in real-world applications where only a small number of qubits could be supported on available NISQ computers at this moment. The key contribution of this paper is to leverage a tensor train network (TTN) to replace the dense layer for dimension reduction, which enables an end-to-end training process fully conducted in a quantum computer.

abstractText: The advent of noisy intermediate-scale quantum (NISQ) computers raises a crucial challenge to design quantum neural networks for fully quantum learning tasks. To bridge the gap, this work proposes an end-to-end learning framework named QTN-VQC, by introducing a trainable quantum tensor network (QTN) for quantum embedding on a variational quantum circuit (VQC). The architecture of QTN is composed of a parametric tensor-train network for feature extraction and a tensor product encoding for quantum embedding. We highlight the QTN for quantum embedding in terms of two perspectives: (1) we theoretically characterize QTN by analyzing its representation power of input features; (2) QTN enables an end-to-end parametric model pipeline, namely QTN-VQC, from the generation of quantum embedding to the output measurement. Our experiments on the MNIST dataset demonstrate the advantages of QTN for quantum embedding over other quantum embedding approaches.

title: Approximation and Estimation Bounds for Artificial Neural Networks

venue: Machine Learning,

authors:
Marcello Benedetti
Erika Lloyd
Stefan Sack
Mattia Fiorentini

title: Parameterized Quantum Circuits as Machine Learning Models

venue: Quantum Science and Technology,

authors:
Ville Bergholm
Josh Izaac
Maria Schuld
Christian Gogolin
M Sohaib Alam
Shahnawaz Ahmed
Juan Miguel Arrazola
Carsten Blank
Alain Delgado
Soran Jahangiri

title: Pennylane: Automatic Differentiation of Hybrid Quantum-Classical Computations

venue: arXiv preprint arXiv:1811.04968,

authors:
Samuel Yen-Chi Chen
Chih-Min Huang
Chia-Wei Hsing
Ying-Jer Kao

title: Hybrid QuantumClassical Classifier Based on Tensor Network and Variational Quantum Circuit

venue: arXiv preprint arXiv:2011.14651,

authors:
Samuel Yen-Chi Chen
Chao-Han Huck Yang
Jun Qi
Pin-Yu Chen
Xiaoli Ma
Hsi-Sheng Goan

title: Variational Quantum Circuits for Deep Reinforcement Learning

venue: IEEE Access,

title: Approximation by Superpositions of A Sigmoidal Function

venue: Mathematics of Control, Signals and Systems,

authors:
Li Deng
Jinyu Li
Jui-Ting Huang
Kaisheng Yao
Dong Yu
Frank Seide
Michael Seltzer
Geoff Zweig
Xiaodong He
Jason Williams

title: Recent Advances in Deep Learning for Speech Research at Microsoft

venue: In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing,

authors:
Yuxuan Du
Min-Hsiu Hsieh
Tongliang Liu
Dacheng Tao

title: Expressive Power of Parametrized Quantum Circuits

venue: Physical Review Research,

authors:
Yuxuan Du
Min-Hsiu Hsieh
Tongliang Liu
Dacheng Tao
Nana Liu

title: Quantum Noise Protects Quantum Classifiers Against Adversaries

title: Inside Quantum Black Boxes

venue: Nature Physics,

authors:
Vedran Dunjko
Hans J Briegel

title: Machine Learning & Artificial Intelligence in the Quantum Domain: A Review of Recent Progress

venue: Reports on Progress in Physics,

authors:
Vedran Dunjko
Jacob M Taylor
Hans J Briegel

title: Quantum-Enhanced Machine Learning

venue: Physical Review Letters,

authors:
Edward Farhi
Jeffrey Goldstone
Sam Gutmann

title: A Quantum Approximate Optimization Algorithm

venue: arXiv preprint arXiv:1411.4028,

title: Hunting for New Drugs with AI

authors:
Yangguang Fu
Mingyue Ding
Chengping Zhou

title: Phase Angle-Encoded and Quantum-Behaved Particle Swarm Optimization Applied to Three-Dimensional Route Planning for UAV

venue: IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans,

authors:
Xun Gao
Zhengyu Zhang
Luming Duan

title: An Efficient Quantum Algorithm for Generative Machine Learning

venue: arXiv preprint arXiv:1711.02038,

authors:
Timur Garipov
Dmitry Podoprikhin
Alexander Novikov
Dmitry Vetrov

title: Ultimate Tensorization: Compressing Convolutional and FC Layers Alike

venue: arXiv preprint arXiv:1611.03214,

authors:
Maxwell Henderson
Samriddhi Shakya
Shashindra Pradhan
Tristan Cook

title: Quanvolutional Neural Networks: Powering Image Recognition with Quantum Circuits

venue: Quantum Machine Intelligence,

authors:
Hsin-Yuan Huang
Michael Broughton
Masoud Mohseni
Ryan Babbush
Sergio Boixo
Hartmut Neven
Jarrod R McClean

title: Power of Data in Quantum Machine Learning

venue: Nature Communications,

authors:
Thomas Huckle
Konrad Waldherr
Thomas Schulte-Herbrüggen

title: Computations in Quantum Tensor Networks

venue: Linear Algebra and its Applications,

authors:
William Huggins
Piyush Patil
Bradley Mitchell
K Birgitta Whaley
E Miles Stoudenmire

title: Towards Quantum Machine Learning with Tensor Networks

authors:
Forrest N Iandola
Song Han
Matthew W Moskewicz
Khalid Ashraf
William J Dally
Kurt Keutzer

title: SqueezeNet: AlexNet-level Accuracy with 50x Fewer Parameters and

venue: MB Model Size. arXiv preprint arXiv:1602.07360,

authors:
Sofiene Jerbi
Casper Gyurik
Simon Marshall
Hans J Briegel
Vedran Dunjko

title: Variational Quantum Policies for Reinforcement Learning

venue: arXiv preprint arXiv:2103.05577,

authors:
Xuanyu Jin
Jiajia Tang
Xianghao Kong
Yong Peng
Jianting Cao
Qibin Zhao
Wanzeng Kong

title: CTNN: A Convolutional Tensor-Train Neural Network for Multi-Task Brainprint Recognition

venue: Proc. IEEE Transactions on Neural Systems and Rehabilitation Engineering,

authors:
John Jumper
Richard Evans
Alexander Pritzel
Tim Green
Michael Figurnov
Olaf Ronneberger
Kathryn Tunyasuvunakool
Russ Bates
Augustin Žı́dek
Anna Potapenko

title: Highly Accurate Protein Structure Prediction with AlphaFold

authors:
Abhinav Kandala
Antonio Mezzacapo
Kristan Temme
Maika Takita
Markus Brink
Jerry M Chow
Jay M Gambetta

title: Hardware-Efficient Variational Quantum Eigensolver for Small Molecules and Quantum Magnets

authors:
Iordanis Kerenidis
Jonas Landman
Anupam Prakash

title: Quantum Algorithms for Deep Convolutional Neural Networks

venue: In Proc. International Conference on Learning Representations,

authors:
Frank Leymann
Johanna Barzen

title: The Bitter Truth About Gate-Based Quantum Algorithms

venue: In the NISQ Era. Quantum Science and Technology,

authors:
Jin-Guo Liu
Lei Wang

title: Differentiable Learning of Quantum Circuit Born Machines

venue: Physical Review A,

authors:
Seth Lloyd
Maria Schuld
Aroosa Ijaz
Josh Izaac
Nathan Killoran

title: Quantum Embeddings for Machine Learning

venue: arXiv preprint arXiv:2001.03622,

authors:
Jarrod R McClean
Jonathan Romero
Ryan Babbush
Alán Aspuru-Guzik

title: The Theory of Variational Hybrid Quantum-Classical Algorithms

venue: New Journal of Physics,

authors:
Jarrod R McClean
Sergio Boixo
Vadim N Smelyanskiy
Ryan Babbush
Hartmut Neven

title: Barren Plateaus in Quantum Neural Network Training Landscapes

authors:
Kosuke Mitarai
Makoto Negoro
Masahiro Kitagawa
Keisuke Fujii

title: Quantum Circuit Learning

authors:
Valentin Murg
Frank Verstraete
Örs Legeza
Reinhard M Noack

title: Simulating Strongly Correlated Quantum Systems with Tree Tensor Networks

venue: Physical Review B,

authors:
Michael A Nielsen
Isaac Chuang

title: Quantum Computation and Quantum Information

authors:
Alexander Novikov
Dmitry Podoprikhin
Anton Osokin
Dmitry Vetrov

title: Tensorizing Neural Networks

venue: In Proc. Advances in Neural Information Processing Systems,

title: Tensor Networks for Complex Quantum Systems

venue: Nature Reviews Physics,

title: Tensor-Train Decomposition

venue: SIAM Journal on Scientific Computing,

authors:
Mateusz Ostaszewski
Lea M Trenkwalder
Wojciech Masarczyk
Eleanor Scerri
Vedran Dunjko

title: Reinforcement Learning for Optimization of Variational Quantum Circuit Architectures

venue: arXiv preprint arXiv:2103.16089,

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga

title: Pytorch: An Imperative Style, High-Performance Deep Learning Library

title: Quantum Computing in the NISQ

venue: Era and Beyond. Quantum,

authors:
Jun Qi
Hu Hu
Yannan Wang
Chao-Han Huck Yang
Sabato Marco Siniscalchi
Chin-Hui Lee

title: Exploring deep hybrid tensor-to-vector network architectures for regression based speech enhancement

venue: arXiv preprint arXiv:2007.13024,

title: Tensor-to-Vector Regression for Multi-Channel Speech Enhancement Based on Tensor-Train Network

authors:
Valeria Saggio
Beate E Asenbeck
Arne Hamann
Teodor Strömberg
Peter Schiansky
Vedran Dunjko
Nicolai Friis
Nicholas C Harris
Michael Hochberg
Dirk Englund

title: Quantum Speed-ups in Reinforcement Learning

venue: In Quantum Nanophotonic Materials, Devices, and Systems 2021,

authors:
Maria Schuld
Nathan Killoran

title: Quantum Machine Learning in Feature Hilbert Spaces

authors:
Maria Schuld
Francesco Petruccione

title: Supervised Learning with

venue: Quantum Computers,

authors:
Maria Schuld
Ilya Sinayskiy
Francesco Petruccione

title: An Introduction to Quantum Machine Learning

venue: Contemporary Physics,

authors:
Pierre Sermanet
David Eigen
Xiang Zhang
Michaël Mathieu
Rob Fergus
Yann LeCun

title: Overfeat: Integrated Recognition, Localization and Detection Using Convolutional Networks

authors:
Andrea Skolik
Sofiene Jerbi
Vedran Dunjko

title: Quantum Agents in The Gym: A Variational Quantum Algorithm for Deep Q-Learning

venue: arXiv preprint arXiv:2103.15084,

title: AI-Powered Drug Discovery Captures Pharma Interest

venue: Nature Biotechnology,

authors:
Andrei N Soklakov
Rüdiger Schack

title: Efficient State Preparation for A Register of Quantum Bits

venue: Physical review A,

authors:
Andros Tjandra
Sakriani Sakti
Satoshi Nakamura

title: Compressing Recurrent Neural Network with Tensor-Train

venue: In Proc. International Joint Conference on Neural Networks,

authors:
Hanrui Wang
Yongshan Ding
Jiaqi Gu
Yujun Lin
David Z Pan
Frederic T Chong
Song Han

title: Quantumnas: Noise-Adaptive Search for Robust Quantum Circuits

venue: arXiv preprint arXiv:2107.10845,

authors:
Chao-Han Huck Yang
Jun Qi
Samuel Yen-Chi Chen
Pin-Yu Chen
Sabato Marco Siniscalchi
Xiaoli Ma
Chin-Hui Lee

title: Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition

authors:
Yinchong Yang
Denis Krompass
Volker Tresp

title: Tensor-Train Recurrent Neural Networks for Video Classification

venue: In Proc. International Conference on Machine Learning,

authors:
R Yu
S Zheng
A Anandkumar
Y Yue

title: Long-term Forecasting Using Tensor-Train RNNs

venue: arXiv preprint arXiv:1711.00073,

heading: 1 INTRODUCTION

text:
The state-of-the-art machine learning (ML), particularly based on deep neural networks (DNN), has enabled a wide spectrum of successful applications ranging from the everyday deployment of speech recognition (Deng et al., 2013) and computer vision (Sermanet et al., 2014) through to the frontier of scientific research in synthetic biology (Jumper et al., 2021). Despite rapid theoretical and empirical progress in DNN based regression and classification (Goodfellow et al., 2016), DNN training algorithms are computationally expensive for many new scientific applications, such as new drug discovery (Smalley, 2017), which requires computational resources that are beyond the computational limits of classical hardwares (Freedman, 2019). Fortunately, the imminent advent of quantum computing devices opens up new possibilities of exploiting quantum machine learning (QML) (Biamonte et al., 2017; Schuld et al., 2015; Schuld & Petruccione, 2018; Schuld & Killoran, 2019; Saggio et al., 2021; Dunjko, 2021) to improve the computational efficiency of ML algorithms in the new scientific domains.
Although the exploitation of quantum computing devices to carry out QML is still in its initial exploratory stages, the rapid development in quantum hardware has motivated advances in quantum neural networks (QNN) to run in noisy intermediate-scale quantum (NISQ) devices (Preskill, 2018; Huggins et al., 2019; Huang et al., 2021; Kandala et al., 2017). A NISQ device means that not enough qubits could be spared for quantum error correction, and the imperfect qubits have to be directly used at the physical layer. Even though, a compromised QNN approach is proposed by employing hybrid quantum-classical models that rely on the optimization of variational quantum circuits (VQC) (Benedetti et al., 2019; Mitarai et al., 2018). The resilience of the VQC based models to certain types of quantum noise errors and high flexibility concerning coherence time and gate requirements (McClean et al., 2018) admit many practical implementations of QNN on NISQ devices (Chen et al., 2020b; Yang et al., 2021; Du et al., 2020; 2021; Skolik et al., 2021; Dunjko et al., 2016; Jerbi et al., 2021; Ostaszewski et al., 2021). One notable limitation in the current QNN training pipeline is that the quantum embedding is not fully realizable in a quantum computer, which may impede the learning of the QNN. Hence, this work proposes QTN-VQC to enable an end-toend trainable QNN, including data embedding to quantum measurements, that are easily realizable in quantum devices, where QTN stands for the quantum tensor network (Orús, 2019; Huckle et al., 2013; Biamonte et al., 2017; Murg et al., 2010) for generating quantum embedding.
As shown in Figure 1, our QNN builds a unitary linear operator that consists of three main components: (1) quantum embedding generation; (2) variational quantum circuit; (3) measurement. Quantum embedding generation, also known as quantum encoding, applies a fixed unitary linear operator Hx transforming classical vectors x to quantum states |ψx〉 in a Hilbert space. This step is an important aspect of designing quantum algorithms that directly impact the entire computation cost of VQC and owns a characteristic of quantum superposition. Moreover, the VQC comprises two types of quantum gates: (1) Controlled-NOT (CNOT) gates; (2) learnable parametric quantum gates. The CNOT gates ensure the property of quantum entanglement through mutually connecting the qubits, and the parametric quantum gates can be adjustable to best fit the quantum input states. The model parameters of VQC should be optimized by employing variants of gradient descent algorithms during the training process. Those parametric quantum gates of VQC are similar to the weights assigned to DNN, and such quantum circuits have been justified to be resilient to quantum noises (Farhi et al., 2014; Kandala et al., 2017; McClean et al., 2016). Besides, the measurement M(|gθ(x)〉) aims at projecting the quantum output states |gθ(x)〉 to one classical output zi. This work focuses on quantum embedding generation because it is quite related to the practical usage in machine learning applications in terms of computational cost and representation capability of classical input features. In particular, we design a novel quantum tensor network (QTN) for quantum embedding generation. More specifically, the QTN consists of a tensor-train network (TTN) for dimension reduction and a quantum tensor encoding framework for outputting quantum embeddings. The dimension reduction is a necessary procedure before the quantum encoding because only a small number of qubits could be supported on available NISQ computers at this moment. A typical approach for dimension reduction relies on a classical fully-connected layer, also known as a dense layer, to convert high-dimensional input vectors y into low-dimensional ones x. However, since a dense layer cannot be physically mapped on a quantum computer, much overhead has to be incurred by frequently communicating between classical and quantum devices during the end-to-end training pipeline.
As shown in Figure 2 (b), one of our contribution is to leverage a tensor train network (TTN) to replace the dense layer in Figure 2 (a). The benefits of applying TTN arise from two aspects: (1) TTN can maintain the representation power of the dense layer, which will be justified in our theorems; (2) TTN is a tensor network and can be flexibly placed in quantum computers, which enables an end-to-end training process fully conducted in a quantum computer. Moreover, in this work, a tensor product encoding (TPE) is delicately designed for generating quantum embedding, which builds the relationship between a classical vector x and the corresponding quantum state |x〉; Besides, we further investigate the representation of QTN-VQC in terms of model size and nonlinear activation function used in TTN. We denote a QTN as the combination of TTN and TPE and utilize QTN-VQC as a genuine end-to-end learning framework for QNN.

heading: 2 RELATED WORK

text:
The work (Schuld & Petruccione, 2018; Biamonte et al., 2017; Dunjko & Briegel, 2018) demonstrate that VQC shows great promise in surpassing the performance of classical ML. Prominent examples of VQC based models include quantum approximate optimization algorithm (QAOA) (Farhi et al., 2014), and quantum circuit learning (QCL) (Mitarai et al., 2018). Various architectures and geometries of VQC have been shown in tasks ranging from image classification (Henderson et al., 2020; Chen et al., 2020a; Kerenidis et al., 2020) to reinforcement learning (Chen et al., 2020b).
As for quantum embedding, basis encoding is the process of associating classical input data in the form of binary strings with the computational basis state of a quantum system (Leymann & Barzen, 2020). Similarly, amplitude encoding is a technique involving encoding data into the amplitudes of a quantum state (Soklakov & Schack, 2006). Unfortunately, the computational cost of both quantum embedding and amplitude encoding becomes exponentially expensive with the increasing number of qubits (Schuld & Killoran, 2019). A new technique of angle embedding makes use of the quantum gates to generate quantum states (Fu et al., 2011), but it cannot deal with the high-dimensional feature inputs. Therefore, this work exploits the use of TTN for dimension reduction followed by a TPE for generating quantum embedding.
In particular, this work employs the TTN for dimensionality reduction. The TTN model based on TT decomposition in neural networks was first proposed in (Oseledets, 2011), and it could be flexibly extended the convolutional neural network (CNN) (Garipov et al., 2016) and recurrent neural network (RNN) (Tjandra et al., 2017). The empirical study of TTN on machine learning tasks shows that TTN is capable of maintaining the DNN baseline results (Qi et al., 2020a; Yu et al., 2017; Yang et al., 2017; Jin et al., 2020). However, to our best knowledge, no existing works have applied TTN to QML. Besides, since the tensor network-based machine learning model like TTN is closely related to quantum machine learning in terms of their model structures (Liu & Wang, 2018; Gao et al., 2017), the QTN-VQC model can be directly regarded as the classical simulation of the corresponding quantum machine learning. In addition to a classical dense layer, more complicated architectures like AlexNet (Lloyd et al., 2020) could be used for dimension reduction, and we also compare the performance between TTN and AlexNet-based models.

heading: 3 NOTATIONS

text:
We denote RI as a I-dimensional real coordinate space, and RI1×I2×···IK refers to a space of Korder tensors. The symbol W ∈ RI1×I2×···×IK represents a K-order multi-dimensional tensor in RI1×I2×···IK , and the symbols v ∈ RI and W ∈ RI×J represent a vector and a matrix, respectively. For the notations of quantum computing, ∀ v ∈ RI , the symbol |v〉 denotes a quantum state associated with a 2I -dimensional vector in a Hilbert space. Particularly, |0〉 = [1 0]T and |1〉 = [0 1]T . The quantum gate RY (θ) means a Pauli-Y gate with a unitary operator as defined in Eq. (1), which implies a qubit rotates the Bloch sphere along the Y-axis by a given angle θ.
RY (θ) =
[ cos θ2 −i sin θ 2
−i sin θ2 cos θ 2
] (1)
Moreover, the operator ⊗ is a tensor product. Given the vectors vi ∈ RI , the tensor product of I vectors is defined as ⊗Ii=1vi, which is a 2I -dimensional vector and can provide a compact represen-
tation for v1 ⊗ v2 ⊗ · · · ⊗ vI . Similarly, the symbol |0〉⊗S means a tensor product of S quantum states of |0〉. Furthermore, for a scalar v, the quantum state |v〉 can be written as:
|v〉 = cos v|0〉+ sin v|1〉 = [ cos v sin v ] . (2)

heading: 4 QTN-VQC: OUR PROPOSED END-TO-END LEARNING FRAMEWORK

text: This section introduces our proposed end-to-end learning framework, namely QTN-VQC in this work. As shown in Figure 3, the QTN model includes two components (a) TTN and (b) TPE, which will be separately introduced in Section 4.1 and Section 4.2. Moreover, Figure 4 illustrates the framework of VQC and Section 4.3 is devoted to discussing the details of VQC.

heading: 4.1 TENSOR TRAIN NETWORK FOR DIMENSION REDUCTION

text:
We leverage TTN (Novikov et al., 2015) for the dimension reduction of input features. TTN relies on the TT decomposition (Oseledets, 2011) and has been commonly employed in machine learning tasks like speech processing (Qi et al., 2020b) and computer vision (Yang et al., 2017). The TT decomposition assumes that given a set of TT-ranks {r0, r1, ..., rK}, a K-order tensor W ∈ RI1×I2×···×IK is factorized into the multiplication of 3-order tensors Xk ∈ Rrk−1×Ik×rk . In more detail, given a set of indices {i1, i2, ..., iK}, X (i1, i2, ..., iK) is decomposed as:
X (i1, i2, ..., iK) = K∏ k=1 Xk(ik), (3)
where ∀ik ∈ [Ik], Xk(ik) ∈ Rrk−1×rk . Since r0 = r1 = 1, the term ∏K k=1 Xk(ik) is a scalar value.
TTN employs the TT decomposition in a dense layer and is explicitly demonstrated in Figure 3 (a). In more detail, for an input tensor X ∈ RI1×I2×···×IK and an output tensor Y ∈ RJ1×J2×···×JK , we achieve
Y(j1, j2, ..., jK) = I1∑ i1=1 I2∑ i2=1 · · · IK∑ iK=1 W ((i1, j1), (i2, j2), ..., (iK , jK))X (i1, i2, ..., iK)
= I1∑ i1=1 I2∑ i2=1 · · · IK∑ iK=1 ( K∏ k=1 Wk(ik, jk) ) · K∏ k=1 Xk(ik)
= K∏ k=1 ( Ik∑ ik=1 Wk(ik, jk)Xk(ik) )
= K∏ k=1 Yk(jk),
(4)
where Xk(ik) ∈ Rrk−1×rk , and Yk(jk) ∈ Rrk−1×rk which results in a scalar ∏K k=1 Yk(jk) because of the ranks r0 = r1 = 1; W((i1, j1), (i2, j2), ..., (iK , jK)) is closely associated with W(m1,m2, ...,mK) as defined in Eq. (3), if each index mk = ik× jk is set. The multi-dimensional tensorW is decomposed into the multiplication of 4-order tensorsWk ∈ Rrk−1×Ik×Jk×rk . A nonlinear activation function, e.g., Sigmoid, Tanh, and ReLU, is imposed upon the tensor Y . Compared with a dense layer with ∏K k=1 IkJk parameters, a TTN owns as few as ∑K k=1 rkrk−1IkJk trainable parameters.
When a TTN is utilized for the dimension reduction, the high-dimensional input vector x ∈ RI is first reshaped into a tensor X ∈ RI1×I2×···×IK , and then we can represent X as a TT format that goes through TTN. The outputs of TTN can be converted back to a tensor Y ∈ RJ1×J2×···×JK , which is further reshaped to a lower dimensional vector y ∈ RJ . Here, we define
∏K k=1 Ik = I and∏K
k=1 Jk = J . Moreover, the computational complexities of TTN and the related dense layer are in the same scale, which is discussed in (Yang et al., 2017).
Eq. (4) suggests that TTN is a multi-dimensional extension of a dense layer, where the trainable weight matrix of a dense layer is changed to the learnable core tensors. Additionally, many empirical studies demonstrate that a TTN is capable of maintaining the baseline results of the dense layer (Qi et al., 2020b; Yang et al., 2017; Novikov et al., 2015; Qi et al., 2020a). More significantly, since TTN can be flexibly mapped into a quantum circuit, the quantumness inherent in TTN brings great advantages over other architectures like the dense layer. In other words, although TTN is treated classically, it is possible to substitute equivalent quantum circuits for TTN when more qubits become available (Du et al., 2020), which implies that QTN-VQC stands for a genuine end-to-end QNN learning architecture on a quantum computer.
Furthermore, since the gradient exploding and diminishing problems are serious issues in the TTN training. To avoid those training problems, we only consider 3-order core tensors and small TT-ranks to configure a simple TTN in our experimental simulations. Our theoretical analysis of QTN-VQC based on Theorem 3 in Section 5 suggests that the representation power is not related to TT-ranks and the tensor order K, thus small TT-ranks and the tensor order K are preferred. In particular, a lower K can significantly reduce the computational cost and speed up the convergence rate.

heading: 4.2 TENSOR PRODUCT ENCODING

text:
In this subsection, we first introduce Theorem 1, and then we derive our TPE associated with the circuits in Figure 3 (b). Theorem 1. Given the classical vector x = [x1, x2, ..., xI ]T ∈ RI , a TPE as shown in Figure 3 (b) can result in a quantum state |x〉 with the following complete vector representation as:(
⊗Ii=1RY (2xi) ) |0〉⊗I = [ cosx1 sinx1 ] ⊗ [ cosx2 sinx2 ] ⊗ · · · ⊗ [ cosxI sinxI ] = |x〉. (5)
Proof. Since each element xi in the vector x can be written as |xi〉 = cosxi|0〉 + sinxi|1〉, the quantum state |x〉 can be written as:
|x〉 = [ cosx1 sinx1 ] ⊗ [ cosx2 sinx2 ] ⊗ · · · ⊗ [ cosxI sinxI ] . (6)
When the vector x goes through the quantum tensor network, which implies the following as: RY (2xi)|0〉 = cosxi|0〉+ sinxi|1〉 = |xi〉. (7)
The preceding equation, in turn, implies that Eq. (5).
Theorem 1 builds a connection between the vector x and the quantum state |x〉, and the resulting |x〉 is taken as the quantum embedding as the inputs to VQC. Since ⊗Ii=1RY (2xi) is a reversely unitary linear operator, there is no information loss incurred during the stage of quantum encoding. Furthermore, if the input is multiplied with a constant π2 , we obtain the following term as:(
⊗Ii=1RY (πxi) ) |0〉⊗I = [ cos(πx1) sin(πx1) ] ⊗ [ cos(πx2) sin(πx2) ] ⊗ · · · ⊗ [ cos(πxI) sin(πxI) ] , (8)
which corresponds to Figure 3 (b).
|0⟩
|0⟩
|0⟩
|0⟩
⨁· · · ·⨁⨁ ⨁
RX(α1) RY(β1) RZ(γ1)
RX(α2) RY(β2) RZ(γ2)
RX(α3) RY(β3) RZ(γ3)
RX(α4) RY(β4) RZ(γ4)
Z1
Z2
Z4
Z3
(a) Variational quantum circuit: the dashed square indicates repeated model with CNOT gates for entangling quantum states, and , , represent Pauli-X, Y, Z gates with free parameters RX(α) RY(β) RZ(γ) α, β, γ
RY(β) = cos β2 −sin β 2
sin β2 cos β 2
RX(α) = [ cos α2 −i sin α 2
−i sin α2 cos α 2
] RZ(γ) = exp(−i γ2 ) 0
0 exp(i γ2 )
Pauli-XPauli-Y Pauli-Z
= 1 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0⨁ · CNOT gate
(b) Matrix representation for the Quantum gates applied in the VQC
Figure 4: A framework of variational quantum circuit.

heading: 4.3 THE FRAMEWORK OF VARIATIONAL QUANTUM CIRCUIT

text:
The framework of VQC is shown in Figure 4 (a), where 4 qubit wires are taken into account, and the CNOT gates aim at mutually entangling the channels such that |x1〉, |x2〉, |x3〉 and |x4〉 lie in the same entanglement state. The Pauli-X, Y, Z gates RX(·), RY (·) and RZ(·) with learnable parameters (α1, β1, γ1), (α2, β2, γ2), (α3, β3, γ3), (α4, β4, γ4) are built to set up the learnable part. Being similar to the unitary operators of RY (α), RX(β) and RZ(γ), which are defined in Figure 4 (b), are separately associated with the rotations along X-axis and Z-axis by the given angles of β and γ. Besides, the quantum circuits in the dash square can be repeatedly copied to compose a deeper architecture. The outputs of VQC are connected to the measurement which projects the quantum states into a certain quantum basis that becomes a classical scalar zi.
As for the end-to-end training paradigm for QTN-VQC, the learnable parameters come from the VQC and TTN models, and they should be updated by applying the back-propagation algorithm based on the Adam optimizer. Given D qubits and H depths, there are totally 3DH trainable parameters for VQC. Consequently, there are ∑K k=1 rk−1rkIkJk + 3DH parameters for QTN-
VQC. On the other hand, the Dense-VQC model possesses more model parameters than QTN-VQC ( ∏K k=1 rk−1rkIkJk + 3DH vs. ∑K k=1 rk−1rkIkJk + 3DH).

heading: 5 CHARACTERIZING REPRESENTATION POWER OF QTN-VQC

text:
This section focuses on analyzing the representation power of QTN-VQC. As shown in Figure 5, given d qubits and a target quantum state |z〉 = ⊗Dd=1|zd〉, sinceHθ is known as a linear operator and Tx is defined as a definite mapping from input x to the unitary matrix Ux, the representation power of QTN-VQC is determined by how TTN can approximate the classical vector T −1x (H−1θ |z〉). To understand the expressiveness of TTN, we first start with the discussion on the expressive capability of Dense-VQC (a dense layer is taken for dimension reduction) and then generalize it to QTN-VQC. Based on the universal approximation theorem (Cybenko, 1989; Barron, 1994) for a feed-forward neural network, we derive the following theorem as: Theorem 2. Given a target vector T −1x (H−1θ |z〉), there exists a feed-forward neural network fdense with a dense layer connecting to D qubits, then
|| T −1x (H−1θ |z〉)− fdense(y) ||1 ≤ C√ D , (9)
where the activation function tanh(·) is imposed upon the dense layer, and C is a constant associated with the target vector T −1x (H−1θ |z〉).
Since TTN is a compact TT representation of a dense layer, by modifying Theorem 2 for TTN, we can also derive the upper bound on the approximation error as follows:
Theorem 3. Given a target vector T −1x (H−1θ |z〉), there exists a TTN, denoted as fTTN , with a TT layer connecting to D qubits, then
|| T −1x (H−1θ |z〉)− fTTN (y) ||1 ≤ K∏ k=1 C√ Dk , (10)
where ∏K k=1Dk = D, the Sigmoid activation function is imposed upon the TTN model, K denotes the multi-dimensional order, C is a constant associated with the target vector T −1x (H−1θ |z〉).
Comparing the two upper bounds, it is observed that TTN can attain an identical upper bound as the dense layer on the approximation error because ∏K k=1Dk = D. That implies that TTN can at least maintain the representation power of a dense layer. Besides, the number of qubits D is a key factor determining the upper bound on the approximation error. However, D is a small fixed number on a NISQ device, and a larger number of qubits D is expected to further improve the representation power of QTN-VQC. However, the computational costs of classical simulation may grow exponentially with the increasing number of qubits, and a small number of qubits have to be considered in practice.

heading: 6 EXPERIMENTS AND RESULTS

heading: 6.1 EXPERIMENTAL SETUPS

text:
We assess our QTN-VQC based end-to-end learning system on the standard MNIST. MNIST is a dataset for the task of 10 digit classification, where there are 50000 and 10000 28 × 28 image data assigned for training and testing, respectively. The full MNIST dataset is challenging for quantum machine learning algorithms, and many works only consider 2-digit classification on the MNIST task (Wang et al., 2021; Chen et al., 2020a). Moreover, the image data are separately reshaped into 784 dimensional input vectors. Dense-VQC and PCA-VQC are taken as our experimental baselines to compare with our QTN-VQC model. Dense-VQC denotes that a dense layer is used for dimension reduction, and PCA-VQC refers to using principal component analysis (PCA) to extract low-dimensional features before training the VQC parameters.
As for the experiments of QTN-VQC, the image data are reshaped into 3-order 7× 16× 7 tensors. We set small TT-ranks as {1, 2, 2, 1} to reduce the computational cost of TTN. the image data are represented as the TT format according to Eq. (3) before going through the TTN model. Since 8 qubits are used for the quantum encoding, the output of TTN needs to configure the tensor format as 2 × 2 × 2, which results in 8 dimensional output vectors. Besides, the model parameters of QTN-VQC are randomly initialized based on the Gaussian distribution, and the back-propagation algorithm is applied to train the models. The Sigmoid function is utilized for the hidden layers of TTN.
To be consistent with QTN-VQC, the weight of the dense layer for Dense-VQC is configured as the shape of 784×8. Although Dense-VQC is a hybrid classical-quantum model, the training process of Dense-VQC can also be set as an end-to-end pipeline and the weights of the dense layer are updated during the training stage. The Sigmoid function is used for the dense layer. On the other hand, PCA
is employed to reduce the feature dimension to 8, and the resulting low-dimensional features are further encoded into quantum states. Consequently, PCA-VQC admits the VQC parameters solely to be updated during the training stage. A standard AlexNet (Iandola et al., 2016) is employed to constitute an AlexNet-VQC to compare the performance.
Moreover, 6 VQC layers are constructed to form a deep model, and the outputs of the VQC model are connected to 10 classes with a non-trainable matrix. The back-propagation algorithm based on the Adam optimizer with a learning rate of 0.001 is employed for the model training. The loss of cross-entropy (CE) is utilized as the objective function during the training stage, and it is also taken as the metric to evaluate the model performance. We leverage the tools of Pennylane (Bergholm et al., 2018) and PyTorch (Paszke et al., 2019) to simulate the model performance. In particular, we separately simulate the model performance with noiseless quantum circuits and noisy quantum circuits corrupted by quantum noises from IBM quantum machines.

heading: 6.2 EXPERIMENTAL RESULTS OF NOISELESS QUANTUM CIRCUIT

text: Table 1 shows the final results of the models on the test dataset. QTN-VQC owns much fewer model parameters than Dense-VQC (328 vs. 6416) and attains even higher classification accuracy than Dense-VQC (91.43% vs. 88.54%) and lower loss values than Dense-VQC (0.3090 vs. 0.4132). However, PCA-VQC with 144 trainable VQC parameters attains the worst performance by all metrics, which implies that a trainable quantum embedding is of significance to boost experimental performance. Although our empirical results cannot reach the state-of-the-art classification performance of classical ML algorithms, our empirical results demonstrate the advantages of QTN-VQC over the PCA-VQC and Dense-VQC counterparts. With the development of more powerful quantum devices supporting more qubits, the representation power of QTN-VQC can be improved and better experimental results could be attained. Moreover, AlexNet-VQC achieves better results than QTN-VQC (92.81%vs.91.43%), but it involves more model parameters than QTN-VQC.

heading: 6.3 EXPERIMENTAL RESULTS OF NOISY QUANTUM CIRCUIT

text: To empirically validate the effectiveness of our proposed VQC algorithm, we proceed with the simulation of the practical experiments with noisy quantum circuits. More specifically, we follow an established noisy circuit experiment with the NISQ device suggested by (Chen et al., 2020b). One major advantage of the setups is to observe the robustness and preserve the quantum advantages of a deployed VQC with physical settings being close to quantum processing unit (QPU) experiments without an executive queuing time. As for the detailed setup, we first use an IBM Q 20-qubit machine to collect channel noise in the real scenario for a deployed VQC and upload the machine noise into our Pennylane-Qiskit simulator (denoted as Accq20. We provide a depolarizing noisy circuit simulation (denoted as Accdepo) based on a depolarizing channel attained from (Nielsen & Chuang, 2010) with a noise level of 0.1. As shown in Table 2, the quantum noise brings about the performance degradation of all models, but our proposed QTN-VQC consistently outperforms PCA-VQC and Dense-VQC in the condition of noisy quantum circuits. In particular, QTN-VQC can even outperform the AlexNet-VQC counterpart in noisy circuit conditions.

heading: 6.4 FURTHER DISCUSSIONS

text: The above experimental results show the advantages of QTN-VQC over Dense-VQC and PCAVQC in the scenarios with noiseless and noisy quantum circuits. Next, we will further discuss the representation power of QTN-VQC based on two factors: (1) the activation function used in TTN; (2) the number of qubits.

heading: 6.4.1 THE ACTIVATION FUNCTION USED IN TTN

text: Table 3 compares the results of QTN-VQC based on different activation functions. Our simulation on noiseless quantum circuits shows that the non-linear activation functions can bring more performance gain than a linear one, but the Sigmoid function attains a better performance than the Tanh and ReLU counterparts in our experiments. Our experiments also correspond to the universal approximation theory for QTN-VQC in Theorem 3.

heading: 6.4.2 THE NUMBER OF QUBITS

text: Finally, we investigate the effects of the number of qubits on the performance of QTN-VQC by increasing the qubits from 8 to 12 and 16. Accordingly, the output of TTN is configured as a tensor format of 2 × 3 × 2, and the model size is increased from 328 to 464 and 600, respectively. Our experiments show that the baseline performance of QTN-VQC can be further improved by increasing the number of qubits, which implies that more qubits are likely to possess higher accuracy.

heading: 7 CONCLUSIONS

text: This work proposes a genuine end-to-end learning framework for quantum neural networks based on QTN-VQC. QTN consists of a TTN for dimension reduction and a TPE framework for generating quantum embedding. The TTN model is a compact representation of a dense layer to classically simulate quantum machine learning algorithms. Our theorem on the representation of QTN-VQC shows that the number of qubits is inversely related to the approximation error of QTN-VQC and the non-linear activation plays an important role. Our experiments compare our proposed QTN-VQC with Res-VQC, Dense-VQC, and PCA-VQC. Our simulated results demonstrate that QTN-VQC obtains better experimental performance than Dense-VQC and PCA-VQC with both noiseless and noisy quantum circuits, and it achieves marginally worse performance than AlexNet-VQC. Besides, our results justify our theorem on the representation power of QTN-VQC.

heading: A APPENDIX

text:
The section of appendix includes the proofs for Theorem 2 and Theorem 3.
A.1 PROOF FOR THEOREM 2
Proof. Theorem 2 is derived from the modification of the universal approximation theory proposed by Barron (1994); Cybenko (1989). The universal approximation theory is shown in Lemma 1, which suggests that a feed-forward neural network with d neurons can approximate any continuous function with arbitrarily small .
Lemma 1. Given a continuous target function f̂ : RI → R, we can employ a 2-layer neural network with a non-linear activation f : RI → R, such that
|f̂ − f | ≤ 1√ D Cf̂ , (11)
where J denotes the number of neurons, and Cf̂ is a constant associated with f̂ . In particular, for r ≥ 1, Cf satisfies the following condition as:
||f̂ ||∞ + ∑
k,1≤ k(k−1)2 ≤r
||Dkf̂ ||∞ ≤ Cf̂ , (12)
where Dkf̂ = [ ∇f̂ ,∇2f̂ , ...,∇kf̂ ]T .
To associate Lemma 1 with our Theorem 2, the target function is replaced with the target vector H−1X H −1 θ |z〉, then there is a neural network with a dense layer connected to D qubits such that
||H−1X H −1 θ |z〉 − fdense(y)||1 ≤ 1√ D C, (13)
where C is related to the target vector H−1X H −1 θ |z〉.
A.2 PROOF FOR THEOREM 3
Proof. Assume that X = fTTN (y), X̂ = H−1X H −1 θ |z〉 and the TT decomposition of target vector is {X̂1, X̂2, ..., X̂K}, then we obtain
||H−1X H −1 θ |z〉 − fTTN (y)||1 = ||X̂ − X ||1 ≤ K∏ k=1 ||X̂k −Xk||1 (14)
On the other hand, we denote vec(Yk) and vec(Xk) as the vectorization of the tensors Yk and Xk, respectively. We also define ∏K k=1 Ik = I , Wk ∈ RDk×Ik×rk−1×rk as the TTN parameters, and also define Wk ∈ RIk×rk−1rkDk as the matricization of Wk. Moreover, σ refers to a non-linear activation function.
Since vec(Xk) = σ(WT vec(Yk)) that corresponds to a dense layer, we can obtain that
||X̂k −Xk||1 ≤ 1√ Dk C. (15)
In sum, we can further obtain
||H−1X H −1 θ |z〉 − fTTN (y)||1 ≤ K∏ k=1 ||X̂k −Xk||1 ≤ K∏ k=1 1√ Dk C, (16)
where ∏K k=1Dk = D.

heading: B APPENDIX

text:
This section includes additional experimental simulations. First, we assess the settings of TT-ranks, and then we compare the convergence rates of QTN-VQC and Dense-VQC in the experiments.
B.1 EXPERIMENTS ON TT-RANKS FOR QTN-VQC
Table 5 corresponds to the experiments of QTN-VQC with 8 qubits and the Sigmoid function. The empirical results suggest that the larger TT-ranks cannot result in better results than the smaller ones. The main reason is that the TT-ranks can correspond to a manifold, and there may potentially exist an optimal manifold with smaller TT-ranks that corresponds to the best performance.
B.2 A COMPARISON OF CONVERGENCE RATES
Next, we analyze the computational complexity for TTN for QTN-VQC. In more detail, given the TT-ranks {r1, r2, ..., rK}, a multi-dimensional tensorW is factorized into several K-order tensors Wk ∈ Rrk−1×Ik×Jk×rk , the computational complexity of the feed-forward process is in the scale of O(Kmaxk Ikmaxk Jk(maxk rk)K). In contrast, the computational overhead for a dense layer is in the scale of O( ∏ k Ik ∏ k Jk). It means that smaller TT-ranks can reduce the computational cost for QTN-VQC, which explains that smaller TT-ranks {1, 2, 2, 1} is configured in our experiments of QTN-VQC.
Empirically, we compare the convergence rates of different models on the test data in our experiments. In our experimental settings with the Tanh activation function and 8 qubits, the QTN-VQC model consistently attains a faster convergence rate than the Dense-VQC and PCA-VQC counterparts. Moreover, Table 6 compares the absolute running time of QTN-VQC with Dense-VQC and AlexNet-VQC. Since our experiments are conducted on the same GPUs and CPUs, the training time of all models can be comparable. Our evaluation shows that QTN-VQC is marginally slower than Dense-VQC, but it is much faster than AlexNet-VQC.

heading: C EXPERIMENTS OF LABELED FACES IN THE WILD (LFW)

text:
C.1 EXPERIMENTAL SETUPS
The LFW is a dataset for the task of unconstrained face recognition, which is composed of 13000 images with the shape of [154, 154, 3]. The shape of We randomly split all the datasets into 11000 training data, 2000 test data. 16 qubits are used for VQC, and the shape of the input tensor is set as 22× 147× 22. The other settings are kept the same as the configurations for the MNIST task.
C.2 EXPERIMENTAL RESULTS
Table 6 presents the simulation results under the noiseless quantum circuit condition, while Table 7 demonstrates the empirical results in the setting of noisy quantum circuits. The QTN-VQC outperforms the Dense-VQC counterpart (92.15 vs. 91.27), and it owns much fewer model parameters (2816 vs. 1.1× 106). Although The experimental results on the LFW dataset also highlight the advantages of QTN-VQC in terms of fewer model parameters and better empirical performance.