\section{Introduction}
Visual captioning aims to help visually impaired people to
quickly understand the image content. Existing methods focus on describing visual objects that have achieved impressive or even super-human performance~\cite{yu2019multimodal,cornia2020meshed,zhang2021consensus,ding2020stimulus,zhang2021consensus,zhang2021tell}. However,  most prominent captioning methods fail to ``read'' the associated texts in the visual scene.  Therefore, text-based image captioning (TextCap)~\cite{sidorov2020textcaps} is proposed in this study to caption models, recognize visual content, read text, understand both modalities jointly, and write a succinct caption. Such task is very practical that encourage machine to generate fine-grained captions (e.g., read the price of commodities in Fig.\ref{fig1}), which is helpful to visually disabled people to understand surroundings more accurately, but put forward new difficulties to existing captioning models meanwhile.

Intuitively, human-made images with text contain more semantic entities (e.g., objects, attributes, and texts) and complex relations (e.g., correlation and constraint) than natural images~\cite{hou2019relational,zhang2020relational,zhang2020photo}. Common single captioning models experience difficulty in representing a holistic image understanding.  These models also hinder the diversity of image descriptions, which is crucial for blind people concerning different aspects of an image (e.g., price comparison, discount, brand and position of commodities in Fig.1). In fact, we humans tend to focus on different aspects of image content by our preferences, personality and sentiment, this applies to the annotators with no exception~\cite{yang2020deconfounded}. Definitely, we could a
sk annotators for more paired data (e.g., \emph{multiview of an image} - \emph{diverse descriptions}), thereby force captioning model to generate varied captions from multiple views of an image. However, in the Textcap task as well as many vision-language applications, such large-scale annotations are not readily available, and are both time-consuming and labor-intensive to acquire. In these scenarios, unsupervised/unpaired methods~\cite{lample2017unsupervised,gu2019unpaired,caron2020unsupervised} that can learn vision to language or conversely from unpaired training data are highly desirable. Based on this insight, our goal is to break away from single and paired captioning conventions and conduct the unpaired captioning paradigm, which encourages TextCap models to generate diverse textual descriptions to comprehend images better.

However, the diverse and unpaired TextCap is challenging and can be summarized as three aspects:  1)~\textbf{Descriptive Diversity}, as shown in Fig.1, three sentences focus on different and interrelated multimodal information. It is hard to determine which fine-grained multimodal details are semantically related and ignore the unconcerned content to mine the diversity of image content from multiple views; 2)~\textbf{Relational Modeling}, how to model complex intra- and cross-relationships among the above multimodal contents, some of which even seem irrelevant intuitively. For instance there are various relational semantics in Fig.\ref{fig1}: (a) \emph{intra-relationship},
a price of ``17.8'' denotes the ``rollback'' while ``19.88'' indicates the original price, a lady is located next to the commodity;
 (b) \emph{cross-relationship}, the word “Hydroxycut” is
  the brand of a commodity and its price of “17.88” is located above a lady. Thus, appropriately model the relational semantic meanings is the premise to generate a plausible caption; 3)~\textbf{Unpaired Learning Paradigm}, there have been few attempts for unpaired captioning~\cite{gu2018unpaired,gu2019unpaired}, mainly relies on aligning/sharing latent semantic space across the image-sentence domain. Although promising,  these methods may not mature enough to apply to unpaired Textcap. The multimodal relationships among images are complex and abundant. The characteristics distribution among images and descriptions are apparently larger than general captioning.

Overall, we propose a novel method for diverse and unpaired TextCap called Multimodal relAtional Graph adversarIal inferenCe (MAGIC)  to address these challenges. As shown in Fig.1, our framework comprises the following parts: 1) \emph{Multimodal Relational Graph Encoder}, this module aims to encode an image to multiple semantic embeddings based on content diversity adaptively.  The intuition behind this is: when human beings composing different image descriptions, they prefer to focus on a few salient regions,  then query the contextually relevant content part and clarify their relationships to describe. To mimic this process, we first introduce a central object-aware pool that adaptively identifies the key objects which are supposed to reflect humans' intentions. Then construct their multimodal relational graphs via query relation-aware multimodal semantic meaning. The relational contexts among graphs are finally modeled by GCN~\cite{berg2017graph} into embeddings to represent varying levels of details of an image;  2) \emph{Sentence Auto-Encoder},  this module aims to reconstruct the sentences by encoding the pre-extracted scene graph from the sentence corpus. Encoded embedding sets can learn the language inductive bias~\cite{yang2020auto} from the sentence corpus to provide explicit signals for subsequent unpaired mapping and decode lexical words appropriately; 3) \emph{Unpaired Adversarial Caption Inference}, last but not least, semantic consistency throughout image and sentence domain is crucial to achieving unpaired learning. For this purpose, we develop a cascaded generative adversarial network(CGAN). CGAN first performs feature alignment from image to sentence to capture the cross-modal semantic consistency, then learns linguistic coherence by a language discriminator to generate more accurate and fluent captions.   To summarize, the major contributions of our paper are as follows:

\begin{itemize}
\item  A novel perspective is presented for the TextCap task to simultaneously liberate from the single and paired captioning training mode.

\item The MAGIC framework can appropriately construct and encode diverse multimodal relational graphs to observe an image from multiview and learn to generate unpaired captions via cascaded adversarial inference.

\item The superiority of MAGIC is demonstrated in terms of both accuracy and diversity using automatic metrics and human judgments.
 \end{itemize}
\vspace{-0.2cm}

\section{Related Work}

\noindent\textbf{Diverse Captioning.} The ability of understanding and reasoning different modalities  is
a longstanding and challenging goal of artificial intelligence~\cite{rennie2017self,zhang2020photo,zhang2019frame,li2021adaptive,li2020unsupervised,li2019walking,chen2021cross}. Image captioning is one of the most important task and
recent works are focus on generating multiple captions to describe an image.  \cite{johnson2016densecap} is the first to propose the dense captioning task, which detects and describes diverse regions in the image.   ~\cite{guo2019mscap} design a multi-style image captioning model that learns to map images into attractive captions of multiple styles. ~\cite{xu2021towards} develops anchor-centered graphs that select different textual contents to generate that multiple captions. However, the works mentioned above are more relevant to holistic visual understanding by paying more attention to unimodal information (e.g., objects, regions or texts). They require much paired data to learn the captioning diversity and may hardly model an image's complex multimodal relational details.

\begin{figure*}[t]
\includegraphics[width=0.9\textwidth]{2.pdf}
\centering\caption{Overview of our framework.  (a) \emph{Multimodal Relational Graph Encoder} encodes multiple multimodal relational graphs based on central object-aware pool adaptively to represent the diversity of image content;
 (b) \emph{Sentence Auto-Encoder} pre-extract the scene graph from the sentence corpus,  then reconstructing the sentences from the latent vectors that are encoded from the scene graph; (c) \emph{Unpaired Adversarial Caption Inference}, a cascaded generative adversarial network developed to reason the accurate and fluent captions with unpaired learning by aligning cross-domain features and distinguishing linguistic structure.}
\end{figure*}
\noindent\textbf{Unpaired Captioning.}
First attempt~\cite{gu2018unpaired} of unpaired image captioning   utilize pivot language corpus and align it to the target language,  connect the pivot language sentences in different domains by shared word embeddings. ~\cite{gu2019unpaired} uses scene graph as representations of the image and the sentence, and maps the scene graphs in their feature space through cycle-consistent adversarial training. The most recent work\cite{liu2021exploring} employs the semantic relationships to bridge the gap between vision and language domains for unpaired caption generation. However, due to the complex multimodal relationships in TextCap task, the discrepancy of feature distribution and semantic gap across image-sentence domain is larger than image captioning. The above methods may fail to generate accurate captions appropriately in an unpaired setting.

\section{Method}
This section describes our diverse and unpaired TextCap framework, we will describe each module of it and introduce the objectives and strategy for training.
\subsection{Problem formulation}
Before presenting our method, we first introduce some basic notions and terminologies. Given a text-based image $\mathcal{I}$,  the goal is to generate diverse natural language sentences $\mathcal{S}=\{\mathcal{S}_i|_{i=1}^{N_k}\}$ with unpaired learning, where $N_k$ is the pre-defined number of generated sentences to satisfy the user’s intention. We use the following training pipeline $C(.,\theta)$  to show how the diverse and unpaired captions are generated:
\begin{equation}
\begin{aligned}
\underbrace{C((\mathcal{S}|\mathcal{I});(\theta_{\mathcal{I}},\theta_{\mathcal{S}}))}_{\rm{TextCap\ Model}}&=  \underbrace{E_{\mathcal{I}}((\mathcal{G}^{\mathcal{I}}|\mathcal{I});\theta_{\mathcal{I}})}_{\rm{Image\ Encoder}} \\&
\!\!\!\!\!\!\!\underbrace{U_{\mathcal{A}}((\mathcal{G}^\mathcal{S}|\mathcal{G}^{\mathcal{I}});\theta_{\mathcal{I}})}_{\rm{Unpaired\ Alignment}}\underbrace{D_{\mathcal{S}}((\mathcal{S}|\mathcal{G}^\mathcal{S});\theta_{\mathcal{S}})}_{\rm{Sentence\ Decoder}}
\end{aligned}
\end{equation}
 $E_{\mathcal{I}}(\cdot; \theta)$ is image encoder aims to model $N_k$  multimodal relational graphs (MRGs) $\mathcal{G}^{\mathcal{I}}=\{\mathcal{G}^{\mathcal{I}}_i|^{N_k}_{i=1}\}$ learned from image $\mathcal{I}$, then translate $\mathcal{G}^{\mathcal{I}}$ to appropriate sentence graph $\mathcal{G}^{\mathcal{S}}$ through unpaired cross-domain alignment $U_{\mathcal{A}}(\cdot; \theta)$, the aligned results can be decoded to diverse captions $\mathcal{S}=\{\mathcal{S}_i|_{i=1}^{N_k}\}$ by sentence decoder $D_{\mathcal{S}}(\cdot; \theta)$. Notice that $E_{\mathcal{I}}(\cdot; \theta)$ and $U_{A}(\cdot; \theta)$ share parameters $\theta_{\mathcal{I}}$  to learn feature alignment across domain, $D_{\mathcal{S}}(\cdot; \theta)$ is a pretrained sentence docoder from textual corpus, $\theta_{\mathcal{S}}$ are fixed as explicit supervised signals to learn $\theta_{\mathcal{I}}$ to generate unpaired captions.

Fig.2 shows the proposed Multimodal relAtional Graph adversarIal  inferenCe (MAGIC) framework $C(.;\theta)$  consists of three main modules: Multimodal Relational Graph Encoder $E_{\mathcal{I}}(\cdot; \theta)$, Sentence Auto-Encoder $D_{\mathcal{S}}(\cdot; \theta)$  and Unpaired Caption Adversarial Inference $U_{A}(\cdot; \theta)$.  We use the TextCap task as an example to illus- trate our method.

\subsection{Multimodal Relational Graph Encoder}
This module aims to select the central objects dynamically as the central object-aware pool, then constructs their MRGs adaptively, which can represent the different fine-grained contents. Last, GCN model intra- and cross-relationships to learn MRG embeddings.

\subsubsection{Multimodal Feature Extractor.}
Given a text-based image $\mathcal{I}$, we firstly initialize it as its corresponding $N_{o}$ visual object features and $N_{t}$ text token features by Faster RCNN~\cite{ren2016faster} and Rosetta OCR~\cite{borisyuk2018rosetta}, respectively. In order to represent visual and text features at a unified level, follow~\cite{sidorov2020textcaps}, we  adopt a single-layer MLP to map the above features as $d$-dimensional vectors, the unified visual and text  embedding sets  of image $\mathcal{I}$ given by $\textbf{O}=\{\textbf{o}_i|_{i=1}^{N_o} \}$ and $\textbf{T}=\{\textbf{t}_j|_{j=1}^{N_t}\}$, respectively.

\subsubsection{Central Object-Aware Pool.}
Inspired by the human-being attention mechanism to describe an image, which usually focuses on a salient object or region, then from ``one-to-many'', queries its relevant information and ignores the unconcerned contents to compose fine-grained sentences.
We imitate this process by introducing a central object-aware pool (COP) that  is supposed to reflect the human preference and select objects that may need additional attention adaptively. In detail, given
$i^{th}$ object embedding $\textbf{o}_i$, we employ soft attention to  each text token embedding in $\textbf{T}$. Then  $\textbf{o}_i$ and attened textual latent vector $\textbf{t}_i^a$ are concatenated as the augmented object embedding $\textbf{o}_i^a=[\textbf{o}_i,\textbf{t}_i^a]$, the embedding set is given by $\textbf{O}^a=\{\textbf{o}_i^a|_{i=1}^{N_o}\}$. To decide the central objects from $\textbf{O}^a$, we adopt a MLP to predict a score for each object:
\begin{equation}
S^{c}_{object} =  {\rm softmax} (l_m^c(\textbf{O}^a))
\end{equation}
$S^{c}_{object}$ contains $N_o$ scores for each element that indicates the importance weight of each object. Then we select $N_k$ objects $\textbf{O}^c=\{\textbf{o}^c_i|_{i=1}^{N_k}\}$ with the highest scores as COP .

\subsubsection{Multimodal Relational Graph Embedding.}
In analogy to the human inference of contextual information with a given focused object, we first query its intra- and cross- relationships for each central object, then construct their corresponding MRGs $\mathcal{G}^{\mathcal{I}}=\{\mathcal{G}^{\mathcal{I}}_i|^{N_k}_{i=1}\}$ to represent the diversity of image $\mathcal{I}$. In detail, for $k^{th}$ central object $\textbf{o}^c_k$,  its MRG defined as $\mathcal{G}_k^\mathcal{I} = (\mathcal{N}^\mathcal{I}, \mathcal{E}^\mathcal{I})$. We devote $\mathcal{N}^\mathcal{I}$ as three relational types according to the contextual information:  relationship nodes $\textbf{n}^r$  attribute nodes $\textbf{n}^a$ and text nodes $\textbf{n}^t$. On the one hand, intra-relational nodes $\textbf{n}^r$ and $\textbf{n}^a$ corresponds to the visually neighbour objects has relationships, attributes for object $\textbf{o}^c_k$, respectively.
\begin{equation} \!\!\mathcal{N}^{\mathcal{I}}\{\textbf{n}^r, \textbf{n}^a\} \!\!= \!\!
\begin{cases}
\{W_r \textbf{o}_i\}_{i=1}^{N_o-1}, \!\!\!& \mbox{relationship nodes};\\
\{W_a [\textbf{o}_i,s_i]\}_{i=1}^{N_o-1},  \!\!\!& \mbox{attribute nodes}.
\end{cases}
\end{equation}
where $W_r$  $\in \mathbb{R}^{ d \times d }$ and $W_a$  $\in \mathbb{R}^{ d \times (d+4) }$ are the relational embedding matrix, we omit the bias term for simplicity. $\textbf{s}_{i}$ the positional embedding to distinguish different attribute nodes connected with the object $\textbf{o}^c_k$.

On the other hand, to capture the cross-relationship among central object $\textbf{o}^c_k$ and  text tokens,  we perform soft attention weight on  texts $\textbf{T}$ to query the semantically textual embeddings as  text nodes $\textbf{n}^t$.
\begin{equation}
\mathcal{N}^{\mathcal{I}}\{\textbf{n}^o\} =
\{a_i^t \textbf{t}_i\}_{i=1}^{N_t-sum(a_i^t > \epsilon)}, \quad \mbox{text nodes}.
\end{equation}
where $a_i^t$ is the attention score, $\epsilon$ is a pre-defined threshold. $sum()$ is counting function aims to connect  more related contextual texts for  $o^c_k$ if attention score $a_j^t$ greater than $\epsilon$.

Thus far, the MRG $\mathcal{G}_k^\mathcal{I} = (\mathcal{N}^\mathcal{I}, \mathcal{E}^\mathcal{I})$ with three relational nodes $\{\textbf{n}^r, \textbf{n}^a, \textbf{n}^t\}$ is obtained.
To model the semantic relations among the above MRG, we introduce a multi-layer GCN~\cite{marcheggiani2017encoding,guo2021semi}, which can capture
mutual relations between neighboured nodes and embed the graph structure into latent vectors.
\begin{equation}
\textbf{g}^{\mathcal{I}}_k=  \delta_{r} (W_{m_0} \textbf{o}^c_k+ \sum\limits_{\hat{\textbf{n}}_i \in \mathcal{N}^{\mathcal{I}}} W_{m_1} \hat{\textbf{n}}_i)
\end{equation}
where  $\delta_{r}$ is the ReLU activation function, $W_{m_0}$ and $W_{m_1}$ $\in \mathbb{R}^{ d \times d }$ are the transformation matrices. In this multi-layer GCN, first layer brings contexts for each node from its direct neighbor nodes. Meanwhile, stacking multiple layers enables the encoding of broad contexts in the graph. After such graph pooling, the MRG embedding $\textbf{g}^{\mathcal{I}}_k$ for object $\textbf{o}^c_k$ is obtained.  As a consequence, the content diversity of image $\mathcal{I}$ can be regarded as the collection $\textbf{g}^{\mathcal{I}}=\{\textbf{g}_k^\mathcal{I}|_{k=1}^{N_k}\}$ of $N_k$ central objects.

\subsection{Sentence Auto-Encoder}
We design a sentence auto-encoder that fully capture the language properties by self-reconstructing sentence. In particular, we extract the scene graph $\mathcal{G}^{\mathcal{S}}$ from the sentence corpus, and then decode the sentence from latent vector that is encoded from the $\mathcal{G}^{\mathcal{S}}$ (sentence $\rightarrow$ $\mathcal{G}^{\mathcal{S}}$  $\rightarrow$ latent vector $g^{\mathcal{S}}$ $\rightarrow$ reconstructed sentence).

\subsubsection{Scene Graph Encoder.}
We use the scene graph parser provided by~\cite{anderson2016spice} for sentences, where a syntactic dependency tree is built by~\cite{klein2003accurate} and then a rule-based method~\cite{schuster2015generating} is applied for transforming the tree to a scene graph.

Given a sentence $\hat{S}$ from TextCap corpus, its corresponding scene graph involved the high-level semantic concepts (e.g., objects, relationships  between two adjacent objects and attributes of objects) and textual tokens, to align subsequently image-sentence domain better, we encode sentences in a similar way in MRG encoding.
In detail, for each conceptual object $\textbf{c}_i^o$ in $\hat{S}$, we define its relational graph as $\mathcal{G}_i^{\mathcal{S}}=(\mathcal{N}^\mathcal{S}, \mathcal{E}^\mathcal{S})$, where $\mathcal{N}^s$ devote three relational type, relationship nodes $\textbf{c}^r$, attribute nodes $\textbf{c}^a$ and associated textual nodes $\textbf{c}^t$. The global semantic meaning of sentence $\hat{S}$  can be regarded as the collection of object relational graphs $\mathcal{G}^{\mathcal{S}}=\{\mathcal{G}^{\mathcal{S}}_i|^{M_o}_{i=1}\}$, where $M_{o}$ is the object number in sentence $\mathcal{S}$. Similarly, the latent embedding of sentence is learned by the multi-layer GCN which using same structure in image graph encoding but independent parameters.
\begin{equation}
g^{\mathcal{S}}=  \frac{1}{M_{o}}\delta_{r} ( \sum\limits_{i=1 }^{M_o}( W_{c0} \textbf{c}^o_i+ \sum\limits_{\hat{\textbf{c}}_i \in \mathcal{N}^{\mathcal{S}} } W_{c1} \hat{\textbf{c}}_i))
\end{equation}
where $W_{c0}$ and $W_{c1}$ $\in \mathbb{R}^{ d \times e }$,   Thus, the  scene graph embedding of sentence $\mathcal{\hat{S}}$ is given by $\textbf{g}^{\mathcal{S}}$.

\subsubsection{Sentence Decoder.}
The decoder part is essential in the way that it translates the semantic latent vector $\textbf{g}^{\mathcal{S}}$ into the textual sentence. However, most text tokens in sentence corpus only appear few times. This long-tailed word distribution specifically demonstrates the large variance in text occurring, which is challenging to model using a fixed word vocabulary. For this challenge, following M4C~\cite{sidorov2020textcaps}, we adopt different classifiers for common vocabulary and candidate texts to predict words. In detail, to capture long-range dependency in sequence modeling, a general LSTM is employed to predict the word $\hat{\textbf{y}}_{t}$ in common vocabulary based on the scene graph embedding
$\textbf{g}^{\mathcal{S}}$. Then a dynamic pointer network that makes final word prediction on the basis of text candidates $\textbf{T}^c$ and $\hat{\textbf{y}}_{t}$.
\begin{equation}
\begin{aligned}
\hat{\textbf{y}}_{t} &= l_m^w({\rm LSTM_{DEC}} (\textbf{h}_{t-1}, \textbf{g}^{\mathcal{S}})),\\
\textbf{y}^t&= {\rm argmax}([\hat{\textbf{y}}_{t}, DPN(\hat{\textbf{y}}_{t},\textbf{T}^c))]);
\end{aligned}
\end{equation}
where $l_m^w(.)$ is a linear classifier for common vocabulary, $DPN(.)$ denotes the dynamic pointer network. We devote notation $\textbf{y}_{0:T}$ refers to a sequence of predicted words $\{\textbf{y}_0, \cdots, \textbf{y}_T \}$,  the captioning loss can be computed by:

\begin{equation}\label{10}
\mathcal{L}(\mathcal{S};\theta_{\mathcal{S}}) =  -\rm{log} \cdot \sum^T_{i=0} (P(\textbf{y}_t| (\textbf{y}_{0:T-1}, \textbf{g}^{\mathcal{S}}, \textbf{T}^c)))
\end{equation}

\subsection{Unpaired Adversarial Caption Inference}
To achieve this unpaired captioning, we develop a CGAN that contains two modules, namely, cross-domain alignment and language discriminator. We first perform the feature alignment across domains and then distinguish whether the semantic structure of decoded caption is real or fake using the language discriminator. Such a cascaded mechanism can train a captioning model in an unsupervised manner and improve coherence among generated sentences.
\subsubsection{Cross-Domain Alignment.}
Given two unpaired graph embeddings sets $\textbf{G}^{\mathcal{I}}=\{\textbf{g}_i^\mathcal{I}|_{i=1}^{N_{\mathcal{I}}}\}$ and $\textbf{G}^{\mathcal{S}}=\{\textbf{g}_i^\mathcal{S}|_{i=1}^{N_{\mathcal{S}}}\}$ from image and sentence domain, respectively, where $N_{\mathcal{I}}$ and $N_{\mathcal{S}}$ are the capacity of corresponding corpus.  The goal of feature alignment is enforce  the $\textbf{G}^{\mathcal{I}}$  to be close to the $\textbf{G}^{\mathcal{S}}$  distribution. We follow ~\cite{zhu2017unpaired,gu2019unpaired} that align feature space through cycle-consistent adversarial training. Differently, considering that the huge domain discrepancy, we leverage recent advances in Optimal Transport~\cite{peyre2019computational} to encourage the cross-modal semantic coherence.  In detail, the  wasserstein distance~\cite{shen2018wasserstein} is employed as the loss value to learn the discriminator, to align domain $X$ to domain $Y$, the distribution distance $W(X, Y)$ can be defined and corresponding loss can be minimized as follows:
 \begin{equation}
\begin{aligned}
W(X, Y) &= \inf_{\gamma \sim \Pi (X, Y)} \mathbb{E}_{(\textbf{x}, \textbf{y}) \sim\gamma} [||\textbf{x}- \textbf{y}||]\\
\mathcal{L}({X \!\!\rightarrow\!\! Y};\theta)&= \mathbb{E}_{\textbf{y} \sim Y}[D_{Y}(\textbf{y})]-\mathbb{E}_{\textbf{x} \sim X}[D_{Y}(\textbf{x})]
\end{aligned}
\end{equation}
 where $\gamma \sim \Pi (X, Y)$ is the set of possible joint distributions for the combination of $\textbf{x}$ and $\textbf{y}$. $D_{Y}$ is a discriminator to distinguish the origin source of the latent vector if from  $Y$.  Such loss is more effectively maintain the semantic coherence between two domain with huge discrepant characteristics. Thus, the cycle loss of feature alignment from $\textbf{G}^{\mathcal{I}}$ to $\textbf{G}^{\mathcal{S}}$ can be defined as follows:
 \begin{equation}
 \begin{aligned}
\mathcal{L}({A};\theta_{\mathcal{I}})=&\lambda_A (\mathcal{L}(M_\mathcal{S}(\textbf{G}^{\mathcal{I}}) \!\! \rightarrow \!\! \textbf{G}^{\mathcal{S}})+\mathcal{L}(M_\mathcal{I}(\textbf{G}^{\mathcal{S}})\\& \!\! \rightarrow \!\! \textbf{G}^{\mathcal{I}})+\lambda_C\mathcal{L}(M_\mathcal{I}(\textbf{G}^{\mathcal{I}})\!\! \leftrightarrow  \!\! M_\mathcal{I}(\textbf{G}^{\mathcal{S}})))
\end{aligned}
\end{equation}
where $\lambda_A$ and $\lambda_C$ are weight hyper-parameters to control the feature alignment, $M_\mathcal{I}$ and $M_\mathcal{S}$ are mapping function to image domain and sentence domain, respectively. $\mathcal{L}(M_\mathcal{I}(\textbf{G}^{\mathcal{I}})\!\! \leftrightarrow  \!\! M_\mathcal{I}(\textbf{G}^{\mathcal{S}}))$ is the  cycle consistency loss to regularize the training. Subsequently, the aligned image-sentence embeddings $\textbf{G}^{\mathcal{\tilde{S}}}=\{\tilde{\textbf{{g}}}_i^\mathcal{S}|_{i=1}^{N_{\mathcal{I}}}\}$ and text candidates $T^c$ are feed into sentence decoder to generate captions.
\subsubsection{Language Discriminator.}

Notably, the cycle loss can minimize the distribution distance among image and sentence domains. However, the characteristic distribution of aligned image embeddings  $\textbf{G}^{\mathcal{\tilde{S}}}=\{\tilde{\textbf{{g}}}_i^\mathcal{S}|_{i=1}^{N_{\mathcal{I}}}\}$ is between image domain and sentence domain. To capture the language characteristics further, we introduce a language discriminator $D_L$ that focuses on the language structure of an individual sentence. Here, we want to ensure fluency and language accuracy that may be lacking in the cross-domain alignment. In addition to generated captions from the sentence decoder,  $D_L$ is given negative inputs with a mixture of randomly shuffled words or repeated phrases within a sentence.
\begin{equation}
\begin{aligned}
\mathcal{L}(L;\theta_{\mathcal{S}})=&  \lambda_L( \mathbb{E}_{\textbf{s} \sim S}[\rm{log}(D_{ {L}}(\textbf{s}))]- \mathbb{E}_{\textbf{s}^r \sim S^r}\\&[1-  \rm{log}(D_{ {L}}(\textbf{s}^r))])
 \end{aligned}
\end{equation}
where $\lambda_L$ is weight hyper-parameter and $D_L$ is a Bidirectional LSTM. $S^r$ is a set in which sentences with random N-grams. This language discriminator verifies not only the semantic fluency among sentences but also improves the captioning accuracy.

\begin{table*}[t]
  \centering
    \begin{tabular}{l |c| c c c c c| c c c c}
    \hline
     &\multicolumn{1}{|c|}{DIC.} &\multicolumn{5}{c|}{ Captioning Evaluation } &\multicolumn{4}{c}{ Diversity Evaluation }\\
    \hline
    Method &  & BLEU & METEOR  & ROUG  &  SPICE &   CIDEr& Div-1  & Div-2   & RE-4 $\downarrow$& SelfCIDEr \\
    \hline
     \hline
    ISM-LSTM      &   & 15.2 &17.6& 38.4 &11.8 &31.1 &25.2 &34.2& 5.3 &41.9\\
    \hline
    MAGIC (w/o CO)$^{\dagger}$      &   & 20.2 &18.1& 39.5& 12.3 &71.3 &28.8 &37.4& 4.8 &45.4\\
     MAGIC (w/o TA)$^{\dagger}$    &\checkmark & \textbf{21.7}&	\textbf{20.3}&	\textbf{42.0}&	\textbf{13.6}&	\textbf{75.9} &\textbf{28.1} &\textbf{40.4}& \textbf{3.6} &\textbf{49.1}\\
    MAGIC (w/o CG)$^{\dagger}$       &\checkmark & 20.1 &18.4& 37.1& 12.2 &65.3 &26.8 &36.2& 5.3 &46.1\\
    MAGIC (w/o LD)$^{\dagger}$                               &\checkmark & 20.8 &20.1& 41.1& 12.8 &73.9 &28.2 &38.3& 4.5 &48.8\\
    \hline
     MAGIC                                          &\checkmark & \textbf{22.2}&	\textbf{20.5}&	\textbf{42.3}&	\textbf{13.8}&	\textbf{76.6} &\textbf{29.7} &\textbf{40.9}& \textbf{3.8} &\textbf{49.6}\\

    \hline
    \end{tabular}
    \caption{Captioning and diversity results on TextCap validation set. DIC. indicates the results from diverse image captions.   $^{\dagger}$ indicates whether the model is a variant of MAGIC. Larger values indicate better performance, except for small value is better for RE-4. The top two scores on each metric are in bold. Acronym notations of each method see in comparison of methods.}
    \vspace{-0.2cm}
\end{table*}
\begin{table}[t]
  \centering
    \begin{tabular}{l |c c c c  }
    \hline
     & \multicolumn{4}{c}{ Results of differnet COP Size}\\
    \hline
    COP Capacity  &BLEU& CIDEr & Div-2 & SelfCIDEr  \\
    \hline
    \hline
    $N_k$=1      &\textbf{22.3}& \textbf{77.3} & \textbf{41.9}& \textbf{50.3} \\
   $N_k$=2      &21.8& {75.8} & \textbf{41.1}& {49.3} \\
      $N_k$=4     &21.6& 75.3 & 39.3& 47.8\\
    $N_k$=5     & 20.8& 73.4 & 38.1& 47.5\\
    \hline
    $N_k$=3    & \textbf{22.2}& \textbf{76.6} & 40.9& \textbf{49.6} \\
    \hline
    \end{tabular}
    \caption{ The captioning and diversity results of different sike $N_k$ of central object-aware pool. }
    \vspace{-0.1cm}
\end{table}
\subsection{Training Algorithm}
Algorithm 1 (in the appendix) presents the pseudocode of our MAGIC algorithm for diverse and unpaired TextCap. First, we pretrain the sentence auto-encoder by reconstructing sentences and the language discriminator as well as disrupting the order of words from the sentence corpus. The MAGIC framework then learns diverse multimodal relational graph embeddings by querying the semantic meaning of central objects. Finally, the developed CGAN aligns cross-modal embeddings from the image domain to the sentence domain and then checks the fluency and accuracy of generated captions using the language discriminator to generate unpaired and diverse captions magically.
\section{Experiments}
We first benchmark our MAGIC for unpaired and diverse captioning on the TextCaps dataset~\cite{sidorov2020textcaps}  to verify its effectiveness, and then discuss MAGIC’s property with controlled studies.
\subsection{Dataset and Setting}
\noindent$\textbf{Dataset.}$ The TextCaps dataset collected 28,408 images with  142,040 captions from Open Image V3 dataset. Each image has five captions with verified texts through the Rosetta OCR system~\cite{borisyuk2018rosetta} and human annotators.  Follows TextVQA ~\cite{singh2019towards}, this dataset splits for training (21,953), validation (3,166), and test (3,289) sets.

\noindent$\textbf{Implementation Details.}$  The number of visual objects is $N_o$ = 100 and the number of OCR tokens is $N_t$ = 50. For TextCap generation, we tokenized the texts on whitespace, and the sentences are ``cut'' at a maximum length of 20 words. We add a \emph{Unknown token} to replace the words out of the vocabulary list. The vocabulary has 6,736 words, and each word is represented by a 300-dimensional vector.

\noindent \textbf{Metrics.} We use the microsoft coco caption evaluation \footnote{https://github.com/tylin/coco-caption/}, which includes BLEU~\cite{papineni2002bleu} , METEOR~\cite{denkowski2014meteor}, ROUG\_L~\cite{rouge2004package},  CIDEr~\cite{vedantam2015cider}  and SPICE~\cite{anderson2016spice} to evaluate the captioning results.  To evaluate the captioning diversity, we use Div-1 and Div-2 scores~\cite{li2015diversity} that measure a ratio of unique N-grams (N=1,2) to the total number of words. We also report RE-4 that captures a degree of N-gram repetition (N=4) in a description and SelfCIDEr~\cite{wang2019describing} to study the semantic level diversity.

\noindent \textbf{Comparison of Methods.}  Diverse and unpaired Textcap is a fire-new task that existing captioning algorithms are not applied to compare. We simplified MAGIC as the baseline and conducted an ablation study to ensure a fair comparison and investigate contributions of individual components in MAGIC using the following versions: \textbf{ISM-LSTM} is a simplified version of MAGIC that maps image features from the image corpus to the sentence corpus using cycle GAN and then decodes the caption with LSTM; \textbf{MAGIC (w/o CO)} is similar to~\cite{yao2018exploring} and encodes the global spatial graph as the image embedding to decode a single caption; \textbf{MAGIC (w/o TA)} considers all text tokens in the MRGs without query attention to examine the impact on the diversity of captions; \textbf{MAGIC (w/o CG)} utilizes the original gan to align the image–sentence domain to investigate the impact on the accuracy of captions.; ~\textbf{MAGIC (w/o LD)} generates captions without the language discriminator to assess the importance of the cascaded adversarial mechanism.

\subsection{Experimental Results}
\noindent$\textbf{Captioning Results.}$ The overall quantitative results of our model and baseline on the validation set of the Textcap dataset are listed in Tab.~1. The trained baseline model (ISM-LSTM) achieves the minimum captioning score, thereby indicating that it fails to describe the text-based image due to missing text awareness. The excellent performance of MAGIC and its variants in comprehending multimodal content compared with the baseline verifies the effectiveness of our proposed model. Notably, the results of other variants are better than those of MAGIC (w/o CO) and MAGIC (w/o CG). Our analysis showed that MAGIC (w/o CO) encodes an image with global content with abundant multimodal details, but its ground truth captions describe the image locally. The CGAN may fail to align the cross-domain appropriately. Compared with MAGIC, MAGIC (w/o CO) with original GAN may fail to capture cross-domain semantic consistency effectively. In contrast, MAGIC with CGAN can improve the quality of unpaired captions in feature alignment and semantic structure levels.
\begin{table}[t]
  \centering
    \begin{tabular}{l |c c c c  }
    \hline
     & \multicolumn{4}{c}{ Results of different COP construction}\\
    \hline
    Central Object  &BLEU& CIDEr & Div-2 & SelfCIDEr  \\
    \hline
    \hline
   Center      & {18.7}& 72.3 & {38.3}& {44.2} \\
     Large    &  \textbf{20.1}& \textbf{75.9} & \textbf{40.2}& \textbf{49.3}\\
      Random     &  {17.1}& 67.8 & {36.9}& {46.6}\\
     \hline
  Top Socre     & \textbf{22.2}& \textbf{76.6} & \textbf{40.9}& \textbf{49.6} \\
    \hline
    \end{tabular}
    \caption{ The captioning and diversity results of different COP construction. Center, large and random means that the central object is selected by different rules.}
    \vspace{-0.2cm}
\end{table}
\begin{figure*}[t]
\includegraphics[width=1\textwidth]{6.pdf}
\centering\caption{Two qualitative cases of MAGIC from TextCaps validation set. For better visualisation, we contruct the multimodal relational graphs based on generated diverse captions. }
\vspace{-0.2cm}
\end{figure*}

\begin{figure}[t]
\includegraphics[width=0.5\textwidth]{4.pdf}
\centering\caption{Visualization of features (1000 samples) 2D space by t-SNE~\cite{van2008visualizing}. }
\vspace{-0.1cm}
\end{figure}
\noindent$\textbf{Diversity Results.}$ Tab.~1 also presents the comparison of diversity results. The top three objects are selected as the central object-aware pool in MAGIC variants to ensure a fair comparison. Notably, diverse captioning models of MAGIC outperform the single captioning baseline of ISM-LSTM by a large margin. Generated various captions effectively respect MRGs, especially in the SelfCider score, which focuses on semantic similarity. We conduct additional diversity analyses in the following section.

\subsection{In-Depth Analysis}
We further validate the several key issues of the proposed method by answering three questions as follows.

$\textbf{Q1: How does the structure of central object-aware}$\\
$\textbf{pool (COP) affect the captioning diversity?}$ On the one hand, we conduct an ablation study to illustrate the impact of different sizes of COP  on diverse and unpaired TextCap.
 The results are presented in Tab.~2. MAGIC with a pool size of $N_k$=1,2,3 presents the solid and approximate TextCap results, and MAGIC with a pool size of $N_k$=4,5 obtains competitive results. For this situation,  we found that there are objective semantic overlaps among annotated image descriptions, i.e., for an image, some ground truths of an image from the sentence corpus are similarly semantic in the sentence domain.  This similarity may cause confusion in the alignment of different multimodal relational graphs to the sentence domain in CGAN and lead to slight performance degradation. On the other hand, the quantitative results according to rule- and learning-based central object selections for OAP are reported. Tab.3 demonstrated that all rule-based methods suffer from low metric performance while the learning-based central object can accurately and diversely describe an image.

$\textbf{Q2: How much improvement of unpaired TextCap}$\\
$\textbf{quality has the MAGIC brought?}$
The MAGIC can improve the unpaired TextCap effectively by feature alignment and semantic coherence distinguishing.  First, we visualize the feature alignment to indicate that our MAGIC can capture the semantic consistency across the image–sentence domain Fig.~4.
To better understand how satisfactory are the sentences generated by CGAN, we  conducted a human study with five experienced workers to evaluate the generated descriptions from MAGIC and MAGIC (w/o CG). Workers evaluated 100 images which are randomly sampled from the validation set for each pairwise comparison.  Fig.~5 shows that employing MAGIC can describe images more diversely, accurately and fluently.

\begin{figure}[t]
\vspace{-0.1cm}
\includegraphics[width=0.5\textwidth]{5.pdf}
\centering\caption{The pie charåts each comparing the two methods in human evaluation.  }
\vspace{-0.2cm}
\end{figure}
$\textbf{Q3: How about the diversity of the visual concepts and}$\\$\textbf{texts for an image with multi-captions?}$ We additionally conduct a qualitative analysis of the descriptive diversity of MAGIC in Fig.~3. We present the multiple MRGs ($N_k$=3) for each image on the basis of generated captions. The excellent results showed that MAGIC produces more accurate and fine-grained captions than baseline. In this case(a), MAGIC models complex multimodal relationships correctly and describes an image with different fine-grained contents adaptively. As presented in third caption, for intra-relationship, MAGIC can capture the semantic relationship ``\emph{can-sit nexts to-bottle}", ``\emph{bottle-on-desk}", and ``\emph{can-on-desk}" correctly; for cross-relationship, MAGIC can model the semantic relationship  ``\emph{bottle-of-energy drink}" and ``\emph{can-of-mountain dew}" appropriately. However, the baseline ISM-LSTM describes the image with error semantic information and semantic structure.

\section{Conclusion}
We focus on diverse and unpaired text-based image caption generation. The MAGIC framework is proposed to achieve these objectives. MAGIC can adaptively build multiple multimodal relational graphs on the basis of selected central objects. A cascaded generative adversarial network is then developed to infer the unpaired caption generation in the image–sentence feature alignment level and linguistic coherence. The experimental results verified that MAGIC can effectively generate very promising and diverse captions without using any paired training data. The results of our study can provide new insights into visual captioning and guidance for future investigations on vision and language.

\section{ Acknowledgments} This work has been supported in Apart by National Key Research and Development Program of China (2018AAA0101900).

\bibliographystyle{aaai21}


\section*{Curriculum development for a holistic view of the field}

Many students begin to enter the field of quantum computing by learning Nelson and Chuang's book entitled {\it Quantum Computation and Quantum Information}\cite{Chuang2010}. This is one of the most representative books on quantum computing, together with another book by Benenti, Casati and Strini entitled {\it Principles of Quantum Computation and Information}\cite{Benenti2004}. The latter is even more accessible as it is specifically written for undergraduate students in physics, mathematics and computer sciences, without too much requirement on prior knowledge on either quantum mechanics or classical compute sciences. We recommend both books to students in our class, especially the second one, as the students who have selected our course in the past two rounds are majored in physics, mathematics, computer science, software, electronics, information security, telecommunication, artificial intelligence, automation control, mechanical engineering, finance, and biology, covering a wide range of engineering disciplines.

However, the above two books were firstly published in 2000 and 2004, respectively. This means they have almost missed the quickly development stage for quantum computing in the past two decades. The quantum computing hardware is much more advanced and scalable. The universal quantum algorithms that the two books emphasize have been developed. For instance, the HHL algorithm\cite{HHL} raised in 2009 becomes an important algorithm for solving linear equations. Furthermore, the adiabatic quantum computing and the hybrid quantum-classical variational quantum computing are now being extensively researched as an alternative to universal quantum computing, and find many quantum advantages in various appplications.

Thanks to the convenience in the Internet, a quickly growing number of new resources for learning quantum computing can be found and accessed. Among them, some are shared by researchers on a certain field, and some are provided by commercial companies, like D-Wave and IBM. There then arises a issue for the resources. They are segmented and not that systematic. The tutorials provided by commercial companies might focus on the quantum computing approach used in their own companies. For instance, D-Wave\cite{DWaveOcean} teaches quantum annealing, while IBM Qiskit\cite{Qiskit} tutorial mainly covers circuit-based quantum computing.

In developing our course, we suggest that we need to build a holistic view for quantum computing. We should not be biased on any quantum computing approach. Instead, we shall teach Grover algorithm, quantum annealing, Variational Quantum Eigensolver (VQE)\cite{VQE}, and Quantum Approximate Optimization Algorithm (QAOA)\cite{QAOA}, and show to students how to apply each of these different quantum algorithms on the same quadratic unconstrained binary optimization (QUBO) task. Through such training, students would have a better understanding of each quantum algorithm and a better vision on how to select a proper quantum algorithm when dealing with a specific computing task.

The teaching for this course lasts 12 weeks, and each week has teaching time lasting 4$\times 45$ minutes (see teaching schedule in Fig.1). We start with a brief history of computers, and some basic knowledge on computer sciences and quantum mechanics. We then introduce the hardware for universal quantum computers using different physical systems, including superconducting qubits, cold atom or ion trap, linear photonics and different spin systems. After that, we begin to teach universal quantum algorithms including the early ones and a few new algorithms using quantum Fourier transform such as HHL. We highlight three sets of tools for studying quantum computing, that is, the underlying physics and hardware, the analytic tool such as matrix or Dirac notations, and the geometric tool such as the Bloch sphere and the angles between vectors used for understanding the Grover search. Especially, linear algebra with matrix calculation is the method we mainly use, without heavy quantum mechanics or classical computation teminologies.

\begin{figure}[b]
\includegraphics[width=0.48\textwidth]{Figure2.pdf}
\caption{\textbf{Interfaces for the quantum cloud platforms used in class.} (\textbf{a}) The graphic interface for universal quantum circuits at IBM Qiskit. (\textbf{b}) The interface for a Jupyter notebook that runs a Max-Cut problem on D-Wave quantum annealer.}
\label{fig:strutturaChip}
\end{figure}

Apart from teaching universal quantum computing, we also teach students analog quantum computing and variational quantum computing. We teach continuous-time quantum walks, a useful tool for analog quantum computing, and show our experimental achievement using integrated photonic chips to make large-scale spatially two-dimensional quantum walks\cite{Tang2018}, as well as the application of quantum walks with quantum speedup advantages such as in quantum fast hitting algorithms\cite{Tang2018b}. The teaching of quantum walks have further meanings in at least three aspects. Firstly, we show that a combination of quantum walk and classical random walk to make a quantum stochastic walk\cite{Whitfield2010} would be useful for quantum simulation of the open quantum systems that widely exist in the reality. Secondly, we extend single-particle quantum walk into multiple-particle quantum walks, and show different behaviors of fermions, bosons and indistinguishable particles. With these knowledge it is much easier for students to understand Boson sampling\cite{BosonSampling}, a very popular task to show impressive quantum advantage, and a further useful variation into Gaussian Boson sampling\cite{GBS}. Thirdly, an understanding of the difference between quantum walk and classical random walk later help the students to learn how the quantum annealing differentiates from the classical simulated annealing. This is because quantum annealing\cite{QuantumAnnealing} or adiabatic quantum computing (AQC)\cite{AQC} can be viewed as an analog quantum evolution similar to quantum walk, but with a time-dependent Hamiltonian.

We teach variational quantum computing after quantum annealing and AQC. We follow that sequence not only because the variational quantum algorithms such as VQE and QAOA were raised many years after quantum annealing and AQC. Furthermore, QAOA, as an adapted adiabatic optimization algorithm that can be implemented in universal quantum circuits, is essentially inspired from AQC. Therefore, AQC sets as the prior knowledge for studying QAOA. On the other hand, variational quantum computing like VQE and QAOA inspires the concept and method of parameterized quantum circuit (PQC)\cite{PQC}, which is then widely used for quantum machine learning. Therefore, it is quite natural to proceed to the session on quantum optimization using different quantum algorithms, and the session on quantum machine learning with a quantum feature map using PQC.

Throughout the course, we raise different oral talk topics for the students (see Table I). The students' grade for the 3-credit teaching course includes three major parts, the in-class performance and homework (30\%), the oral talk (35\%) and the final-term exam (35\%). The setting for oral talks is a very helpful way to extend the curriculum to a more comprehensive one, with the contribution from the students themselves according to their interest and strength. Some of the oral topics ask students go deeper into some quantum hardware, so that students from physics department always would like to go for them, while some other oral topics for quantum machine learning methods draw lots of interests from the computer science students. In another sense, the setting of oral talks also helps to relieve the discrepancy of students' prior knowledge and hence is very necessary for courses open to multidisciplinary students. It is also worth to mention that we do not list all oral talk topics at once, but announce two or three topics in every class with the topics related to what we learn in class. This helps students gain better understanding of the meaning of the oral talk topics.

\begin{table}[b!]
\caption{\textbf{Some practical projects and required quantum techniques}}
\begin{center}
\begin{tabular}{p{4.6cm}<{\raggedright}p{3.9cm}<{\raggedright}}
\hline\noalign{\vskip 0.5mm}
		\hline\noalign{\vskip 0.14cm}
Project topic & Quantum technique \\ [0.1cm]
\hline

Protein folding  & Quantum annealing/VQE \\ [0.3cm]

Medical image classification & Quantum autoencoder\\ [0.3cm]

Computer Vision for 3D phase matching & Quantum annealing \\ [0.6cm]

Quantum frustration for spin liquid &VQE\\[0.6cm]

Reinforcement learning & Variational parameterized quantum circuit\\ [0.3cm]

Natural language processing for music & Variational parameterized quantum circuit\\ [0.1cm]
\hline\noalign{\vskip 0.5mm}
		\hline\noalign{\vskip 0.14cm}
\end{tabular}
\end{center}
\end{table}

\section*{Practical module: tailored to industry needs}
Throughout the 3-credit teaching course, we emphasize a practical skill. Each time after learning a quantum algorithm, the students are asked to set up the quantum algorithm on their own in a quantum computing platform. Initially, for simple universal quantum algorithms, we use the graphic interface at IBM Qiskit (see Fig.2a), which is of fun and brings an easy start for students who had little coding experience. Gradually, we will use a Jupyter note book to show how to set quantum algorithms with programming. The Jupyter notebook also works for teaching other quantum approaches, for instance, operating quantum annealing on D-Wave hardware (see Fig.2b). There is much work involved for our teaching assistants. They are expected to be those who know quantum computing well themselves, and are familiar with programming. The teaching assistant would prepare the Jupyter notebooks for quantum algorithms or case studies that will be used in class. They also address the questions that students raise on operating the programs, together with the teacher. As for the quantum computing cloud platform, we mainly use Amazon Braket as it has broad access to quantum annealers, universal quantum computers and classical computing resources. For the analog quantum computing part, we use the software FeynmanPAQS\cite{FeynmanPAQS} that we develop ourselves as a photonic analog quantum simulator. For the session on quantum machine learning, we use the framework DeepQuantum developed by a startup TuringQ.

Our teaching content in the teaching course also shows a practical spirit. We have specific teaching sessions on quantum computing for optimization, machine learning, finance and chemistry. They are a good review and practise of all quantum algorithms that have been taught, and prepare the students with practical knowledge for the upcoming 1-credit practical module. In quantum optimization, we show how to map various real-life optimization problems such as Max-cut, vertex cover, knapsack, graph partitioning and maximal independent set into a QUBO Hamiltonian and then solve it using quantum algorithms. In quantum machine learning, we take quantum support vector machine (QSVM)\cite{QSVM} as an example, we show the original method of using a swap gate to create an inner product of the quantum state, and the derivation process is a good practise of deriving quantum states. We then show that the quantum feature map made from parameterized quantum circuits can also be used to form a QSVM\cite{QSVM2}. We explain main aspects that need to be considered for quantum machine learning, including data encoding, quantum feature maps and the parameter shift rule\cite{Crooks2019} as a preferred method for gradient descent in quantum circuits. For finance and chemistry, we show that these fields develop with computer sciences in history and now quantum approaches can help in various way. For instance, derivative pricing may need quantum amplitude estimation algorithm, portfolio optimization may resort to quantum annealing, and stock market prediction may involve some quantum machine learning. Similarly, molecular energy calculation, molecular docking and protein folding can also be solved using VQE, Gaussian Boson sampling or quantum annealing.

Then we come to the 1-credit practical module during Week 13 and 16 after the 3-credit teaching course. The students are asked to form teams of three and solve a real problem using any quantum approach. This is quite similar to the mathematical modeling competition which many students have participated in before, so that they generally get used to this form very well. The topics are flexible and can be suggested by the students themselves. See Table II for a list of projects that the students have done for the practical module.

There are some common requirements for all teams. Firstly, real input data is required. They can be real finance market portfolio, real protein structure, real medical image database (breast cancer, skin disease, et al), and so on. Students may try to reproduce the published work that has the real stock portfolio or protein data; They may also find data in Yahoo finance, protein bank, Github AI resources, et al. There is no strict lower bound for the data size. Secondly, at least one quantum approach shall be demonstrated. It will be even better if students compare quantum and classical approaches, or compare two or more quantum approaches (e.g. quantum annealing vs. QAOA, etc). They do not necessarily show quantum advantage over classical method, and of course it will be better and even publishable if they do show the advantage. Thirdly, the report should be well written in Latex and figures well presented.

\begin{figure*}[ht!]
\includegraphics[width=0.85\textwidth]{Figure3.pdf}
\caption{\textbf{The setup for Jiao Tong Global Virtual Classroom.} (\textbf{a}) The three 85-inch screens on the back of the classroom. (\textbf{b}) The 75-inch screen in the front of the classroom. (\textbf{c}) The setting for the Zoom meeting by the teaching assistant.  }
\label{fig:Concept}
\end{figure*}

We give a careful thought on how to organize the practical classes during the four weeks. Students in one team will sit together for easier discussion. The class will include three main parts. Part One is the student report of their progress. For instance, in Week 13 they report on the topic and a basic plan for their project. In Week 14 one team member reports on how they find the data and encode the data in quantum approaches. In Week 15 another team member reports on how they set their quantum model, and in Week 16 the third team member reports on some initial results and how they plan to write their report. Part Two is some helpful expert tips given by researchers or engineers in the real fields or industries like quantum computing for ETF trading, drug design or computer vision. The knowledge goes deeper and more focused than what is taught in the teaching course. The Part Three are free Q\&A session, where the teacher, teaching assistants,and some guest expert speaker will help students sort out any questions they encounter in the project.

\section*{Jiao Tong Global Virtual Classroom}

The Jiao Tong Global Virtual Classroom (GVC) Initiative\cite{GVC} encourages Shanghai Jiao Tong University (SJTU) faculty to establish collaborative online courses with partner universities across the globe, or share undergraduate/postgraduate-level courses to provide international students the opportunities to enhance cross-cultural competence, and deepen their understanding of the subject.

Our course is selected into the Jiao Tong GVC Initiative. Since the autumn semester of 2021, it is open to global students in Association of Pacific Rim Universities at the same time with the offline students. Global students that have selected this course are undergraduate students in several top universities such as Hong Kong University, University College London and INSA Lyon, majored in several engineering subjects.

This course now becomes an online-offline blended teaching. The mode of blended teaching was proposed a few decades ago and now an online and offline blended learning mode has become prevalent\cite{Sharma2010, Graham2006}. Especially, since the outbreak of coronavirus disease of 2019 (COVID-19), the demands for online learning soars and the online meeting software gets more and more convenient. For this course, we use Zoom meeting for online students to get into the class. We also use Zoom for weekly Q\&A session after class.

In order to create a good virtual classroom for online students and ensure good interactions between online and offline students, the classroom has been updated specifically. The main difference is the additional use of four large screens and two HD webcams. As shown in Fig.3a and 3b, we have three 85-inch screens on the back of the classroom, and one 75-inch screen on the front-left side of the classroom.

The three screens 85-inch screens are prepared for teachers. The left one will show the courseware controlled by teacher, which will also be shown at the front side of the classroom by a projector. The middle screen will show all the students. The right screen will show the same sight as the middle one but when teacher interacts with students online, this screen will show the close-up view of them (See Fig.3a). The 75-inch screen in the front is set for the offline students. It is set as the same with the right screen on the back, so that students in classroom can interact with those online face to face.

While the screens help teachers and students in the classroom interact with the offline students, the two HD webcams are set for online students. One webcam captures the view of the front, including the teacher desk, projection of teaching slides and the blackboard. Especially, it can track the teacher so that when teacher writes on the blackboard it will give the writing a close-up view automatically and everyone can see the writing clearly. The other webcam shows a panoramic view of the class with every offline student shown in the scene. In addition, the audio equipment in the classroom is well set to ensure online students can hear the voice in class clearly.

There is extra work and training for the teaching assistant (TA) than in usual classes. The TA needs to help set and control the Zoom account during the class. We need three Zoom accounts for the class. The first one is for teachers to show their courseware and the webcam view of the teacher. The second is connected to the HD webcam at the front of the classroom for the panoramic view of the class. These two accounts are set before class and don’t need to be managed during the class. The last one is controlled by the TA. When teacher asks student A to answer a question online or student A online put his/her hands up, TA will set A’s windows as ‘focus for everyone’ to give a close-up view of A for people in the classroom. If teacher wants student B to answer the next part of the question, TA will select ‘replace focus video’ to give a close-up view of B (See Fig.3c). If teacher wants both student A and B to have a discussion, TA will select ‘add focus video’ to give a close-up view of both A and B. After interaction, TA will reset all the settings. In the meeting, TA is the host and teacher would be co-host if needed.

In our class for quantum information technologies, such settings for the classroom hardware and Zoom meeting ensures effective interaction between each pair of parties among the three, the teacher, the offline students and the online students. Meanwhile, all people, including teachers, TAs, online and offline students, have joined one discussion group on Wechat. During and after class, the students actively share their progress on quantum algorithms, and TA shares the link to some practical Jupyter notebooks in the Wechat discussion group. This works better than Canvas as the former provides an easier access on each's mobilephone.

\section*{Discussion}

We manage to launch the course {\it Quantum Information Technologies and a Practical Module} owing to many aspects. First of all, the background of the quick development of quantum technologies and strong demand on relevant education gives the fundamental motivation. Besides that, the Zhiyuan Honors Program in the university encourages setting new innovative multidisciplinary course modules, which gives the opportunity to launch this course. Furthermore, the strong engineering research atmosphere in the university and the advice from Academic Affairs Office help to strengthen the style of the course with emphasis on practical skills and a practical module for engineering application.

The development for an updated, holistic and systematic curriculum on quantum sciences and technologies is very necessary and urgent. We are writing up our teaching content on quantum computing into a tutorial book. It will show links between different quantum approaches, and sections on quantum computing for optimization, machine learning, finance, chemistry and biology. The book is with a hope to inspire readers from different fields to use quantum computing more widely, which is consistent with the purpose of the course.

Furthermore, the involvement of more technical support is helpful for courses on quantum sciences and technologies. We have shown the use of quantum cloud platforms and the setup for Jiao Tong Global Virtual Classroom that ensure a good effect for international, multidisciplinary and practical studies. In the future, when various quantum hardware get more mature and scalable, they may more commonly appear as a teaching tool in class, and that will give a further considerable boost for the teaching on quantum sciences and technologies.

In all, we show our efforts in curriculum development, practical module setting and blended learning for our new course, and hope this can be a useful case study for state-of-the-art education on quantum sciences and technologies.

\section*{Acknowledgements}
The authors thank Zhiyuan College at Shanghai Jiao Tong University for supporting the establishment of this course, the Academic Affairs Office at   Shanghai Jiao Tong University for helpful advices, and the International Affairs Division at Shanghai Jiao Tong University for launching the Jiao Tong Global Virtual Classroom Initiative. This course is supported by a funding on course development from Zhiyuan College (WF620160207/001/008) and a funding from Jiao Tong Global Virtual Classroom Initiative (WF610561701/003/006).

\section*{Author Contributions}
H.T. and X.M.J. are teachers for the course reported in this letter. T.Y.W. and R.S. are the teaching assistants for the course. T.Y.W. takes more responsibilities on monitoring the Zoom account and operations for the virtual classroom, while R.S. is more responsible for preparing for the practical Jupyter notebooks and address programming problems. H.T. writes this letter with input from all authors.