\section{Introduction}

Image inpainting \cite{bertalmio2000image} aims at reconstructing damaged regions or removing unwanted regions of images while improving the visual aesthetics of the inpainted images, which has been widely used in low-level vision tasks, such as restoring corrupted photos and object removal. The main challenge of image inpainting is how to generate reasonable structures and realistic textures. Traditional image inpainting, such as patch-based methods \cite{efros2001image, barnes2009patchmatch}, fill out the hole with the most similar patch as the to-be-inpainted patch by searching on undamaged region. Since the similarity is computed on pixel domain of the image, these methods fail to generate objects with strong semantic information.

Deep learning-based methods have shown remarkable performance on image inpainting tasks \cite{pathak2016context, nazeri2019edgeconnect, ren2019structureflow, xiong2019foreground}. They can generate semantically consistent results by understanding high-level features of the images. Among these methods, the encoder-decoder architecture has been widely developed since this architecture can extract semantic features and generate visually pleasing contents even if the hole is large. The variants of encoder-decoder architecture like U-Net  \cite{yan2018shift} are also developed for image inpainting, which can enhance the feature connection between the encoder and the decoder. For irregularly corrupted images, Liu \textit{et al.} \cite{liu2018image} and  Yu \textit{et al.} \cite{yu2019free} proposed to force the network to exploit valid pixels only, obtaining considerable performance. Nevertheless, the above methods do not take full use of the image structure features so that their generative networks are difficult to generate the reasonable structures. This results in blur and artifacts around the hole boundaries.

\begin{figure}[t!]
\begin{center}
\includegraphics[width=1\linewidth]{sample_ours1.pdf}
\end{center}
\vspace{-3mm}
\caption{Visual results of the proposed method. From left to right: (a) Input images with holes. (b) Inpainted images without SDFF. (c) Inpainted images without BPFA. (d) Inpainted images with SDFF and BPFA. (e) Ground-truth images. We can observe that the inpainted images by our method have visually pleasing content.}
\label{fig:sample_frames}
\end{figure}

To overcome the above problems, several methods were proposed to use structural knowledge. For example, Nazeri \textit{et al.} \cite{nazeri2019edgeconnect} designed EdgeConnect with a two-stage model: edge generator and image completion generator. The edge generator is trained to predict the full edge map, while the image completion generator exploits the edge map as the structure prior to reconstruct the final image. Similar to EdgeConnect, Ren \textit{et al.} \cite{ren2019structureflow} designed  StructureFlow by using the edge-preserved smoothing technique \cite{xu2012structure}, making the structure reconstruction contain more information, such as image color. However, these methods do not simultaneously employ the structure and texture features, thereby leading to the inconsistent structures and textures of the output images. Based on this observation, Liu \textit{et al.} \cite{liu2020rethinking} designed MED to learn structures and textures separately by a mutual encoder-decoder, in which the deep-layer features are learned as structures, meanwhile the shallow-layer features are learned as textures. Although the consistency between structures and textures is improved, there have still been two major shortcomings: 1) The relationship between the structures and textures is not fully considered, resulting in limited consistency between them. 2) The context of an image is not sufficiently utilized, leading to  insufficient connection from local feature to overall consistency.

Motivated by these two shortcomings, we propose a multi-feature co-learning method for image inpainting. Specifically, we present two novel modules: 1) A Soft-gating Dual Feature Fusion (SDFF) module is designed to reorganize structure and texture features so that their consistency is greatly enhanced.  2) A Bilateral Propagation Feature Aggregation (BPFA) module is designed to capture the connection between contextual attention, channel-wise information, and feature space. This BPFA greatly enhances the connection from local feature to overall consistency. The major contributions of the proposed method are as follows:
\begin{itemize}
  \item We propose a novel SDFF module. With SDFF, the blur and artifacts around the holes are significantly reduced.
  \item We propose a novel BPFA module. With BPFA, the inpainted images show more rational structures and more detailed textures.
  \item Extensive experimental results demonstrate the superiority of the proposed method over the state-of-the-art.
\end{itemize}

\section{Related Works}
Traditional image inpainting methods are usually divided into two main categories: diffusion-based methods \cite{efros2001image, bertalmio2000image} and patch-based methods \cite{hays2007scene, darabi2012image}.  The first one propagates the appearance of adjacent content to fill out the missing regions. However, due to the limitation of search mechanism on adjacent content, there are obvious artifacts within images when facing large area masks. The second one fills in the missing region with the most similar patch as the to-be-inpainted patch. Although they can capture long-distance information, it is difficult to generate semantically reasonable images due to the lack of high-level structure understanding.

Deep learning-based methods \cite{yu2018generative, lahiri2020prior, zeng2020high, liu2021pd, peng2021generating, liao2021image} have been widely explored in the field of image inpainting. Pathak \emph{et al} \cite{pathak2016context} firstly developed encoder-decoder architecture and adversarial training for image inpainting. Iizuka \emph{et al} \cite{iizuka2017globally} overcame the information bottleneck defect by introducing a series of dilated convolution layers. Recently, Nazeri \emph{et al} \cite{nazeri2019edgeconnect} proposed EdgeConnect to generate possible edges and fill in the holes with precondition information. Like EdgeConnect, Xiong \emph{et al} \cite{xiong2019foreground} designed a similar model by adopting a contour generator as structure prior instead of the edge generator. Ren \emph{et al} \cite{ren2019structureflow} utilized the edge-preserving smoothing method to obtain sharp edges and low-frequency structures. Yang \emph{et al} \cite{yang2020learning} proposed a multi-task learning framework by introducing structure embedding to generate refined structures. Liu \emph{et al} \cite{liu2020rethinking} designed a mutual encoder-decoder network to learn the structure and texture features separately. Peng \emph{et al} \cite{peng2021generating} presented the conditional autoregressive network and structure attention module, which can learn the distribution of structure features and capture the distance relationship between structures, respectively. However, the above-reviewed methods do not fully consider the relationship between the structure and texture features, it is difficult to generate the images with reasonable structures and sophisticated textures.

\section{Proposed method}

\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1\linewidth]{backbone1.pdf}
\end{center}
\vspace{-6mm}
\caption{Description of the proposed pipeline. \textbf{Generator}: We propose a variant of U-Net architecture to jointing learn image structures and textures. The Soft-gating Dual Feature Fusion (SDFF) module and Bilateral Propagation Feature Aggregation (BPFA) module are designed to refine the generated structures and textures. \textbf{Discriminator}: We adopt the local and global discriminators to ensure from local to global content consistency.}
\label{fig:system}
\end{figure*}

The overall pipeline of the proposed method is shown in Fig. \ref{fig:system}, which is built upon the generative adversarial network. The generative network consists of mutual encoder-decoder, structure and texture branches, Soft-gating Dual Feature Fusion (SDFF), and Bilateral Propagation Feature Aggregation (BPFA). The discriminative network consists of global discriminator and local discriminator. By convention, the generative network aims to generate the inpainted images by co-learning image structures and textures, while the discriminative network aims to distinguish between the inpainted images and real ones. In the following, we describe the proposed network architecture and loss functions in detail.

\subsection{Generator} \label{tube_extraction}
The generator can be divided into the five parts: 1) The encoder consists of six convolutional layers. The three shallow-layer features are reorganized as texture features to represent image details. Meanwhile, the three deep-layer features are reorganized as structure features to represent image semantics. 2) We adopt two branches to separately learn the structure and texture features. 3) We design a SDFF module to fuse the structure and texture features generated by the above two branches. 4) We design a BPFA module to equalize the features between contextual attention, channel-wise information, and feature space. 5) The skip connection is used to supplement decoder features, which helps synthesize structure and texture branches to produce more sophisticated images.

\textbf{Structure and Texture Branches.}
The texture feature reorganized by shallow-layer convolution is denoted as $\boldsymbol{F}_{te}$ and the structure feature reorgnized by deep-layer convolution is denoted as $\boldsymbol{F}_{st}$. In each branch, three parallel streams are used to fill out the corrupted regions at multiple scales. For each stream, we replace all the vanilla convolutions with padding based partial convolution in order to better fill in irregular holes. Note that each stream consists of five convolutional layers, and the convolutional kernel sizes of the three streams are $3\times 3$, $5\times 5$ and $7\times 7$, respectively. We can obtain the filled features by first combining the output feature maps of the three streams and then mapping the combined features into the same size of the input feature. Here, we denote $\boldsymbol{F}_{cst}$ and $\boldsymbol{F}_{cte}$ as the outputs of the structure and texture branches, respectively. To ensure that the two branches focus on structures and textures respectively, we use two reconstruction losses, denoted as $\mathcal{L}_{\textrm{rst}}$ and $\mathcal{L}_{\textrm{rte}}$ respectively. The pixel-wise loss is defined as:
\begin{equation} \label{con:structureBranch}
\footnotesize{
 \mathcal{L}_{\textrm{rst}}=\left\|g\left(\boldsymbol{F}_{cst}\right)-\boldsymbol{I}_{st}\right\|_1,
}
\end{equation}
\begin{equation} \label{con:textureBranch}
\footnotesize{
 \mathcal{L}_{\textrm{rte}}=\left\|g\left(\boldsymbol{F}_{cte}\right)-\boldsymbol{I}_{gt}\right\|_1,
}
\end{equation}
where $g(\cdot)$ is the convolution operation with the kernel size of 1, which aims to map $\boldsymbol{F}_{cst}$ and $\boldsymbol{F}_{cte}$ to two color images respectively. $\boldsymbol{I}_{gt}$ and $\boldsymbol{I}_{st}$ denote the ground-truth image and its structure image, respectively. We follow \cite{ren2019structureflow} by using an edge-preserving smoothing method \cite{xu2012structure} to generate $\boldsymbol{I}_{st}$.

\begin{figure}
	\centering
	\includegraphics[width=1\linewidth, trim={0.5cm, 0.5cm, 0.5cm, 0.5cm}]{SDFF.pdf}
	\vspace{-4mm}
	\caption{Description of Soft-gating Dual Feature Fusion, which can effectively fuse the structure and texture features.}
	\label{fig:SDFF module}
\end{figure}

\textbf{Soft-gating Dual Feature Fusion (SDFF).}
This module is designed to better exchange the structure features $\boldsymbol{F}_{cst}$ and texture features $\boldsymbol{F}_{cte}$ generated by the above two branches, respectively. The exchange is implemented by utilizing a soft gating way to dynamically adjust the fusion ratio between the structure and texture features. Fig. \ref{fig:SDFF module} illustrates the proposed SDFF module. Specifically, in order to construct the structure-guided texture features, we utilize the soft gating $\boldsymbol{G}_{te}$ to control a degree of refining the texture information. The soft gating can be defined as:
\begin{equation}
\footnotesize{
 \boldsymbol{G}_{te}=\sigma\left(SE\left(h\left(\left[\boldsymbol{F}_{cst}, \boldsymbol{F}_{cte}\right]\right)\right)\right),
}
\end{equation}
where $h(\cdot)$ is a convolution layer with the kernel size of 3, $SE(\cdot)$ is a squeeze and excitation operation \cite{hu2018squeeze} to capture important channel information, and $\sigma(\cdot)$ is the Sigmoid activation function. With soft gating $\boldsymbol{G}_{te}$, we can dynamically merge $\boldsymbol{F}_{cst}$ into $\boldsymbol{F}_{cte}$ by
\begin{equation}
\footnotesize{
 \boldsymbol{F^{\prime}}_{cte}=\alpha\left(\beta\left(\boldsymbol{G}_{te}\odot\boldsymbol{F}_{cte}\right)\odot\boldsymbol{F}_{cst}\right)\oplus\boldsymbol{F}_{cte},
}
\end{equation}
where $\alpha$ and $\beta$ are two learnable parameters, $\odot$ and $\oplus$ denote element-wise multiplication and addition respectively.

Similarly, the texture-guided structure features $\boldsymbol{F^{\prime}}_{cst}$ can be calculated as follows:

\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1\linewidth]{BPFA1.pdf}
\end{center}
\vspace{-4mm}
\caption{Description of the Bilateral Propagation Feature Aggregation (BPFA) module. It can enhance the connection from local feature to overall consistency.}
\label{fig:BPFA module}
\end{figure*}

\begin{equation}
\footnotesize{
 \boldsymbol{G}_{st}=\sigma\left(SE\left(k\left(\left[\boldsymbol{F}_{cst}, \boldsymbol{F}_{cte}\right]\right)\right)\right),
}
\end{equation}
\begin{equation}
\footnotesize{
 \boldsymbol{F^{\prime}}_{cst}=\gamma\left(\boldsymbol{G}_{st}\odot\boldsymbol{F}_{cte}\right)\oplus\boldsymbol{F}_{cst},
}
\end{equation}
where $k(\cdot)$ is a convolution layer with the kernel size of 3 and $\gamma$ is a learnable parameter. Thus, we can concatenate $\boldsymbol{F^{\prime}}_{cte}$ and $\boldsymbol{F^{\prime}}_{cst}$, and use a convolution layer $v$ with the kernel size of 1 to generate the integrated feature map $\boldsymbol{F}_{fu}$:
\begin{equation}
\footnotesize{
 \boldsymbol{F}_{fu}=v\left(\left[\boldsymbol{F^{\prime}}_{cst}, \boldsymbol{F^{\prime}}_{cte}\right]\right).
}
\end{equation}

\textbf{Bilateral Propagation Feature Aggregation (BPFA).}
This module is designed to co-learn contextual attention, channel-wise information, and feature space so as to enhance the overall consistency. Fig. \ref{fig:BPFA module} illustrates the proposed BPFA module in detail. Specifically, to capture channel-wise information, we use the Selective Kernel Convolution module of SKNet \cite{li2019selective} to generate the feature map $\boldsymbol{F^{\prime}}_{fu}$. Inspired by \cite{yu2018generative}, we introduce the Contextual Attention (CA) module to capture the correlation between feature patches. For a given input feature $\boldsymbol{F^{\prime}}_{fu}$, we divide it into non-overlapping patches with size 3$\times$3 and calculate the cosine similarity between these patches as:
\begin{equation}
\footnotesize{
 \boldsymbol{S}^{i,j}_{contextual}=\left \langle \frac{\boldsymbol{p}_{i}}{\left\|\boldsymbol{p}_{i}\right\|_{2}}, \frac{\boldsymbol{p}_{j}}{\left\|\boldsymbol{p}_{j}\right\|_{2}} \right \rangle,
}
\end{equation}
where $\boldsymbol{p}_{i}$ and $\boldsymbol{p}_{j}$ are the $i$-th and $j$-th patches of the input feature $\boldsymbol{F^{\prime}}_{fu}$, respectively. We utilize the Softmax function to get the attention score of each pair of patches:
\begin{equation}
\footnotesize{
 \boldsymbol{\hat{S}}^{i,j}_{contextual}=\mathrm{exp}\left(\frac{\boldsymbol{S}^{i,j}_{contextual}}{\sum_{j=1}^N\mathrm{exp}\left(\boldsymbol{S}^{i,j}_{contextual}\right)}\right),
}
\end{equation}
where $N$ is the total number of patches of the input feature $\boldsymbol{F^{\prime}}_{fu}$. Next, the attention score is used to compute  the feature patches $\widetilde{\boldsymbol{p}}_{i}, i=1,...,N$ by
\begin{equation}
\footnotesize{
 \widetilde{\boldsymbol{p}}_{i}=\sum_{j=1}^N\boldsymbol{p}_{j} \cdot \boldsymbol{\hat{S}}^{i,j}_{contextual}
}.
\end{equation}
The reconstructed feature map $\widetilde{\boldsymbol{F}}_{fu}$ can be obtained by directly reorganize all the feature patches.

In the range and spatial domains, we introduce the Bilateral Propagation Activation (BPA) module to generate the feature maps based on the range and spatial distances. The feature map of the range domain can be calculated as:

\begin{equation}
\footnotesize{
 \mathbf{y}^{r}_{i}=\frac{1}{C(x)}\sum_{j\in{v}}f\left(\mathbf{x}_{i},\mathbf{x}_{j}\right)\mathbf{x}_{j}, \label{con:range-domain}
}
\end{equation}
\begin{equation}
\footnotesize{
 f\left(\mathbf{x}_{i},\mathbf{x}_{j}\right)=(\mathbf{x}_{i})^{T}(\mathbf{x}_{j}), \label{con:dot-product}
}
\end{equation}
where we use the unfold function of PyTorch to reshape $\widetilde{\boldsymbol{F}}_{fu}$ to two kinds of vectors, which are of $HW \times 3 \times 3 \times C$ and $HW \times 1 \times 1 \times C$ dimensions. Here, $\mathbf{x}_{i}$ is the $i$-th output channel of the input feature $\widetilde{\boldsymbol{F}}_{fu}$, $\mathbf{x}_{j}$ is a neighboring channel at position $j$ around position $i$. The pairwise function $f(\cdot)$ is dot-product similarity. The operations from the unfold function to $P_{1}$ represent Eq. (\ref{con:dot-product}). $v$ is a neighboring region of position $i$ and its size is set to $3\times3$. For a given $i$, $\frac{1}{C(x)}f\left(\mathbf{x}_{i},\mathbf{x}_{j}\right)$ can be seen as the Softmax computation along dimension $j$. $C(x)$ is the number of channels in $\widetilde{\boldsymbol{F}}_{fu}$. Similarly, Eq. (\ref{con:range-domain}) represent the operations from on $\widetilde{\boldsymbol{F}}_{fu}$ until the position $P_{2}$. The feature map of the spatial domain can be calculated as:

\begin{equation}
\footnotesize{
 \mathbf{y}^{s}_{i}=\frac{1}{C(x)}\sum_{j\in{s}}g_{\alpha_{s}}\left(\left\|j-i\right\|_{1}\right)\mathbf{x}_{j}, \label{con:spatial-domain}
}
\end{equation}
where we use the unfold function of PyTorch to reshape $\widetilde{\boldsymbol{F}}_{fu}$ to a vector, which is of $HW \times C \times H \times W$ dimension. $g_{\alpha_{s}}$ is a Gaussian function to adjust the spatial contributions from neighboring region. We explore $j$ in a neighboring region $s$ for global propagation. In the experiment, $s$ is set to the same size as the input feature. Eq. \ref{con:spatial-domain} represent the operations from the unfold function to $P_{3}$. Therefore, we can obtain the feature maps $\mathbf{y}^{s}_{i}$ and $\mathbf{y}^{r}_{i}$ by the spatial and range similarity measurement methods, respectively. We can see that the bilateral propagation considers both local consistency via $\mathbf{y}^{s}_{i}$ and global consistency via $\mathbf{y}^{r}_{i}$. Each feature channel can be computed by
\begin{equation}
\footnotesize{
 \mathbf{y}_{i}=q\left(\left[\mathbf{y}^{s}_{i},\mathbf{y}^{r}_{i}\right]\right),}
\end{equation}
where $q$ denotes a convolution layer with the kernel size of 1. Next, the reconstructed feature map $\widetilde{\boldsymbol{F}}_{sr}$ is obtained by aggregating all feature channels $\mathbf{y}_{i}$ ($i=1,...,C(x)$). Finally, $\boldsymbol{F^{\prime}}_{fu}$ and $\widetilde{\boldsymbol{F}}_{sr}$ are concatenated and mapped to $\boldsymbol{F}_{sc}$ by
\begin{equation}
\footnotesize{
 \boldsymbol{F}_{sc}=z\left(\left[\boldsymbol{F^{\prime}}_{fu},\widetilde{\boldsymbol{F}}_{sr}\right]\right),
}
\end{equation}
where $z$ is a convolution layer with the kernel size of 1.

\def0.325\linewidth{0.124\linewidth}
\begin{figure*}[!t]
\renewcommand{\tabcolsep}{0.5pt}
\centering
\begin{tabular}{cccccccc}
\includegraphics[width=0.325\linewidth]{damaged_Places2.pdf}&
\includegraphics[width=0.325\linewidth]{GC_Places2.pdf}&
\includegraphics[width=0.325\linewidth]{EC_Places2.pdf}&
\includegraphics[width=0.325\linewidth]{MED_Places2.pdf}&
\includegraphics[width=0.325\linewidth]{RFR_Places2.pdf}&
\includegraphics[width=0.325\linewidth]{DSI_Places2.pdf}&
\includegraphics[width=0.325\linewidth]{Ours_Places2.pdf}&
\includegraphics[width=0.325\linewidth]{GT_Places2.pdf}\\
\includegraphics[width=0.325\linewidth]{damaged_PSV.pdf}&
\includegraphics[width=0.325\linewidth]{GC_PSV.pdf}&
\includegraphics[width=0.325\linewidth]{EC_PSV.pdf}&
\includegraphics[width=0.325\linewidth]{MED_PSV.pdf}&
\includegraphics[width=0.325\linewidth]{RFR_PSV.pdf}&
\includegraphics[width=0.325\linewidth]{DSI_PSV.pdf}&
\includegraphics[width=0.325\linewidth]{Ours_PSV.pdf}&
\includegraphics[width=0.325\linewidth]{GT_PSV.pdf}\\
(a) Input &(b) GC~\cite{yu2019free}&(c) EC~\cite{nazeri2019edgeconnect}&(d) MED~\cite{liu2020rethinking}&(e) RFR~\cite{li2020recurrent}&(f) DSI~\cite{peng2021generating}&(g) Ours&(h) GT
\end{tabular}
\vspace{-2mm}
\caption{Visual comparison on the Places2 (the first row) and  Paris StreetView (the second row) datasets.}
\label{imgirre}
\end{figure*}

\def0.325\linewidth{0.124\linewidth}
\begin{figure*}[!t]
\renewcommand{\tabcolsep}{0.5pt}
\centering
\begin{tabular}{ccccccc}
\includegraphics[width=0.325\linewidth]{celeba_Damaged.pdf}&
\includegraphics[width=0.325\linewidth]{celeba_CE.pdf}&
\includegraphics[width=0.325\linewidth]{celeba_CA.pdf}&
\includegraphics[width=0.325\linewidth]{celeba_SH.pdf}&
\includegraphics[width=0.325\linewidth]{celeba_MED.pdf}&
\includegraphics[width=0.325\linewidth]{celeba_Ours.pdf}&
\includegraphics[width=0.325\linewidth]{celeba_GT.pdf}\\
(a) Input &(b) CE \cite{pathak2016context} &(c) CA~\cite{yu2018generative}&(d) SH~\cite{yan2018shift}&(e) MED~\cite{liu2020rethinking}&(f) Ours&(g) GT
\end{tabular}
\vspace{-2mm}
\caption{Visual comparison on the CelebA dataset.}
\label{imgcent}
\end{figure*}

\subsection{Discriminator} The discriminator consists of the global critic network and local critic network, which can ensure the local and global content consistency. Each critic network includes six convolution layers with the kernel size of 4 and stride of 2, and use the Leaky ReLu with the slope of 0.2 for all but the last layer. Furthermore, the spectral normalization is adopted in our network to achieve stable training.

\subsection{Loss Function}
Our network is trained with a series of loss functions, including pixel reconstruction loss, perceptual loss, style loss, and adversarial loss so that the finally generated image looks more visually realistic.

\textbf{Pixel Reconstruction Loss.} We adopt the $l_1$ distance as the reconstruction loss from two aspects. The first aspect is to supervise the structure and texture branches. The corresponding loss functions are formulated in Eqs. (\ref{con:structureBranch}) and (\ref{con:textureBranch}), respectively. The second aspect is to measure the similarity between the final output result $\boldsymbol{I}_{out}$ and the ground-truth image $\boldsymbol{I}_{gt}$ by
\begin{equation}
\footnotesize{
 \mathcal{L}_{\textrm{rec}}=\left\|\boldsymbol{I}_{out}-\boldsymbol{I}_{gt}\right\|_1.
}
\end{equation}

\textbf{Perceptual Loss.} We utilize the perceptual loss $\mathcal{L}_{\textrm{perc}}$ to capture the high-level semantics \cite{johnson2016perceptual} by computing the $l_1$ distance between the feature spaces of $\boldsymbol{I}_{out}$ and $\boldsymbol{I}_{gt}$  through ImageNet-pretrained VGG-16 backbone, which can be written as
\begin{equation}
\footnotesize{
 \mathcal{L}_{\textrm{perc}}=\mathbb{E}\left[\sum_{i}\left\|\phi_{i}\left(\boldsymbol{I}_{out}\right)-\phi_{i}\left(\boldsymbol{I}_{gt}\right)\right\|_1\right],
}
\end{equation}
where $\phi_{i}(\cdot), i=1,...,5$ denote the five activation maps from VGG-16, which are ReLu1\_1, ReLu2\_1, ReLu3\_1, ReLu4\_1 and ReLu5\_1.

\textbf{Style Loss.} We introduce the style loss $\mathcal{L}_{\textrm{style}}$ to mitigate style differences, which is defined as:
\begin{equation}
\footnotesize{
 \mathcal{L}_{\textrm{style}}=\mathbb{E}\left[\sum_{i}\left\|\varphi_{i}\left(\boldsymbol{I}_{out}\right)-\varphi_{i}\left(\boldsymbol{I}_{gt}\right)\right\|_1\right],
}
\end{equation}
where $\varphi_{i}(\cdot) = \phi_{i}^\mathrm{T}\phi_{i}$ denotes the Gram matrix constructed from the above-mentioned five activation maps.

\textbf{Adversarial Loss.} The relativistic average least squares adversarial loss $\mathcal{L}_{\textrm{adv}}$ is to ensure the local and global content consistency. For the generator, the adversarial loss is defined as:
\begin{equation}
\footnotesize{
 \mathcal{L}_{\textrm{adv}}=-\mathbb{E}_{\mathbf{x}_{r}}\left[\textrm{log}\left(1-D_{ra}\left(\mathbf{x}_{r},\mathbf{x}_{f}\right)\right)\right]-\mathbb{E}_{\mathbf{x}_{f}}\left[\textrm{log}\left(D_{ra}\left(\mathbf{x}_{f},\mathbf{x}_{r}\right)\right)\right],
}
\end{equation}
where $D_{ra}\left(\mathbf{x}_{r},\mathbf{x}_{f}\right) = \textrm{Sigmoid}\left(D_{gl}\left(\mathbf{x}_{r}\right)-\mathbb{E}_{\mathbf{x}_{f}}\left[D_{gl}\left(\mathbf{x}_{f}\right)\right]\right)$, $D_{gl}$ denotes the local or global discriminator without the last Sigmoid function, and $\left(\mathbf{x}_{r}, \mathbf{x}_{f}\right)$ denotes a pair of the ground-truth and output images.

\textbf{Total Loss.} The total loss of the proposed method can be obtained by
\begin{equation}
\footnotesize{
 \mathcal{L}_{\textrm{total}}=\lambda_{r}\mathcal{L}_{\textrm{rec}}+\lambda_{p}\mathcal{L}_{\textrm{prec}}+\lambda_{s}\mathcal{L}_{\textrm{style}}+\lambda_{adv}\mathcal{L}_{\textrm{adv}}+\lambda_{te}\mathcal{L}_{\textrm{rte}}+\lambda_{st}\mathcal{L}_{\textrm{rst}},
}
\end{equation}
where $\lambda_{r}$, $\lambda_{p}$, $\lambda_{s}$, $\lambda_{adv}$, $\lambda_{te}$ and $\lambda_{st}$ are the tradeoff parameters, and we empirically set $\lambda_{r} = 1$, $\lambda_{p} = 0.2$, $\lambda_{s} = 250$, $\lambda_{adv} = 0.2$, $\lambda_{te} = 1$ and $\lambda_{st} = 1$.

\section{Experimental Results}
\vspace{-1mm}
In our experiment, three public datasets are used for the verification, including Places2 \cite{zhou2017places},  Paris StreetView \cite{doersch2012makes}, and CelebA \cite{liu2015deep}. We follow the training, testing, and validation splits of these three datasets themselves. The irregular masks are taken from PConv \cite{liu2018image} and classified based on the ratios of the hole to the entire image with an increment of 10\%. Our network is built on the PyTorch framework, trained on a single NVIDIA 2080 Ti GPU (11GB), and optimized by the Adam optimizer with a learning rate of $2 \times 10^{-4}$. The CelebA, Paris StreetView, and Places2 models require around 10, 55, and 100 epochs, respectively. All the masks and images are resized to $256 \times 256$.

\begin{table*}[t!]
\setlength{\abovecaptionskip}{0mm}
\renewcommand\arraystretch{1.2}
\footnotesize
\caption{Performance comparison on the Places2 dataset.}
\vspace{-2mm}
\begin{center}
\setlength{\tabcolsep}{1mm}{
\scalebox{0.94}{
\begin{tabular}{|c|cccc|cccc|cccc|cccc|}
\hline
  Metrics &\multicolumn{4}{c|}{$\textrm{PSNR}^\mathrm{\uparrow}$}&\multicolumn{4}{c|}{$\textrm{SSIM}^\mathrm{\uparrow}$}&\multicolumn{4}{c|}{$\textrm{MAE}^\mathrm{\downarrow}$}&\multicolumn{4}{c|}{$\textrm{FID}^\mathrm{\downarrow}$}
  \\
  \hline
  Mask Ratio & 10-20\% & 20-30\% & 30-40\% & 40-50\% & 10-20\% & 20-30\% & 30-40\% & 40-50\% & 10-20\% & 20-30\% & 30-40\% & 40-50\% & 10-20\% & 20-30\% & 30-40\% & 40-50\%  \\
  \hline
  GC \cite{yu2019free} &29.16&26.35&24.24&22.35&0.928&0.880&0.791&0.704&0.0116&0.0195&0.0283&0.0391&28.31&39.01&49.81&62.73\\
  \hline
  EC \cite{nazeri2019edgeconnect} &30.44&27.46&25.69&24.02&0.939&0.886&0.830&0.759&0.0109&0.0186&0.0259&0.0350&18.41&31.00&42.03&54.85\\
  \hline
  MED \cite{liu2020rethinking} &30.88&27.54&25.40&23.55&0.945&0.890&0.828&0.750&0.0101&0.0180&0.0262&0.0365&20.18&36.68&50.14&65.72\\
  \hline
  RFR \cite{li2020recurrent} &31.04&27.77&25.77&24.09&0.945&0.893&0.836&0.768&0.0091&0.0166&0.0241&0.0330&16.22&29.32&41.05&\bf{53.22}\\
  \hline
  DSI \cite{peng2021generating} &31.02&27.59&25.45&23.61&0.945&0.890&0.829&0.752&0.0100&0.0179&0.0260&0.0359&\bf{15.79}&\bf{28.94}&\bf{40.87}&54.73 \\
  \hline
  Ours  &\bf{32.24}&\bf{28.68}&\bf{26.23}&\bf{24.32}&\bf{0.957}&\bf{0.912}&\bf{0.854} &\bf{0.783} &\bf{0.0086} &\bf{0.0153} &\bf{0.0233} &\bf{0.0326} &16.13&29.14&42.23 &58.01\\
  \hline
\end{tabular}
}
}
\end{center}
\label{tab:Places2_comparison}
\end{table*}

\begin{table*}[t!]
\setlength{\abovecaptionskip}{0mm}
\renewcommand\arraystretch{1.2}
\footnotesize
\caption{Performance comparison on the Paris StreetView dataset.}
\vspace{-2mm}
\begin{center}
\setlength{\tabcolsep}{1mm}{
\scalebox{0.94}{
\begin{tabular}{|c|cccc|cccc|cccc|cccc|}
\hline
  Metrics &\multicolumn{4}{c|}{$\textrm{PSNR}^\mathrm{\uparrow}$}&\multicolumn{4}{c|}{$\textrm{SSIM}^\mathrm{\uparrow}$}&\multicolumn{4}{c|}{$\textrm{MAE}^\mathrm{\downarrow}$}&\multicolumn{4}{c|}{$\textrm{FID}^\mathrm{\downarrow}$}
  \\
  \hline
  Mask Ratio & 10-20\% & 20-30\% & 30-40\% & 40-50\% & 10-20\% & 20-30\% & 30-40\% & 40-50\% & 10-20\% & 20-30\% & 30-40\% & 40-50\% & 10-20\% & 20-30\% & 30-40\% & 40-50\%  \\
  \hline
  GC \cite{yu2019free} &30.77&27.71&25.58&23.81&0.943&0.897&0.839&0.782&0.0158&0.0234&0.0313&0.0409&28.44&39.91&52.72&69.26\\
  \hline
  EC \cite{nazeri2019edgeconnect} &31.08&28.08&25.95&24.30&0.949&0.906&0.848&0.789&0.0110&0.0194&0.0283&0.0390&22.22&38.69&55.85&72.43\\
  \hline
  MED \cite{liu2020rethinking} &32.28&28.75&26.08&24.32&0.966&0.925&0.877&0.813&0.0105&0.0190&0.0282&0.0387&20.29&30.79&48.91&67.42\\
  \hline
  RFR \cite{li2020recurrent} &31.63&28.61&26.63&24.89&0.957&0.920&0.872&0.825&0.0152&0.0220&0.0296&0.0388&18.07&30.81&\bf{42.61}&\bf{51.80}\\
  \hline
  DSI \cite{peng2021generating} &31.42 &28.21 &26.04 &24.07 &0.953 &0.912 &0.866 &0.799 &0.0121 &0.0197 &0.0285 &0.0393 &18.09 &33.36 &47.80 &60.15 \\
  \hline
  Ours  &\bf{33.68}&\bf{30.29}&\bf{27.34}&\bf{25.28}&\bf{0.971}&\bf{0.940}&\bf{0.889} &\bf{0.828} &\bf{0.0090} &\bf{0.0152} &\bf{0.0240} &\bf{0.0336} &\bf{13.83} &\bf{27.32}&45.44 &56.07\\
  \hline
\end{tabular}
}
}
\end{center}
\label{tab:ParisStreetView_comparison}
\end{table*}

\begin{table}[t!]
\setlength{\abovecaptionskip}{0mm}
\renewcommand\arraystretch{1.2}
\footnotesize
\caption{Performance comparison on the CelebA dataset.}
\vspace{-2mm}
\begin{center}
\scalebox{0.94}{
\begin{tabular}{|c|c|c|c|c|}
  \hline
  Metrics &$\textrm{PSNR}^\mathrm{\uparrow}$ &$\textrm{SSIM}^\mathrm{\uparrow}$ &$\textrm{MAE}^\mathrm{\downarrow}$ &$\textrm{FID}^\mathrm{\downarrow}$\\
  \hline
  CE \cite{pathak2016context} &25.12 &0.899 &0.0352 &3.84\\
  \hline
  CA \cite{yu2018generative} &24.54 &0.887 &0.0286 &5.01 \\
  \hline
  SH \cite{yan2018shift} &26.03 &0.901 &0.0245 &1.71 \\
  \hline
  MED \cite{liu2020rethinking} &25.92 &0.914 &0.0241 &1.59 \\
  \hline
  Ours  &\bf{26.38}&\bf{0.922}&\bf{0.0223}&\bf{1.40}\\
  \hline
\end{tabular}
}
\end{center}
\label{tab:CelebA_comparison}
\end{table}

\subsection{Performance comparison with state-of-the-art}
We compare the proposed method with eight state-of-the-art methods, including GC \cite{yu2019free}, EC \cite{nazeri2019edgeconnect}, MED \cite{liu2020rethinking}, RFR \cite{li2020recurrent}, DSI \cite{peng2021generating}, CE \cite{pathak2016context}, CA \cite{yu2018generative} and SH \cite{yan2018shift}. For the evaluation metrics, we use four common metrics: PSNR, SSIM, MAE and FID (Fr\'{e}chet Inception Distance) \cite{heusel2017gans}. For fair comparison, we use the same experimental setup for all the compared methods. The experiments are conducted on two types of damaged images containing center hole and irregular hole. In the following, we present the experimental results and analysis in detail.

For images with irregular holes, we take the Places2 \cite{zhou2017places} and Paris StreetView \cite{doersch2012makes} datasets for evaluation, and select several representative methods GC \cite{yu2019free}, EC \cite{nazeri2019edgeconnect}, MED \cite{liu2020rethinking}, RFR \cite{li2020recurrent} and DSI \cite{peng2021generating} for performance comparison. Meanwhile, we use the same validation images as the MED method. Experimental results are shown in Tables \ref{tab:Places2_comparison} and \ref{tab:ParisStreetView_comparison}. We can see from the two tables that our method achieves significant improvement over the compared methods in terms of the PSNR, SSIM, and MAE metrics. This is due to the fact that our method designs two effective multi-feature fusion techniques, which not only exploits the connection between the structure and texture features, but also considers the relationship within image context. For the FID metric, our method still shows competitive performance on the images with small damaged region. When evaluated on the images with large damaged region, our method exhibits minor performance degradation. The reason might be as follows. The FID metric aims to measure the distance between the feature spaces of two groups of images. While the proposed method does not introduce the structure prior in the encoding stage, thus enlarging the distance between the feature spaces of the generated output and ground-true images. Besides, we can observe from Fig. \ref{imgirre} that the images inpainted by the proposed method have better visual quality than those of all the compared methods.

For images with center hole, we compare the proposed method with four typical methods including CE \cite{pathak2016context}, CA \cite{yu2018generative}, SH \cite{yan2018shift} and MED \cite{liu2020rethinking}. The performance is evaluated on 10,000 images selected randomly from the CelebA \cite{liu2015deep} validation dataset. The result is shown in Table \ref{tab:CelebA_comparison}. We can see that the proposed method still performs best. Especially for the FID metric, our method can better infer the missing structure and texture of an image compared with its competitors.
The subjective quality of the inpainted images is shown in Fig. \ref{imgcent}. It can be observed that our method demonstrates its effectiveness in dealing with the center hole images.

\begin{table}[t!]
\setlength{\abovecaptionskip}{0mm}
\renewcommand\arraystretch{1.2}
\footnotesize
\caption{Ablation study of different modules on Paris StreetView. Here, random masks with mask ratio 30\%-40\% are used.}
\vspace{-2mm}
\begin{center}
\scalebox{0.94}{
\begin{tabular}{|c|c|c|c|c|}
  \hline
  Metrics &$\textrm{PSNR}^\mathrm{\uparrow}$ &$\textrm{SSIM}^\mathrm{\uparrow}$ &$\textrm{MAE}^\mathrm{\downarrow}$ &$\textrm{FID}^\mathrm{\downarrow}$\\
  \hline
  w/o SDFF &27.15 &0.883 &0.0244 &46.33 \\
  \hline
  w/o CA &27.08 &0.879 &0.0249 &\bf{44.56} \\
  \hline
  w/o SKNet &27.17 &0.885 &0.0242 &45.61 \\
  \hline
  w/o BPFA &26.99 &0.880 &0.0247 &46.39 \\
  \hline
  w/o PC &26.93 &0.877 &0.0250 &46.92 \\
  \hline
  Baseline &26.08 &0.877 &0.0282 &48.91 \\
  \hline
  Ours  &\bf{27.34}&\bf{0.889}&\bf{0.0240}&45.44\\
  \hline
\end{tabular}
}
\end{center}
\label{tab:Ablation_comparison}
\end{table}

\iffalse
\begin{table}[t!]
\setlength{\abovecaptionskip}{0mm}
\renewcommand\arraystretch{1.2}
\footnotesize
\caption{Ablation study on the Paris StreetView dataset with different loss. The statistics are based on irregular masks with mask ratio 30\%-40\%.}
\vspace{-2mm}
\begin{center}
\scalebox{0.94}{
\begin{tabular}{|c|c|c|c|c|}
  \hline
  Metrics &$\textrm{PSNR}^\mathrm{\uparrow}$ &$\textrm{SSIM}^\mathrm{\uparrow}$ &$\textrm{MAE}^\mathrm{\downarrow}$ &$\textrm{FID}^\mathrm{\downarrow}$\\
  \hline
  w/o $\mathcal{L}_{\textrm{perc}}$ &26.98 &0.896 &0.0254 &47.96 \\
  \hline
  w/o $\mathcal{L}_{\textrm{style}}$ & & & & \\
  \hline
  w/o $\mathcal{L}_{\textrm{rst}}$ &27.16 &0.883 &0.0251 &45.95 \\
\hline
  w/o $\mathcal{L}_{\textrm{rte}}$ &27.23 &0.887 &0.0252 &\bf{43.96} \\
  \hline
  Ours  &\bf{27.34}&\bf{0.889}&\bf{0.0240}&45.44\\
  \hline
\end{tabular}
}
\end{center}
\label{tab:Ablation_comparisonLoss}
\vspace{-6mm}
\end{table}
\fi

\subsection{Ablation study}
To verify the contribution of each individual component of our network, the ablation study is performed on the Paris StreetView dataset. The result is shown in Table \ref{tab:Ablation_comparison}. We can see that each module shows its effectiveness. Specifically speaking, the partial convolution based padding (PC) module \cite{liu2018image} improves the performance by making the model pay more attention to visible pixels (undamaged regions). Moreover, our designed BPFA module contributes to the whole network the best. This is because BPFA enhances the connection from local feature to overall consistency, thus reducing visible artifacts and unpleasant contents. In addition, we can find that the SDFF module has the second contribution. As expected, SDFF captures the correlation between the structure and texture features. Interestingly, the CA module can learn contextual feature representations and the SKNet module can effectively equalize the structure and texture features generated by the multi-scale filling stage, thereby benefiting the proposed network.

To better visulization, we show the outputs and ground-truths for Eqs. (1) and (2) in Fig. \ref{eqs1and2}. Fig. \ref{eqs1and2} (a) and (c) are our structure and texture feature maps obtained after the multi-scale filling stage, respectively. Fig. \ref{eqs1and2} (b) and (d) are the structure and texture of ground-turth, respectively. We can see that deep-layer convolution focuses more on image structure (Fig. \ref{eqs1and2} (a) and (b)) and shallow-layer convolution focuses more on image texture (Fig. \ref{eqs1and2} (c) and (d)).

\def0.325\linewidth{0.35\linewidth}
\begin{figure}[h!]
\centering
\begin{tabular}{cccc}
\includegraphics[width=0.325\linewidth]{st14.pdf}&
\includegraphics[width=0.325\linewidth]{014_imst.pdf}
\\
(a) Our structure & (b) GT structure & \\
\includegraphics[width=0.325\linewidth]{de14.pdf}&
\includegraphics[width=0.325\linewidth]{014_imde.pdf}\\
(c) Our texture & (d) GT texture
\end{tabular}
\caption{Visual outputs for  Eqs. (1) and (2).}
\label{eqs1and2}
\end{figure}

In addition, according to our ablation study, our method with SDFF performs better than that without SDFF. This shows that our SDFF is a better channel attention module than existing channel attention modules. An example of the feature output is shown in Fig. \ref{imgGate}. It can be observed Fig. \ref{imgGate} that these two soft gates can respectively response the structure and texture well. It is worth emphasizing that our method use $\mathcal{L}_{\textrm{rst}}$ and $\mathcal{L}_{\textrm{rte}}$ to make deep-layer and shallow-layer convolutions focus on structure and texture features, respectively. Meanwhile, $\mathcal{L}_{\textrm{perc}}$ and $\mathcal{L}_{\textrm{style}}$ are used to improve perceptual quality and mitigate style differences, respectively. Clearly, $\mathcal{L}_{\textrm{rec}}$ and $\mathcal{L}_{\textrm{adv}}$ are necessary for pixel reconstruction and adversarial training. The ablation experiments on these losses are shown in Table \ref{tab:AblationLoss_comparison}. It is clear that each loss contributes to our method.

\def0.325\linewidth{0.325\linewidth}
\begin{figure}[h!]
\renewcommand{\tabcolsep}{1pt}
\begin{tabular}{ccccccc}
\includegraphics[width=0.325\linewidth]{st21.pdf}&
\includegraphics[width=0.325\linewidth]{de21.pdf}&
\includegraphics[width=0.325\linewidth]{021_im.pdf}\\
(a) Structure & (b) Texture & (c) Ground Truth
\end{tabular}
\caption{Visualization of the feature maps for the structure and texture information.}
\label{imgGate}
\end{figure}

\begin{table}[h!]
\setlength{\abovecaptionskip}{0mm}
\renewcommand\arraystretch{1.2}
\footnotesize
\caption{Ablation study of different losses on Paris StreetView. Here, random masks with mask ratio 30\%-40\% are used.}
\vspace{-1mm}
\begin{center}
\scalebox{1}{
\begin{tabular}{|c|c|c|c|c|}
  \hline
  Metrics &$\textrm{PSNR}^\mathrm{\uparrow}$ &$\textrm{SSIM}^\mathrm{\uparrow}$ &$\textrm{MAE}^\mathrm{\downarrow}$ &$\textrm{FID}^\mathrm{\downarrow}$\\
  \hline
  w/o $\mathcal{L}_{\textrm{perc}}$ &26.99 &0.879 &0.0254 &46.96 \\
  \hline
  w/o $\mathcal{L}_{\textrm{style}}$ &26.98 &0.879 &0.0250 &47.55 \\
  \hline
  w/o $\mathcal{L}_{\textrm{rst}}$ &27.16 &0.883 &0.0251 &45.95 \\
\hline
  w/o $\mathcal{L}_{\textrm{rte}}$ &27.23 &0.887 &0.0252 &\bf{43.96} \\
  \hline
  Ours  &\bf{27.34}&\bf{0.889}&\bf{0.0240}&45.44\\
  \hline
\end{tabular}
}
\end{center}
\label{tab:AblationLoss_comparison}
\vspace{-4mm}
\end{table}

\section{Conclusion}
In this paper, we have presented a deep multi-feature co-learning network for image inpainting, which can yield the detailed textures and reasonable structures. Our network designs two new fusion modules:  Soft-gating Dual Feature Fusion (SDFF) and Bilateral Propagation Feature Aggregation (BPFA). SDFF can control the fusion ratio through a soft gating technique to refine the structure and texture features,  making the structures and textures more consistent. BPFA can co-learn contextual attention, channel-wise information and feature space. By the co-learning strategy, the inpainted images preserve the from local to overall consistency.

Note that the soft gating operation is commonly used to channel attention. But our SDFF module exploits two soft gates ($\boldsymbol{G}_{st}$ and $\boldsymbol{G}_{te}$) to make effective response to structure and texture information, respectively. Through $\boldsymbol{G}_{st}$ and $\boldsymbol{G}_{te}$, our SDFF can control the degree of integrating structure and texture information. We also tried to first concatenate $\boldsymbol{F}_{cst}$ and $\boldsymbol{F}_{cte}$, and then used channel attention module (SENet [8]) directly to output feature. However, this will lead to dynamic output changes and impair the training of GAN. Our experiment also shows that this strategy achieves only limited improvement for our model. Therefore, we propose to multiply reweighted $\boldsymbol{F}_{cte}$ by $\boldsymbol{F}_{cst}$ to selectively transmit useful features, resulting in significant performance improvement. In the future, we will explore the performance of introducing the structure prior in the encoding stage.

{\small
\nocite{*}
\bibliographystyle{ieee}


\section{Introduction}

The Anti-de-Sitter/boundary conformal field theory (AdS/BCFT) correspondence \cite{Takayanagi:2011zk,Fujita:2011fp} is a generalization of AdS/CFT correspondence \cite{Maldacena:1997re}
in the sense that the space-time manifold admits a boundary. (See also \cite{Karch:2000gx}.)
There are
various works which confirms this duality in terms of the entanglement
entropy and the conformal anomaly both from the gravity sides and the BCFT sides, a small selection being \cite{Takayanagi:2011zk,Fujita:2011fp,Nozaki:2012qd,McAvity:1995zd,Collier:2021ngi,Kusuki:2021gpt,Numasawa:2022cni}.
The AdS/BCFT plays an important role
in the recent studies of
the replica wormhole and the island formula \cite{Almheiri:2019qdq,Penington:2019kki}
in relation to the Page curve \cite{Page:1993df}.

In the AdS/BCFT correspondence we impose the Neumann boundary condition
on the metric on a boundary, which is called the end-of-the-world brane (ETW brane):
\begin{equation}
K_{ab}=(K-T)h_{ab},\label{eq:1}
\end{equation}
where the $K$ is an extrinsic curvature, $T$ is a constant, called the tension of the ETW brare, and $h_{ab}$
is an induced metric on the boundary.\footnote{
In this paper, the ETW brane simply means that
we impose
the boundary condition \eqref{eq:1} on a codimension one time-like hypersurface,
which ends on a fixed hypersurface in the asymptotic boundary of the AdS space,
and only consider one of the spacetime regions separated by the hypersurface.
}
To see why we call it as Neumann boundary condition, we work
in the Gaussian normal coordinate where the boundary is on $r=0$ and
\begin{equation}
ds^{2}=dr^{2}+h_{ab}dx^{a}dx^{b}.
\label{eq:15}
\end{equation}
Then, we can see that
\begin{equation}
K_{ab}=\frac{1}{2}\frac{\partial h_{ab}}{\partial r},\label{eq:16}
\end{equation}
which is the Neumann boundary condition on the metric. Here we note
that the indices $a,b$ run the tangential direction to an ETW brane.
The gravity with boundary conditions have a long history, a small selection being \cite{Anderson:2006lqb,vanNieuwenhuizen:2005kg,Vassilevich:2003xt,Miao:2017gyt,Witten:2018lgb,Suzuki:2021pyw,Chu:2021mvq,Suzuki:2022xwv}. The meaning of fixing the extrinsic curvature as in the (\ref{eq:1}) can be found in \cite{Witten:2018lgb,Park:2015xoa}.

One of the important subjects of the AdS/BCFT correspondence is, of course, the dynamics, which have not been considered
even for the perturbation around the vacuum.
In the bulk gravity picture, this can be done by just studying the gravitational perturbations around the vacuum
with the boundary condition (\ref{eq:1}).

In this paper, we explicitly obtain the mode expansion of the gravitational perturbations around the AdS vacuum in the Poincare patch
with the boundary condition (\ref{eq:1}) for the tensionless case, $T=0$.
Using these results, we can obtain the bulk reconstruction formulas \cite{Bena} \cite{HKLL} assuming the BDHM relation \cite{Banks:1998dd} in principle, although we leave such work as a future problem.
The bulk wave packets can be also considered in the AdS/BCFT correspondence following \cite{Te1,Te2}, which used an alternative bulk-boundary map given in \cite{Te3, Te4}.
This will be also interesting future work.

In this paper, we also show that if a $d$-dimensional BCFT can couple to a $d$-dimensional gravity with a boundary,
the energy-momentum tensor should satisfy nontrivial constraints other than the ones for the boundary conformal symmetry.\footnote{
Note that
even if we consider the AdS/BCFT,
this $d$-dimensional gravity is different from the
$d+1$-dimensional bulk gravity.
}
More precisely, we need to choose the boundary condition of the gravity.
We consider two known consistent boundary conditions,
namely, the Neumann and the conformally Dirichlet boundary conditions. We note that the Dirichlet boundary condition may be incosistent\cite{Witten:2018lgb}, therefore in this paper we do not consider the Dirichlet case.
We find that the necessary conditions for energy-momentum tensor in the BCFT which couples to the gravity with the Neumann boundary condition are
\begin{equation}
T_{x a}|_{x=0}=0,  \,\,\,
\partial_xT_{ab} |_{x=0}=0, \,\,\,
\partial_xT_{xx} |_{x=0}=0,
\label{nbci}
\end{equation}
which are satisfied for the free scalar BCFT with the Neumann and also the  Dirichlet boundary conditions.
The necessary conditions for energy-momentum tensor in the BCFT which couples to the gravity with the conformal Dirichlet boundary condition are
\begin{equation}
T_{x a}|_{x=0}=0,  \,\,\,
T_{ab} |_{x=0}= - \frac{1}{d-1} \eta_{ab} \, T_{xx}|_{x=0}, \,\,\,
\partial_xT_{xx} |_{x=0}=0,
\label{dbci}
\end{equation}
which are satisfied for the free scalar BCFT with the Dirichlet boundary condition.

For the BCFT in the AdS/BCFT,
we find that
\eqref{nbci} are satisfied for the AdS/BCFT with tensionless ETW brane and
\eqref{dbci} are satisfied for the AdS/BCFT with non-zero tension ETW brane
by explicitly deriving the constraints for the energy-momentum tensor
using holographic relations.
Thus,
the BCFT in the AdS/BCFT with the tensionless brane can couple to
the gravity with the Neumann boundary condition,
but the BCFT for the nonzero tension brane can couple to the gravity with the conformally Dirichlet boundary condition.
These results might be surprising because the ETW brane in the AdS/BCFT
imposes the Neumann boundary conditions for the bulk gravity theory,
even for the non-zero tension case.
We expect that the key to this problem is that we impose the conformally Dirichlet boundary conditions at the asymptotic AdS boundary.
It will be interesting to understand these results clearly. We will expect that our results can also be examined by the holographic renormalization of the bulk geometry \cite{Guijosa:2022jdo}.

Note: This work has a overlap with \cite{Izumi:2022opi} partially. In that work they solve Einstein equation with the condition (\ref{eq:1}). Technically the assumptions they set are a little different from ours.

This paper is organized as follows:
In section 2
we give a review of the mode expansion
of the metric in the AdS/CFT in the Poincare patch.
In section 3 we consider the mode expansion of the metric in AdS/BCFT
and obtain the explicit form of the mode expansion for the tensionless case.
In section 4
we show that
the energy-momentum tensor in a $d$-dimensional BCFT should satisfy
nontrivial constraints
if the BCFT can couple to a $d$-dimensional gravity
with a specific boundary condition.
For the BCFT in the AdS/BCFT,
we find that
the BCFT can couple to
the gravity with the Neumann boundary condition for the tensionless brane,
but the BCFT can couple to the gravity with the conformally Dirichlet boundary condition
for the nonzero tension brane,
by using holographic relations.
Section 5 is devoted to the conclusions.
In the Appendix, we consider the
mode expansion
of the metric in the AdS/BCFT in the gauge invariant formalism,
although we only obtain partial results.

\section{ Free theory limit of gravity
in Poincare $AdS$ space}
\label{s1}

In this section, in preparation for the study of the dynamical degrees of freedom of AdS/BCFT,
we will review the
free theory limit of gravity
in Poincare $AdS_{d+1}$ space, which is supposed to be dual to
the generalized free limit of the holographic CFT on the Minkowski space.\footnote{
Assuming the BDHM relation \cite{Banks:1998dd},
the reconstruction of the bulk local operators can be done in principle.}
This will be done by knowing
the mode expansion of the gravitational
perturbation $g_{\mu\nu}^{\left(1\right)}$
around the $AdS_{d+1}$
background $g_{\mu\nu}^{\left(0\right)}$ in the Poincare coordinate:
\begin{equation}
ds^{2}=g_{MN}^{\left(0\right)} d x^M dx^N
=\frac{dz^{2}+\eta_{\mu \nu} dx^\mu dx^\nu}{z^{2}},
\end{equation}
where $\mu,\nu=1,\ldots,d$.
In this paper, we assume $d \geq 3$ because for $d=2$ there is no usual gravitons
in $AdS_3$ and there are only the boundary gravitons.
The mode expansion for this was already done in \cite{He} \cite{Kabat} and
we will follow their studies.

We will take the Fefferman-Graham coordinate, i.e.
\begin{equation}
g_{z M}^{\left(1\right)}=0.
\end{equation}
Then, a part of the linearized Einstein equations with the normalizable condition for $z \rightarrow 0$
gives the following conditions:
\begin{equation}
g_{ \mu}^{\,\,\, \mu \left(1\right)}=0, \,\,\, \partial^\mu g_{\mu \nu}^{\left(1\right)}=0.
\end{equation}
The remaining linearized Einstein equations become
\begin{equation}
\left( \partial^\alpha \partial_\alpha+\partial_z^2 +\frac{5-d}{2} \partial_z- \frac{2(d-2)}{z^2}
\right)  g_{\mu \nu}^{\left(1\right)} =0.
\end{equation}
These equations can be solved by the Bessel functions as
\begin{equation}
g_{\mu \nu}^{\left(1\right)} =
\int d \omega d k_\mu d \zeta_{\mu \nu} \,  e^{i (\omega t+k_\mu x^\mu) } \, \zeta_{\mu \nu} \, z^{\frac{d}{2}-2}(a_{\omega, k,\zeta} J_{d/2}(\sqrt{ \omega^2 -k_\mu k^\mu
} \, \, z)+b_{\omega, k,\zeta}Y_{d/2}(\sqrt{ \omega^2 -k_\mu k^\mu
} \, \, z)),
\label{mode1}
\end{equation}
where
the integrations are constrained with $\zeta_{\mu \nu}=\zeta_{\nu \mu},\zeta_{\mu}^{ \,\,\, \mu}=0, k^\mu \zeta_{\mu \nu}=0, \zeta_{\mu \nu} \zeta^{\mu \nu}=1$.
Because $J_{\nu}(Z) \sim Z^\nu$ and $Y_{\nu}(Z) \sim Z^{-\nu}$ near $z=0$,
the coefficients  $b_{\omega, k,\zeta} =0$ so that the field should be (delta-function) normalized.
Furthermore, we need to take $a_{\omega, k,\zeta}=0$ if
$\omega^2 <\sum_{p=1}^{n+1} (k_p)^2$ because  $J_{\nu}(Z) \sim  e^{|Z|}$   in the limit
 $Z \rightarrow \infty$ for the region.
Finally, by the free field quantization, the coefficients $a_{\omega, k,\zeta}$ will be the creation operators or
the annihilation operator for $\omega>0 $  or  $\omega < 0 $, respectively
with a suitable normalization constant factor.

\section{Gravitational perturbations in the AdS/BCFT }

In this section, we consider the mode expansion of the
gravitational perturbations in the AdS/BCFT
with the boundary condition  (\ref{eq:1}) ,
\begin{equation}
K_{ab}=(K-T)h_{ab}.
\label{eq:1a}
\end{equation}
Note that this implies $K=T \frac{d}{d-1}$ and $K_{ab}=\frac{T}{d-1} h_{ab}$.
The mode expansion has been done for the case without the boundary condition
in the previous section, and then we will search which combinations of
the modes satisfy the boundary condition

In order to specify the boundary condition explicitly,
we first write the background $AdS_{d+1}$ metric as,
 \begin{equation}
     ds^2=d\eta^2+\frac{\cosh^2{\eta}}{\zeta^2} (d\zeta^2-dt^2+\sum_{i=1}^{n} dw_{i}^{2})
 \end{equation}
where $n=d-2$,
which becomes
the familiar form of the metric in the Poincare coordinate,
 \begin{equation}
ds^{2}=g_{M N}^{\left(0\right)} d x^M dx^N
=\frac{dz^{2}-dt^{2}+dx^{2}+\sum_{i=1}^{n} dw_{i}^{2}}{z^{2}},
\end{equation}
by the following coordinate transformation:
  \begin{eqnarray}
  z&=&\frac{\zeta}{\cosh{\eta}},\nonumber\\
  x&=&\zeta\tanh{\eta}.
  \end{eqnarray}
For this background, the
boundary condition (\ref{eq:1}) is satisfied by restricting the spacetime
to the region $-\infty<\eta< \eta^*$,
where $\eta^*$ is determined by the tension $T$ as $T=(d-1) \tanh \eta^*$.
In the usual coordinate, the boundary is at
$x/z=\sinh \eta^*$.
In particular, for tensionless case $T=0$, $\eta^*=0$ which means that the boundary is at  $x=0$.

Note that, if we choose the Gaussian normal coordinate like (\ref{eq:15}),
the boundary condition (\ref{eq:1}) becomes just a partial derivative \cite{Nozaki:2012qd},
 \begin{equation}
     \frac{\partial h_{ab}}{\partial r}=\frac{2T}{d-1}h_{ab}=2 \tanh \eta^* h_{ab}.\label{eq:17}
 \end{equation}

Note also that in this paper  we require the asymptotic AdS boundary condition
where we can take the FG gauge and
the metric which includes the perturbations near $z=0$ can be written by
 \begin{equation}
ds^{2}
=\frac{dz^{2}+h_{\mu \nu}(x^\mu,z)  dx^\mu dx^\nu }{z^{2}},
\end{equation}
where
 \begin{equation}
h_{\mu \nu}(x^\mu,z)  dx^\mu dx^\nu = \eta_{\mu \nu} + T_{\mu \nu} (x^\rho) z^{d} +{\mathcal O}(z^{d+1}),
\end{equation}
and $T_{\mu \nu} (x^\rho) $ is the energy momentum tensor of the corresponding
BCFT up to a constant.
We require this is valid on $\eta=\eta^*$.
In particular, this implies that $ T_{\mu \nu} (x^\rho)  |_{ x=0}$
is well-defined and does not depend on how we approach the corner point $z=x=0$.
Our requirement is different from the condition imposed in \cite{Izumi:2022opi}
where $ T_{\mu \nu} (x^\rho)  |_{ x=0}$ is allowed to be divergent for some modes.

\subsection
Tensionless case}

Let us consider the tensionless case.
The perturbation of the metric can be written as

\begin{eqnarray*}
g_{MN}=g_{M N}^{\left(0\right)}+g_{MN}^{\left(1\right)}.\\
\end{eqnarray*}
We will place the tensionless ETW brane at $x=0$ in this Poincare $AdS_{d+1}$
background $g_{M N}^{\left(0\right)}$.
First we  assume that
the ETW brane is on $x=0$
even after including the perturbations of the metric.
We will reconsider the case where the ETW brane is not on $x=0$ later.

The unit normal vector for the surface is given by
\begin{eqnarray*}
n_M=\delta_{x M} /\sqrt{(g^{-1})^{xx}}
\simeq\left(\frac{1}{z}+\frac{z g_{xx}^{(1)}}{2}\right) \delta_{x M},
\end{eqnarray*}
up to the first order of the perturbation.
The extrinsic curvature is defined by
\begin{eqnarray*}
K_{ab}=e_{a}^{M} e_{b}^{N}\nabla_{M}n_{N}
=\frac{\partial n_b}{\partial x^a} - \Gamma^M_{ab} n_M=- \Gamma^M_{ab} n_M \simeq -\frac{1}{z} \delta \Gamma^x_{ab},
\end{eqnarray*}
where $x^a=\{ t,z, w_i  \}$
and the projection tensor is
\begin{eqnarray*}
e_{a}^{M}=\frac{\partial x^M}{\partial x^a}=\delta_{a}^{M}.
\end{eqnarray*}
Here, the deviation of the Christoffel symbol $\delta {\Gamma^{x}_{ab}}$ from the background metric in the linearized approximation is
\begin{eqnarray*}
\delta {\Gamma_{ab}^{x}} &=& \frac{1}{2} \delta g^{xM} \left(g_{Ma,b}+g_{Mb,a}-g_{ab,M} \right)
+\frac{1}{2} g^{x M} \left( \delta g_{Ma,b} + \delta g_{Mb,a} - \delta g_{ab,M} \right) \\
&=& z^3 \left( g^{(0)}_{bz} g^{(1)}_{xa} +g^{(0)}_{az} g^{(1)}_{xb} -g^{(0)}_{ab} g^{(1)}_{xz} \right)
+ \frac12 z^2 \left(  g^{(1)}_{xa,b} + g^{(1)}_{xb,a} - g^{(1)}_{xa,x} \right).
\end{eqnarray*}
Substituting these altogether, the boundary conditions in components become
\begin{eqnarray*}
&K_{tt}=-z\left(g_{xt,t}^{(1)}-\frac{1}{2}g_{tt,x}^{(1)}\right)-g^{(1)}_{xz}=0,\\
&K_{tz}=K_{zt}=-\frac{z}{2}\left(g_{xt,z}^{(1)}+g_{xz,t}^{(1)}-g_{zt,x}^{(1)}\right)-g^{(1)}_{xt}=0,\\
&K_{zz}=-z\left(g_{xz,z}^{(1)}-\frac{1}{2}g_{zz,x}^{(1)}\right)+g^{(1)}_{xz}=0,\\
&K_{tw_i}=K_{w_i t}=-\frac{z}{2}\left(g_{xw_i,t}^{(1)}+g_{xt,w_i}^{(1)}-g_{tw_i,x}^{(1)}\right)=0,\\
&K_{zw_i}=K_{w_iz}=-\frac{z}{2}\left(g_{xz,w_i}^{(1)}+g_{xw_i,z}^{(1)}-g_{zw_i,x}^{(1)}\right)-g^{(1)}_{xw_i}=0,\\
&K_{w_{i}w_{i}}=-\frac{z}{2}\left(2g_{xw_{i},w_{i}}^{(1)}-g_{w_{i}w_{i},x}^{(1)}\right)+g^{(1)}_{xz}=0,\\
&K_{w_{i}w_{j}}=-\frac{z}{2}\left( g_{xw_{i},w_{j}}^{(1)}+g_{xw_{j},w_{i}}^{(1)}-g_{w_{i}w_{j},x}^{(1)} \right) =0, \,\,\, (i \neq j),
\label{eq:-6}
\end{eqnarray*}
where they are evaluated on $x=0$ and
the trace of the extrinsic curvature $K$ was set to be zero since we focus on the tensionless
case.

Next, we take the Fefferman-Graham gauge and
substitute the mode expansions of the metric perturbations obtained in the previous section to
the above boundary conditions.
Note that
the ETW brane is on $x=0$ by the assumption.
Then, from $K_{tz}=0$ we find $z g_{xt,z}^{(1)}+2 g_{xt}^{(1)}=0 $, which means $g_{xt}^{(1)} = z^{-2} G(t,w_i)$ on $x=0$.
This can not be possible for the normalizable mode, thus we find  $g^{(1)}_{xt} |_{x=0}= 0$.
With this and $K_{tt}=0$, we find $\frac{\partial g^{(1)}_{tt}}{\partial x} |_{x=0}= 0$.
Similarly, we find
$g^{(1)}_{x w_i} |_{x=0}=\frac{\partial g^{(1)}_{t w_i}}{\partial x} |_{x=0}= \frac{\partial g^{(1)}_{w_i w_j}}{\partial x} |_{x=0}=0 $.

In summary, the boundary conditions in the Fefferman-Graham gauge $g^{(1)}_{z M}=0$ are
$g^{(1)}_{x a} |_{x=0}=\frac{\partial g^{(1)}_{a b}}{\partial x} |_{x=0}=0 $.
Thus, we find that the modes which satisfy the boundary condition using the results without the boundary, \eqref{mode1},
are
\begin{eqnarray*}
g_{\mu \nu}^{\left(1\right)}  &\sim &
\left(
 \cos (k_x x) \, \zeta^N_{\mu \nu}
+ \sin (k_x x)  \, \zeta^D_{\mu \nu}
\right) e^{i (\omega t+k_a x^a) } \, z^{\frac{d}{2}-2} J_{d/2}(\sqrt{ \omega^2 -k_\mu k^\mu
} \, \, z)
,
\label{mode2}
\end{eqnarray*}
where
$\zeta^N_{x \alpha} =\zeta^N_{\alpha x}=0 $
and
$\zeta^D_{\alpha \beta} =0$
for $\alpha \neq x$ and $\beta \neq x$.
Moreover,
$\zeta^N_{\mu \nu} $ and $\zeta^D_{\mu \nu} $ are symmetric traceless, which implies $\zeta^D_{x x}=0$, and they should satisfy
\begin{eqnarray*}
-k^x  \zeta^N_{x \nu}
+ i k^a \zeta^D_{a \nu} =0, \,\,\,
k^x  \zeta^D_{x \nu}
+ i k^a \zeta^N_{a \nu} =0,
\label{mode3}
\end{eqnarray*}
which come from $\partial^\mu g_{\mu \nu}^{\left(1\right)} =0$.
The non-trivial parts of the equations of \eqref{mode3}
are $k^x  \zeta^D_{x b}
+ i k^a \zeta^N_{a b} =0$
and $-k^x  \zeta^N_{x x}
+ i k^a \zeta^D_{a x} =0$, which implies $k^a k^b \zeta^N_{a b} = (k^x )^2 \zeta^N_{x x }= -  (k^x )^2 \zeta^{N \,\, c}_{ \,\, \,\,c }$.

Note that we can regard the independent components of $\zeta^N$ and $ \zeta^D$ as $\zeta^{N}_{ab}$ constrained by
$k^a k^b \zeta^N_{a b} = -  (k^x )^2 \zeta^{N \,\, c}_{ \,\, \,\,c }$.
Thus, the number of the degrees of freedom is $d(d-1)/2 -1=(d-2)(d+1)/2$ which is same as the one for the theory without the boundary, as expected because the gravitational waves far from the boundary will be same as ones for the theory without the boundary.
Note also that in the Gaussian normal coordinate, the boundary condition seems to be given by the Neumann boundary condition on the plane wave along $x$ direction only, however, we have chosen the different gauge and the mode expansions given here are different
from that.

\subsubsection{No mode if the ETW brane is not on $x=0$}

We have assumed that the ETW brane is stretching on $x=0$ in the FG gauge.
This means that we searched the modes which satisfying this condition and
the modes we obtained might not  be complete.
Below we will consider the case that the ETW brane is not stretching on $x=0$ and
will find that there are no additional modes, thus \eqref{mode2} are complete.

Let us assume that we pick
the gauge choice above, and adding the perturbations.
Then correspondingly the brane deviates a little from $x=0$ in general and we denote the hypersurface  of the brane as $F \equiv x-f(z,t,w_i)=0$
where $f(z,t,w_i)$ is the order of the perturbation.
Let us linearize the boundary condition (\ref{eq:1}).
The unit normal vector $n+ \delta n$ of the ETW brane is given by,
 \begin{equation}
n_M+\delta n_M=
\frac{\partial_M F}{\sqrt{ (g^{-1})^{MN} \partial_M F \partial_N F}},
\end{equation}
which implies
\begin{equation}
\delta n_M=
-\frac{1}{z} \partial_M f  + \delta_{x M} \frac1z \partial_M f+ \cdots,
\end{equation}
where we keep the terms of the liner order of the perturbations.
Below, we will omit the higher order of the perturbations in the equations
for the notational simplicity.
The extrinsic curvature is defined by
\begin{eqnarray*}
K_{ab}=e_{a}^{M} e_{b}^{N} \nabla_{M} (n_{N}+ \delta n_N )
=\frac{\partial (n_{b}+ \delta n_b )}{\partial x^a} - \Gamma^M_{ab} (n_{M}+ \delta n_M )
+\partial_a f  \delta_{b z}  /z^2  ,
\label{Kab}
\end{eqnarray*}
where $x^a=\{ t,z, w_i  \}$
and the projection tensor is
\begin{eqnarray*}
e_{a}^{M}=\frac{\partial x^M}{\partial x^a}=\delta_{a}^{M} + \delta^{M}_{x} \partial_a f.
\end{eqnarray*}
Here, we have approximated the terms comes from the deviation of the projection tensor, $  \partial_a f   \nabla_{x} n_{b} +\partial_b f   \nabla_{a} n_{x}=
\partial_a f  \delta_{b z}  /z^2
$, in \eqref{Kab}
in the linear order in the perturbation.
Then, the deviation of it proportional to $\delta n$ is
$\delta K_{ab}=\frac{\partial  \delta n_b }{\partial x^a} - \Gamma^M_{ab}  \delta n_M
+\partial_a f  \delta_{b z}  /z^2  $, and
we can calculate the extrinsic curvature as
\begin{eqnarray}
K_{zz}&=& g_{xz}^{(1)}-\frac{z}{2}\left(2 g_{xz,z}^{(1)}- g_{zz,x}^{(1)}\right)
-\frac{\partial}{\partial z} \left( \frac1z \frac{\partial f}{\partial z}\right)

\label{eq:21}
\end{eqnarray}
\begin{eqnarray}
K_{zt}=-z \frac{\partial}{\partial z} \left( \frac{1}{z^2}  \frac{\partial f}{\partial t}\right)-\frac{z}{2}\left( g_{xt,z}^{(1)}+ g_{xz,t}^{(1)}- g_{zt,x}^{(1)}\right)-g_{xt}^{(1)},\label{eq:20}
\end{eqnarray}
\begin{eqnarray}
K_{zw_i}=-\frac{1}{z}\left(\frac{\partial^2f}{\partial w_i\partial z}\right)
-\frac{z}{2}\left( g_{xw_i,z}^{(1)}+ g_{xz,w_i}^{(1)}- g_{zw_i,x}^{(1)}\right)-g_{xw_i}^{(1)},
\end{eqnarray}
\begin{eqnarray}
K_{tt}&=& g_{xz}^{(1)}-\frac{z}{2}\left(2 g_{xt,t}^{(1)}- g_{tt,x}^{(1)}\right)
-\frac{1}{z}\left(\frac{\partial^2f}{\partial t^2}\right)-\frac{1}{z^2}\left(\frac{\partial f}{\partial z}\right),\label{eq:22}
\end{eqnarray}
\begin{eqnarray}
K_{tw_i}=-\frac1z \frac{\partial^2f}{\partial t\partial w_i}-\frac{z}{2}\left( g_{xt,w_i}^{(1)}+ g_{xw_i,t}^{(1)}- g_{w_it,x}^{(1)}\right),
\end{eqnarray}
\begin{eqnarray}
K_{w_iw_i}&=& -g_{xz}^{(1)}-\frac{z}{2}\left(2 g_{xw_i,w_i}^{(1)}- g_{w_iw_i,x}^{(1)}\right)
-\frac{1}{z}\left(\frac{\partial^2f}{\partial w_i^2}\right)+\frac{1}{z^2}\left(\frac{\partial f}{\partial z}\right),
\end{eqnarray}
\begin{eqnarray}
K_{w_iw_j}=-\frac1z \frac{\partial^2f}{\partial w_i\partial w_j}-\frac{z}{2}\left( g_{xw_i,w_j}^{(1)}+ g_{xw_j,w_i}^{(1)}- g_{w_iw_j,x}^{(1)}\right),
\label{eq:25}
\end{eqnarray}
where we have used the fact that non-zero $\Gamma_{MN}^P$ are
\begin{eqnarray*}
\frac{1}{z}=\Gamma_{ii}^z \,\, =-\Gamma_{tt}^z \,\, =-\Gamma_{zz}^z \,\,
=-\Gamma_{iz}^i \,\, =-\Gamma_{zi}^i \,\, =\Gamma_{tz}^t \,\, =\Gamma_{zt}^t \,\,
\end{eqnarray*}
for the Poincare AdS space in the coordinate system.

The boundary conditions are $K_{ab}|_{x=f}=0$.
In the Fefferman-Graham gauge $g^{(1)}_{z M}=0$,
$K_{zz}|_{x=f}=0$ implies $f =c_1 z^2+ c_0$.
Here, $c_0$ should be zero
because of
the asymptotic boundary condition $f \rightarrow 0$ for $z \rightarrow 0$.
Furthermore,
$K_{tt}|_{x=f}$ contains $-\frac{1}{z}\left(\frac{\partial^2f}{\partial t^2}\right)-\frac{1}{z^2}\left(\frac{\partial f}{\partial z}\right)= -2 c_1/z+{\cal O}(z)$,
which should be canceled by other terms.
Thus, we find $f=0$ and
then we conclude that
there are no additional modes with $f \neq 0$.

\subsection{Non-zero tension case}

In this subsection we consider the non-zero tension case although
we can not obtain a closed form of the mode expansion.

Let us consider the perturbations in the FG gauge, $g_{z M}^{(1)}=0$, and we denote the position of the brane as
$F \equiv x-k z-f(z,t,w_i)=0$, where $k =\sinh \eta^*$.
We note that $f(z,t,w_i)$ is the order of the perturbation ${\cal O}(g^{(1)})$.
The unit normal vector $n$ of the ETW brane is given by,
 \begin{equation}
n_M=
\frac{\partial_M F}{\sqrt{ (g^{-1})^{MN} \partial_M F \partial_N F}},
\end{equation}
where $\partial_M F= \delta_{x M}-k \delta_{z M}-\partial_M f$ and
\begin{eqnarray*}
(g^{-1})^{M N} \partial_M F \partial_N F =z^2(1+k^2){\eta^*}+2z^2k\partial_z f-z^4g_{xx}^{(1)}
\end{eqnarray*}
Then, we find
\begin{equation}
n_M=
\frac{1}{z\sqrt{1+k^2}} \left(1-\frac{k}{1+k^2}\partial_z f+\frac{z^2}{2(1+k^2)}g_{xx}^{(1)}\right)\left(\delta_{x M}-k \delta_{z M}-\partial_M f\right)
\end{equation}
In particular, we obtain
\begin{eqnarray}
n_z&=&\frac{1}{z\sqrt{1+k^2}}(-
-\frac{1}{1+k^2}\partial_z f-\frac{kz^2}{2(1+k^2)}g_{xx}^{(1)}),\nonumber\\
n_x&=& \frac{1}{z\sqrt{1+k^2}}(1-\frac{k}{(1+k^2)}\partial_z f+\frac{z^2}{2(1+k^2)}g_{xx}^{(1)}),\nonumber\\
n_t&=&-\frac{1}{z\sqrt{1+k^2}}\partial_t f,\nonumber\\
n_{w_i}&=&-\frac{1}{z\sqrt{1+k^2}}\partial_{w_i} f.
\end{eqnarray}

The extrinsic curvature is defined by
\begin{eqnarray*}
K_{ab}&=&e_{a}^{M} e_{b}^{N} \nabla_{M} n_{N}\nonumber\\
&=&\frac{\partial n_{b}}{\partial x^a} - \Gamma^M_{ab} n_{M}
+ (k \delta_{a}^{z}+\partial_a f)  \nabla_{x} n_{b} + (k  \delta_{b}^{z} +\partial_b f ) \nabla_{a} n_{x}+(k \delta_{a}^{z}+\partial_a f) (k  \delta_{b}^{z} +\partial_b f )\nabla_{x} n_{x}   ,
\end{eqnarray*}
where $x^M=\{ x=k z +f( t,z, w_i  ), x^a=\{ t,z, w_i  \}\}$ and the projection tensor is
\begin{eqnarray*}
e_{a}^{M}=\frac{\partial x^M}{\partial x^a}=\delta_{a}^{M}+ k \delta_{x}^{M} \delta_{a}^{z}   + \delta^{M}_{x} \partial_a f.
\end{eqnarray*}
The boundary condition for the non-zero tension case is
\begin{eqnarray}
K_{ab} &=&\tanh \eta^*  h_{ab},
\label{bcg}
\end{eqnarray}
and we will firstly compute the $zz$ component.
This is evaluated as
\begin{eqnarray*}
K_{zz}=\frac{1}{\sqrt{1+k^2}}\left(\frac{k}{z^2}+\frac{k^3}{z^2}-\frac{1}{z}\partial^2_z f+\frac{1}{z^2}\frac{2k^4+3k^2+1}{k^2+1}\partial_z f-\frac{3k}{2}g_{xx}^{(1)}-g_{xx,z}^{(1)}(zk+\frac{zk^3}{2})-g_{xx,x}^{(1)}\frac{zk^2}{2}\right).
\end{eqnarray*}
Similarly we can expand the right hand side of (\ref{bcg})
and we obtain
\begin{equation}
    \tanh \eta^* h_{zz}=\frac{1}{\sqrt{1+k^2}}\left(\frac{k}{z^2}+\frac{k^3}{z^2}+k^3g_{xx}^{(1)}+\frac{2k^2}{z^2}\partial_z f\right),
\end{equation}
which implies that $f$ is ${\cal O}(z^{d+1})$  and completely determined
by
\begin{eqnarray}
\frac{1}{z}\partial^2_z f - \frac{1}{z^2}\partial_z f-\frac{(2k^2+3)k}{2}g_{xx}^{(1)}-g_{xx,z}^{(1)}(zk+\frac{zk^3}{2})=0,
\end{eqnarray}
where we omit $zg_{xx,x}^{(1)}$ because it is the order of $z^{d-1}$.
In particular, for $g^{(1)}_{xx} = T_{xx} z^{d-2} + {\cal O}(z^{d-1})$,
we find
\begin{eqnarray}
f= -\frac{dk^3+(2d-1)k}{2(d^2-1)}T_{xx}z^{d+1}  + {\cal O}(z^{d+2}).
\label{fz}
\end{eqnarray}
We note that there could be ${\cal O}(z^{2})$ or ${\cal O}(z^{0})$ terms in $f$
other than the those determined by $g_{xx}^{(1)}$.
However, such terns are forbidden by
the other components of the boundary conditions as we will see below.
Note that  ${\cal O}(z^{0})$ term is not allowed because $f$ should become zero in the limit
$z \rightarrow 0$ because the ETW brane is should be terminated at $x=z=0$.

For the boundary conditions for other components, we obtain
\begin{eqnarray}
zt&:&\frac{1}{\sqrt{1+k^2}}\left(\frac{k^2}{z^2}\partial_t f-\frac{1}{z}\partial_z\partial_t f-\frac{z(1+k^2)}{2}g_{xt,z}^{(1)}-(1+k^2)g_{xt}^{(1)}-\frac{zk}{2}g_{xx,t}^{(1)}\right)=0,\nonumber\\
zw_i&:&\frac{1}{\sqrt{1+k^2}}\left(\frac{k^2}{z^2}\partial_{w_i} f-\frac{1}{z}\partial_z\partial_{w_i} f-\frac{z(1+k^2)}{2}g_{xw_i,z}^{(1)}-(1+k^2)g_{xw_i}^{(1)}-\frac{zk}{2}g_{xx,w_i}^{(1)}\right)=0,\nonumber\\
tt&:&\frac{1}{\sqrt{1+k^2}}\left(-\frac{1}{z^2(1+k^2)}\partial_z f-\frac{1}{z}\partial_t^2 f-\frac{k}{2(1+k^2)}g_{xx}^{(1)}-\frac{zk}{2}g_{tt,z}^{(1)}+\frac{z}{2}g_{tt,x}^{(1)}-kg_{tt}^{(1)}-zg_{xt,t}^{(1)}\right)=0,\nonumber\\
tw_i&:&\frac{1}{\sqrt{1+k^2}}\left(-\frac{1}{z}\partial_{w_i}\partial_t f-\frac{zk}{2}g_{w_it,z}^{(1)}-kg_{tw_i}^{(1)}-\frac{z}{2}(g_{xt,w_i}^{(1)}+g_{xw_i,t}^{(1)}-g_{tw_i,x}^{(1)})\right)=0,\nonumber\\
w_iw_i&:&\!\frac{1}{\sqrt{1+k^2}}\!
\left(\frac{1}{z^2(1+k^2)}\partial_z f-\!\frac{1}{z}\partial_{w_i}^2 f+\!\frac{k}{2(1+k^2)}g_{xx}^{(1)}
\right.
\nonumber\\
&& \!\ -
\left.
\frac{zk}{2}g_{w_iw_i,z}^{(1)}+\!\frac{z}{2}g_{w_iw_i,x}^{(1)}-\!kg_{w_iw_i}^{(1)}-\!zg_{xw_i,w_i}^{(1)}
\right) =0,
\nonumber\\
w_iw_j&:&\frac{1}{\sqrt{1+k^2}}\left(-\frac{1}{z}\partial_{w_i}\partial_{w_j} f-\frac{zk}{2}g_{w_iw_j,z}^{(1)}-kg_{w_iw_j}^{(1)}-\frac{z}{2}(g_{xw_j,w_i}^{(1)}+g_{xw_i,w_j}^{(1)}-g_{w_jw_i,x}^{(1)})\right)=0.
\label{bcf1}
\end{eqnarray}
If $f= \alpha z^2+ \cdots$,
these are dominant in $z \rightarrow 0$
in the above $tt$ component and we find
$-\frac{1}{(1+k^2)} 2 \alpha -\partial_t^2 \alpha z^2=0$, which implies $\alpha=0$

We would like to find the mode which satisfies the boundary conditions.
Note that each mode of the $g_{M N}^{(1)}$ is proportional to the Bessel function
$J_{d/2}(\sqrt{\omega^2 - k^\mu k_\mu } z)$ in the Fefferman-Graham gauge, however, $f$ is completely different from the Bessel function.
Thus, in order to solve the boundary conditions generically we need to
consider linear combinations of infinitely many modes with different $k_\mu$, but same $\omega$.
In principle, there will be only the technical problem to do this, however,
in this paper we leaves this problem open for completion in the future.

Finally, in order to obtain the constraints for the energy-momentum tensor,
we can extract the coefficient of $z^{d-2}$ in the above equations \eqref{bcf1}. Then we obtain
\begin{equation}
T_{xt}=T_{xw_i}=T_{tw_i}=T_{w_iw_j}=0,
\end{equation}
where we define
\begin{equation}
g^{(1)}_{ij} = T_{ij} z^{d-2} + {\cal O}(z^{d-1}).
\end{equation}
For the $tt$ and $w_iw_i$ components we can substitute the form of (\ref{fz}) we find that
\begin{eqnarray}
T_{tt}&=&\frac{1}{d-1}T_{xx},\\
T_{w_iw_i}&=&-\frac{1}{d-1}T_{xx}.
\end{eqnarray}
Therefore we can combine these conditions as
\begin{equation}
    T_{ij}=-\frac{1}{d-1}\eta_{ij}T_{xx},
\end{equation}
where the $\eta_{ij}$ is the Minkowski metric.
We will see what these conditions mean in the next section.

\label{s3}

\section{Constraints of the energy-momentum tensor in BCFT }

In this section, we consider general BCFT in flat space and derive constraints of the energy-momentum tensor of it by considering the coupling to the gravity.
We will also find that the energy-momentum tensor in AdS/BCFT satisfies the constraints.

First, by the definition of the $d$-dimensional BCFT, the boundary conformal symmetry should be kept, namely,
the presence of the boundary breaks the conformal symmetry $SO(2,d)$ to $SO(2,d-1)$, which means
the condition on the following energy-momentum tensor:
\begin{equation}
T_{x a}|_{x=0}=0,
\label{cbc}
\end{equation}
where we put the boundary on the $x=0$ surface and
$a,b,\cdots$ run the transverse directions to the boundary surface $x=0$.
Note that \eqref{cbc} implies
\begin{equation}
\partial_x T_{x x}|_{x=0}=0,
\label{cbc2}
\end{equation}
by the current conservation $\partial^x T_{xx}+\partial^a T_{xa}=0$.
This boundary condition requires that the energy flux
does not spread out from the boundary of the CFT region. This is just
a generalization of the two-dimensional case $(T-\bar{T})|_{x=0}=0$ \cite{Cardy:1984bb}.

Then, we will consider a $d$-dimensional BCFT which can couple to a $d$-dimensional  gravity theory with the boundary
and the necessary conditions for the properties of such a BCFT.
Here, we need to specify the boundary condition of the gravity theory\footnote{
Similar considerations were given in \cite{Prochazka:2018bpb}.}
and we will choose the Neumann boundary condition for simplicity
and later we will consider the conformal Dirichlet boundary condition.\footnote{
The Dirichlet  boundary condition may be inconsistent \cite{Witten:2018lgb}.
There might be other types of consistent boundary conditions, which are not considered in this paper.
}
In order to fix the location of the boundary in the BCFT,
we choose the Gaussian normal coordinate so that the boundary is fixed at $x=0$. Then, the Neumann boundary condition means $\partial_{x}g_{ab}|_{x=0}=0$  and,
using Fourier transformation, the metric can be represented as
\begin{equation}
    \delta g_{ab}=g_{ab} -\eta_{ab} \sim\int dk f_{ab}(k)\cos{(kx)}.
\end{equation}
Thus, in order to couple the gravity consistently,\footnote{
Here, we assume that the nonzero energy momentum tensor should be a source for the gravity. For a system which has a subsystem which
is completely decoupled from the other subsystem and the gravity,
we can consider only the subsystem which couples to the gravity.
We thank M. Shigemori for the useful discussions on this point.
}
the energy-momentum tensor in the BCFT should also satisfy
the Neumann boundary condition,
\begin{equation}
    \partial_xT_{ab} |_{x=0}=0,
\label{cons1}
\end{equation}
because of
the coupling of the energy-momentum tensor to the perturbations of the metric $\delta g$:
 \begin{equation}
\int T_{\mu\nu} \delta g^{\mu\nu}.
\label{eq:60}
 \end{equation}
(The gauge dependence is of course absent by the current conservation.)
Using the traceless condition,
we find that the necessary conditions for energy-momentum tensor in the BCFT which couples to the gravity with the Neumann boundary condition are
\begin{equation}
T_{x a}|_{x=0}=0,  \,\,\,
\partial_xT_{ab} |_{x=0}=0, \,\,\,
\partial_xT_{xx} |_{x=0}=0.
\label{nbc}
\end{equation}
Note that for two dimensional BCFT,
$T_{x a}|_{x=0}=0$ implies others by using the current conservation
$\partial_x T_{xx} -\partial_t T_{tx}=0$ and the traceless condition.

Although a BCFT does not need to satisfy the boundary conditions \eqref{nbc}, we will see that the BCFT which is realized as a free scalar field theory
with a boundary indeed satisfy them.
Let us consider a free scalar field on a half of $\mathbb{R}^{1,d-1}$.
Here we set the boundary at $x=0$ and we impose
the Neumann or Dirichlet boundary conditions at this boundary.
The Lagrangian of the conformally coupled scalar filed
is
\begin{equation}
    L=-\frac{1}{2}\sqrt{-g}\left(g^{\mu\nu}\partial_{\mu}\phi\partial_{\nu}\phi-\xi R \phi^2\right),
\end{equation}
where $\xi=\frac{1}{4}\frac{d-2}{d-1}$. The energy-momentum tensor on general background is given by
\begin{eqnarray}
T_{\mu\nu}&=&\frac{2}{\sqrt{-g}}\frac{\delta S}{\delta g^{\mu\nu}}\nonumber\\
&=&(1-2\xi)\partial_{\mu}\phi\partial_{\nu}\phi+(2\xi-\frac{1}{2})g_{\mu\nu}g^{\rho\sigma}\partial_{\rho}\phi\partial_{\sigma}\phi-2\xi\phi\partial_{\mu}\partial_{\nu}\phi+\frac{2}{d}\xi g_{\mu\nu}\phi\Box\phi\nonumber\\
&-&\xi\left(R_{\mu\nu}-\frac{1}{2}Rg_{\mu\nu}+\frac{2(d-1)}{d}\xi Rg_{\mu\nu}\right)\phi^2.
\end{eqnarray}
For the flat space , we obtain
\begin{equation}
   T_{\mu\nu}=\partial_{\mu}\phi\partial_{\nu}\phi-\frac{1}{2} \eta_{\mu\nu}\partial^{\alpha}\phi\partial_{\alpha}\phi+\frac{d-2}{4(d-1)}\left((\eta_{\mu\nu}\partial^{\alpha}\partial_{\alpha}-\partial_{\mu}\partial_{\nu})\phi^2+(\frac{2}{d}-2) \eta_{\mu\nu}\phi\Box\phi\right).
\end{equation}
The last term proportional to  $\Box\phi$ vanishes by the equations of motion and this is equivalent to the usual improved energy-momentum tensor.
Then, for the Neumann boundary condition $\partial_x \phi|_{x=0}=0$,
which implies $\partial_a \partial_x \phi|_{x=0}=0$, we can easily see that
\eqref{nbc} are satisfied using $\partial^{\alpha}\partial_{\alpha} \phi=0$.
For the Dirichlet boundary condition $\phi|_{x=0}=const.$,
which implies $\partial_a  \phi|_{x=0}=0$, we can easily see\footnote{
Note that $\partial^{x}\partial_{x} \phi |_{x=0} =-\partial^{a}\partial_{a} \phi |_{x=0} =0$.
}
that \eqref{nbc} is satisfied if we require $\phi|_{x=0}=0$ which is obviously needed for
the conformal symmetry.

Next, we will
consider a $d$-dimensional BCFT which can couple to a $d$-dimensional  gravity theory with the conformally Dirichlet boundary \cite{Anderson:2006lqb}.
For this, the metric fluctuations should be $\delta g_{a b} |_{x=0} \sim \eta_{a b}$,
which implies $T_{ab} |_{x=0} \sim \eta_{a b}$.
Thus,
the necessary conditions for energy-momentum tensor in the BCFT which couples to the gravity with the conformally Dirichlet boundary condition are
\begin{equation}
T_{x a}|_{x=0}=0,  \,\,\,
T_{ab} |_{x=0}= - \frac{1}{d-1} \eta_{ab} \, T_{xx}|_{x=0}, \,\,\,
\partial_xT_{xx} |_{x=0}=0.
\label{dbc}
\end{equation}
where we have used the traceless condition.
For the free scalar field theory, we can see that
these conditions are not
satisfied for the Neumann boundary condition,
but are satisfied for the Dirichlet boundary condition.

\subsection{Constraints of the energy-momentum tensor in AdS/BCFT }

We will study the constraints of the energy-momentum tensor in AdS/BCFT,
where the Neumann boundary condition is imposed on the bulk $(d+1)$-dimensional metric,
and will determine which kind of BCFT is realized in AdS/BCFT.

In the Fefferman-Graham gauge
the bulk metric becomes the following form:
\begin{equation}
ds^{2}=\frac{dz^{2}}{z^{2}}+g_{\mu\nu}dx^{\mu}dx^{\nu}.
\end{equation}
In this gauge the bulk metric admits a Taylor series expansion in
the neighborhood of the asymptotic boundary:
\begin{equation}
g_{\mu\nu}=g_{\mu\nu\left(0\right)}+z g_{\mu\nu\left(1\right)}+\cdots+z^{d-2}g_{\mu\nu\left((d-2)/2\right)}+\cdots,
\end{equation}
where the subscripts of the metric represent the order of the $z$ expansion. The boundary stress energy corresponds to the $g_{\mu\nu
\left((d-2)/2\right)}$
, so in terms of $g_{\mu\nu}$ we can rewrite as \cite{Banks:1998dd}:

\begin{equation}
T_{\mu\nu}=\underset{z\rightarrow0}{\lim}z^{-(d-2)}(g_{\mu\nu}-g_{\mu\nu}^{(0)} ), \label{eq:-4}
\end{equation}
where $g_{\mu\nu}^{(0)}$ is the AdS metric and we assumed
there is no source term.

Then, for the tensionless ETW brane case, $T=0$, from the mode expansion \eqref{mode2},
we find $T_{\mu \nu}$ in the BCFT should satisfy \eqref{nbc} at the boundary $x=0$.\footnote{
In \cite{Suzuki:2022xwv} the authors mentioned the condition $T_{xi}=0$ in AdS/BCFT. }
This can be seen also from the linearized boundary condition (\ref{eq:21})-(\ref{eq:25}) directly.
Note that near $z=0$ any metric should be close to the AdS metric,
thus we can use the linearized analysis.
Thus, the BCFT${}_d$ realized in the AdS/BCFT with the tensionless ETW brane
can couple to the $d$-dimensional gravity with Neumann boundary condition.

For the nonzero tension ETW brane case, $T \neq 0$, we derive the constraints in subsection \ref{s3}, which are given by
\begin{equation}
    T_{ij}=-\frac{1}{d-1}\eta_{ij}T_{xx}.
\end{equation}
Thus, the BCFT${}_d$ realized in the AdS/BCFT with the nonzero tension ETW brane
can couple to the $d$-dimensional gravity with conformally Dirichlet boundary condition.
It will be a future problem to resolve why the Neumann like boundary condition in the AdS reduces to the conformally Dirichlet boundary condition in the BCFT.

 \section{Conclusion and Discussions}

In this paper we consider the AdS/BCFT correspondence for the pure gravity, especially in the vacuum AdS space. We analyze the perturbation of the metric and the boundary condition on the ETW brane in the sourceless case.
In the tensionless case we find that the position of the ETW brane
is fixed at the original hyperplane ($x=0$) in the FG gauge.
We confirm that after imposing these boundary conditions the perturbation of the metric survives in the tensionless case. We also consider the boundary condition of the energy-momentum tensor in BCFT side and we find nontrivial constraints.
These conditions are necessary to define the BCFT which can  couple to a dynamical gravity with the Neumann or the conformally Dirichlet boundary conditions on the metric.
We find that the BCFT${}_d$ realized in the AdS/BCFT with the tensionless ETW brane
can couple to the $d$-dimensional gravity with Neumann boundary condition.
For the AdS/BCFT with the nonzero tension ETW brane,
we find that
the BCFT${}_d$
can couple to the $d$-dimensional gravity with conformally Dirichlet boundary condition.

\section*{Acknowledgement}
YS thank A.Ishibashi, Y.Okuyama, T.Ugajin and Z.Wei for helpful discussions. We also thank K.Izumi, T.Kawamoto, Y.Kusuki, T.Shiromizu, K.Suzuki, T.Takayanagi, N.Tanahashi, T.Tsuda and Y.Wang for giving useful comments on the draft of this paper and Y.Sato for  correcting our manuscript.
This work was supported by JSPS KAKENHI Grant Number 17K05414.
This work was supported by MEXT-JSPS Grant-in-Aid for Transformative Research Areas (A) ``Extreme Universe'', No. 21H05184.