\section{Introduction}
Many model architectures have been proposed for image classification with increasing levels of performance~\cite{krizhevsky2012imagenet,simonyan2014very,he2016deep,szegedy2014going,huang2017densely}. These advances have been fueled partially by the availability of common benchmarks that have allowed researchers to quickly iterate over model designs~\cite{lecun1998gradient,krizhevsky2009cifar,deng2009imagenet}. While the Imagenet dataset has brought significant advances, more compact datasets such as CIFAR-10 have allowed the quick exploration of novel ideas that are later validated in larger scale datasets. We aim to capture the simplicity of CIFAR-10 but for object segmentation. Current datasets for segmentation often require models that solve simultaneously object localization and object recognition on top of predicting segmentation masks (e.g.~Mask-RCNN~\cite{he2017mask}). We posit that as a result there is considerably less work in creating specialized segmentation networks that are solely designed to produce high quality segmentation masks (e.g.~U-Net~\cite{ronneberger2015u}).

\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.84\linewidth]{figures/ChairSeg.pdf}
\end{center}
   \caption{Images in the \ChairSegments dataset are created semi-synthetically by relying on images of chairs with transparent backgrounds and compositing them on background images to encourage diversity, and reasonably challenging testing conditions. Unlike other datasets, ground truth segmentation masks are pixel-level accurate as they are not hand annotated.}
\vspace{-0.2in}
\label{fig:lead-figure}
\end{figure}

In this work, we introduce Chair Segments, a semi-synthetic dataset that is compact, reasonably challenging, and has pixel-accurate ground truth segmentations (Figure~\ref{fig:lead-figure}). We focus on chairs, because this object is challenging both structurally and semantically. Chairs have also long interested perceptual psychologists and computer vision scientists because recognizing them requires cues that go beyond shapes and colors but also need to account for affordances~\cite{lloyd1997categorizing,lupyan2008chair,grabner2011makes,ordonez2014furniture,Polania_2020_CVPR_Workshops}, i.e. chairs are defined by their functionality.  In terms of structure chairs have elongated and complex shapes that could be hard to group.
Moreover, we show that complex models such as U-Net can be reliably trained to full convergence in under an hour using a single GPU. We also demonstrate that these models are reliably modeling real-world data by showing state-of-the-art accuracy using transfer learning on the Object Discovery dataset (airplanes, horses, and cars)~\cite{Rubinstein13Unsupervised}. Finally, we use Chair Segments to replicate findings in the image classification domain where it was shown that models fine-tuned from a set of pretrained weights lie in the same basin of the optimization landscape~\cite{neyshabur2020being}.

Collecting data for image segmentation is considerably costlier than for image classification. Accurate segmentation masks are laborious to annotate even for a single image. Many prior works rely on clicking to obtain object segmentations, where annotators approximate segment masks with a polygon. In our work, we rely on images with transparent backgrounds which are commonly used for e-commerce websites to display shopping items. These images are often captured using a chroma-key backdrop in order to easily and accurately separate the object from its background. By using these type of images we are able to collect thousands of object foregrounds with pixel accurate masks.
We generate a dataset with these chair segments by compositing them on a diverse array of background images of scenes, making sure that neither segments nor backgrounds have any overlap between training, validation and testing. A sample of images in this benchmark are shown in Figure~\ref{fig:chair_grid}.

\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.88\linewidth]{figures/chair_grid.pdf}
\end{center}
   \caption{Sample images from the \ChairSegments dataset. Neither foregrounds nor backgrounds are shared across training, validation and test splits.}
   \vspace{-0.2in}
\label{fig:chair_grid}
\end{figure}

Our paper also serves to validate some of the findings of Neyshabur et al~\cite{neyshabur2020being} in transfer learning but for image segmentation. Including that models fine-tuned under the same initial pretrained model lie closer in parameter space than models trained from a random set of parameters, and that those fine-tuned models also fall within the same optimization basin compared to models trained from random initialization. As in image classification, fine-tuning also seems to bring benefits both in accuracy through feature reuse and convergence rates due to good initialization.

Our contributions can be summarized as follows:

\begin{compactitem}
\vspace{0.02in}
\item We introduce \ChairSegments, a semi-synthetic dataset that can be used to benchmark and quickly develop neural network architectures for object segmentation.

\vspace{0.02in}
\item Results showing that \ChairSegments is a good source of pretraining for real images by showing state-of-the-art results on the Object Discovery dataset~\cite{Rubinstein13Unsupervised}.

\vspace{0.02in}
\item Analysis of transfer learning confirming previous empirical results on transfer learning for image classification, signaling that gains in transfer learning for classification area are likely to be beneficial for image segmentation as well.

\end{compactitem}

Our paper is organized as follows: Section~\ref{sec:related} discusses related segmentation datasets and use of synthetic datasets in various applications, Section~\ref{sec:dataset} discusses the construction of Chair Segments, Section~\ref{section:analysis} states the main claims of our work with respect to our provided resource and our findings on transfer learning for segmentation, Section~\ref{section:experiments} presents our experiments on using Chair Segments and initial benchmarks, Section~\ref{section:analysistransfer} discusses our findings in transfer learning analysis, and finally Section~\ref{sec:conclusion} concludes our paper.

\section{Related Work}
\label{sec:related}

Our work is related to many previous efforts in collecting datasets for segmentation, and the use of semi-synthetic data for the development of image segmentation models.

\vspace{0.02in}
\noindent{\bf Compact Datasets} Compact datasets such as CIFAR-10 and CIFAR-100 \cite{krizhevsky2009cifar} have been heavily used to develop novel architectures for image classification. We argue that such equivalent dataset is missing for the task of image segmentation. As a result, many developments in models for segmentation have happened as separate efforts e.g. U-Net~\cite{ronneberger2015u} in the medical image domain, and fully convolutional networks (FCNs)~\cite{long2015fully} in semantic segmentation. We extensively test FCNs and U-Net models in our proposed \ChairSegments benchmark and confirm their relative strength as evidenced in larger scale benchmarks.

\vspace{0.02in}
\noindent{\bf Semantic Segmentation Datasets} There have been many datasets proposed in the past for semantic segmentation. These datasets usually extend the task of object detection to also include masks, such as the PASCAL VOC Dataset~\cite{everingham2010pascal}, the LabelMe Dataset~\cite{russell2008labelme}, and more recently the COCO Dataset~\cite{lin2014microsoft}, the KITTI dataset\cite{geiger2013vision}, and the ADE20K Dataset~\cite{zhou2017scene}, among others. Unlike these datasets \ChairSegments is built to be compact and pixel-accurate, and does not require to solve the semantic labeling task together with the pixel grouping task. This will allow researchers to concentrate in obtaining good grouping for a specific category while bypassing the need to solve together other problems such as the long-tail problem in segmentation datasets. Moreover, while Chair Segments is smaller in scale, due to its conception, it contains ground truth masks that are of considerably higher quality than existing datasets (see Figure~\ref{fig:chaircomparison}). Other compact datasets such as ObjectDiscovery~\cite{Rubinstein13Unsupervised} with excellent pixel accuracy are not large enough to allow for pretraining and explore deeply feature representation learning for segmentation. There are other datasets in specialized domains such as the INRIA Aerial Image Labeling dataset~\cite{maggiori2017dataset} in remote sensing, and the EM segmentation challenge in medical imaging~\cite{arganda2015crowdsourcing}.

\begin{figure}[t!]
\begin{center}
   \includegraphics[width=0.8\linewidth]{figures/ChairsComparisonv2.pdf}
\end{center}
\vspace{-0.1in}
   \caption{Images in the Chair Segments dataset are composites but have high resolution masks (a), while other datasets such as COCO Segments use polygon annotations which do not strictly follow object boundaries (b). Chair Segments allows designing models optimized for mask quality and are thus complementary.}
   \vspace{-0.15in}
\label{fig:chaircomparison}
\end{figure}

\vspace{0.02in}
\noindent{\bf Synthetic Datasets} There have been several previous efforts to use synthetic data to bypass the need to manually annotate images. For instance, dense vector field annotations for benchmarking optical flow algorithms are almost infeasible to be annotated by hand. Datasets such as the Flying Chairs dataset~\cite{dosovitskiy2015flownet} consisting of synthetic scenes, have enabled great progress in this area. For image segmentation, the SAIL-VOS Dataset used a graphics engine from a videogame to render scenes with segmentations~\cite{hu2019sail}. Unlike these datasets based on computer generated imagery, we propose composites of images with transparent backgrounds as supported by the portable network graphics format (PNG) which is widely used on the web to advertise products in image catalogs. Closest to our approach is the previous work of Jin~et~al~\cite{jin2017webly} that queried images on a search engine for various objects with white backgrounds.

\section{Data Collection: \ChairSegments}
\label{sec:dataset}

Our objective was to collect a compact dataset where model architectures can be tested in a short amount of time, while maintaining a reasonable level of difficulty. We also wanted to isolate the task of segmentation and grouping from issues of categorization. However, we do acknowledge that pixel grouping is a holistic process that requires both low-level features and top-down reasoning~\cite{arbelaez2009contours}. As a result, a generic segmentation model that is category independent is unlikely to be realized, therefore we aimed to collect a category specific segmentation resource that is instead diverse in terms of shapes and challenging in terms of visual complexity. Chairs have challenging and highly intricate shapes for image segmentation, as they often contain thin and elongated parts, hollow segments, and self-occluding parts. Moreover chairs are the hardest categories in current benchmarks such as the Pascal VOC segmentation task\footnote{Pascal VOC Segmentation task. Retrieved on November 2020. \url{http://host.robots.ox.ac.uk:8080/leaderboard/displaylb_main.php?challengeid=11&compid=6}.} where the top performer achieves an average overall precision of $90.5$ but only $57.5$ precision for the chair category.

We queried images using the Google search engine and restrict the search results to only images with transparent backgrounds. In order to ensure diversity we issue queries for chairs by relying on pairs of words from the LEVAN ontology~\cite{divvala2014learning} which was compiled in a data driven fashion. Queries included things such as ``arm chair'', ``accent chair'', ``swivel chair'', ``office chair'', ``dining chair'', etc. Then we manually filtered out images that did not contain transparent backgrounds, did not contain relevant images, or contained more than one chair in the same image. Most of the remaining chairs have masks that are of the highest accuracy as these are often used in e-commerce and are captured in controlled conditions with chroma-key backdrops and/or multiple camera settings for clean and effective background removal.
Using this process we ended up with $900$ chair segments by dropping the pixels indicated by the transparent background.  Similarly, we selected a random subset of $10,000$ background scene images following a similar protocol as in the Houzz Dataset~\cite{poursaeed2018vision} but also ensuring diversity across different types of indoor and outdoor scenes in a house environment. In this way, we encourage our foregrounds and backgrounds to be somewhat semantically consistent. Our final dataset will be released publicly along with code to replicate our current benchmarks.

We split the chair segments into $500$ for the training split, $200$ for the validation split, and $200$ for the testing split. Similarly, we split the background images into three groups containing 60\% for the training split, 20\% for the validation split and 20\% for the testing split. Then we create composites for the training set, by pairing each of the $500$ chair segments in this split with $100$ randomly selected backgrounds in the training split for every chair, resulting in a total of $50,000$ image composites for the training set. We follow a similar process to generate composite images for the validation and testing set, to obtain $20,000$ for each of them.

We generate composites at several resolutions by first resizing the background image into a target $d\times d$ resolution, and then alpha blending (with the transparent background mask) with a given chair segment that is first resized to a $(d-k)\times (d-k)$ resolution where $k$ is a padding size that is left around the chair segment. We create several versions of our dataset at various resolutions: $32\times32$, $64\times64$, $128\times128$, and $256\times256$, using padding sizes of $3$, $6$, $13$, and $26$ pixels respectively for each resolution.

\section{Main Claims}
\label{section:analysis}
In the rest of the paper we aim to demonstrate three main claims related to the proposed resource and transfer learning in image segmentation more broadly:
\vspace{0.02in}
\begin{compactitem}
    \vspace{0.02in}
    \item \ChairSegments is challenging while allowing fast benchmarking (in a few hours) for modern neural network architectures designed for image segmentation such as U-Net~\cite{ronneberger2015u} and FCNs~\cite{long2015fully}.
    \vspace{0.02in}
    \item \ChairSegments despite being semi-synthetic, represents well the type of learning required for real datasets as evidence by state-of-the-art results on the Object Discovery Dataset~\cite{Rubinstein13Unsupervised} through transfer learning. We also compare pretraining on chair images from the existing COCO dataset vs pretraining on our proposed Chair Segments dataset with favorable results.
    \vspace{0.02in}
    \item Transfer learning in image segmentation exhibits similar properties to those recently found in the transfer learning process for image classification~\cite{neyshabur2020being}.
\end{compactitem}
\vspace{0.03in}
\begin{table*}[t]
\begin{center}
\begin{tabular}{l|cc|cc|cc|cc }
\hline
& \multicolumn{8}{c}{Resolution}\\
 & \multicolumn{2}{c}{32$\times$32}& \multicolumn{2}{c}{64$\times$64}& \multicolumn{2}{c}{128$\times$128} & \multicolumn{2}{c}{256$\times$256}\\
\hline
Model & time   & IoU & time   & IoU  & time   & IoU  & time  &  IoU \\ \hline

U-Net & 0.4 &  77.59 & 0.5 &  85.08 & 1 &  89.79 & 6.4  &91.68\\
FCN-VGG-16 & 2.1  & 60.75 & 2.7   & 61.09  & 3.6  & 74.55 & 9.1   & 68.38\\
FCN-ResNet-50 & 5.5   & 53.59 & 19.3   & 60.19 & 10.1  & 70.94  & 33   & 82.38\\

\hline
\end{tabular}
\end{center}
\caption{This table shows the time in hours  and the number of epochs that each model required to converge on \ChairSegments under various resolutions. For instance, a model such as U-Net can be trained to convergence in about $30$ minutes on images at a $64\times64$ resolution.}
\label{table:timings}
\vspace{-0.1in}
\end{table*}
The first claim is important as a trivial dataset could be potentially limited in its lifetime. However at this point it is worth noting that even datasets such as MNIST~\cite{lecun1998gradient} have withstood the test of time despite being considered largely ``solved'' -- where many modern approaches can reach accuracies close to the maximum possible. That said, we do show that Chair Segments is reasonably challenging with the top performing model obtaining an IoU of $85.08$ at our target standard resolution of $64 \times 64$ pixels. We further demonstrate the validity and usefulness of Chair Segments by showing that models that are more performant on large scale benchmarks are also more performant on Chair Segments (e.g. U-Net outperforms vanilla FCNs).

The second claim is demonstrated by showing that by simply fine-tuning models pretrained on Chair Segments for a smaller dataset of real images. We demonstrate this by fine-tuning models on the Object Discovery Dataset~\cite{Rubinstein13Unsupervised} which contains three categories unrelated to chairs: horses, cars, and airplanes. A fine-tuned U-Net model yields state-of-the-art results on all metrics.

The third claim is demonstrated by replicating the experiments from Neyshabur~et~al~\cite{neyshabur2020being} that demonstrate that under transfer learning, models tuned on a smaller dataset share certain properties such as fine-tuned models lying in the same optimization basin despite being trained under stochastic gradient descent (SGD). Our work is the first to empirically demonstrate these properties for transfer learning in image segmentation.

\section{Experiments}
\label{section:experiments}

We consider two approaches for image segmentation resulting in four different models. The first is the fully-convolutional network (FCN) approach~\cite{long2015fully} consisting of standard neural network architectures for classification where the last layers are replaced with an upsampling layer that produces a 2D output instead of classification scores. The second approach we consider is U-Net~\cite{ronneberger2015u}, consisting of encoder-decoder modules containing multiple downsampling and upsampling operators respectively with skip-connections to preserve high-resolution information across the network which has proved essential for segmentation. Our implementation of U-Net also relies on zero padding in order to be used with low resolution inputs. For the FCN approach we consider three variants leveraging various neural network architecture backbones, VGG-16~\cite{simonyan2014very}, ResNet-50, and ResNet-101~\cite{he2016deep}. We denote the resulting four models in our experiments as: FCN-VGG-16, FCN-ResNet-50, FCN-ResNet-101, and U-Net.

We use three standard metrics for image segmentation in our experiments:
\textbf{Precision} also known as per-pixel accuracy, which simply consists in the percentage of pixels in the image that were classified correctly; \textbf{Jaccard Index} also known as Intersection-Over-Union (IoU), which is the area of overlap between the predicted segmentation and the ground truth divided by the area of their union; and \textbf{Dice Coefficient}, also known as F1 score, which can be computed as $2\times$ the area of overlap divided by the total number of pixels between the ground truth segmentation and the segmentation predicted by the model. These are all three widely used metrics in this area.

\subsection{Training Time Trade-Offs}
First, we measure how quickly can we train to full convergence some of our models in the Chair Segments dataset and whether they are able to achieve a reasonable accuracy. We train U-Net, FCN-VGG-16 and FCN-ResNet-50 models on the \ChairSegments dataset for all resolutions and train until convergence by early stopping if the IoU metric has not changed in the past few training steps. We show results on Table~\ref{table:timings} including time to convergence and the maximum IoU score achieved.

We particularly found that U-Net can reach full convergence in four epochs in about $30$ minutes for the $64 \times 64$ resolution, and in four epochs in about $1$ hour for the $128 \times 128$ resolution. Based on these results, we decided to use the $64\times64$ resolution for the remaining experiments on the paper and this will be the recommended resolution to be used for future benchmarking. Note that this is only four times the resolution of images in the CIFAR-10 dataset that are scaled at a resolution of $32 \times 32$.

\begin{table}[b]
\begin{center}
\begin{tabular}{l|c|c|c}

\hline
Model & Prec. & IoU  & Dice \\ \hline

U-Net & 97.18 & 85.08 & 91.25 \\

FCN-VGG-16
& 91.73 & 61.09  & 74.09\\

FCN-ResNet-50
& 92.04 & 60.19  & 72.58\\

FCN-ResNet-101
 & 92.19 & 61.62 & 73.96 \\ \hline

\end{tabular}
\end{center}
\caption{Benchmarking results of different models on the \ChairSegments dataset. U-Net is the top performing method while fully convolutional networks (FCNs) trail in all metrics depending on the feature backbone used.}
\label{table:chairsegments}
\end{table}

\subsection{Benchmarking on \ChairSegments}
We benchmark all four models on \ChairSegments at the fixed resolution of $64 \times 64$. For FCN-VGG-16 we train with a learning rate of 1e-4, using standard stochastic gradient descent and a per pixel binary cross entropy loss. For the FCN-ResNet-50 and FCN-ResNet-101 models, we use a learning rate of 1e-5,  with an RMSprop optimizer, momentum of $0.9$, weight decay of 1e-6, and a per pixel binary cross entropy loss. For the U-Net model, we use a learning rate of 1e-3 with an Adam optimizer, and a per pixel binary cross entropy loss.

\begin{figure*}[t!]
\begin{center}

   \includegraphics[width=0.9\linewidth]{figures/Testdataset2.pdf}
\end{center}
   \caption{Benchmark of common generic segmentation backbones used in the literature. Results on the \ChairSegments dataset show that under several resolutions, the relative strength of these methods is preserved. We envision \ChairSegments to play the role of the CIFAR-10 dataset for quick development of algorithms for segmentation.}
\label{fig:test}
\end{figure*}

\vspace{0.02in}
\noindent \textbf{Results.} Table~\ref{table:chairsegments} shows the results for all these models across all metrics. We observe that U-Net is the top performing model by a considerable margin, followed by FCN-ResNet-101, FCN-ResNet-50 and FCN-VGG-16. Thus, demonstrating that our benchmark represents well the difficulties of larger benchmarks with respect to top performing models for image segmentation. Figure~\ref{fig:test} shows example results for these models along with the input images and ground truth segmentation masks. We observe that FCNs coupled with classification networks seem to be learning the right masks but they fail to capture finer level details as these networks were mostly designed for image classification. This is especially clear for the ResNet networks which due to their depth they inherently tend to lose high-resolution information along their computation path and multiple steps of downsampling. On the other hand, U-Net having multiple downsample and upsample steps coupled with skip connections manages to retain high-resolution information much better than the other models. This implementation of U-Net also avoids boundary issues through zero padding.

\vspace{0.04in}
\noindent \textbf{Is Chair Segments solved?} The top performing U-Net model achieves a precision of $97.18$, however a zero-knowledge baseline that blindly assigns every pixel as background would achieve a precision of $78.01$ on this dataset. The IoU metric (intersection over union) is therefore much more informative and even the best model achieves $85.08$ in this score. Models that are not able to generate precise high resolution masks lag considerably behind on this metric. We consider that models achieving IoUs squarely above $90$ will be an open area of investigation and this benchmark is unlikely to saturate soon. We expect that this leaves room for future improvements both in terms of stronger model architectures for segmentation and faster models for real time deployment.

\subsection{Fine-tuning on the Object Discovery Dataset}

\begin{table*}[h]
\begin{center}
\begin{tabular}{l|c|ccc|ccc|ccc}

\textbf{} &Pretraining &\multicolumn{3}{c}{{Car}} & \multicolumn{3}{|c|}{{Horse}} & \multicolumn{3}{c}{{Airplane}}\\

Model &  Dataset & {Prec.}  & {IoU} & {Dice} & {Prec.} & {IoU}  & {Dice}  & {Prec.} & {IoU} & {Dice}\\ \hline

FCN-VGG-16 &
--& 78.70 & 46.47 & 62.27
& 80.54 & 36.19 & 51.57
& 81.31 & 36.38 & 52.23\\

FCN-ResNet-50 &
--& 79.97 & 35.03  & 45.13
& 86.46 & 26.44  & 36.79
& 88.48 & 20.84  & 28.72\\

FCN-ResNet-101 &
-- & 94.57 & 82.73 & 88.99
& 91.44 & 57.68 & 70.03
& 93.12 & 60.64 & 73.08  \\

U-Net &
-- & 95.70 & 86.16 & 91.72
& 93.52 & 66.41 & 74.99
& \textbf{94.57} & 64.93 & 73.99 \\ \hline

FCN-VGG-16  &
Chair Segments & 83.80 & 51.08 & 64.14
& 85.69 & 33.33 & 46.37
& 87.27 & 33.89 & 47.30\\

FCN-ResNet-50 &
Chair Segments& 85.59 & 53.61  & 63.44
& 89.26 & 41.08 & 53.61
& 90.79 & 38.82 & 50.93\\

FCN-ResNet-101 &
Chair Segments & 93.05 & 81.34 & 87.77
& 90.38 & 57.73 & 70.28
& 91.60 & 51.54 & 51.54\\

\hline

U-Net &
COCO Chairs & 95.96 & 87.17 & 91.51
& 94.20 & 68.61 & 77.43
& 94.19 & 68.05 & 77.53\\

U-Net &
Chair Segments& \textbf{96.43} & \textbf{88.22} & \textbf{92.23}
& \textbf{94.46} & \textbf{70.47} & \textbf{78.47}
& 94.49 & \textbf{71.59} & \textbf{80.42}\\

\end{tabular}
\end{center}
\caption{Pretraining on \ChairSegments demonstrates a clear advantage overall and almost across all metrics in all subsets.}
\label{table:obj-discovery-results}
\end{table*}

\begin{table*}[h]
\begin{center}
\begin{tabular}{l|cc|cc|cc}
\hlin
\textbf{} & \multicolumn{2}{|c|}{{Car}} & \multicolumn{2}{c|}{{Horse}} & \multicolumn{2}{c}{{Airplane}}\\

Model & Prec.  & IoU & Prec. & IoU   & Prec. & IoU \\ \hline

Rubinstein et al.\cite{Rubinstein13Unsupervised}
& 85.4 & 0.64
& 82.8 & 0.52
& 88.0 & 0.56 \\

Quan et al.\cite{Quan_2016_CVPR}
& 88.5 & 0.67
& 89.3 & 0.58
& 91.0 & 0.56  \\

Jerripothula et al.\cite{jerripothula2016image}
& 88.0 & 0.71
& 88.3 & 0.61
& 90.5 & 0.61  \\

Yuan et al.\cite{yuan2017deep}
& 90.4 & 0.72
& 90.2 & 0.65
& 92.6  & 0.66 \\

U-Net
& 95.7 & 0.86
& 93.5 & 0.66
& \textbf{94.6} & 0.65\\

U-Net (w/ pretraining)
& \textbf{96.4} & \textbf{0.88}
& \textbf{94.5} & \textbf{0.71}
& 94.5 & \textbf{0.72} \\ \hline

\end{tabular}
\end{center}
\caption{U-Net pretrained on \ChairSegments obtains the state-of-the-art in the Object Discovery dataset. }
\label{table:object-discovery-state-of-the-art}
\end{table*}

The Object Discovery dataset introduced by Rubinstein~et~al~\cite{Rubinstein13Unsupervised} for object segmentation contains images with ground truth masks for three types of objects: cars, horses and airplanes. This dataset although small, has been used over the years to test from handcrafted methods based on energy-based minimization and graphical models, to more recent neural network architectures for segmentation. We train our four considered models on this dataset in two conditions: from randomly initialized weights, and from weights pretrained on \ChairSegments. Additionally, we train the best model (U-Net) with chair segments obtained from the COCO Segments dataset (COCO Chairs) instead of our dataset. Our goal is to measure whether \ChairSegments provides useful transfer learning information for this task.

For FCN-VGG-16 we train with a learning rate of 1e-4, an SGD optimizer with per pixel binary cross entropy loss.
For the FCN-ResNet-50 model we use a learning rate of 1e-4, an SGD optimizer with momentum of 0.9, weight decay of 1e-5, and a per pixel binary cross entropy loss.
For the FCN-ResNet-101 model we use a learning rate of 1e-5, an RMSprop optimizer with a momentum of $0.9$, weight decay of 1e-6, and a per pixel binary cross entropy loss.
Finally, for U-Net we use a learning rate of 1e-3, and an Adam optimizer with a per pixel binary cross entropy loss. These hyperparameters were determined using the validation split through standard hyperparameter tuning.

\vspace{0.04in}
\noindent \textbf{Results and Discussion.} Table~\ref{table:obj-discovery-results} shows results for all the models both when training from scratch (from randomly initialized weights), and when fine-tuning from a model pretrained on the \ChairSegments dataset. We additionally show results for U-Net when pretrained on COCO Chairs. We observe as in the \ChairSegments dataset, that U-Net is the superior model across all experiments, and fine-tuning helps in almost all subsets of this dataset under all metrics, and often by a considerable margin especially on the IoU metric which is the most important for segmentation quality. Moreover, U-Net pretrained on the \ChairSegments dataset outperforms U-Net pretrained on COCO Chairs, further demonstrating that \ChairSegments is a valuable resource and is able to represent well a segmentation task with real-world images despite being semi-synthetic.

Table~\ref{table:object-discovery-state-of-the-art} shows our top performing models U-Net and U-Net pretrained on \ChairSegments when compared against all previous methods in the research literature, establishing these two models as the state-of-the-art under this benchmark. Figure~\ref{fig:test2} shows qualitative results on this benchmark for a few examples along with results released with this dataset~\cite{Rubinstein13Unsupervised} using a hand-crafted approach. We can see that FCN-ResNet-101 still struggles with finer-level details while U-Net is able to produce masks at a much higher resolution. However, even U-Net seems to be assigning some pixels as foreground when they are clearly disconnected or dissimilar from the object, which leaves room for future improvement in model designs that can better preserve finer-level structures.

\begin{figure*}[t]
\begin{center}
   \includegraphics[width=0.88\linewidth]{figures/ObjDiscovery_finetuning.pdf}
\end{center}
   \caption{Qualitative results for segmentation models on the Object Discovery dataset, along with their ground truth segmentation masks.}
   \vspace{-0.1in}
\label{fig:test2}
\end{figure*}

\section{Transfer Learning Analysis}
\label{section:analysistransfer}
There has been recent work in image classification to analyze the extent to which transfer learning helps in tasks that have access to less amounts of training data~\cite{ngiam2018domain,he2019rethinking,kornblith2019better,neyshabur2020being}. More specifically, there is a growing interest to analyze to what extent does transfer learning help with providing a better starting point for optimization and to what extent feature reuse is the driving factor. We adopt the recently proposed framework of Neyshabur~et~al~\cite{neyshabur2020being} which allows us to directly draw analogies between transfer learning in image classification and transfer learning in segmentation.

We show in Figure~\ref{fig:finetuning-test} a plot averaging two runs of our U-Net model training from scratch, and fine-tuned from pretrained weights across training steps until convergence. We can observe that not only the pretrained model is reaching a higher IoU score but it also reaches a value close to convergence much faster than the same network trained from random parameters. Hence we can see that both feature reuse and good initialization are driving factors -- confirming results obtained for image classification. This result might seem trivial but we would like to emphasize the discrepancy in the source and target domains, where the source domain only contains chair images and chair segments, while the target domain only contains, horses, cars and airplanes.

\begin{figure}[t!]
\begin{center}
   \includegraphics[width=0.74\linewidth]{figures/iou_finetuned_vs_randominit.pdf}
\end{center}
\vspace{-0.1in}
   \caption{Fine-tuning from a pretrained model on \ChairSegments vs training from scratch using randomly initialized weights averaged across epochs and two runs. \ChairSegments demonstrates gains on a real dataset --ObjectDiscovery-- despite it being a semi-synthetic dataset.}
\label{fig:finetuning-test}
\end{figure}

There are two additional findings that we present in this section: (1) Analyzing how far apart are comparable pretrained models and models trained from scratch in terms of $\ell_2$ distances, and (2) Exploring the linear interpolation space between models trained from pretrained weights, and models trained from random initialization. In both cases, we found that the results we obtain confirm those obtained for image classification in Neyshabur~et~al~\cite{neyshabur2020being}. These results could lead to better algorithms for initializing neural networks for segmentation and for optimizing ensembles of models that lie within close areas in the optimization landscape.

\begin{table}[h]
\begin{center}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{c|c|c|c|c}
\hline
Model & Ft \& Ft & Ri \& Ri & Ft \& P & Ri \& P \\ \hline

Classification~\cite{neyshabur2020being} & 193.45 & 815.08 & 164.83 & 796.80\\

Segmentation (ours)
& 373.94 & 637.05  & 313.06 & 549.94\\ \hline

\end{tabular}
\end{center}
\caption{We examine the $\ell_2$ distance between a pretrained model (P), fine-tuned models (Ft), and models trained from random initialization (Ri). Following~\cite{neyshabur2020being}, we find that two models trained from scratch lie further apart than two models fine-tuned from a common pretrained model, confirming their findings but for image segmentation. In this experiment, model P is pretrained on \ChairSegments and fine-tuned on ObjectDiscovery. }
\label{table:distances}
\vspace{-0.1in}
\end{table}

\subsection{Analyzing Model Distances in Parameter Space}
Table~\ref{table:distances} shows $\ell_2$ distances in parameter space between two models fine-tuned from the same set of pretrained weights (Ft \& Ft) for both image classification and image segmentation (first column), similarly we compute $\ell_2$ distances between two models trained from randomly initialized weights (Ri \& Ri) for both tasks (second column), and between a model fine-tuned from pretrained weights and the pretrained network itself (Ft \& P) for both tasks (column 3), and between a network trained from randomly initialized weights and a pretrained network (Ri \& P) (last column). For all segmentation models we used the U-Net neural network architecture.

The results are strikingly similar as those obtained for image classification by~\cite{neyshabur2020being}, suggesting that improved methods for transfer learning in image classification are likely to lead to gains in image segmentation as well. And more importantly for us, that our fine-tuned models behave similarly as models fine-tuned from Imagenet pretrained models for image classification on a smaller dataset -- which suggests that \ChairSegments is useful as a proxy for real world data. To summarize these results, it seems models that are trained from the same initialization from pretrained weights end up lying closer in parameters space, than models that started from random weights instead. As a sanity check, unsurprisingly, models that are fintuned, are closer in parameter space from their starting point pretrained network (Ft \& P) compared to models trained from scratch (Ri \& P).

\subsection{Analyzing the Loss Landscape of Models}
Finally, we analyze the behavior of our pair of models (Ft \& Ft), both trained from the same checkpoint $P$, we shall denote them more formally here $F_1$ and $F_2$, and the two models trained from randomly initialized weights (Ri \& Ri), we shall denote them more formally here as $S_1$ and $S_2$. As in the previous section, for all models we used the U-Net neural network architecture.

Let $W_1$ and $W_2$ be two different set of weights for models $F_1$ and $F_2$, then we can define a new model $F_\alpha$ that is a model with a set of parameters $W_\alpha = \alpha W_1 + (1 - \alpha) W_2$, where $\alpha$ is an interpolation scalar parameter betwen 0 and 1. Therefore $F_\alpha$ is a model that lies in the linear interpolation space between $F_1$ and $F_2$. We define in the same way a model $S_\alpha$ with respect to $S_1$ and $S_2$. The question we are trying to assess is: Are models that lie in between these two models found through stochastic gradient descent any good? If the loss landscape where the two models ($S_1$, $S_2$) or ($F_1$, $F_2$) lie within a flat landscape, then the transition must be smooth and intermediate models should be performant. We found this to be the case only for the pair of models $(F_1, F_2)$ as evidence in our Figure~\ref{fig:testxx} where we can clearly observe that randomly initialized models quickly degrade when moving away from the starting points, while fine-tuned models transition smoothly without loss in IoU scores. These results confirm the results in transfer learning obtained for Imagenet pretrained networks on transfer learning as evidenced in~\cite{neyshabur2020being}.

\begin{figure}[h!]
\begin{center}
\vspace{-0.1in}
   \includegraphics[width=0.7\linewidth]{figures/basin_figure_trimmed.pdf}
\end{center}
\vspace{-0.1in}
   \caption{We show that two models that are fine-tuned from the same pretrained model lie within the same optimization basin with respect to a linear interpolation, while the same is not true for two models trained from scratch.}
   \vspace{-0.1in}
\label{fig:testxx}
\end{figure}

\section{Conclusions}
\label{sec:conclusion}

We can conclude that our \ChairSegments benchmark is a useful dataset that presents consistent results over different image segmentation methods and can be a useful resource to develop new models and algorithms for image segmentation that can later be applied to semantic image segmentation or semantic instance segmentation. Also, our study of transfer learning over image segmentation give us the opportunity for a better understanding of how deep neural networks behave when fine-tuned for a target task with more constrained numbers of training samples -- mirroring results observed in the more well studied problem of image classification. As a result of our experiments we are planning to apply variations of our current U-Net model to either obtain more performant models or devising new methods for optimizing models that take better advantage of the properties of transfer learning.

\paragraph{Acknowledgments} We thank the UVA Department of Computer Science in Partnership with DREU CRA-W, for hosting and funding I.K.T and R.G. through NSF \#CNS-1246649, IAAMCS. This work was also partially funded with generous gifts from Leidos and Adobe.

{\small
\bibliographystyle{ieee_fullname}


\section{\label{sec:intro}Introduction}

Quantum correlations have been a central field of research since the inception of quantum mechanics  \cite{einstein_podolsky,schrodinger_1935}. They are a fundamental feature of the quantum theory and give rise to many interesting phenomena like those observed in the fields of quantum cryptography \cite{nielsen_chuang,quantum_cryptography,QuantumSystemsChannelsInformation,watrous_2018}, quantum teleportation \cite{teleportation1,teleportation2,teleportation3}, quantum phase transitions in many body systems \cite{sachdev2001quantum,quantum_phase_transitions_review,bond_charge_Hubbard}, etc.

They can be studied from different points of view. For instance,
from a many body perspective, it is known that, if we solve a many body Hamiltonian through mean field techniques, such as the Hartree-Fock or Hartree-Fock-Bogoliubov method, the ground state will, in general, not preserve the symmetries of the Hamiltonian \cite{ring_schuck}. This spontaneous breaking of symmetries is interpreted as a way  to catch important correlations of the exact ground state while preserving the simple mean field picture. From a quantum information point of view, quantum correlations show up when we analyze the state of a given partition in a quantum system. For instance, if we have a pure state living in a Hilbert space with some tensor product structure, we can use the von Neumann entropy of one of the marginals in order to quantify the amount of entanglement between parties \cite{nielsen_chuang}.
However, when we try to join both perspectives, some subtleties arise. For example, when we try to quantify quantum correlations through partial traces, we usually need a Hilbert space with a tensor product structure. Nonetheless, in dealing with many body systems it is customary to deal with identical fermionic particles, and due to the antisymmetrization principle, the tensor structure is lost \cite{reasonable}. Moreover, as it is physically impossible to distinguish among identical particles, it is inconsistent to compute correlations between them through, for example, the von Neumann entropy of the marginals. While it's true that those subtleties are solved by defining a fermionic partial trace \cite{friis}, or quantifying the particle entanglement through the one body entropy \cite{yoshiko}, in general quantum information concepts are not directly applicable in fermionic many body systems. Some efforts with Helium-like systems have been done in \cite{helium_like_systems,helium_like_systems_2} and references therein. Moreover, concepts as the multipartite concurrence under the context of identical particles \cite{multipartite_concurrence}, the Lo Franco and Compagno’s approach \cite{LoFranco2016,entanglement_indistinguisahble} or the antisymmetric negativity \cite{experimental_entanglement_antisymmetrization} have been proposed, among others.

A very powerful measure of quantum correlations is the so-called quantum discord \cite{quantum_discord}. It quantifies all quantum correlations beyond entanglement and allows us to differentiate between classical and quantum correlations \cite{quantum_discord}. However, its calculation requires to perform arbitrary projective measurements in one of the subsystems and therefore it implicitly requires the existence of a tensor product structure. Moreover, it is in general hard to compute, both analytically and numerically, due to the variational process implicit in its definition \cite{discord_hard_analytically,discord_hard_numerically}.

In this work, we aim to study correlations in many body fermionic systems from a quantum information perspective through the analysis of the quantum discord. Specifically, we derive a very simple expression in order to compute the quantum discord between two fermionic orbitals in a arbitrary mixed state, and we apply it the characterization of some well known benchmarking models. The paper is structured as follows. In Sec. \ref{sec:QD} we briefly introduce the concept of quantum discord. We refer to \cite{review_quantum_discord,Adesso_2016} for a complete review of the quantum discord concept and quantum correlations. In Sec. \ref{sec:two_orbital} we derive an expression to compute the quantum discord between pair of orbitals in fermion systems. In \ref{sec:natural_orbitals} and \ref{sec:HF_HFB} we discuss some properties of quantum discord related to the orbital basis. In \ref{sec:results} we compute it in the context of the Agassi and Lipkin-Meshkov-Glick models. Finally, in Sec. \ref{sec:conclusions} and the appendix we summarize the results obtained and we discuss some connections with other results in the literature, respectively.

\section{\label{sec:QD}Quantum Discord}

Given the Hilbert space $\mathcal{H}$ of a quantum system, let us assume there exists a bi-partition
 $\mathcal{H} = \mathcal{H^{(A)}} \otimes  \mathcal{H^{(B)}}$. The quantum discord, introduced by Ollivier and Zurek \cite{quantum_discord}, is a measure of the purely-quantum correlations beyond entanglement between both parts $A$ and $B$. It is defined as the discrepancy between two classically equivalent measures $I(A,B)$ and $J(A,B)$ of the mutual information
\begin{equation*}
\delta(A,B) = I(A,B) - J(A,B).
\end{equation*}
They are given by
\begin{equation}
\label{eq:I}
I(A,B) = S(\rho^{(A)})+S(\rho^{(B)})-S(\rho^{(A,B)})
\end{equation}
and
\begin{equation}
\label{eq:J}
J(A,B) = \max_{\{\Pi_{k}^{(B)}\}} S(\rho^{(A)})-S(\rho^{(A,B)}|\{\Pi_{k}^{(B)}\})
\end{equation}
While $I(A,B)$ is a measure of all kind of correlations, $J(A,B)$ quantifies the classical part. The measurement-based conditional entropy entering the definition of  $J(A,B)$ is defined as
\begin{equation*}
S(\rho^{(A,B)}|\{\Pi_{k}^{(B)}\}) = \sum_{k} p_k S(\rho_k^{(A,B)})
\end{equation*}
where $\rho_k^{(A,B)} = \frac{1}{p_k}\Pi_{k}^{(B)}\rho^{(A,B)}\Pi_{k}^{(B)}$ is the measured-projected state and $p_k = \tr (\Pi_{k}^{(B)}\rho^{(A,B)}\Pi_{k}^{(B)})$ is the associated probability. The measurement and the associated projector $\Pi_{k}^{(B)}$ are defined only in the sector $B$ of the bi-partition.

 Due to the variational process involved in Eq. (\ref{eq:J}) the quantum discord is hard to compute in general either  analytically \cite{discord_hard_analytically} or numerically \cite{discord_hard_numerically}. Some results exist for two qubit systems \cite{two_qubit, x-states}, qubit-qudit systems \cite{qubit_qudit}, and there is also some work related to the quantum discord in fermionic systems \cite{superconducting_systems, majorana_fermions, PhysRevA.100.052331, bond_charge_Hubbard}. The calculation is simplified (that is, there is no variational process involved) if there exist in the model considered some kind of selection rule that reduces drastically the variational space (i.e. the set of valid projective measurements in $B$). In this work, we derive a very simple expression for the two orbital quantum discord in a general fermionic system by using a number-parity selection rule, and we apply it under the context of the Agassi model. Although this type of approach is a novelty, similar equations were obtained in \cite{information_loss} considering the information loss due to a measurement of a single mode in a fermion system.

\section{\label{sec:two_orbital}Two-Orbital fermionic system}

Consider a system formed by $\Omega$ orbitals occupied by fermions where number-parity symmetry \cite{ring_shuck} is preserved and can be considered as a selection rule (NPSR) \footnote{Wave functions mixing configurations with an even and odd number of fermions are not allowed.}. Since we are dealing with fermions, the single-orbital occupation may be $0$ (if there is no fermion in the orbital) or $1$ (if there is a single fermion in the orbital). We divide the system in three subsystems: $A$, $B$ and $C$. $A$($B$) corresponds to the $i$($j$)-th orbital, and C corresponds to the orthogonal complement of $AB$. Since all pure and mixed states must fulfill the NPSR, the density matrix corresponding to the $AB$ subsystem will have the following structure in the occupation basis:
\begin{equation}
\label{eq:rho_AB}
\rho^{(A,B)} =
\begin{pmatrix}
\rho_{1} & 0 & 0 & \alpha \\
0 & \rho_{2} & \gamma & 0 \\
0 & \gamma^* & \rho_{3} & 0 \\
\alpha^* & 0 & 0 & \rho_{4} \\
\end{pmatrix}
\end{equation}
with $\sum_{i = 1}^4 \rho_{i} = 1$. Now we must find a complete set of projectors in the $B$ subspace. For this purpose, one would be tempted to follow the path as in \cite{two_qubit}, this is, performing $U(2)$ rotations on the two ``computational" local projectors. Nonetheless, as the NPSR must be fulfilled, a projective measurement that mix the occupied and non-occupied state of just one orbital would be unphysical. In fact, a self-adjoint operator must commute with the super-selection rule in order to be an observable. A measurement that do not respect the super-selection rule can not be related to any observable, so it would be unrealizable. Indeed, ignoring superselection rules, under the context of fermionic quantum information measures, could lead to a vast overestimation of the correlation or entanglement of the system \cite{ding_1,ding_2}. Then, the only possible projectors in the $j$-th orbital's occupation space are
\begin{equation}
\label{eq:projectors}
\begin{split}
\Pi_0 = a_ja^\dagger_j \\
\Pi_1 = a^\dagger_ja_j
\end{split}
\end{equation}
Since the set of possible projective measurements for part $B$ has just one pair of elements instead of infinite, no optimization process is involved in Eq. (\ref{eq:J}) and  quantum discord can be easily computed. The measured-projected states will be
\begin{equation*}
\begin{split}
\rho_0^{(A,B)} = \frac{1}{\rho_{1}+\rho_{3}}
\begin{pmatrix}
\rho_{1} & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & \rho_{3} & 0 \\
0 & 0 & 0 & 0 \\
\end{pmatrix} \\
\rho_1^{(A,B)} = \frac{1}{\rho_{2}+\rho_{4}}
\begin{pmatrix}
0 & 0 & 0 & 0 \\
0 & \rho_{2} & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & \rho_{4} \\
\end{pmatrix}
\end{split}
\end{equation*}
and, straightforwardly, the conditional entropy can be written as
\begin{equation*}
S(\rho^{(A,B)}|\{\Pi_{k}^{(B)}\}) = S\big(\mathcal{Z}(\rho^{(A,B)})\big)-S(\rho^{(B)})
\end{equation*}
where $\mathcal{Z}(\rho)$ is the de-phasing channel, i.e, the quantum channel that destroys the off-diagonal elements of $\rho$. Finally the quantum discord can be written as
\begin{equation}
\label{eq:quantum_discord}
\delta(A,B) = S\big(\mathcal{Z}(\rho^{(A,B)})\big) - S(\rho^{(A,B)})
\end{equation}
or, more explicitly, in terms of the two-orbital reduced matrix elements:
\begin{equation*}
\delta(A,B) = \sum_k \lambda_k\ln\lambda_k-\rho_{k}\ln \rho_{k}
\end{equation*}
with
\begin{equation}
\label{eq:eigenvalues_rho_AB}
\left\lbrace
\begin{aligned}
\lambda_0 &= \frac{\rho_{1}+\rho_{4}}{2}+\sqrt{\bigg(\frac{\rho_{1}-\rho_{4}}{2}\bigg)^2+\abs{\alpha}^2} \\
\lambda_1 &= \frac{\rho_{1}+\rho_{4}}{2}-\sqrt{\bigg(\frac{\rho_{1}-\rho_{4}}{2}\bigg)^2+\abs{\alpha}^2} \\
\lambda_2 &= \frac{\rho_{2}+\rho_{3}}{2}+\sqrt{\bigg(\frac{\rho_{2}-\rho_{3}}{2}\bigg)^2+\abs{\gamma}^2} \\
\lambda_3 &= \frac{\rho_{2}+\rho_{3}}{2}-\sqrt{\bigg(\frac{\rho_{2}-\rho_{3}}{2}\bigg)^2+\abs{\gamma}^2}
\end{aligned}
\right.
\end{equation}
The quantum discord between the $i$ and $j$ orbitals grows with the off-diagonal matrix elements of $\rho^{(A,B)}$ in Eq \ref{eq:rho_AB}, whose value reflects the amount of quantum coherence of the state. This fact can be easily seen if we write a pair of diagonal elements, i.e, $\rho_{1}$ and $\rho_{4}$ (the same discussion applies to $\rho_{2}$ and $\rho_{3}$) as $\rho_{1} = \frac{1}{4}+\epsilon$ and  $\rho_{4} = \frac{1}{4}-\epsilon$ with $\epsilon \in [0,\frac{1}{4}]$. This corresponds to the case in which $S\big(\mathcal{Z}(\rho^{(A,B)})\big)$ is maximum, and we perturb it via the parameter $\epsilon$. The eigenvalues are, in this case, $\lambda_0 = \frac{1}{4}+\sqrt{\epsilon^2+\abs{\alpha}^2}$ and $\lambda_1 = \frac{1}{4}-\sqrt{\epsilon^2+\abs{\alpha}^2}$. For all allowed values of $\epsilon$, $S(\rho^{(A,B)})\leq S\big(\mathcal{Z}(\rho^{(A,B)})\big)$ with equality when the off-diagonal elements are zero (in this case, $\alpha = 0$). If $\alpha$ increases, then the quantum discord increases too, revealing that the exclusively quantum correlations increase as the coherence grows, which is an expected and intuitive result.

\section{\label{sec:natural_orbitals}Quantum discord and natural orbitals}

The two-orbital reduced density matrix $\rho^{(A,B)}$ can be written in terms of three well known quantities in many-body theory: the one-body density matrix, the two-body density matrix and the pairing tensor, defined as $\gamma_{i,j} = \langle a^\dagger_ja_i\rangle$,  $\gamma_{i,j,i,j} = \langle a^\dagger_ia^\dagger_ja_ja_i\rangle$ and $\kappa_{i,j} = \langle a_ja_i\rangle$, respectively. It has the same structure as Eq. (\ref{eq:rho_AB}), with:
\begin{equation}
\label{eq:rho_AB_with_densities}
\left\lbrace
\begin{aligned}
\rho_{1} &= 1-\gamma_{i,i}-\gamma_{j,j}+\gamma_{i,j,i,j} \\
\rho_{2} &= \gamma_{j,j}-\gamma_{i,j,i,j} \\
\rho_{3} &= \gamma_{i,i}-\gamma_{i,j,i,j} \\
\rho_{4} &= \gamma_{i,j,i,j} \\
\alpha &= \kappa_{j,i}^* \\
\gamma &= \gamma_{j,i}
\end{aligned}\right.
\end{equation}
Together with Eq. (\ref{eq:quantum_discord}), we see that the off-diagonal elements of the one-body density matrix and the pairing tensor are directly related with the quantum discord between $i$ and $j$ orbitals: if at least one of them is non-zero, there exist quantum correlations between $i$ and $j$. Inversely, it can be easily seen that there are two conditions for the quantum discord to be zero for all pairs of orbitals: $\gamma_{j,i} = 0$ and $\kappa_{j,i} = 0$. According with the results obtained in \cite{information_loss}, the first condition, $\gamma_{j,i} = 0$, is fulfilled for all $i,j$ if and only if the orbitals are the natural ones, i.e, those who diagonalize the one-body density matrix. Additionally, if the state commutes with the particle number operator, then $\kappa_{j,i} = 0$ for all $i,j$. Thus, the two conditions for the vanishing of quantum discord between all orbital pairs are:
\begin{enumerate}
    \item The orbital basis is the natural one
    \item The state commutes with the particle number operator
\end{enumerate}

Additionally, it is known that the natural orbitals are the ones that minimize the overall entropy, defined as the sum of all the one-orbital entropies \cite{rosignolli}. Since this quantity is used to quantify the amount of total correlation in a state (the total entanglement if the state is pure) \cite{szalay}, then, if the number of particles is well defined and the state is pure, a non-zero overall entropy implies that all correlations between pairs of natural orbitals will be purely classical (if they exists) and the entanglement must be manifested between three orbitals or more.

\section{\label{sec:HF_HFB}General orbital basis}

It is important to remark that the quantum discord is measured between orbitals, and not between particles. For this reason, a change in the orbital basis may induce a change in the correlations between them. So, in order to study the quantum discord of a state, it is fundamental to specify properly the orbital basis. A natural orbital basis of a state which commutes with the particle number operator implies that those orbitals are constructed so that they can keep the intrinsic quantum correlation of the state without needing quantum correlation by pairs between them (this will be clearer in Sec. \ref{sec:lipkin}). Therefore the following question arises: which is the value of the quantum discord of a given state in a general orbital basis?

Suppose that we have a general orbital basis and the natural orbital basis (of the given state), related by the most general linear canonical transformation between creation/annihilation operators (Bogoliubov transformation \cite{ring_schuck}):
\begin{equation*}
\beta_k^\dagger = \sum_l U_{l,k}c^\dagger_l+V_{l,k}c_l
\end{equation*}
where $\{\beta_k^\dagger\}$ are the fermionic creation operators for the general basis, $\{c^\dagger_l\}$ are the fermionic creation operators for the natural basis. The following relations among the Bogoliubov amplitudes $U$ and $V$
\begin{equation}
\label{eq:U_V_relations}
\begin{split}
U^\dagger U+V^\dagger V &=U U^\dagger + V^* V^T = I \\
U^T V + V^T U &= UV^\dagger + V^* U^T = 0 \\
\end{split}
\end{equation}
hold. Then, the one-body matrix and the pairing tensor elements read \cite{ring_schuck}:
\begin{equation*}
\begin{split}
\gamma_{k,k'} &= \sum_lV^\dagger_{k,l}V_{l,k'}+(U^\dagger_{k,l}U_{l,k'}-V^\dagger_{k,l}V_{l,k'})p_l \\
\kappa_{k,k'} &= \sum_lV^\dagger_{k,l}U^*_{l,k'}+(U^\dagger_{k,l}V^*_{l,k'}-V^\dagger_{k,l}U^*_{l,k'})p_l \\
\end{split}
\end{equation*}
where $p_l = \langle c^\dagger_lc_l \rangle$. In this general case, the quantum discord will be nonzero and orbital dependent except for the case $p_l = \frac{1}{2}$ $\forall l$. This can be checked using the relations in (\ref{eq:U_V_relations}). Since $\sum_lp_l = N$ where $N$ is the number of particles, this case can only exist when the number of particles is exactly half the number of orbitals. Less restrictive is the case of a  transformation among particles, i.e, $V = 0$. Again, the quantum discord will be nonzero except for the case $p_l = \frac{N}{\Omega}$ $\forall l$ where $\Omega$ is the number of orbitals. In both cases, the quantum discord will be zero and orbital independent if the occupation of the system in the natural orbital basis is equally distributed.

A common measure of entanglement among particles is the entanglement entropy of the one-body density matrix, defined as $S(\gamma) = -\gamma\log\gamma = -\sum_lp_l\log p_l$ \cite{yoshiko}. It is interesting to note that the orbital-independent zero discord case in general corresponds with the maximum $S(\gamma)$, which is indeed related to the overall entropy in the natural orbital basis since $S_{ov}^{nat} = -\sum_l p_l\ln p_l -\sum_l (1-p_l)\ln (1-p_l)$ \cite{rosignolli}, and it reaches its maximum value when $p_l = \frac{1}{2}$. In other words, for a pure system with half filling, i.e, $\Omega = 2N$, if the particles are equally distributed between all natural orbitals, the entanglement is maximum but there is no quantum correlations between orbital pairs.

\section{\label{sec:results}Results}

As an example of how the quantum discord between pairs of fermionic orbitals can be used to characterize the correlations in the system, we  apply the previous concepts to the HFB ground state of the Agassi model and the exact ground state of the Lipkin-Meshkov-Glick (LMG) model. Both models are composed by a two-level fermionic system, each having a $\Omega$-fold degeneracy. The difference between them lies in the interaction terms of their respective Hamiltonians.

The LMG model \cite{lipkin} has been widely used over the years as a benchmark in the characterization of different approximations to the many-body problem. The model is simple enough to be exactly solvable and at the same time is sufficiently rich to catch some non-trivial properties of  many-body systems, mainly, the quantum phase transition to a `deformed' state through a spontaneous symmetry breaking of the mean field approximation. As we will see, its Hamiltonian is composed by two terms: the non-interacting one, and the so-called monopole-monopole interaction, which mixes the high and low-lying orbitals of the same degeneracy. The model is very well known in the nuclear physics literature -- see Ref \cite{robledo92} for a study of the model adequate for the present purposes. Also, their entanglement properties have been widely studied \cite{vidal_1,lipkin_rossignoli}, as well as their correlation properties in a finite temperature context \cite{vidal_2} and the solution under the thermodynamical limit \cite{vidal_3,vidal_4}.

The Agassi model \cite{agassi}, is an extension of the LMG one where a separable pairing interaction has been added. The pairing interaction induces the creation and annihilation of particles by pairs with the same (and different) energies. When treated at the mean field level, the Agassi model contains a superfluid phase (treated using the Bardeen-Cooper-Schriefer theory of superconductivity) as well as a deformed one where the broken symmetry is parity (see \cite{robledo92} for a thorough discussion). The model can also be solved exactly using group theory techniques and it is also often used as a benchmark of different approximations in the context of nuclear physics.

\subsection{\label{sec:Agassi}The HFB ground state of the Agassi model as a benchmark}

The Agassi model \cite{agassi} is a two-level system, each of them with a degeneracy $\Omega$ (even). The system is filled with $N=\Omega$ fermions, and the Hamiltonian is given by
\begin{equation}
\label{eq:agassi_hamiltonian}
H = \epsilon J_0 -g\sum_{\sigma, \sigma^\prime}A_\sigma^\dagger A_{\sigma^\prime}
-\frac{1}{2}V[ (J_+)^2+(J_-)^2]
\end{equation}
with
\begin{equation}
\label{eq:SO5_generators}
\begin{split}
J_0 &= \frac{1}{2}\sum_{\sigma, m}\sigma c^\dagger_{\sigma, m}c_{\sigma, m} \\
J_+ &= (J_-)^\dagger = \sum_m c^\dagger_{1,m}c_{-1,m} \\
A_\sigma &= \sum_{m>0} c_{\sigma,-m}c_{\sigma,m}
\end{split}
\end{equation}
where $\sigma = \pm 1$ labels the upper/lower level, $m = \pm 1, \pm 2, ..., \pm \frac{\Omega}{2}$ labels the states within a level, and $c^\dagger_{\sigma, m}$ is the fermionic creation operator of the single particle state labelled by $(\sigma,m)$\footnote{Those states are also called `Hamiltonian orbitals' throughout this work.}. This model is exactly solvable using group theory methods \cite{agassi_solution}, and the HFB ground state solution can be easily obtained. For this reason, we are going to analyze the quantum correlation properties of the HFB ground state as a benchmark of the proposed measure of quantum discord between pairs of fermionic orbitals (Eq. (\ref{eq:quantum_discord})).

Following reference \cite{agassi_solution}, the one body density matrix and the pairing tensor of the HFB ground state can be written as
\begin{equation}
\label{eq:variables_agassi}
\begin{split}
\gamma_{\sigma m,\sigma^\prime m^\prime} &= \gamma_{\sigma,\sigma^\prime}\delta_{m,m^\prime} \\
\kappa_{\sigma m,\sigma^\prime m^\prime} &= \text{sgn}(m)\frac{1}{2}\sin \alpha \delta_{\sigma,\sigma^\prime} \delta_{m,-m^\prime}
\end{split}
\end{equation}
with
\begin{equation}
\label{eq:variables_agassi_2}
\begin{split}
\gamma_{\sigma,\sigma} &= \frac{1}{2}(1-\sigma \cos\phi\cos\alpha) \\
\gamma_{\sigma,-\sigma} &= -\frac{1}{2}\sin\phi\cos\alpha
\end{split}
\end{equation}
The values of $\phi$ and $\alpha$ depend on the parameters of the Hamiltonian, that is:
\begin{equation*}
     \begin{cases}
       \phi = \alpha = 0 &\quad \text{if } \chi, \Sigma_0<1\\
       \cos\phi = \frac{1}{\chi}, \alpha = 0 &\quad \text{if } \chi>\Sigma_0\\
       \phi = 0, \cos\alpha = \frac{1}{\Sigma_0} &\quad \text{if } \chi<\Sigma_0\\
     \end{cases}
\end{equation*}
with $\chi = \frac{(\Omega-1)V}{\epsilon}$, $\Sigma = \frac{(\Omega-1)g}{\epsilon}$ and $\Sigma_0 = \Sigma +\frac{V}{\epsilon}$. As can be seen, there are three differentiated regions in the parameters space: the HF spherical phase, the HF deformed phase and the BCS phase. The first one corresponds to the conditions $\chi, \Sigma_0<1$, and the HFB ground state is the non interacting exact ground state, i.e, all the lower levels occupied. The second one corresponds to the conditions $\chi>\Sigma_0$ and $\chi>1$. In this case the HFB ground state breaks the parity symmetry\footnote{In the context of the Agassi model, particles in the upper (lower) level are assumed to have positive (negative) parity.} (that's why it's called `deformed'). The last region corresponds to $\chi<\Sigma_0$ and $\Sigma_0>1$. It preserves the parity symmetry but it breaks the particle number symmetry and represents a superfluid system described by the BCS approximation. Since in all regions the ground state is defined as a quasi-particle vacuum, the two body density is separable and the diagonal elements can be written as
\begin{equation*}
\gamma_{i,j,i,j} = \gamma_{i,i}\gamma_{j,j} + \Delta_{i,j}
\end{equation*}
with $\Delta_{i,j} = \abs{\kappa_{i,j}}^2-\abs{\gamma_{i,j}}^2$ and, using Eq. (\ref{eq:rho_AB_with_densities}) and (\ref{eq:rho_AB}), we can write the two orbital reduced density matrix as
\begin{equation*}
\left\lbrace
\begin{aligned}
\rho_{1} &= (1-\gamma_{i,i})(1-\gamma_{j,j}) + \Delta_{i,j} \\
\rho_{2} &= (1-\gamma_{i,i})\gamma_{j,j} - \Delta_{i,j}  \\
\rho_{3} &= \gamma_{i,i}(1-\gamma_{j,j}) - \Delta_{i,j} \\
\rho_{4} &= \gamma_{i,i}\gamma_{j,j} + \Delta_{i,j}  \\
\alpha &= \kappa_{j,i}^* \\
\gamma &= \gamma_{j,i}
\end{aligned} \right .
\end{equation*}

With those results and using Eq. (\ref{eq:quantum_discord}) we can easily compute the quantum discord between a pair of orbitals in the HFB ground state solution, which is
\begin{equation}
\label{eq:quantum_discord_agassi}
    \begin{cases}
        \delta(m,\sigma ; m,-\sigma) = h(\chi) &\quad \text{in the deformed HF region} \\
        \delta(m,\sigma ; -m,\sigma) = h(\Sigma_0) &\quad \text{in the BCS region} \\
        \delta(m,\sigma ; m^\prime, \sigma^\prime) = 0 &\quad \text{otherwise} \\
    \end{cases}
\end{equation}
with $h(x) = -\frac{1}{2}(1-\frac{1}{x})\ln \frac{1}{2}(1-\frac{1}{x})-\frac{1}{2}(1+\frac{1}{x})\ln \frac{1}{2}(1+\frac{1}{x})$. This solution is shown in Figs. \ref{fig:up_down_quantum_discord} and \ref{fig:pairing_quantum_discord}.

\begin{figure}[ht]
\includegraphics[width=0.5\textwidth]{up_down_quantum_discord}
\caption{Quantum discord between an orbital pair with $m=m^\prime$ and $\sigma = -\sigma^\prime$ as a function of the two Hamiltonian parameters $\chi$ and $\Sigma$. The quantum correlations in this case are zero for the spherical HF and BCS regions, while is non-zero only in the deformed HF region. $\Omega = 20$.}
\label{fig:up_down_quantum_discord}
\end{figure}

\begin{figure}[ht]
\includegraphics[width=0.5\textwidth]{pairing_quantum_discord}
\caption{Quantum discord between an orbital pair with $m=-m^\prime$ and $\sigma = \sigma^\prime$ as a function of the two Hamiltonian parameters $\chi$ and $\Sigma$. The quantum correlations in this case are zero for the spherical and deformed HF regions, while is non-zero only in the BCS region. $\Omega = 20$.}
\label{fig:pairing_quantum_discord}
\end{figure}

The structure of the quantum discord is the same as the phase diagram \cite{agassi_solution}. In the deformed HF phase, there are quantum correlations only between orbitals with the same $m$ and opposite $\sigma$ due to the monopole-monopole interaction. In the same way, there are quantum correlations in the BCS phase only between orbitals with the same $\sigma$ and opposite $m$ due to the pairing interaction in the Hamiltonian. In fact, if we compute the mutual information, defined in Eq. (\ref{eq:I}), we obtain $I(A,B) = 2\delta(A,B)$. This is expected when the state $\rho^{(A,B)}$ is pure \cite{two_qubit} and, indeed, it is the case within the HFB solution\footnote{It can be checked that the eigenvalues of the two-body density matrix, i.e, $\lambda_i$ in Eq. (\ref{eq:eigenvalues_rho_AB}) acquire the values $\lambda_i = 0,1$.}. Specifically, the two orbital reduced state between orbitals with the same $m$ and opposite $\sigma$ is pure in the deformed HF region and mixed in the BCS one (the inverse happens with same $\sigma$ and opposite $m$ orbitals). This result serves as a benchmark of the results obtained in Sec. \ref{sec:two_orbital}. Moreover, we note that the quantum discord between a transition from a spherical HF state to a deformed HF or BCS state is continuous, while the quantum discord between a transition from a deformed HF state to a BCS one is discontinuous. Since the quantities $\rho = -\frac{1}{2}\sin\phi\cos\alpha$ and $\kappa = \frac{1}{2}\sin\alpha$ from Eqs. (\ref{eq:variables_agassi}) and (\ref{eq:variables_agassi_2}) can be considered as order parameters of the model \cite{agassi_solution}, the quantum discord shows the behaviour of a combined order parameter.

It is interesting to note that there is no quantum discord between the Hamiltonian orbital pairs considered when the state is the exact ground state, since the corresponding one-body matrix elements are zero and the particle number is well defined (see Appendix \ref{sec:exact_agassi} for details). As we will discuss in next section, a low quantum discord implies a better adaptation of the orbitals in order to describe the state. This indicates that the Hamiltonian orbitals are suited to describe the exact ground state better than the HFB ground state, as expected.

\subsection{\label{sec:lipkin}The exact ground state of the LMG model}

Finally, we analyze the quantum discord between orbital pairs within the exact ground state of the LMG model. The LMG Hamiltonian is the same as Eq. (\ref{eq:agassi_hamiltonian}), with $g = 0$, i.e, there are only monopole-monopole interaction. For this reason, we only consider the quantum discord between orbitals with same $m$ and opposite $\sigma$.

Since the Hamiltonian commutes with the particle number and parity operators, the Hamiltonian orbitals, represented by the creation/annihilation operators ($c^\dagger_{\sigma, m}$ and $c_{\sigma, m}$ respectively) in Eq. (\ref{eq:agassi_hamiltonian}) are the natural ones ($\gamma_{i,j} = 0$ for $i\neq j$), and the pairing tensor is zero. Thus, as explained in Sec. \ref{sec:natural_orbitals}, the quantum discord is zero for all pairs.

But this is not true if we change the orbital basis. In general, a low quantum discord implies a better adaptation of the orbitals in order to describe the exact ground state, while a high quantum discord reflects the contrary case. If we compute the quantum discord between an up-down pair of HF orbitals, we obtain the result shown in Fig. \ref{fig:QD_vs_kappa_Nmax_20}. It is interesting to analyze the behaviour of the quantum discord of the exact ground state between those orbitals since they are defined in order to catch the maximum correlations as possible within a mean field scenario.

\begin{figure}[ht]
\includegraphics[width=0.5\textwidth]{QD_vs_kappa_Nmax_20}
\caption{Quantum discord between an up-down orbital pair as a function of the Hamiltonian parameter $\chi$ with $N$ from $3$ to $20$.}
\label{fig:QD_vs_kappa_Nmax_20}
\end{figure}

For $\chi<1$ the quantum discord is zero, since the HF orbitals in this region coincide with the natural ones. For $\chi>1$ there are two different regimes. First, as $\chi$ is big enough and it grows, the quantum discord decreases. This decrease is more drastic if the particle number is bigger, in coherence with the mean field description, in which more accuracy is obtained when the number of particles is big enough.
The other regime is manifested when $\chi>1$ acquires intermediate values, this is, near the quantum phase transition point ($\chi=1$) and far from the asymptotic limit. In this region, the quantum discord grows fast until reaching the maximum. Then, it decreases exponentially until the asymptotic regime. This intermediate region is where the Hartree-Fock approximation becomes less accurate, and this is reflected as a high quantum discord between the HF orbitals: since the orbitals are less optimum in order to encode the exact ground state, more quantum correlation is needed between them for that task. In this intermediate region it is necessary to consider linear combinations of mean field Slater determinants to catch the physics of the exact ground state \cite{robledo92}.

Until now, we have discussed the quantum discord between an up-down HF orbital pair for the exact ground state of the LMG model. We argued that, since the Hamiltonian orbitals are the natural ones, the same quantity between those is zero. The same argument apply for the HF orbitals in a HF ground state. However, we can ask ourselves what is the quantum discord between an up-down Hamiltonian orbital pair of the HF ground state (which is the `inverse' case with respect to the results in Fig. (\ref{fig:QD_vs_kappa_Nmax_20})). Since the LMG model is a particular case of the Agassi model, we find that this quantity is given by Eq. (\ref{eq:quantum_discord_agassi}) when $\Sigma = 0$ (Fig. (\ref{fig:QD_vs_kappa_HF_GS})).

\begin{figure}[ht]
\includegraphics[width=0.5\textwidth]{QD_vs_kappa_HF_GS}
\caption{Quantum discord between an up-down Hamiltonian orbital pair for the HF ground state as a function of the Hamiltonian parameter $\chi$.}
\label{fig:QD_vs_kappa_HF_GS}
\end{figure}

Unlike Fig. (\ref{fig:QD_vs_kappa_Nmax_20}), now the quantum discord approaches to the value $\ln 2$ when $\chi \rightarrow \infty$ and doesn't depend on the particle number. This different behaviour is coherent with the fact that the HF ground state is rather different than the exact one: although the HF orbitals are better adapted when $\chi$ is high, as discussed previously, the structure of the HF ground state remains a Slater determinant, which in general is far from being exact. Thus, the Hamiltonian orbitals require higher quantum correlations in order to describe this state when the interaction is large. So, within the context of the models considered, a low quantum discord between pairs of orbitals can be related to an optimal orbital adaptation when describing a given state.

\section{\label{sec:conclusions}Conclusions}

The quantum discord is a measure of  quantum correlations in a given state. It is defined as the minimum difference between two classically equivalent but quantumly different versions of the mutual information. This definition is based in the fact that, given a bi-partition $A|B$ of a system, a measurement on $A$ may break the quantum correlations between $A$ and $B$. In this manner, a projective measurement may be performed in one of the subsystems. Nonetheless, a fermion system must satisfy the parity superselection rule, so not all the projective measurements are physical.

In this work, we use this property and we propose a simple expression (see Eq (\ref{eq:quantum_discord})) in order to compute the quantum discord between two orbitals in a general fermionic pure or mixed state. This expression does not require an optimization procedure and is directly related to two central many-body quantities: the one body density matrix and the pairing tensor. Thus, we have shown that the natural orbital basis, which is defined as the one that diagonalize the one body density matrix, reduces the quantum discord between any pair of orbitals to zero when the state commutes with the particle number operator. Moreover, when the system's orbitals are half filled, there is no quantum correlations between pairs for any arbitrary orbital basis. Finally, we compute and discuss the quantum discord between pairs of orbitals in the HFB ground state of the Agassi model, and in the exact ground state of the LMG model. Our results may be useful in order to analyze quantum correlations in more complicated and realistic many body fermionic systems.

\acknowledgments{}
The authors want to thank the Madrid regional government, Comunidad
Aut\'onoma de Madrid, for the project Quantum Information Technologies:
QUITEMAD-CM P2018/TCS-4342.
The  work of LMR was supported by Spanish Ministry
of Economy and Competitiveness (MINECO) Grants No.
PGC2018-094583-B-I00.