## Permutation-invariant Feature Restructuring for Correlation-aware Image Set-based Recognition

Xiaofeng Liu 1 2 , , Zhenhua Guo 1 4 , , Site Li 1 , Lingsheng Kong , Ping Jia , Jane You , B.V.K. Kumar 3 3 5 1 ∗ 1 Carnegie Mellon University; 2 Harvard University; 3 CIOMP, Chinese Academy of Sciences; 4 Graduate School at Shenzhen, Tsinghua University; 5 The Hong Kong Polytechnic University ∗ Corresponding author: kumar@ece.cmu.edu

## Abstract

We consider the problem of comparing the similarity of image sets with variable-quantity, quality and un-ordered heterogeneous images. We use feature restructuring to exploit the correlations of both inner &amp; inter-set images. Specifically, the residual self-attention can effectively restructure the features using the other features within a set to emphasize the discriminative images and eliminate the redundancy. Then, a sparse/collaborative learning-based dependency-guided representation scheme reconstructs the probe features conditional to the gallery features in order to adaptively align the two sets. This enables our framework to be compatible with both verification and open-set identification. We show that the parametric self-attention network and non-parametric dictionary learning can be trained endto-end by a unified alternative optimization scheme, and that the full framework is permutation-invariant. In the numerical experiments we conducted, our method achieves top performance on competitive image set/video-based face recognition and person re-identification benchmarks.

## 1. Introduction

Many research studies focus on using a single image [33, 35, 32] or video [27, 30] as shown in Fig. 1. However, in many practical applications, a set of images of a subject (consisting of still images, or frames from a video, or a mixture of both) can usually be collected from different checkpoints, segments in videos, mugshot history of a criminal and lifetime enrollment images for identity documents etc [28]. These can contain extreme rotations, complex expressions and illumination variations [25, 40, 61]. This setting is more similar to the real-world biometric scenarios [15].

Compared to the case of a single image [24, 37, 31, 38, 29, 26], richer and complementary information can be expected in a set, because the samples are captured from multiple views [36]. However, it also poses several challenges

Figure 1. Illustration of the single image-based, video sequencebased and image set-based open-set face identification.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01174v1-figure-6d0ec779-dd1c-4178-805d-a030a64439d0.jpg)

including a) variable number of samples within a set, b) larger inner set variability -than its video-based recognition counterpart, and c) order less data -.

The conventional approaches to set-based recognition aggregate all images X = { x n } N n =1 in a set with N images to a single feature vector. The aggregation is achieved through max/average pooling [21, 43, 5]. More formally, f (X) = ρ max/average pooling ( ( { ϕ x ( 1 ) , · · · , ϕ ( x N ) } ) , where ρ, ϕ are any appropriate mapping functions. Considering that the importance of each image maybe different, their weights are calculated independently by ϕ as the soft attention score in pooling operation [61, 40]. By doing this, the high quality ( i.e., clear and frontal) faces are favored in these models. These methods are not sensitive to the order of images with in a set (i.e., permutation-invariant), but their weight selection procedures for each image in the set do not pay attention to the other images. Without inner-set correlation can result in redundant weights as in Fig. 2.

The inter-set correlation is also important for similarity measurements [62]. If we are given a query set consisting of clear profile face images, it should be more reasonable to emphasize the profile face images in the gallery set rather than the high-quality frontal face images.

How to model the inner/inter-set correlations in image sets with arbitrary number of order-less images has long been a challenging [25].

For exploring the inner-set correlations, the typical

Figure 2. Examples in IJB-A dataset showing the weights of images calculated by independent image quality assessment (NAN [61]) that does not take into account inner-set correlations and RL-based inner-set relationship exploring (DAC [25]). The NAN suggests a similar score for a discriminative image and its blurry version in the same pose (dashed red box). However, there is little additional information in the latter. When we change the order of images from the 1st row to the 2nd row, the output of DAC is different. If the inferior samples are processed ahead by DAC, their weights can be larger, and the performance would be degraded.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01174v1-figure-d26dea88-ebf8-4692-be72-601590d30dd8.jpg)

permutation-sensitive methods, e.g., recurrent neural networks, are totally not applicable for order-less data [25]. The reinforcement learning (RL) was introduced to estimate the weights of each image sequentially [25]. Fig. 2 shows that RL is still sensitive to the order.

| Method   | permutation invariant   | inner-set correlation   | inter-set correlation   | verif- ication   | open-set identification   |
|----------|-------------------------|-------------------------|-------------------------|------------------|---------------------------|
| AvePool  | yes                     |                         |                         | yes              | yes                       |
| NAN[61]  | yes                     |                         |                         | yes              | yes                       |
| DAC[25]  |                         | yes                     | pose-only               | yes              | yes                       |
| CNet[56] | yes                     | pose-only               | pose-only               | yes              |                           |
| PIFR     | yes                     | yes                     | yes                     | yes              | yes                       |

For exploring the inter-set correlations, the parametric pair-wise input network (takes two sets as input and output the similarity score using a sigmoid unit directly) usually computationally inefficient for open-set identification [56, 61]. [25] proposes to represent and compare frontal and profile faces separately. When we start using separate representations for expressions, lighting, occlusion, makeup etc ., the enormous number of final representations and hyperparameters needed to balance them will be uncontrollable.

To address the challenge of taking inner-set and inter-set correlations into account in a computationally efficient way, we propose the permutation-invariant feature restructuring ( PIFR ) framework to reduce the redundancy and maintain the benefit of diversity information. It fully considers the inner/inter-set correlation by restructuring a sample in the probe set with the samples in probe/gallery set to explore their complementarity and substitutability.

Specifically, a parametric redundancy-eliminating selfattention (RSA) module allows all samples in a set to contribute to produce a refined feature vector based on feature pair-wise affinity. Its residual attention mechanism not only learns to emphasize discriminative samples while repelling inferior ones, but it also explicitly reduces the redundancy. Moreover, spatial clues are incorporated using the fully convolutional feature extractor and Gaussian similarity metric.

To take all possible inter-set variations into account and scale to open-set identification, we develop a nonparametric dependency-guided feature alignment (DFA) module for inter-set interaction. It assumes that each probe image feature vector can be sparsely/collaboratively restructured on the basis of gallery feature set [34]. In testing, the gallery features can be pre-computed and stored, and its reconstruction coefficients can be computed in parallel.

The parametric deep learning-based RSA and non-

Table 1. Comparisons between proposed PIFR and other state-ofthe-art approaches for set-based recognition.

parametric dictionary learning-based DFA can be alternatively optimized in our PIFR. Table 1 compares the capabilities of the proposed PIFR to other set-based approaches. The main contributions of this paper are

- · We introduce PIFR that takes into account the correlations among variable number of order-less samples by restructuring the feature with the other inner &amp; inter-set features in a permutation-invariant manner. It emphasizes the discriminative images, reduces the redundancy, adaptively aligns the probe-gallery set, and is efficient for both verification and open-set identification tasks.
- · We show how self-attention and dictionary learning can be applied to set-based recognition. The spatial cues and explicit redundancy-reduction are further incorporated in RSA. A novel reconstruction-based sparse/collaborative similarity and loss are proposed in DFA.
- · The proposed PIFR with alternate optimization can be a general framework to integrate the parametric deep learning and non-parametric dictionary learning.

We apply the proposed method to several image set-based verification/identification tasks and empirically demonstrate its effectiveness and generality.

## 2. Related Works

Representation of sets. The most straightforward permutation-invariant operations are the max/average pooling over a set of embeddings X= { x n } N n =1 [21, 43, 5]. Zaheer et al. [63] has proven that the function that can be represented the

×

̅

̅

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01174v1-figure-47864d1f-bb85-4f3f-a317-8a3d34d2c629.jpg)

̅

̅

W

W

W

W

Figure 3. Our permutation-invariant feature restructuring ( PIFR ) framework for image set-based recognition, which consists of the fully convolutional image feature extractor (IFE), redundancy-eliminating self-attention blocks for inner-set restructuring (RSA × L ), global pooling layer and sparse/collaborative representation-based dependency-guided feature alignment (DFA) module for inter-set restructuring.

following form is permutation-invariant:

$$f ( X ) = \rho ( \sum _ { x \in X } ( \varphi ( x ) ) \quad \ \ ( 1 ) \quad \ \ \ \ P _ { x } \\ \text{ a}$$

However, ϕ acts independently on each element, which does not allow an exploration of the inter-sample relationships. This limitation is also in the methods in [13, 10].

tion, and the parametric pair-wise input network in [56] is computationally inefficient for open-set identification. Our PIFR is end-to-end trained given only the set-level identity annotations, and is compatible with multiple inter-set variations and is scalable for open-set identification.

Image set based recognition. A video can be considered as an image set with ordered images and has been actively studied, while a general image set is orderless and usually contains more challenging variations. Thus the methods relying on temporal dynamics will not be covered here. Traditional methods usually model the set/video as appearance subspaces or manifolds [51, 19, 49].

Along a different axis, [61, 40] are based on computing a score for each image with neural image assessment modules. Then, a set of features are aggregated to a fixed size feature vector via weighted pooling. Without inner-set interactions, this may result in redundancy and not use the information diversity present in a set.

[25] proposes to exploit the inner-set relationship using the RL. However, it is permutation-variant since the weight of the first image is computed based on [ x 1 ; x 2 + ··· + x N N -1 ] , while if we move that image to the second position, its weight will be based on the [ x 2 ; a x 1 1 + x 3 + ··· + x N N -2+ a 1 ] , where [ ; · · ] denotes the concatenation and a 1 is the updated weight of the first image. Besides, the RL itself is usually unstable [17]. The feature dimension is compressed from 1024 in its GoogleNet backbone [45] to 128 which unavoidably weakens its representation ability. Different from these extensions, we address the inner-set redundancy by constructing a feature vector that takes into account all of its neighbors and results in a set of redundancy-reduced features.

Using a single set-level feature vector is inadequate for representing inter-set interactions. [25, 56] target the interset pose-aware alignment relying on pose/landmark detec-

Self-attention and non-local filtering. As attention models grow in popularity, [48] develops a self-attention mechanism for machine translation. It calculates the response at one position as a weighted sum of all positions in the sentences. A similar idea is also inherited in the non-local algorithm [2], which is a classical image denoising technique. The interaction networks were also developed for modeling pair-wise interactions [1, 18, 53, 60]. Moreover, [52] proposes to bridge self-attention to the more general nonlocal filtering operations. [65] proposes to learn temporal dependencies between video frames at multiple time scales. Inspired by above works, we further adapt this idea to the set-based problem, the difference with the aforementioned studies includes, but is not limited to incorporating spatial clues with Gaussian similarity matrix, explicit redundancyeliminating residual term and single-layer pairwise affinity. Sparse/collaborative representation based classification (SRC/CRC). SRC [54] and CRC [64], and their many extensions [34, 7, 55] have been widely studied. They achieve promising single image and video-based [6] face recognition performance under occlusions and illumination variations. Our DFA is inspired by them, but using a novel similarity measurement instead of reconstruction coefficients.

## 3. Methodology

We consider the image-set based recognition as matching the probe set P containing N images ( i.e., P = { p n } N n =1 ) with the gallery set G which consist of M images ( i.e., G = { g m m } M =1 ), n, m are the image/video frames indexes. As illustrated in Fig. 3, our permutation-invariant feature restructuring (PIFR) framework consists of three ma-

̅

̅

Table 2. The structure details of our IFE and RSA modules. We adopt the pre-trained ResNet-50 backbone, and utilize the output of 3 rd resblock as our { x n } N n =1 on IJB-A&amp;Celebrity-1000.

| Layers          | Layers                                                       | Output size                          |
|-----------------|--------------------------------------------------------------|--------------------------------------|
| conv            | 7 × 7 , 64 , stride 2 max pool, 3 × 3, stride 2              | 112 × 112 × 64 × N 56 × 56 × 64 × N  |
| pool            |   conv 1 × 1 , 64 conv 3 × 3 ,                             |                                      |
| 1-3 res blocks  | 64 conv 1 × 1 , 256   ×                                    | 56 × 56 × 256 × N                    |
| 4-7 res blocks  |   conv 1 × 1 , 128 conv 3 × 3 , 128 conv 1 × 1 , 512   × | 28 × 28 × 512 × N                    |
| 8-13 res blocks |   conv 1 × 1 , 256 conv 3 × 3 , 256 conv 1 × 1 , 1024    | 14 × 14 × 1024 × N ( H × W × D × N ) |
| RSA             | [ RSAblock ] × L                                             | 14 × 14 × 1024 × N                   |
| Global pool     | Global pool, 14 × 14                                         | 1 × 1 × 1024 × N                     |

jor components: fully convolutional image feature extractor (IFE), a group of redundancy-eliminating self attention (RSA) blocks and a dependency-guided feature alignment (DFA) module. To represent the spatial correlations, we choose the convolution layers of a deep convolutional neural network (CNN) networks with high-end recognition performances ( e.g. , GoogleNet[45], ResNet50[16]) as our IFE, instead of either hand-crafted extractors or CNN with fully connected layers used in [25, 61]. It embeds the images into latent space independently and generates the corresponding feature sets X = { x n } N n =1 and Y = { y m m } M =1 from P and G respectively. x , y n m ∈ R H × W D × where H W , and D are the height, width and channel dimension, respectively of our representation. Its parameters are fixed after pretraining by a large scale image-based recognition datasets.

## 3.1. Redundancy-eliminating Self Attention

A shown in Fig. 4, we cascade L × RSA blocks, which works as a residual self-attention module receiving all extracted features { x n } N n =1 and restructuring them based on inner-set correlations. Since the features { x n } N n =1 are deterministically computed from the probe set { p n } N n =1 , they also inherit and display large variations and redundancy. Ideally, the RSA should display four attributes: Ability to handle arbitrary number of samples, inner-set correlationawareness, permutation invariance to the features, and sensitivity to relative locations in the image plane.

We propose to harvest the spatially-aware inner-set correlations by exploiting the affinity of point-wise feature vectors. We use i = 1 , · · · , H × W to index the position in HW plane and the j is the index for all D -dimensional feature vectors other than the i th vector ( j = 1 , · · · , H × W × N -1) . Specifically, our RSA block can be formulated as

$$\dot { \nu } \quad \dot { \nu } \\ x _ { n, i } ^ { l } = x _ { n, i } ^ { l - 1 } + \frac { \Omega ^ { l } } { C _ { n, i } } \sum _ { \forall n, j } \omega ( x _ { n, i } ^ { 0 }, x _ { n, j } ^ { 0 } ) ( x _ { n, j } ^ { l - 1 } - x _ { n, i } ^ { l - 1 } ) \Delta _ { i, j } \quad \text{ for a } \\ C _ { n, i } = \sum _ { \forall n, j } \omega ( x _ { n, i } ^ { 0 }, x _ { n, j } ^ { 0 } ) \Delta _ { i, j } \quad \text{ (2)} \quad \text{Pro} \quad \text{the f} \quad \text{(2)} \text{ if }$$

where Ω l ∈ R 1 × × 1 D is the weight vector to be learned, l = 0 1 , , · · · , L with L being the number of stacked sub-

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01174v1-figure-54aca501-68d7-4b8e-b9fa-9580a1b0d27b.jpg)

W

Figure 4. Redundancy-eliminating self attention blocks.

self attention blocks and x 0 n = x n . The pairwise affinity ω ( · , · ) is an scalar. The response is normalized by C n i .

To explore the spatial clues, we propose to use a Gaussian similarity metric ∆ i,j = exp( ‖ hw n i -hw n j ‖ 2 2 σ ) , where hw n i , hw n j ∈ R 2 represent the position of i th and j th vector in the HW-plane of x n , respectively.

The residual term is the difference between the neighboring feature ( i.e., x l -1 n j ) and the computed feature x l -1 n i . If x l -1 n j incorporates complementary information and has better imaging/content quality compared to x l -1 n i , then RSA will erase some information of the inferior x l -1 n i and replaces it by the more discriminative feature representation x l -1 n j . Compared to the method of using only x l -1 n j [52, 65], our setting shares more common features with diffusion maps [9, 46], graph Laplacian [8] and non-local image processing [3, 14]. All of them are non-local analogues [12] of local diffusions, which are expected to be more stable than its original non-local counterpart [2, 52] due to the nature of its inherit Hilbert-Schmidt operator [12].

The operation of ω in Eq. (2) is not sensitive to many function choices [52, 65]. We simply choose the embedded Gaussian given by

$$\lambda ^ { \quad } \omega ( \stackrel { \tilde { x } } { x _ { n, i } ^ { 0 } }, x _ { n, j } ^ { 0 } ) = e ^ { \psi ( x _ { n, i } ^ { 0 } ) ^ { T } \phi ( x _ { n, j } ^ { 0 } ) } \quad \quad ( 3 )$$

where ψ x ( 0 n i ) = Ψ x 0 n i and φ x ( 0 n j ) = Φ x 0 n j are two embeddings, and Ψ Φ , are matrices to be learned.

Different from [52], our affinity ω is pre-computed given the input feature set { x n } N n =1 and stays the same through the propagation within RSA, which reduces the computational cost while ω x ( 0 n i , x 0 n j ) can still represent the affinity between x l -1 n i and x l -1 n j to some extent. This design reduces the number of parameters and speeds up the learning.

Definition 1. Let π be an arbitrary permutation function for a sequence. We say a function f (X) is permutation -invariant iff for ∀ π f ; (X ) = f ( π (X)).

Property 1. The RSA is permutation invariant -, due to the fact that all features in { x n } N n =1 are considered in Eq. (2) in an order-less manner (sum operation).

The formulation in Eq. (2) supports the inputs of variable number of features, and maintains the corresponding number and dimension of features in the output tensor X L after several blocks are calculated consecutively. We apply the global pooling (GP) [23] to summarize the information in HW-plane, and get the final representations:

$$\begin{cases} \int r w - \text{plane}, \, \text{and} \, \text{get} \, \int \, \text{in} \, \text{parameters} \, \text{ions} \colon & \quad 1 \, L \\ \left \{ \begin{array} { c } \overline { X } ^ { L } = \left \{ \overline { x } ^ { L } _ { 1 }, \cdots, \overline { x } ^ { L } _ { N } \right \}, \overline { x } ^ { L } _ { n } = \text{GP} ( x ^ { L } _ { n } ) \in \mathbb { R } ^ { 1 \times 1 \times D } \\ \end{array} \\ \begin{array} { c } \overline { Y } ^ { L } = \left \{ \overline { y } ^ { L } _ { 1 }, \cdots, \overline { y } ^ { L } _ { M } \right \}, \overline { y } ^ { L } _ { m } = \text{GP} ( y ^ { L } _ { m } ) \in \mathbb { R } ^ { 1 \times 1 \times D } \\ ( 4 ) & s \\ \end{array} \\ R S A \, \text{learn} \, \text{to a\,o\,o\,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \, o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \,o \} \end{cases}$$

RSA learns to aggregate inner-set correlations for each individual position of a sample's feature adaptively. It can be flexibly attached to any pre-trained IFE (by initializing Ω l as zero) and possible to be fine-tuned jointly with IFE.

## 3.2. Dependency-guided Feature Alignment

Beyond specifically aligning the poses, illuminations, expressions etc ., [25, 62], we propose to measure the dependency-guided similarity between a pair of feature sets with different feature numbers leveraging the nonparametric dictionary learning methods [34].

x L n is expected to be represented using a linear combination of { y L 1 , · · · , y L M } , which means that we can search similar feature vectors in Y L to reconstruct x L n . It transforms the matching task into a sparse/collaborative representation problem:

$$\min _ { a _ { n } } \left \| \, a _ { n } \, \right \| _ { p } \ s. t. \ \bar { x } _ { n } ^ { L } = \bar { Y } ^ { L } a _ { n } \, \quad \ \ ( 5 ) \quad \ \text{int} \\ \text{set}$$

where a n ∈ R M × 1 is the sparse/collaborative coding vector of x L n w.r.t. Y L . The glyph[lscript] p -norm is used to constrain a n , which leads to fewer features in Y L to reconstruct the x L n . p = 1 and 2 are adopted in sparse and collaborative representation-based classification, respectively. However, SRC/CRC is proposed to reconstruct a probe image based on all of the gallery images and uses the coefficient a n to rank similar samples. Our method is different from that, since the objectives are the samples in the sets and we do not use ‖ a n ‖ p directly for similarity measurement. We can relax the constraint term and rewrite Eq. (5) as

$$\underset { a _ { n } } { \min } \left \| \overline { x } _ { n } ^ { L } - \overline { \Upsilon } ^ { L } a _ { n } \right \| _ { 2 } ^ { 2 } + \frac { \lambda } { M } \right \| a _ { n } \left \| _ { p } \quad \quad ( 6 ) \quad \text{$ \quad$}.$$

where ‖ x L n -Y L a n ‖ 2 2 is the glyph[lscript] 2 distance of x L n and Y L , and λ controls the sparsity of coefficients a n . For a probe set with N samples, we need a sparse/collaborative reconstruction coefficient matrix A = { a , 1 · · · , a N } ∈ R M N × , and the set-to-set verification is reformulated to calculate the set-level similarity, which can be measured by 1 N ‖ X L -Y A L ‖ 2 2 . It is essentially different from previous metric learning-based algorithms for two sets.

As such, DFA is essentially accounting for all possible variations to further boost the performance at the extremes of these variations, and there is no need for addi-

Algorithm 1: Bi-level training of PIFR

Input: Training sets and labels; λ ; Pre-trained IFE

Output: Updated RSA parameters θ

- 1 Level 1: Pre-train RSA ;
- 2 for all IFE extracted training sets X and Y do
- 3 Reconstruct { x n } N n =1 , { y m m } M =1 to X L , Y L using RSA
- 4 X L = Global pool (X L ); ˜ X L = Average pool (X L )
- 5 Compute cross entropy loss and update RSA by SGD

6

end

- 7 Level 2: Alternative optimization with RSA and DFA ; 8 for all X and Y pairs do
- 9 1) Compute index function α with traning set labels
- 10 2) Compute { a n } N n =1 in DFA following Eq. (8)
- 11 3) Compute the reconstruction error loss in Eq. (7)
- 12 4) Update the parameters θ by SGD following Eq. (9)

13

end tional pose/landmark annotations. Since there are no additional constraints for the relations of the elements in A, it is also permutation-invariant.

In the testing stage of an open-set identification task with K gallery sets, we need K times verification, where K can be a few hundreds ( e.g., IJB-A) or even millions in real world applications.The pair-wise input network [56] for inter-set interaction need to process K query-gallery image set pairs. Considering the limited memory of GPUs, we usually cannot put in 2 K image sets and several network models. Its processing time equals to K times of verification, which is impractical. However, our gallery Y L s can be pre-computed and stored. We only need to extract X L from a query set, and the K times similarity measurements is applied to X L and Y L vector set pairs instead of original image. Unlike parametric network-based scheme, our DFA does not take a lot of memory. Therefore, we can process numerous X L and Y L pairs in parallel. Actually, a , 1 · · · a N are also solved using a parallel algorithm [20, 64]. The separated and non-parametric DFA enables our framework to be practicable for open-set identification.

## 3.3. Bi-level Training and Alternative Optimization

Wenote that the DFA is nonparametric and does not have a training phase. For the closed-set identification, there are no inter-set interaction, and the RSA only model is applied. The nonparametric frame-wise average pooling is applied to X L , which aggregates features into a fixed-size representation ˜ X L ∈ R 1 × × 1 D , followed by a softmax output layer to calculate the cross-entropy loss. For the verification or open-set identification, the RSA only model is trained by contrastive loss as in [61] using ˜ X L , while the PIFR is trained following a bi-level scheme using paired X &amp;Y L L .

First, we pre-train the RSA module following closed-set

Figure 5. Examples of DFA. The coefficients almost align the pose variation, but essentially take all possible inter-set variations into account. Following [25], we rotate images to frontal/right face in pre-processing stage. The value of red points a n represents the weight of each gallery feature and is a row of the DFA matrix in Fig. 3. The CRC has fewer 0 values than SRC.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01174v1-figure-f5f6318b-e3db-44c7-8455-28d847fb5e05.jpg)

identification setting. In the second stage, the DFA module is added, and the dictionary learning similarity is used as the fine-tunning signal. The loss function is defined as

$$\mathcal { L } ( \theta, \mathbf A ) = \frac { \alpha } { N } \| \, \overline { X } ^ { L } - \overline { Y } ^ { L } \, \| _ { 2 } ^ { 2 } + \frac { \lambda } { N \times M } \| \, A \, \| _ { p } \quad ( 7 ) \quad \text{II} \quad \text{cien}$$

where θ (Ω l , Ψ Φ) , denotes the parameters in RSA and implements as in [52, 65], α is an indicator function which takes value 1 when the probe and gallery set have the same identity and value -1 for different identity. λ is usually set to 1 for simply. This objective explicitly encourages the feature sets from the same subject to be close while the feature sets of different subjects are far away from each other.

To minimize the loss in Eq. (7), we use the following alternative optimization algorithm to incorporate a parametric free module into deep models:

a) Fix (initialize) θ and minimize L w.r.t. A. The task in this step is to solve the optimal sparse/collaborative coding matrix A. In practice, this is achieved by solving a , 1 · · · a N with parallel computing mode. Eq. (6) can be further rewritten as follows:

$$\underset { a _ { n } } { \min } \frac { 1 } { 2 } a _ { n } ^ { T } \overline { \mathbf Y } ^ { L } \overline { \mathbf Y } ^ { L } a _ { n } - \overline { x } _ { n } ^ { T } \overline { \mathbf Y } ^ { L } a _ { n } + \frac { \lambda } { M } \| a _ { n } \| _ { p } \quad ( 8 ) \quad \text{woff}$$

The feature-sign searching algorithm [20, 64] is utilized to find the optimal a n .

b) Fix A and minimize L w.r.t. θ . One of the remarkable properties of L is differentiable. We update the parameters of RSA using the standard Stochastic Gradient Descent (SGD) by the following gradients of L w.r.t. X and Y

$$\begin{cases} \frac { \partial \mathcal { L } ( \theta ) } { \partial \overline { X } } = \frac { 2 \alpha } { N } ( \overline { X } ^ { L } - \overline { Y } ^ { L } \text{A} ) & \text{avc} \\ \frac { \partial \mathcal { L } ( \theta ) } { \partial \text{Y} } = \frac { - 2 \alpha } { N } ( \overline { X } ^ { L } - \overline { Y } ^ { L } \text{A} ) \text{A} ^ { T } & \text{shc} \\ \text{The overall bi-level training is outlined in Algorithm 1.} & \text{cur} \end{cases}$$

The overall bi-level training is outlined in Algorithm 1. The similarity value learned through PIFR are self-adaptive and sensitive to spatial location and inner/inter set variations. The benefits of utilizing both of the RSA and DFA restructuring modules are demonstrated in experiments.

Analysis. Since IFE extracts { x n } N n =1 from { p n } N n =1 independently, RSA used to reconstruct the representation set is permutation-invariant and the computing of the a n in DFA is a permutation-invariant operation. Combined with the fact that the similarity measurement is permutationinvariant as well, we have the following:

## Proposition 1. The PIFR is permutation-invariant.

B

B

1 2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 1 2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 𝑎1 𝑎2 Normalized Value Normalized Value 𝑎1 𝑎2 Note that we can deconstruct PIFR into two operation stages to align with Eq. (1). In RSA, the residual term is our ϕ and sum operation, which does not act independently on each item but considers the whole set to obtain the embeddings. In the second stage, the RSA can be regarded as ϕ . The sum operator is inherent in ‖ · ‖ 2 2 and glyph[lscript] p -norm. Then ρ is simply the sqrt function. Additionally, instead of a fixed function such as pooling, our inner-set relationship function is parameterized and can adapt to the task at hand.

3

3

𝑎3

𝑎3

A

A

## 4. Numerical Experiments

4

4

𝑎4

𝑎4

5 6 5 6 𝑎5 𝑎6 𝑎5 𝑎6 In this section, we first explore the meaning of the coefficients in DFA (see Fig. 5). Based on the above understanding, we evaluate PIFR on several benchmarks via exhaustive ablation studies. RSA is implemented on our Titan Xp GPU. In testing stage, our gallery Y L s are pre-computed and stored. The average verification time of collaborative DFA is only 93ms for IJB-A, when we use a single Xeon E5 v4 CPU . The detailed architectures for the individual modules are provided in the Supplementary file.

## Unconstrained Set-based face recognition

The IARPA Janus Benchmarks (IJB)-A is a set-based face recognition dataset, where all images and videos are collected in unconstrained environments with large variants in imaging condition and viewpoints. It contains 5,397 still images and 20,412 video frames from 500 identities, and the number of samples in a set ranges from 1 to 190 with an average of 11.4 images and 4.2 videos per subject.

To take advantage of the long-tested image-based network structure and millions of available training images, the backbone of IFE is adapted from the GoogleNet [45] /ResNet50 [16] and trained on 3M Indoor face images [61] or VGGFace2 [4] (3.31M images) respectively. We choose the same training details as [25, 57] for fair comparisons.

In the baseline methods, CNN+Mean glyph[lscript] 2 measures the glyph[lscript] 2 similarity of all X L and Y L pairs. The CNN+AvePool is the average pooling along each feature dimension for aggregation, followed by a linear classifier [61, 25].

For the 1:1 verification task, the performances are shown using the receiver operating characteristics (ROC) curves in Fig. 6. We also report the true accept rate (TAR) vs. false positive rates (FAR) in Table 3.

The performance of 1:N identification is characterized by the cumulative match characteristics (CMC) curves shown in Fig. 6. It is an information retrieval metric, which

-

-

B

Table 3. Evaluation on the IJB-A dataset. For verification, the true accept rates (TAR) vs. false positive rates (FAR) are presented. For identification, the true positive identification rate (TPIR) vs. false positive identification rate (FPIR) and the Rank-1 accuracy are reported.

| Method                                                                                                                             | IFE Backbone                                                | Dataset                                                           | 1:1 Verification TAR                                                                      | 1:1 Verification TAR                                                                      | 1:1 Verification TAR                                                                | 1:N Identification TPIR                                                             | 1:N Identification TPIR                                                             | 1:N Identification TPIR                                                                   |
|------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------|-------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|
|                                                                                                                                    |                                                             |                                                                   | FAR=0.001                                                                                 | FAR=0.01                                                                                  | FAR=0.1                                                                             | FPIR=0.01                                                                           | FPIR=0.1                                                                            | Rank-1                                                                                    |
| Pose-model[41] Masi et al. [42] Adaptation[11] DR-GAN[47] QAN[40] NAN[61] DAC[25]                                                  | VGGNet VGGNet VGGNet CASIA-Net - GoogleNet GoogleNet        | WebFace WebFace VGGNet WebFace - Indoor(3M) Indoor(3M)            | 0.652 ± 0.037 0.725 0.836 ± 0.027 0.539 ± 0.043 0.893 ± 0.039 0.881 ± 0.011 0.893 ± 0.010 | 0.826 ± 0.018 0.886 0.939 ± 0.013 0.774 ± 0.027 0.942 ± 0.015 0.941 ± 0.008 0.954 ± 0.010 | - - 0.979 ± 0.004 - 0.980 ± 0.006 0.978 ± 0.003 0.981 ± 0.008                       | - - 0.774 ± 0.049 - - 0.817 ± 0.041 0.855 ± 0.042                                   | - - 0.882 ± 0.016 - - 0.917 ± 0.009 0.934 ± 0.009                                   | 0.840 ± 0.012 0.906 0.928 ± 0.010 0.855 ± 0.015 0.955 ± 0.006 0.958 ± 0.005 0.973 ± 0.011 |
| RSA DFA( glyph[lscript] 1 ) DFA( glyph[lscript] 2 ) PIFR( glyph[lscript] 1 ) PIFR( glyph[lscript] 2 ) PIFR( glyph[lscript] 2 ) × 2 | GoogleNet GoogleNet GoogleNet GoogleNet GoogleNet GoogleNet | Indoor(3M) Indoor(3M) Indoor(3M) Indoor(3M) Indoor(3M) Indoor(3M) | 0.926 ± 0.011 0.912 ± 0.017 0.914 ± 0.013 0.933 ± 0.015 0.937 ± 0.016 0.938 ± 0.013       | 0.964 ± 0.010 0.957 ± 0.011 0.958 ± 0.006 0.966 ± 0.007 0.968 ± 0.005 0.968 ± 0.006       | 0.986 ± 0.008 0.983 ± 0.005 0.984 ± 0.003 0.987 ± 0.005 0.988 ± 0.004 0.988 ± 0.006 | 0.882 ± 0.042 0.873 ± 0.045 0.875 ± 0.037 0.894 ± 0.039 0.896 ± 0.035 0.897 ± 0.031 | 0.954 ± 0.009 0.940 ± 0.015 0.940 ± 0.010 0.958 ± 0.013 0.958 ± 0.007 0.958 ± 0.009 | 0.984 ± 0.011 0.976 ± 0.009 0.978 ± 0.008 0.985 ± 0.010 0.988 ± 0.006 0.988 ± 0.007       |
| VGG Face2[4] MNet[57]                                                                                                              | ResNet50 ResNet50                                           | VGGFace2 VGGFace2                                                 | 0 . 895 ± 0.019 0.920 ± 0.013                                                             | 0.950 ± 0.005 0.962 ± 0.005                                                               | 0.980 ± 0.003 0.989 ± 0.002                                                         | - -                                                                                 | - -                                                                                 | - -                                                                                       |
| RSA PIFR( glyph[lscript] 2 )                                                                                                       | ResNet50 ResNet50                                           | VGGFace2 VGGFace2                                                 | 0.943 ± 0.008 0.955 ± 0.010                                                               | 0.976 ± 0.006 0.983 ± 0.004                                                               | 0.991 ± 0.002 0.993 ± 0.003                                                         | 0.886 ± 0.041 0.908 ± 0.028                                                         | 0.962 ± 0.009 0.969 ± 0.009                                                         | 0.987 ± 0.005 0.990 ± 0.005                                                               |

( ⊗⊗〉⊕˜∐{(⊕

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01174v1-figure-aae80b26-2d62-4e35-9d03-d9e0410e54c4.jpg)

⊔∐⌈√˜(∑}√]√]√˜(∫∐√˜

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01174v1-figure-266de3dd-79ba-45e7-8c39-e08d25f6545c.jpg)

∫∐{⌋

Figure 6. Average ROC (Left) and CMC (Right) curves of the proposed method (with GoogleNet backbone) and its baselines on the IJB-A dataset over 10 splits.

plots identification rates corresponding to different ranks. A rankk identification rate is defined as the percentage of probe searches whose gallery match is returned within the topk matches. The true positive identification rate (TPIR) vs. false positive identification rate (FPIR), as well as the rank-1 accuracy, are shown in Table 3.

The GoogleNet-based RSA, DFA( glyph[lscript] 2 ), PIFR( glyph[lscript] 2 ) outperforms DAC [25] by 2.6%/3.3%, 2.0%/2.1%, 4.2%/4.5% in terms of TPIR@FPIR= 10 -2 /TAR@FAR= 10 -3 respectively. We note that the training and testing time of PIFR( glyph[lscript] 2 ) are 3 × and 1.5 × smaller than DAC [25]. These results not only show that our model improve both the verification and identification performance significantly, but also indicates the RSA and DFA can be well combined in our alternative optimization framework and are complimentary to each other. The improvement over the previous state-ofthe-art w.r.t. rank-1 accuracy is also more notable than of recent works [61, 25]. Using the ResNet50 backbone, our PIFR( glyph[lscript] 2 ) achieves state-of-the-art 99.0% and outperforms [57] by 3.5% for TAR@FAR= 10 -3 .

The performance of PIFR( glyph[lscript] 2 ) slightly exceeds PIFR( glyph[lscript] 1 ), indicating that the collaborative representation is better for set feature reconstructions. A possible reason is that SRC usually assigns the coefficient strictly to 0 (see Fig. 5), which is similar to hard rather than soft attention [58].

The PIFR( glyph[lscript] 2 ) × 2 represents the symmetric dictionary learning which reconstructs x L n using { y L 1 , · · · , y L M } ,

⊕

Figure 7. On IJB-A dataset, selecting the output layer of IFE (left) and the number of RSA blocks (right) matters. The low-level features are less discriminative, while the higher-level features lose too much spatial information. The performance usually saturated with more than 5 RSA blocks.

and reconstructs y L m using { x L 1 , · · · , x L N } simultaneously. However, the doubled complexity does not lead to improved performance. Although the GoogleNet and ResNet50 are designed in different structures and trained on different datasets, PIFR can be applied on top of either of them.

Then PIFR's sensitivity to the level of feature extraction and the number of RSA blocks is analyzed in Fig. 7.

## Unconstrained video-based face recognition

The Celebrity-1000 dataset is a video-based unconstrained face identification dataset, but used as a typical setbased benchmark. It contains 2.4M frames from 159,726 videos of 1,000 subjects. We follow the protocol in [61, 25] and report the performances in Table 4 &amp; 5 and Fig. 8.

In the closed-set testing, we choose the restructured representations from the 5 th RSA blocks, and use the softmax output of our RSA only model. The RSA outperforms all of the previous methods consistently following the setting without fine-tuning the IFE module on Celebrity-1000 dataset. The video sequences are obviously redundant, and the inner-set correlation modeling does contribute to the improvements over [61]. The low quality of the several former frames also have negative impact of [25].

For the open-set protocol, we further measure the collaborative representation similarity to rank the gallery subjects. We see that our GoogleNet based PIFR outperforms the average pooling baseline and DAC by more than 9% and 3%

Table 4. Closed-set identification performance (rank-1 accuracies), on the Celebrity-1000 dataset. This setting does not have inter-set comparison, and only RSA is necessary.

| Method             | Temp   |   Number of Subjects( closed ) |   Number of Subjects( closed ) |   Number of Subjects( closed ) |   Number of Subjects( closed ) |
|--------------------|--------|--------------------------------|--------------------------------|--------------------------------|--------------------------------|
| Method             | info   |                       100      |                       200      |                       500      |                      1000      |
| NAN(GoogleNet)[61] | no     |                         0.9044 |                         0.8333 |                         0.8227 |                         0.7717 |
| DAC(GoogleNet)[25] | no     |                         0.9137 |                         0.8783 |                         0.8523 |                         0.8353 |
| RSA(GoogleNet)     | no     |                         0.9385 |                         0.9047 |                         0.8859 |                         0.8572 |
| RSA(ResNet50)      | no     |                         0.9458 |                         0.914  |                         0.9943 |                         0.8671 |

Table 5. Open-set identification performance (rank-1 accuracies) on the Celebrity-1000 dataset

| Method                                | Temp   |   Number of Subjects( open ) |   Number of Subjects( open ) |   Number of Subjects( open ) |   Number of Subjects( open ) |
|---------------------------------------|--------|------------------------------|------------------------------|------------------------------|------------------------------|
|                                       | info   |                     100      |                     200      |                     500      |                     800      |
| CNN+Mean glyph[lscript] 2 (GoogleNet) | no     |                       0.8488 |                       0.7988 |                       0.7676 |                       0.7067 |
| AvePool(GoogleNet)                    | no     |                       0.8411 |                       0.7909 |                       0.784  |                       0.7512 |
| NAN(GoogleNet)[61]                    | no     |                       0.8876 |                       0.8521 |                       0.8274 |                       0.7987 |
| DAC(GoogleNet)[25]                    | no     |                       0.9004 |                       0.8715 |                       0.8428 |                       0.8264 |
| RSA(GoogleNet)non-local               | no     |                       0.8987 |                       0.868  |                       0.8376 |                       0.8251 |
| RSA(GoogleNet)w/o ∆                   | no     |                       0.9104 |                       0.8812 |                       0.8545 |                       0.842  |
| RSA(GoogleNet)- x l - 1 n j           | no     |                       0.9097 |                       0.8791 |                       0.8603 |                       0.8388 |
| RSA(GoogleNet)- (ΨΦ) L                | no     |                       0.9297 |                       0.899  |                       0.8764 |                       0.8556 |
| RSA(GoogleNet)                        | no     |                       0.9297 |                       0.8991 |                       0.8763 |                       0.8558 |
| DFA( glyph[lscript] 2 )(GoogleNet)    | no     |                       0.9105 |                       0.8807 |                       0.8562 |                       0.8383 |
| PIFR( glyph[lscript] 2 )(GoogleNet)   | no     |                       0.9355 |                       0.9189 |                       0.8876 |                       0.8604 |
| PIFR( glyph[lscript] 2 )(ResNet50)    | no     |                       0.9497 |                       0.9308 |                       0.8922 |                       0.8713 |

at all measurements. More appealingly, the use of ResNet50 can further enhance our framework and achieve the state-ofthe-art results with more than 4% improvement over [25].

We did not manage to get competitive results using original non-local [52] in our RSA. We note that non-local is also first introduced to set-based recognition in this paper, and can be regarded as a baseline. For ablation study, in RSAw/o ∆ , we excise ∆ in Eq. (2). x l -1 n j refers to using x l -1 n j as residue term instead of the difference in Eq. (2). Their inferior performance demonstrates the effectiveness of our choices. (ΨΦ) L , denotes using embedded Gaussian pair-wise affinity for every RSA blocks, which has similar performance but takes more than 2 × training time.

## Person re-identification

To verify the effectiveness of PIFR on full body images, we carry out additional experiments on the iLIDS-VID [50] benchmark which was created at an airport arrival hall under a multi-camera CCTV network, and contains 300 people in total. Its image sequences (rangings from 23 to 192 images) were accompanied by clothing similarities among people, lighting and viewpoint variations, cluttered background and occlusions. We follow the evaluation protocol from [22] for 10-fold cross evaluation. We sample the frames with a stride of 5, and the remaining frames, in the end, are simply discarded. Notice that since the data pre-processing, training setting and network structure vary in different state-of-theart methods, we only list recent best-performing methods in the tables just for reference. For Re-ID, the widely used Cumulative Match Curve (CMC) is adopted.

Weuse the same IFE as in [22], and set L = 5 . The PIFR with ResNet50 backbone is comparable to the TCP which is based on more powerful ResNet101 and extra training

ψ PIFR↼GoogleNet↽

ψ PIFR↼ResNet↽

Figure 8. CMC curves of different methods on Celebrity 1000. (left) Close-set identification on 1000 subjects, (right) Open-set identification on 800 subjects.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01174v1-figure-428382c6-7bf0-43af-b9c9-f7c053a49928.jpg)

Table 6. Experimental results of the proposed and other comparisons on iLIDS-VID re-identification dataset. The best are in bold while the second best are underlined.

| Method                             | Temp   | iLIDS-VID   | iLIDS-VID   | iLIDS-VID   | iLIDS-VID   |
|------------------------------------|--------|-------------|-------------|-------------|-------------|
|                                    | info   | CMC-1       | CMC-5       | CMC-10      | CMC-20      |
| TAM+SRM[66]                        | yes    | 0.552       | 0.865       | -           | 0.970       |
| ASTPN[59]                          | yes    | 0.620       | 0.860       | 0.940       | 0.980       |
| QAN[40]                            | no     | 0.680       | 0.868       | 0.954       | 0.974       |
| RQEN[44]                           | no     | 0.771       | 0.932       | 0.977       | 0.994       |
| STAN(ResNet50)[22]                 | no     | 0.802       | -           | -           | -           |
| TCP(ResNet101)[39]                 | no     | 0.717       | 0.951       | 0.983       | 0.993       |
| PIFR( glyph[lscript] 2 )(ResNet50) | no     | 0.825       | 0.950       | 0.980       | 0.997       |

data. Benefiting from the more efficient inner-set complementary/diversity exploring and additional inter-set correlation module, PIFR achieves 82.5% rank-1 accuracy dispense with the optical flow information (see Table 6). It outperforms the STAN and TCP by 2.3% and 10.8% respectively, and 2 × faster than STAN w.r.t. training.

## 5. Conclusions

Wepresent a novel permutation-invariant feature restructuring framework (PIFR) for set-based representation and similarity measurement. Both the RSA and DFA module inherit the restructuring idea (in a parametric/non-parametric manner) to take into account the inner/inter-set correlations respectively. It can emphasize the discriminative images, reduces the redundancy, adaptively aligns the probe-gallery set and is scalable to open-set identification. The proposed alternative optimization can be a unified way to train the deep learning and dictionary learning jointly. The PIFR can be a general framework for the task with variable number of order-less samples. We plan to apply it to vehicle re-id, action recognition, relation reasoning etc., in the future.

## 6. Acknowledgement

The funding support from Hong Kong Government General Research Fund GRF (No.152202/14E), Natural Science Foundation of China (NSFC) (No. 61772296, 61627819, 61727818) and Shenzhen fundamental research fund (Grant No. JCYJ20170412170438636) are greatly appreciated.

## References

- [1] P. Battaglia, R. Pascanu, M. Lai, D. J. Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems , pages 4502-4510, 2016. 3
- [2] A. Buades, B. Coll, and J.-M. Morel. A non-local algorithm for image denoising. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on , volume 2, pages 60-65. IEEE, 2005. 3, 4
- [3] A. Buades, B. Coll, and J.-M. Morel. A review of image denoising algorithms, with a new one. Multiscale Modeling &amp;Simulation , 4(2):490-530, 2005. 4
- [4] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman. Vggface2: A dataset for recognising faces across pose and age. In Automatic Face &amp; Gesture Recognition (FG 2018), 2018 13th IEEE International Conference on , pages 67-74. IEEE, 2018. 6, 7
- [5] J.-C. Chen, R. Ranjan, A. Kumar, C.-H. Chen, V. M. Patel, and R. Chellappa. An end-to-end system for unconstrained face verification with deep convolutional neural networks. In IEEE CVPRW , pages 118-126, 2015. 1, 2
- [6] Y.-C. Chen, V. M. Patel, P. J. Phillips, and R. Chellappa. Dictionary-based face recognition from video. In European conference on computer vision , pages 766-779. Springer, 2012. 3
- [7] Y. Chi and F. Porikli. Classification and boosting with multiple collaborative representations. IEEE transactions on pattern analysis and machine intelligence , 36(8):1519-1531, 2014. 3
- [8] F. R. Chung and F. C. Graham. Spectral graph theory . Number 92. American Mathematical Soc., 1997. 4
- [9] R. R. Coifman and S. Lafon. Diffusion maps. Applied and computational harmonic analysis , 21(1):5-30, 2006. 4
- [10] A. Cotter, M. Gupta, H. Jiang, J. Muller, T. Narayan, S. Wang, and T. Zhu. Interpretable set functions. arXiv preprint arXiv:1806.00050 , 2018. 3
- [11] N. Crosswhite, J. Byrne, C. Stauffer, O. Parkhi, Q. Cao, and A. Zisserman. Template adaptation for face verification and identification. In FG , pages 1-8. IEEE, 2017. 7
- [12] Q. Du, M. Gunzburger, R. B. Lehoucq, and K. Zhou. Analysis and approximation of nonlocal diffusion problems with volume constraints. SIAM review , 54(4):667-696, 2012. 4
- [13] H. Edwards and A. Storkey. Towards a neural statistician. arXiv preprint arXiv:1606.02185 , 2016. 3
- [14] G. Gilboa and S. Osher. Nonlocal linear image regularization and supervised segmentation. Multiscale Modeling &amp; Simulation , 6(2):595-630, 2007. 4
- [15] P. Grother and M. Ngan. Face recognition vendor test (frvt). Performance of face identification algorithms , 2014. 1
- [16] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 4, 6
- [17] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560 , 2017. 3
- [18] Y. Hoshen. Vain: Attentional multi-agent predictive modeling. In Advances in Neural Information Processing Systems , pages 2701-2711, 2017. 3
- [19] Z. Huang, J. Wu, and L. Van Gool. Building deep networks on grassmann manifolds. arXiv preprint arXiv:1611.05742 , 2016. 3
- [20] H. Lee, A. Battle, R. Raina, and A. Y. Ng. Efficient sparse coding algorithms. In Advances in neural information processing systems , pages 801-808, 2007. 5, 6
- [21] H. Li, G. Hua, X. Shen, Z. Lin, and J. Brandt. Eigen-pep for video face recognition. In Asian Conference on Computer Vision , pages 17-33. Springer, 2014. 1, 2
- [22] S. Li, S. Bak, P. Carr, and X. Wang. Diversity regularized spatiotemporal attention for video-based person reidentification, 2018. 8
- [23] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprint arXiv:1312.4400 , 2013. 5
- [24] X. Liu. Research on the technology of deep learning based face image recognition. In Thesis , 2019. 1
- [25] X. Liu, K. B.V.K, C. Yang, Q. Tang, and J. You. Dependency-aware attention control for unconstrained face recognition with image sets. In European Conference on Computer Vision , 2018. 1, 2, 3, 4, 5, 6, 7, 8
- [26] X. Liu, Y. Ge, L. Kong, P. Jia, Y. Jane, and J. Lu. Unimodal regularized neuron stick-breaking for ordinal regression. In ArXiv , 2019. 1
- [27] X. Liu, Y. Ge, C. Yang, and P. Jia. Adaptive metric learning with deep neural networks for video-based facial expression recognition. Journal of Electronic Imaging , 27(1):013022, 2018. 1
- [28] X. Liu, Z. Guo, J. Jia, and B. Kumar. Dependency-aware attention control for imageset-based face recognition. In ArXiv , 2019. 1
- [29] X. Liu, X. Han, Y. Qiao, Y. Ge, S. Li, and J. Lu. Unimodaluniform constrained wasserstein training for medical diagnosis. In ArXiv , 2019. 1
- [30] X. Liu, L. Kong, Z. Diao, and P. Jia. Line-scan system for continuous hand authentication. Optical Engineering , 56(3):033106, 2017. 1
- [31] X. Liu, B. V. Kumar, Y. Ge, C. Yang, J. You, and P. Jia. Normalized face image generation with perceptron generative adversarial networks. In 2018 IEEE 4th International Conference on Identity, Security, and Behavior Analysis (ISBA) , pages 1-8, 2018. 1
- [32] X. Liu, B. V. Kumar, P. Jia, and J. You. Hard negative generation for identity-disentangled facial expression recognition. Pattern Recognition , 88:1-12, 2019. 1
- [33] X. Liu, S. Li, L. Kong, W. Xie, P. Jia, J. You, and B. Kumar. Feature-level frankenstein: Eliminating variations for discriminative recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 637-646, 2019. 1
- [34] X. Liu, Z. Li, L. Kong, Z. Diao, J. Yan, Y. Zou, C. Yang, P. Jia, and J. You. A joint optimization framework of lowdimensional projection and collaborative representation for discriminative classification. In 2018 24th International Conference on Pattern Recognition (ICPR) , pages 14931498. 2, 3, 5
- [35] X. Liu, B. Vijaya Kumar, J. You, and P. Jia. Adaptive deep metric learning for identity-aware facial expression recognition. In CVPR Workshops , pages 20-29, 2017. 1
- [36] X. Liu, Y. Zou, T. Che, J. You, and K. B.V.K. Conservative wasserstein training for pose estimation. In ICCV , 2019. 1
- [37] X. Liu, Y. Zou, L. Kong, Z. Diao, J. Yan, J. Wang, S. Li, P. Jia, and J. You. Data augmentation via latent space interpolation for image classification. In 24th International Conference on Pattern Recognition (ICPR) , pages 728-733, 2018. 1
- [38] X. Liu, Y. Zou, Y. Song, C. Yang, J. You, and B. K Vijaya Kumar. Ordinal regression with neuron stick-breaking for medical diagnosis. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 0-0, 2018. 1
- [39] Y. Liu, G. Song, J. Shao, X. Jin, and X. Wang. Transductive centroid projection for semi-supervised large-scale recognition. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 70-86, 2018. 8
- [40] Y. Liu, J. Yan, and W. Ouyang. Quality aware network for set to set recognition. In Proc. IEEE Int. Conf. Comput. Vision Pattern Recognit. , pages 5790-5799, 2017. 1, 3, 7, 8
- [41] I. Masi, S. Rawls, G. Medioni, and P. Natarajan. Pose-aware face recognition in the wild. In IEEE CVPR , pages 48384846, 2016. 7
- [42] I. Masi, A. T. Trn, T. Hassner, J. T. Leksut, and G. Medioni. Do we really need to collect millions of faces for effective face recognition? In ECCV , pages 579-596. Springer, 2016. 7
- [43] O. M. Parkhi, A. Vedaldi, A. Zisserman, et al. Deep face recognition. In BMVC , volume 1, page 6, 2015. 1, 2
- [44] G. Song, B. Leng, Y. Liu, C. Hetang, and S. Cai. Regionbased quality estimation network for large-scale person reidentification, 2017. 8
- [45] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2818-2826, 2016. 3, 4, 6
- [46] Y. Tao, Q. Sun, Q. Du, and W. Liu. Nonlocal neural networks, nonlocal diffusion and nonlocal modeling. arXiv preprint arXiv:1806.00681 , 2018. 4
- [47] L. Tran, X. Yin, and X. Liu. Disentangled representation learning gan for pose-invariant face recognition. In CVPR , volume 3, page 7, 2017. 7
- [48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998-6008, 2017. 3
- [49] R. Wang, H. Guo, L. S. Davis, and Q. Dai. Covariance discriminative learning: A natural and efficient approach to image set classification. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on , pages 2496-2503. IEEE, 2012. 3
- [50] T. Wang, S. Gong, X. Zhu, and S. Wang. Person reidentification by video ranking. Lecture Notes in Computer Science , page 688703, 2014. 8
- [51] W. Wang, R. Wang, Z. Huang, S. Shan, and X. Chen. Discriminant analysis on riemannian manifold of gaussian dis-
- tributions for face recognition with image sets. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2048-2057, 2015. 3
- [52] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 3, 4, 6, 8
- [53] N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti. Visual interaction networks: Learning a physics simulator from video. In Advances in Neural Information Processing Systems , pages 4539-4547, 2017. 3
- [54] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma. Robust face recognition via sparse representation. IEEE transactions on pattern analysis and machine intelligence , 31(2):210-227, 2009. 3
- [55] Y. Wu, V. Jarich, M. Mukunoki, and M. Minoh. Collaborative representation for classification, sparse or non-sparse? arXiv preprint arXiv:1403.1353 , 2014. 3
- [56] W. Xie, L. Shen, and A. Zisserman. Comparator networks. In European Conference on Computer Vision , 2018. 2, 3, 5
- [57] W. Xie and A. Zisserman. Multicolumn networks for face recognition. 2018. 6, 7
- [58] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning , pages 2048-2057, 2015. 7
- [59] S. Xu, Y. Cheng, K. Gu, Y. Yang, S. Chang, and P. Zhou. Jointly attentive spatial-temporal pooling networks for video-based person re-identification. 2017 IEEE International Conference on Computer Vision (ICCV) , Oct 2017. 8
- [60] F. S. Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales. Learning to compare: Relation network for fewshot learning. 2018. 3
- [61] J. Yang, P. Ren, D. Zhang, D. Chen, F. Wen, H. Li, and G. Hua. Neural aggregation network for video face recognition. In IEEE CVPR , pages 4362-4371, 2017. 1, 2, 3, 4, 5, 6, 7, 8
- [62] X. Yin and X. Liu. Multi-task convolutional neural network for pose-invariant face recognition. IEEE Transactions on Image Processing , 2017. 1, 5
- [63] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. In Advances in Neural Information Processing Systems , pages 3391-3401, 2017. 2
- [64] L. Zhang, M. Yang, and X. Feng. Sparse representation or collaborative representation: Which helps face recognition? In Computer vision (ICCV), 2011 IEEE international conference on , pages 471-478. IEEE, 2011. 3, 5, 6
- [65] B. Zhou, A. Andonian, and A. Torralba. Temporal relational reasoning in videos. In ECCV , 2018. 3, 4, 6
- [66] Z. Zhou, Y. Huang, W. Wang, L. Wang, and T. Tan. See the forest for the trees: Joint spatial and temporal recurrent neural networks for video-based person re-identification. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Jul 2017. 8
Xiaofeng Liu Zhenhua Guo Site Li Lingsheng Kong Ping Jia Jane You B. V. K. Kumar
Permutation-invariant Feature Restructuring for Correlation-aware Image Set-based Recognition
We consider the problem of comparing the similarity of image sets with
variable-quantity, quality and un-ordered heterogeneous images. We use feature
restructuring to exploit the correlations of both inner$\&$inter-set images.
Specifically, the residual self-attention can effectively restructure the
features using the other features within a set to emphasize the discriminative
images and eliminate the redundancy. Then, a sparse/collaborative
learning-based dependency-guided representation scheme reconstructs the probe
features conditional to the gallery features in order to adaptively align the
two sets. This enables our framework to be compatible with both verification
and open-set identification. We show that the parametric self-attention network
and non-parametric dictionary learning can be trained end-to-end by a unified
alternative optimization scheme, and that the full framework is
permutation-invariant. In the numerical experiments we conducted, our method
achieves top performance on competitive image set/video-based face recognition
and person re-identification benchmarks.