summaries: This paper presents a model for detecting irregularities in images, such as would occur in manufacturing acceptance testing.  As a form of anomaly detection, the claims made are for a principled modeling approach and an adaptive algorithm quick to learn from just a few samples.   Based on "energy based models" for modeling data densities, the paper claims improvements to avoid the need for retraining for new tasks, such as, instead of synthesizing negative samples rom noise, using a more targeted method of “learning from inpainting” operation to learn anomalies.    The paper proposes a framework for anomaly detection and localization that allows fast adaptation to new tasks.  Specifically, the authors propose an energy-based model (EBM) with an adaptive sparse coding layer directly trained with normal features of a target task.  A meta-learning process is followed to extract common knowledge across tasks enabling few shots adaptation.  Shrinkage functions, sparse coding with large receptive fields, and learning by inpainting are introduced to improve and accelerate the EBM training.  This paper proposes an image classification system, where a set of normal patterns are stored as a "dictionary," and the degree of deviation is used as the anomaly score.    The overall architecture seems to feature a conventional deep encoder and a pattern matching module that compares the encoded latent vector against the pattern dictionary. The pattern matching is done by solving the lasso regression problem.   The authors propose a certain online learning approach, following existing few-shot learning methods, and also a synthetic sample generation approach using random perturbation combined with a gradient method.  The paper presents an anomaly detection algorithm that uses an EBM (Energy Based Model) to distinguish between 'normal' and 'anomaly'. This model generates pseudo-anomaly instances on-the-fly for each 'normal' instance and then learns to assign low energy to normal instances and high energy to pseudo-anomaly instances. The model is further designed to be able to adapt quickly (few 'normal' labeled examples) to new tasks.
abstractText: The ability to detect anomaly has long been recognized as an inherent human ability, yet to date, practical AI solutions to mimic such capability have been lacking. This lack of progress can be attributed to several factors. To begin with, the distribution of “abnormalities” is intractable. Anything outside of a given normal population is by definition an anomaly. This explains why a large volume of work in this area has been dedicated to modeling the normal distribution of a given task followed by detecting deviations from it. This direction is however unsatisfying as it would require modeling the normal distribution of every task that comes along, which includes tedious data collection. In this paper, we report our work aiming to handle these issues. To deal with the intractability of abnormal distribution, we leverage Energy Based Model (EBM). EBMs learn to associates low energies to correct values and higher energies to incorrect values. As its core, the EBM employs Langevin Dynamics (LD) in generating these incorrect samples based on an iterative optimization procedure, alleviating the intractable problem of modeling the world of anomalies. Then, in order to avoid training an anomaly detector for every task, we utilize an adaptive sparse coding layer. Our intention is to design a plug and play feature that can be used to quickly update what is normal during inference time. Lastly, to avoid tedious data collection, this mentioned update of the sparse coding layer needs to be achievable with just a few shots. Here, we employ a meta learning scheme that simulates such a few shot setting during training. We support our findings with strong empirical evidence.
references.authors: Jinwon An Sungzoon Cho
references.title: Variational autoencoder based anomaly detection using reconstruction probability
references.venue: Special Lecture on IE,
references.authors: Abdourrahmane M Atto Dominique Pastor Gregoire Mercier
references.title: Smooth sigmoid wavelet shrinkage for non-parametric estimation
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing,
references.authors: Yuequan Bao Zhiyi Tang Hui Li Yufeng Zhang
references.title: Computer vision and deep learning–based data anomaly detection method for structural health monitoring
references.venue: Structural Health Monitoring,
references.authors: Paul Bergmann Sindy Löwe Michael Fauser David Sattlegger Carsten Steger
references.title: Improving unsupervised defect segmentation by applying structural similarity to autoencoders
references.authors: Paul Bergmann Michael Fauser David Sattlegger Carsten Steger
references.title: Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Richard J Bolton David J Hand
references.title: Statistical fraud detection: A review
references.venue: Statistical science,
references.authors: Louis Clouâtre Marc Demers
references.title: Figr: Few-shot image generation with reptile
references.venue: arXiv preprint arXiv:1901.02199,
references.authors: Niv Cohen Yedid Hoshen
references.title: Sub-image anomaly detection with deep pyramid correspondences
references.venue: arXiv preprint arXiv:2005.02357,
references.authors: Ingrid Daubechies Michel Defrise Christine De Mol
references.title: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint
references.venue: Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences,
references.authors: David Dehaene Oriel Frigo Sébastien Combrexelle Pierre Eline
references.title: Iterative energy-based projection on a normal data manifold for anomaly localization
references.authors: Yilun Du Igor Mordatch
references.title: Implicit generation and generalization in energy-based models
references.venue: NeurIPS,
references.authors: Yilun Du Shuang Li Joshua Tenenbaum Igor Mordatch
references.title: Improved contrastive divergence training of energy based models
references.venue: International Conference on Machine Learning,
references.authors: Chelsea Finn Pieter Abbeel Sergey Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In International Conference on Machine Learning,
references.authors: Chelsea Finn Kelvin Xu Sergey Levine
references.title: Probabilistic model-agnostic meta-learning
references.venue: arXiv preprint arXiv:1806.02817,
references.authors: Ergin Utku Genc Nilesh Ahuja Ibrahima J Ndiour Omesh Tickoo
references.title: Energy-based anomaly detection and localization
references.venue: arXiv preprint arXiv:2105.03270,
references.authors: Spyros Gidaris Nikos Komodakis
references.title: Generating classification weights with gnn denoising autoencoders for few-shot learning
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Jonathan Gordon John Bronskill Matthias Bauer Sebastian Nowozin Richard E Turner
references.title: Metalearning probabilistic inference for prediction
references.venue: arXiv preprint arXiv:1805.09921,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Geoffrey E Hinton
references.title: Training products of experts by minimizing contrastive divergence
references.venue: Neural computation,
references.authors: Yuling Jiao Bangti Jin Xiliang Lu
references.title: Iterative soft/hard thresholding with homotopy continuation for sparse recovery
references.venue: IEEE Signal Processing Letters,
references.authors: Ming-Yu Liu Xun Huang Arun Mallya Tero Karras Timo Aila Jaakko Lehtinen Jan Kautz
references.title: Few-shot unsupervised image-to-image translation
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: W. Liu D. Lian W. Luo S. Gao
references.title: Future frame prediction for anomaly detection – a new baseline
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Wenqian Liu Runze Li Meng Zheng Srikrishna Karanam Ziyan Wu Bir Bhanu Richard J Radke Octavia Camps
references.title: Towards visually explaining variational autoencoders
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Cewu Lu Jianping Shi Jiaya Jia
references.title: Abnormal event detection at 150 fps in matlab
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Yiwei Lu K Mahesh Kumar Seyed shahabeddin Nabavi Yang Wang
references.title: Future frame prediction using convolutional vrnn for anomaly detection
references.venue: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS),
references.authors: Yiwei Lu Frank Yu Mahesh Kumar Krishna Reddy Yang Wang
references.title: Few-shot scene-adaptive anomaly detection
references.venue: In European Conference on Computer Vision,
references.authors: Weixin Luo Wen Liu Shenghua Gao
references.title: A revisit of sparse coding based anomaly detection in stacked rnn framework
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Vijay Mahadevan Weixin Li Viral Bhalodia Nuno Vasconcelos
references.title: Anomaly detection in crowded scenes
references.venue: IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
references.authors: Biswanath Mukherjee L Todd Heberlein Karl N Levitt
references.title: Network intrusion detection
references.venue: IEEE network,
references.authors: Eric Nalisnick Akihiro Matsukawa Yee Whye Teh Dilan Gorur Balaji Lakshminarayanan
references.title: Do deep generative models know what they don’t know
references.venue: arXiv preprint arXiv:1810.09136,
references.authors: Radford M Neal
references.title: Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo
references.authors: Trong-Nguyen Nguyen Jean Meunier
references.title: Anomaly detection in video sequence with appearancemotion correspondence
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Afrooz Purarjomandlangrudi Amir Hossein Ghapanchi Mohammad Esmalifalak
references.title: A data mining approach for fault diagnosis: An application of anomaly detection
references.venue: algorithm. Measurement,
references.authors: Siyuan Qiao Chenxi Liu Wei Shen Alan L Yuille
references.title: Few-shot image recognition by predicting parameters from activations
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Prajit Ramachandran Barret Zoph Quoc V Le
references.title: Searching for activation functions
references.venue: arXiv preprint arXiv:1710.05941,
references.authors: Sachin Ravi Hugo Larochelle
references.title: Optimization as a model for few-shot learning
references.venue: International Conference on Learning Representations,
references.authors: Andrei A Rusu Dushyant Rao Jakub Sygnowski Oriol Vinyals Razvan Pascanu Simon Osindero Raia Hadsell
references.title: Meta-learning with latent embedding optimization
references.venue: International Conference on Learning Representations,
references.authors: Thomas Schlegl Philipp Seeböck Sebastian M Waldstein Ursula Schmidt-Erfurth Georg Langs
references.title: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery
references.venue: In International conference on information processing in medical imaging,
references.authors: Vikash Sehwag Mung Chiang Prateek Mittal
references.title: Ssd: A unified framework for self-supervised outlier detection
references.venue: arXiv preprint arXiv:2103.12051,
references.authors: Shelly Sheynin Sagie Benaim Lior Wolf
references.title: A hierarchical transformation-discriminating generative model for few shot anomaly detection
references.venue: arXiv preprint arXiv:2104.14535,
references.authors: Jake Snell Kevin Swersky Richard S Zemel
references.title: Prototypical networks for few-shot learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Ricardo Vilalta Youssef Drissi
references.title: A perspective view and survey of meta-learning
references.venue: Artificial intelligence review,
references.authors: Oriol Vinyals Charles Blundell Timothy Lillicrap Koray Kavukcuoglu Daan Wierstra
references.title: Matching networks for one shot learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Max Welling Yee W Teh
references.title: Bayesian learning via stochastic gradient langevin dynamics
references.venue: In Proceedings of the 28th international conference on machine learning
references.authors: Jianwen Xie Yang Lu Song-Chun Zhu Yingnian Wu
references.title: A theory of generative convnet
references.venue: In International Conference on Machine Learning,
references.authors: Haowen Xu Wenxiao Chen Nengwen Zhao Zeyan Li Jiahao Bu Zhihan Li Ying Liu Youjian Zhao Dan Pei Yang Feng
references.title: Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications
references.venue: In Proceedings of the 2018 World Wide Web Conference,
references.authors: Jaesik Yoon Taesup Kim Ousmane Dia Sungwoong Kim Yoshua Bengio Sungjin Ahn
references.title: Bayesian model-agnostic meta-learning
references.venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,
references.authors: Qiang Zhang Aldo Lipani Shangsong Liang Emine Yilmaz
references.title: Reply-aided detection of misinformation via bayesian deep learning
references.venue: In The world wide web conference,
references.authors: Bin Zhao Li Fei-Fei Eric P Xing
references.title: Online detection of unusual events in videos via dynamic sparse coding
references.authors: Junbo Zhao Michael Mathieu Yann LeCun
references.title: Energy-based generative adversarial network
references.venue: arXiv preprint arXiv:1609.03126,
references.authors: Yang Zhao Jianwen Xie Ping Li
references.title: Learning energy-based generative models via coarse-to-fine expanding and sampling
references.venue: In International Conference on Learning Representations,
sections.heading: 1 INTRODUCTION
sections.text: Anomaly detection is an important area of study in the field of artificial intelligence. It has found utility in computer vision applications such as industrial inspection (Bergmann et al., 2019) and video surveillance (Liu et al., 2018; Zhao et al., 2011; Nguyen & Meunier, 2019), in the context of abuse prevention such as misinformation, fraud and network intrusion detection (Zhang et al., 2019; Bolton & Hand, 2002; Mukherjee et al., 1994), and others such as system health monitoring and fault detection (Bao et al., 2019; Purarjomandlangrudi et al., 2014). In this paper, we propose an approach for detecting anomaly in images, where we have carefully designed steps to handle some of the bigger issues that have prevented the deployment of image anomaly detection in the real-world.
  Image anomaly detection can generally be defined as the identification of abnormalities in a given image. An exact definition of abnormality in this case is elusive because abnormality can be derived from any unknown distribution outside of a normal population. Many studies have hence focused on modeling the normal population instead of learning irregularities, where the goal is to capture the shared concept among all of the normal data as one or several reference models. This process usually will require investing significant efforts in curating a large enough set of normal samples for each task, after which anomaly is detected as deviations from the reference model(s) (An & Cho, 2015; Xu et al., 2018). Recent work from (Sheynin et al., 2021) provides algorithms that utilize only a few normal samples to train models from scratch. However, the models still have to be provisioned for each new task, which requires considerable human efforts and expertise, and thus lack the fast deployment criterion that is often time critical for real-world applications. In view of these challenges, our goals for this work are threefold. We are interested in designing an anomaly detection system that is capable of: (G1) modeling the normal population while at the same time has a principled approach towards modeling the abnormalities; (G2) quickly adapting to a new task at inference time; and (G3) requiring only a few normal shots to update itself to the new task at hand.
  For (G1), we introduce the class of Energy Based Model (EBM), which is an important family of generative models (Zhao et al., 2016; Du & Mordatch, 2019; Xie et al., 2016). EBMs have been shown to demonstrate superior capability on modeling data density and localizing anomaly (Genc et al., 2021). For our purpose, the EBM we adopted learn to assign low energy to normal samples but high energy to abnormal samples. More importantly, the abnormal samples are generated with a procedure known as Langevin Dynamics (LD) (Welling & Teh, 2011), which, in its original form, starts with a noise image (see App. Fig C) and gradually samples from the distribution along the direction of lower energy. This lends itself gracefully to utilizing the generated intermediate samples as negative/abnormal. The LD procedure is then coupled with a contrastive divergence loss (Hinton, 2002) that aims to maximize the energy differences between the normal and abnormal samples.
  To achieve (G2), we propose an adaptive sparse coding layer that is attached to the deep feature extractor in the EBM. The deep features are projected into a set of feature vectors along the spatial axes, after which each vector is forwarded to the sparse coding layer, where the dictionary is constructed with the features of a few normal samples of the given task. In essence, the input representation has been decomposed into a linear combination of normal features with the sparsity constraint imposed. The final energy score is measured by the distance between the original and the reconstructed features (after the sparse coding layer). Under this scheme, the dictionary for a particular task is not obtained by learning, but instead is constructed by the feature representations of a few normal samples during inference. As a result, this simple “plug-and-play” trick allows the model to be adapted to novel tasks promptly without re-training. Further, we expect that the dictionary, which is formed by normal features, will not be able to explain the abnormal samples well, causing relatively high reconstruction error that lends itself for subsequent detection. As a bonus, a backward pass from the reconstruction error to the image is also additionally useful for localizing the abnormal regions.
  Towards (G3), we utilize meta learning (Vilalta & Drissi, 2002; Finn et al., 2017) to simulate the scenario of being given a new task with a few normal shots to update the dictionary, followed by training the EBM. This is accomplished by episodic training, where in each episode the model is adapted to a held back task that is given a few normal samples. To accelerate the EBM training, we introduce “learning from inpainting”, a simple yet effective strategy for synthesizing hard abnormal samples quicker by starting the LD procedure with a synthesized image that is simply a normal sample with a noise patch injected as opposed to a noise image that is traditionally what is used.
  We show the proposed framework is able to efficiently adapt to a novel task (e.g., a new object category or scenes from a new camera) with a few normal samples on both industrial inspection and video surveillance tasks. We provide both qualitative and quantitative results to demonstrate that our method outperforms other adaptive frameworks and is comparable to methods that rely on large amount of normal samples.
sections.heading: 2 BACKGROUNDS
sections.text: We briefly introduce two key ingredients of the proposed method: EBMs and sparse coding.
  Energy-based Model In EBMs, the goal is to learn an energy function Eθ(x) : Rd → R which parametrizes the data density pθ(x) as:
  pθ(x) = exp(−Eθ(x))∫ x exp(−Eθ(x)) , (1)
  where θ is the parameter of the energy function and Zθ = ∫ x exp(−Eθ(x)) is the partition function. Approximating the true data distribution pdata(x) is equivalent to minimizing the expected negative log-likelihood function over the data distribution, defined by the loss function:
  LML = Ex∼pdata(x)[− log pθ(x)] = Ex∼pdata(x)[Eθ(x) + logZθ]. (2) As the computation of LML involves an intractable term Zθ, the common practice is to represent the gradient of LML as,
  ∇θLML = Ex+∼pdata(x)[∇θEθ(x+)] − Ex−∼pθ(x)[∇θEθ(x−)]. (3) This objective is commonly referred to as the contrastive divergence (Hinton, 2002), which decreases the energy of positive data samples x+ from the true distribution (normal samples in our use case) and increases the energy of negative samples x− from the model pθ (synthesized abnormal samples). In practice, the synthesized negative samples are achieved through Langevin dynamics (Welling &
  Teh, 2011), which samples along the direction of decreasing energy score, typically starting from a given noise image:
  x̃k = x̃k−1 − β 2 ∇xEθ(x̃k−1) + ωk, ωk ∼ N (0, βI), (4)
  where β is the step size. The synthesizing ability of EBMs enables generating abnormal samples to help in learning a more accurate data density, and is often touted as the one of the advantages of using an EBM.
  Sparse coding. Approximating a signal z ∈ Rd with the sparse linear combination over a dictionary D ∈ Rd×k can be expressed as:
  min α
  1 2 ||z−Dα||22 + λ||α||1, (5)
  where α is the sparse coefficients, with its sparsity (l1 norm) and λ is the weight of the sparsity constraint. Dα is a sparse approximation to the original signal z. In practice, finding the dictionary atoms and the sparse coefficients is usually formulated as an optimization problem.
  In this paper, we adopt Iterative Soft Thresholding Continuation (ISTC) (Jiao et al., 2017) to convert this optimization problem into linear operations with a non-linear shrinkage function, which allows sparse coding to be seamlessly integrated into the deep neural networks. To compute a sparse coefficient α, ISTC performs iterations of gradient steps on reconstruction ||z −Dα||2 and a proximal projection step to increase coefficient sparsity.
  Formally, initializing the coefficients at the first step α0 with all zeros, each step of ISTC refines the sparse code with descending values of λ from λmax to λ?: each step of ISTC is expressed as:
  αn+1 = σ(αn + D >(z−Dαn), λn), with λn = λmax λmax λ?
  −n/N , (6)
  where σ(·, ·) here is a shrinkage function that truncates small values (lower than λ) of the coefficients to 0 to enforce sparsity, and can be easily implemented by a customized ReLU activation function:
  σ(z, λ) = sgn(z)(max(|z| − λ, 0)) = sgn(z)ReLU(|z| − λ). (7)
sections.heading: 3 PROPOSED METHOD
sections.text: The objective of our proposed method, when given an input image or video frame, is to output an anomaly score indicating how likely this input deviates from the normal, and additionally, a pixel map grounding abnormal regions. In this section, we describe the proposed fast adaptive anomaly detection framework in details. In Section 3.1, we introduce the adaptive EBM which consists of a deep feature extractor followed by an adaptive sparse coding layer. From there, we further show that utilizing larger receptive field in the sparse coding could improve training robustness (Section 3.1.1), and applying smoothed shrinkage functions could help speed up convergence (Section 3.1.2). In Section 3.2, we describe the episodic training regime on various anomaly detection tasks that mimics few-shot adaptation in the meta-testing stage while learning common knowledge across tasks. Finally, Instead of synthesizing negative samples (anomaly) directly from noise, we introduce a simple but effective “learning from inpainting” operation to accelerate the training in Section 3.3.
sections.heading: 3.1 ADAPTIVE ENERGY-BASED MODEL
sections.text: An EBM is a form of generative model and it is widely used for modeling data density and sampling. While there has been recent work (Genc et al., 2021) applying EBM to anomaly detection, it still requires re-training for each new task. To efficiently adapt the EBM to novel tasks, we introduce an adaptive sparse coding layer which is conditioned on the dictionary constructed by the features of normal samples. Specifically, as illustrated in Fig 1, given an input image, x ∈ R3×h×w, we first obtain the corresponding feature z ∈ Rd×h′×w′ from the deep feature extractor Ψ with parameters θ, so that z = Ψ(x; θ). All feature vectors along spatial axes of z are then sparsely decomposed through the sparse coding layer over a task-specific dictionary D ∈ Rd×Kh′w′ , which contains the features of K normal samples of the current task as shown in the Fig 1(a). Each feature vector
  of the normal sample feature is then directly used as an atom in the task dictionary. The decomposed coefficients are α = S(z;D), where α ∈ RKh′w′×h′×w′ and S denotes the iterative sparse decomposition process of (6). By multiplying the coefficient α with the dictionary D, we obtain the reconstructed features z′ = Dα. The sparsity regularization to α is important, as it encourages input features to be reconstructed by simple combinations of dictionary atoms (normal features), so that it would be difficult for features of abnormal samples to be well-approximated, therefore producing higher reconstruction errors that make it conducive for detecting anomalies. From here, the final energy score is formulated as the mean squared error (MSE) between the original and the reconstructed features:
  Eθ(x;D) = MSE(z, z′) = ||Ψ(x; θ)−DS ( Ψ(x; θ);D ) ||2. (8)
  In effect, Eqn. 8 depicts a conditional EBM, which is conditioned on the task-specific D formed by normal features. In the following sections, we will discuss how to make the training of this adaptive structure more robust.
sections.heading: 3.1.1 SPARSE CODING WITH RECEPTIVE FIELD.
sections.text: As discussed in Section 3.1, the input feature z is represented as h′ × w′ of d-dim feature vectors and they are treated independently while passing through the sparse coding layer. The region of the input image that affects one feature vector is determined by the receptive field of the feature extractor. The trade-off is that a small receptive field may not capture enough contextual information, while applying a large receptive field would make feature maps spatially coarse and make it hard to spot small anomaly regions. To solve this dilemma, instead of carefully tuning the receptive field of each layer of the feature extractor, we introduce a simple yet effective technique of applying the receptive field on the sparse coding layer. Specifically, as Fig 2 shows, rather than performing sparse coding to each individual d-dim feature vectors, we apply it on d × l × l volumes centered around each feature vector, where l is the receptive field. This is equivalent to applying a l × l, sliding window on spatial axes of the feature map and can be easily implemented by image to column (Im2Col) operation. Then we flatten the feature volumes into dl2-dim vectors and adjust the shape of the dictionary accordingly. In this way, we are able to capture contextual information without needing to carefully tune the architecture of the feature extractor and we show in the later experiments that this technique improves the robustness of the network on different types of objects.
sections.heading: 3.1.2 SHRINKAGE FUNCTION
sections.text: The effectiveness of training the EBM for localizing anomaly regions heavily depends on the gradient propagation from later to earlier layers. It is shown in (Du et al., 2021) that smooth activation functions like Swish Ramachandran et al. (2017) could be beneficial here. Notably, the gradients of
  …
  𝐳
  𝑑
  𝑙
  𝑙
  Im2Col 𝑑×𝑙×𝑙
  Figure 2: Illustration of sparse coding layer with l× l receptive filed. 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 X
  1.00
  0.75
  0.50
  0.25
  0.00
  0.25
  0.50
  0.75
  1.00
  Y
  Shrinkage functions ( = 0.25) Hard Shrinkage Linear SigShrink =1.0 SigShrink =2.5 SigShrink =5.0 SigShrink =10.0 SigShrink =100.0 SigShrink =1000.0
  Figure 3: Shrinkage functions.
  the dictionary D are determined by the sparse coding coefficients α as shown in Eqn. 6. However, the sparsity constraint of α would turn off the gradient computation of many elements in D and this could be detrimental during the early stage of the training. To alleviate the sparse gradient issue, we replace the RELU-like shrinkage function in Eqn. 7 with its smoothed counterparts by introducing the Sigmoid based shrinkage functions (SigShrink). The SigShink is originally proposed for non-parametric signal estimation in Atto et al. (2008), and can be defined as:
  στ (z, λ) = z
  1 + exp(−τ(|z| − λ)) , (9)
  where τ is the hyperparameter of smoothness. We present visualizations of the hard shrinkage function Eqn. 7 and SigShrink with different values of τ in Fig 3. Comparing to the hard shrinkage function which truncates small values into zeros, the SigShrink with a large τ can sharply force small values to near-zeros. Therefore, the SigShrink will guarantee non-zero gradients everywhere.
sections.heading: 3.2 EPISODIC TRAINING
sections.text: To train the proposed adaptive EBM, we perform episodic training that is widely adopted by metalearning based few-shot learning tasks Finn et al. (2017); Snell et al. (2017). Following the terminology of few-shot learning, in each training episode, the model is adapted and tested with a task sampled from the underlying task distribution. Specifically, the model is adapted to a support set of the given task, then a query set with ground truth labels is applied to evaluate the adaptation, which is used to update the model parameters accordingly. As shown in Fig 4, the support set contains K normal samples {xk}Kk=1 of the current task, where K is usually a small number. The feature representations zk = Ψ(xk; θ) of these normal samples are plugged into the dictionary Di ∈ Rd×Kh ′w′ corresponding to the i-th task during the i-th episode to adapt the dictionary to the normal samples of the task. After that, the adapted model is measured by a query set consisting of M normal samples {x̂m}Mm=1 andM abnormal samples {x̂′m}Mm=1. Note that there is no actual abnormal samples given during training, instead, they are iteratively sampled from the EBM and we will discuss the initialization and sampling of these synthetic samples in details in Section 3.3. Recall that the training of EBM with contrastive divergence described in Eqn. 3 requires the estimation of energy scores of both positive samples from the true data distribution and negative samples sampled from the model distribution. The positive energy can be estimated empirically with normal samples from the query set. The negative energy can be estimated by performing the MCMC-based (Markov Chain Monte Carlo) sampling technique (Neal et al., 2011; Welling & Teh, 2011), typically Langevin Dynamics as described in Eqn. 4. Denoting the output of Langevin dynamics (sampled abnormal samples) with the initialization x̂′m as LD(x̂′m), we now have the empirical estimation of the contrastive divergence of the i-th episode as:
  Lcd = 1
  m M∑ m=1 [ Eθ(xm;Di)− Eθ(LD ( x̂′m);Di )] . (10)
  With the energy score equivalent to the feature reconstruction error in Eqn. 8, minimizing Lcd encourages normal features to be well-reconstructed by a sparse linear combination of dictionary atoms while the features from abnormal samples tend to produce relatively higher reconstruction errors so that they can be easily spotted.
sections.heading: 3.3 SYNTHESIZING NEGATIVE SAMPLES
sections.text: Typical EBM training with contrastive divergence conducts negative sampling from the modeled density using techniques such as Langevin Dynamics, which applies gradient descent to a noise initialization (App. Fig C) with small step size and large number of steps Du & Mordatch (2019). Such negative sampling steps can be costly and we argue that it is unnecessary in our case. Instead, we introduce a new strategy of “learning by inpainting”. Starting from a positive query sample x̂m, we synthesize the corresponding negative sample x̂′m by randomly placing a small uniform noise patch on the image. The Langevin Dynamics procedure is then initialized with the resulting image instead of a noise image. As the Langevin Dynamics proceeds, synthesized abnormal samples LD(x̂′m) are inpainted along the direction of “normal”, x̂m, and we introduce the following reconstruction loss:
  Lrec = MSE(LD(x̂′m), x̂m). (11)
  The gradient map from Lrec reveals anomaly regions, which helps with localization. We show in Fig 4(a) that, starting from a synthesized abnormal sample, only 5 steps of Langevin dynamic would be sufficient to make it visually close to the corresponding normal sample during training, serving as “hard negatives” that further facilitates the learning. The final loss of the episodic training is simply:
  L = η0Lrec + η1Lcd, (12) where η0 and η1 are hyperparameters balancing two loss terms.
sections.heading: 4 RELATED WORK
sections.heading: 4.1 ANOMALY DETECTION
sections.text: Sparse coding. Early efforts on adopting sparse coding in anomaly detection are based on optimization (with L1 penalty) (Lu et al., 2013; Zhao et al., 2011). Recent advances on iterative sparse thresholding algorithms (Daubechies et al., 2004; Jiao et al., 2017) allow seamless integration of online sparse coding with deep neural networks, and (Luo et al., 2017) formulates the sparse coding as stack RNNs for video anomaly detection.
  Generative models. Generative models are widely utilized in anomaly detection due to the capability in modeling the density of desired data distribution. Early efforts on variational autoencoders (VAE) based methods (An & Cho, 2015; Xu et al., 2018) are arguably having hard time calibrating uncertainties in novel samples (Nalisnick et al., 2018), accurately localizing abnormal regions through reconstruction errors (Dehaene et al., 2020). Recent efforts have explored variant generative architectures like energy-based models (EBM) (Genc et al., 2021), GANs (Sheynin et al., 2021), and combining VAE with EBM (Dehaene et al., 2020). Various methods also exploit intra-image structures (Cohen & Hoshen, 2020; Bergmann et al., 2018), cross-frame consistency (Lu et al., 2019), and motion-appearance consistency in videos (Nguyen & Meunier, 2019) while detecting anomaly.
sections.heading: 4.2 FEW-SHOT LEARNING
sections.text: Few-shot learning is extensively explored in classification tasks. Proposed methods are based on optimization (Finn et al., 2017; Rusu et al., 2019; Finn et al., 2018; Yoon et al., 2018; Ravi & Larochelle, 2016), learning metric (Snell et al., 2017; Vinyals et al., 2016) and parameter prediction (Gordon et al., 2018; Qiao et al., 2018; Gidaris & Komodakis, 2019). These technologies are further applied in other tasks like image generation (Clouâtre & Demers, 2019; Liu et al., 2019) and out-ofdistribution detection (Sehwag et al., 2021).
sections.heading: 5 EXPERIMENTS
sections.text: In this section, we conduct evaluation on the industrial inspection task with the benchmark MVTecAD dataset Bergmann et al. (2019) (Section 5.1). Even though our proposed framework is imagebased, we further demonstrate it’s efficacy on the video anomaly detection task in Section 5.2. In Section 5.3, we show ablations and insights relating to the adaptive sparse coding components. We also show additional ablations on robustness to pose variations (e.g. rotation) while applying fewshot adaptation (App. B.1) and comparing with naive background subtraction to detect anomaly (App. B.2).We provide implementation details in App. A.
sections.heading: 5.1 INDUSTRIAL INSPECTION
sections.text: The goal of this anomaly detection task is to predict whether a manufactured component contains any defects. The MVTec-AD dataset includes 15 categories of object. To demonstrate the fast adaptation capability of the proposed method, we adopt a leave-one-out training strategy. Specifically, samples of each target category are reserved for testing only, and the episodic training is performed on the remaining categories. During the training stage, the model will not see any samples from the target category. During testing, we first adapt the model to the target category with 10 randomly selected
  normal samples, then measure the performance with the entire testing set. We run the test 5 times, each time the model is adapted to random sets of 10 normal samples from the target category. The final result is the average of the 5 runs. Following common practice (Bergmann et al., 2019; Liu et al., 2020), we report the performance of pixel-wise anomaly localization with AUC-ROC and mIoU (mean intersection over union). IoU is the area of overlap between the predicted map and the ground truth divided by the area of union between these two.
  To the best of our knowledge, no existing methods feature fast adaptation like ours. We first show the performance of fully supervised methods, which train each category with normal samples from scratch, as the “upper-bounds”. Specifically, (Bergmann et al., 2018; 2019) trains auto-encoders (AE) on massive number of normal samples and measure the reconstruction errors during the inference; AnoGAN (Schlegl et al., 2017) adopts a generative adversarial network (GAN) to learn a manifold of normal; VE-VAE (Liu et al., 2020) presents a visually explainable variational autoencode through gradient-based attention. Furthermore, we create a strong baseline by applying model-agnostic meta-learning (Finn et al., 2017) on an AE (denoted as MAML-AE, detailed in App. Sec. A.3). Numerical results of pixel-wise anomaly localization are in Table 1. Note that all results of our methods are obtained without any data augmentation. Our proposed method outperforms MAML-AE by a large margin and is competitive with the “upper-bounds”. We show the localized anomaly regions from our method in Fig 5. Additional visualizations are in the App. Fig A.
sections.heading: 5.2 VIDEO SURVEILLANCE
sections.text: In video anomaly detection, a common goal is to detect abnormal events captured by surveillance cameras (e.g., a motorcycle on the sidewalk). A model trained on videos from one camera might not generalize well on other cameras due to different locations / mounting heights / lightning conditions, and it is not feasible to train one model for every new camera in practice. The ability to quickly adapt to new scenes is a significant contribution to the task of video surveillance. We are only aware of the work in (Lu et al., 2020) (r-GAN) that has such adaptation capability. Specifically, the model adapts to a new scene using gradient descent with several beginning frames of a query video, after which a GAN is applied to generate future frames. Anomaly is then detected via the discrepancy between predicted future frames and the original frames.
  We follow the same evaluation regime as r-GAN by training with normal samples in all 13 scenes from SH-Tech Liu et al. (2018) and testing on UCSD Pedestrian 1, UCSD Pedestrian 2 (Mahadevan et al., 2010), and CUHK Avenue Lu et al. (2013). Note that since our method is image-based, it predicts the video frames independently without leveraging any temporal information as in r-GAN.
  Category Leather Grid Hazelnut Cable Pill
  l = 1 0.41 0.98 0.11 0.80 0.36 0.91 0.21 0.85 0.10 0.85 l = 3 0.42 0.98 0.12 0.81 0.40 0.94 0.24 0.87 0.22 0.88
  Table 3: Comparison of different sparse coding receptive fields. We report both mIoU (left) and AUC-ROC (right) values.
  Category Leather Hazelnut Cable
  Ours 0.42 0.98 1.6e-4 0.40 0.94 2.4e-4 0.24 0.87 2.0e-4 No sparsity 0.32 0.90 0.9e-4 0.24 0.80 1.7e-4 0.12 0.68 1.5e-4
  Table 4: Performance w/ and w/o sparsity constraint. From left to right: mIoU; AUC-ROC; the difference of averaged reconstruction errors between abnormal/normal samples.
  In each episode, we adapt our model with a support set containing a few normal frames randomly sampled from the target scenes. In Table 2, we compare our method against r-GAN pre-trained on SH-Tech only (r-GAN Pre-train), fine-tuned on target datasets (r-GAN Fine-tune), and with one step gradient descent with meta-learning (r-GAN MAML). We also show the performance of MAML-AE as a baseline for image-based meta-learning method. In the last section of Table 2, we present intradataset results as well by training with 6 scenes of SH-Tech and testing on remaining 7. We follow common evaluation protocol and measure the frame-level AUC-ROC. Without leveraging temporal information and performing re-training (gradient descent), our method achieves comparable results to r-GAN MAML and outperforms image-based meta-learning method by a large margin.
sections.heading: 5.3 ABLATION STUDIES
sections.text: Sparse coding receptive fields. To evaluate the effectiveness of using large receptive fields in the sparse coding layer, we conduct additional experiments on the MVTec-AD dataset, and select 5 representative categories with different levels of difficulties to present the comparisons with l = 1 and l = 3 (Sec. 3.1.1) in Table 3. Sparse coding with large receptive field clearly benefits more complex structural objects (hazelnut, cable, and capsule), while the improvements are limited for the texture objects (leather and grid), where contextual regularization is intuitively less important. Shrinkage functions. To show the benefits of smooth shrinkage function, we plot the loss curves of models trained with smooth SigShrink (Eqn. 9) and non-smooth RELU-like shrinkage (Eqn. 7) functions in Fig 7. The model with smooth shrinkage function converges notably faster in the early training stage and achieves lower loss. Sparsity constraint. As discussed in Section 3.1, we impose sparsity constraint to the feature decomposition in the adaptive sparse coding layer, in order to prevent abnormal features from being well-approximated by the linear combinations of normal features, so that the reconstruction errors are effective for detecting anomaly. To validate this, we conduct experiments by removing the shrinkage function σ in the sparse coding stage (Eqn. 6). We show comparison in Table 4 with mIoU, AUC-ROC, and the difference of averaged reconstruction errors between abnormal and normal samples. Without sparsity, the performance drops dramatically, and reconstruction errors of normal and abnormal samples become closer.
sections.heading: 6 CONCLUSION
sections.text: In this paper, we introduced a novel framework for anomaly detection and localization that allows fast adaptation to new tasks. We formulated our model as an energy based model with an adaptive sparse coding layer, of which the dictionary is directly formed by normal features of a target task. We adopted episodic meta-learning to extract common knowledge across tasks, which has the effect of enabling few shots adaptation. We further introduced smooth shrinkage functions, sparse coding with large receptive fields, and learning by inpainting to improve and accelerate the EBM training. It’s worthy to note that when evaluating our method’s performance on industrial inspection and video anomaly detection, our method is comparable and even boasts better performance than methods trained with a large amount of normal samples. Through this work, we hope to have made a significant contribution to the important problem of anomaly detection by shedding light on our findings that anomaly detection can indeed be generalized to new tasks.
sections.heading: B ADDITIONAL ABLATIONS
sections.text: B.1 ROBUSTNESS TO POSE VARIATIONS
  To validate the robustness of our method to pose variations, we perform additional experiments with pose variations and provide the quantitative comparisons in Table B. We report results by performing random ±90o rotation to samples during testing stage when adapting the trained model to novel categories. We observe that the textures categories (leather and grid) are robust to any pose variations in both the query and support samples. For the hazelnut category, where there are intrinsically significant pose variations across samples, performing any new rotations does not influence the results. On the other hand, for the pill category, where all samples are aligned horizontally (see examples in Appendix, Figure A), performing rotation to only query samples results in performance drop. Performing rotation to both query and support samples help to recover the performance. The above results suggest the robustness of our methods under the condition of sufficient sample diversity in the support set.
  Category Leather Grid Hazelnut Cable Pill
  Original 0.42 0.98 0.12 0.81 0.40 0.94 0.24 0.87 0.22 0.88 Rotate query 0.41 0.98 0.12 0.80 0.40 0.94 0.21 0.82 0.10 0.71
  Rotate query and support 0.42 0.98 0.11 0.81 0.40 0.94 0.23 0.85 0.19 0.86
  Table B: Robustness to pose variations. We report both mIoU (left) and AUC-ROC (right) values.
  B.2 COMPARISON TO IMAGE DIFFERENCING
  Image differencing and its variances are an intuitive approach for detecting image anomaly. We perform image differencing and background subtraction on the industrial inspection and video anomaly detection task, respectively, and report performance in Table C. Our method demonstrates clear advantages.
  Category Leather Hazelnut Cable UCSD Ped1 UCSD Ped2 CUHK
  Image differencing 0.02 0.57 0.12 0.77 0.07 0.65 65.34 59.03 57.12 Ours 0.42 0.98 0.40 0.94 0.24 0.87 77.42 91.22 75.32
  Table C: Comparing with image differencing. We report mIoU (left) and AUC-ROC (right) for industrial inspection task and AUC-ROC for video anomaly detection task.
  B.3 ROBUSTNESS AGAINST CONTAMINATED TRAINING DATA
  While it is a common practice among machine learning practitioners to assume clean training data, this may not be true in real world applications. In this section, we evaluate the robustness of our method against contaminated training data by inserting certain amount of abnormal samples. As shown in Table D, we progressively contaminate the normal training data by increasing the amount of abnormal data from 1% to 10%. Our proposed method is in general robust to data contamination, where contamination under 5% only decreases the performance slightly. The network is still able to perform decently in the extreme case of 10% contamination.
  Contamination Leather Grid Hazelnut Cable Pill
  0% 0.42 0.98 0.12 0.81 0.40 0.94 0.24 0.87 0.22 0.88 1% 0.42 0.98 0.11 0.80 0.40 0.94 0.23 0.86 0.21 0.88 2% 0.41 0.98 0.11 0.80 0.40 0.93 0.23 0.84 0.21 0.87 5% 0.40 0.96 0.11 0.78 0.39 0.93 0.22 0.84 0.20 0.87
  10% 0.38 0.95 0.10 0.76 0.37 0.90 0.21 0.82 0.19 0.85
  Table D: Performance evaluation against data contamination. We contaminate the normal training data by inserting increasing percentages of abnormal data.
  B.4 LEVERAGING TEMPORAL INFORMATION
  An EBM is agnostic to the underlying backbone network architecture, therefore it is straightforward to incorporate temporal information into the image-based anomaly detection framework described in Section 5.2. We replace the 2D convolutions in the first four layers with 3D convolutions, and have the model accepts a 5-frame input instead. Without heavy tuning, we report 5-shot performance on both ’1 frame’ (original, without temporal) and ’5 frames’ (with temporal) in Table E. The results show that when temporal information is added, it leads to further improvements.
  Category UCSD Ped1 UCSD Ped2 CUHK Sh-Tech
  1 frame 78.12 92.00 83.41 79.64 5 frames 79.06 91.94 84.27 80.80
  Table E: Performance comparisons with temporal information incorporated.
sections.heading: C QUALITATIVE RESULTS
sections.text: We present in Figure 1 additional anomaly localization results of categories with different anomalies in the MVTec-AD dataset.
sections.heading: D ADDITIONAL FIGURES
sections.text: Common practice of sampling from EBMs is computationally costly, requiring as many as 50 steps of Langevin Dynamics when initialized with full noise, instead of a synthesized negative sample as shown in Fig. B. Some examples are provided in Fig. C.
  Input GT Prediction Input GT Prediction Input GT Prediction
  Figure A: Visualizations of anomaly localization on industrial inspection data. All results are obtained by adapting the model using 10 normal samples only.
  Normal sample Synthesized negative Step 1 Step 2 Step 3 Step 4 Step 5
  Figure B: Sampling outputs after a few steps of Langevin Dynamics starting from each synthesized negative sample. 5 steps of Langevin dynamics are sufficient to quickly generate hard negative samples with minor artifacts.
  Figure C: Figure from (Zhao et al., 2020). Generated sequences from the process of Langevin Dynamics. Initialization from noise usually requires around 50 steps to synthesize the images with desired quality (images are shown every 5 steps).