We propose a neural network for unsupervised anomaly detection with a novel robust subspace recovery layer (RSR layer). This layer seeks to extract the underlying subspace from a latent representation of the given data and removes outliers that lie away from this subspace. It is used within an autoencoder. The encoder maps the data into a latent space, from which the RSR layer extracts the subspace. The decoder then smoothly maps back the underlying subspace to a "manifold" close to the original inliers. Inliers and outliers are distinguished according to the distances between the original and mapped positions (small for inliers and large for outliers). Extensive numerical experiments with both image and document datasets demonstrate state-of-the-art precision and recall.
After reading all the reviews and the comments, I feel more positive about the paper. I appreciate the feedback of the Authors and I have decided to increase the rating.
============================
The paper proposes using Robust Subspace Recovery in combination with an autoencoder (and possibly GANs) for anomaly detection. The encoder maps input data to the latent space of dimensionality D, which then is linearly projected to a subspace of dimensionality d (d < D). The projection of the latent space then goes to a decoder that reconstructs the input.
A transformation matrix A is trained jointly with the autoencoder. Two additional terms are added to the loss: one to encourage the subspace of A^TA to approximate the latent space z and the second one to force it to be an orthogonal projector.
 
The paper claims to generalize the existing RSR framework to the nonlinear case. However, the linear RSR is applied to the latent space of the autoencoder. In addition to that, all the following discussion and proofs are limited to the linear case. 
Since the proposed method is using RSB as it’s core part, and claims to be a non-linear extension of it, it would be crucial to have a comparison with RSB, at least on those experimental setups, where high-level features are used (Tiny Imagenet with ResNET features, Reuters-21578, and 20 Newsgroups). However, there is no such comparison.
Since autoencoders can potentially learn any, arbitrary entangled latent space, it is not clear why outliers should necessarily have such embedding that is outside of the learned subspace. In the case of the original RSR it happens due to the dimensionality reduction by the orthogonal projector. However, autoencoders already perform dimensionality reduction at each layer down to the bottleneck layer.
The matrix A and the parameters of the AE are trained jointly. So, it can be seen that two processes can occur:
- The AE in order to minimize the reconstruction error would learn such latent space z, that would fit into the subspace of A^TA, so that projection \tilde z =Az doesn’t cause data loss.
- The AE in order to minimize the reconstruction error would learn such A, so that the subspace that z approximates is the best possible.
It is not clear, which of the two cases would take place. If the first one would dominate, then it is not clear if such method would have any discriminating capabilities.
 
My point is mainly that the presented work is not really a generalization of RSR as it claims to be, but rather it is just using RSR on a leaned embedding of the data. 
Some citations are missing, as well as it is missing a comparison to some state-of-the art methods such as OCNN ‘Robust, Deep and Inductive Anomaly Detection’ ECML 2017;   ‘Adversarially Learned One-Class Classifier for Novelty Detection’ CVPR 2017; DSVDD ‘Deep one-class classification.’ ICML, 2018; ODIN  ‘Enhancing The Reliability  Of Out-of-distribution Image Detection  In Neural Networks’ ICLR 2018; ‘Generative Probabilistic Novelty Detection with Adversarial Autoencoders’ NeurIPS 2018.