We propose RAPP, a new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder. Precisely, RAPP compares input and its autoencoder reconstruction not only in the input space but also in the hidden spaces. We show that if we feed a reconstructed input to the same autoencoder again, its activated values in a hidden space are equivalent to the corresponding reconstruction in that hidden space given the original input. We devise two metrics aggregating those hidden activated values to quantify the novelty of the input. Through extensive experiments using diverse datasets, we validate that RAPP improves novelty detection performances of autoencoder-based approaches. Besides, we show that RAPP outperforms recent novelty detection methods evaluated on popular benchmarks.
UPDATE: 
I acknowledge that I‘ve read the author responses as well as the other reviews. 
I appreciate the improvements and clarifications the authors have made, especially adding an ablation study to see the benefits of adding additional layers. I updated my score to Weak Accept (6). 
####################
This paper considers deep autoencoders (AEs) for the unsupervised novelty/anomaly detection task and proposes to extend the standard AE anomaly score, given by the reconstruction error between the input and output in the original data space, to also utilize the reconstruction errors of the hidden activations in the AE network. The proposed method, Reconstruction along Projection Pathway (RaPP), specifically compares the hidden activations of all encoder layers given by the original input $x$ with the activations of the same units given by feeding the reconstruction $\hat{x}$ back into the AE. Thus RaPP compares the activation statistics of the original input $x$ and its reconstruction $\hat{x}$ along the encoder projection pathway from original data space to latent code space. Two ways for aggregating those reconstruction errors to a final anomaly score are presented: (1) Simple Aggregation Along Pathway (SAP) which simply computes the sum of reconstruction errors, and (2) Normalized Aggregation Along Pathway (NAP) which computes the sum of reconstruction errors after normalization via Singular Value Decomposition (SVD). The paper conclusively presents experiments on eight datasets from various domains, in which SAP and NAP are compared to the reconstruction error baseline for vanilla AE, VAE, and AAE, as well as experiments on MNIST and Fashion-MNIST in which NAP is compared to state-of-the-art deep anomaly detectors.
Though this work is well presented and indicates promising results, I think the paper should not yet be accepted due to the following main reasons: 
(i) The experimental evaluation indicates promising, but not yet convincing results; 
(ii) The computational complexity of NAP seems to be a major limitation of RaPP which is not addressed in the text; 
(iii) The added value/insights from the theoretical Section 4 (Motivation of RaPP) are not clear.
(i) I think the experimental section shows promising, but not yet convincing results. To judge the significance of results, I think the paper should address the following:
(ia) The experiments on the eight non-image datasets should include other baselines (e.g. OC-SVM, Isolation Forest) besides the standard AE reconstruction error. One should expect SAP and NAP to improve over the standard AE since both methods include the original data space reconstruction errors as well. Moreover, the advantage of deep approaches on such non-image datasets is less clear [7] why a comparison to well-known baselines should be given.
(ib) The main motivation for deep approaches to anomaly detection are large and complex datasets [6, 5, 4, 2]. I think the comparison to recent, state-of-the-art deep competitors should at least include another dataset more complex than MNIST or Fashion-MNIST, e.g. CIFAR-10 as reported in the previous works or MVTec [1]. 
(ic) I think the proposed method begs for an ablation study of subsequently adding the reconstruction errors of additional layers. This would clearly demonstrate the potential benefits of adding the hidden reconstructions.
(ii) The experiments indicate that a proper normalization of the hidden activation reconstruction errors is crucial for improving detection performance. NAP shows consistent improvements, whereas SAP often performs similar to the AE baseline. However, the current SVD normalization procedure on a matrix with dimensions number of samples × number of hidden encoder units seems extremely costly to me and appears to be a major limitation towards larger datasets or networks. Could you comment on this since this is not yet addressed in the manuscript. Have you tried using Batch Normalization (after activation) together with per-layer averaging? To me, this seems the natural first choice to normalize unit scores and to account for different layer widths. Do you apply SVD on mini-batches?
(iii) The additional insights from the theoretical Section 4 are not clear to me. I think the presented reconstruction property for the hidden layers follows somewhat directly per definition for symmetrically constructed deep autoencoders (specifically if weights would be shared in addition). For a theoretical contribution, on the other hand, the proof and proposition should be fully rigorous in my mind, i.e. stating all the necessary assumptions on the function class (e.g. you implicitly assume invertibility and thus some smoothness of the $g_i$'s which Conv+ReLU modules do not satisfy for instance). As of now, I think this section does not add to intuition, but on the other hand is not completely rigorous. Maybe I am missing something?
The overall presentation of the paper is good (clear writing and structure, polished Figures and Tables). The work is well motivated and properly placed in the literature. Maybe since the approach is rather simple (which I don’t find negative), the author felt the need to add some rigor to the paper, which I think would not be necessary for a significant contribution if the experimental results hold up against the additional baselines and more complex datasets as described in (i).
####################
*Additional Feedback*
*Positive Highlights*
1. Simple idea that does not require autoencoder modification or retraining that indicates improved anomaly detection results.
2. The work is well placed in the literature. The related work includes all relevant and recent major works on the subject matter.
3. I appreciate the evaluation on both anomaly/novelty detection setups, unimodal and multimodal.
4. Comparison to recent OC-NN [3], GPND [5], Deep SVDD [6], and GT [4].
5. The writing, structure and overall presentation is good.
*Ideas for Improvement*
6. Include additional baselines and more complex datasets as described in (i).
7. Address the computational complexity of RaPP as in described in (ii).
8. Maybe cut the methodical/theoretical parts in Section 3.2 and Section 4 a bit. I think they are rather straightforward. Maybe combine Figures 1+2 as well. Extend the experimental evaluation instead.
9. Report the AUROC standard deviations over the trials as well to better infer statistical significance of the results (defer to appendix if space is a constraint).
*Minor comments*
10. Section 2: “Unsupervised and semi-supervised learnings” » “Unsupervised and semi-supervised learning approaches”.
11. Section 2: “Variational Autoencoders (VAE) was reported ...” » “Variational Autoencoders (VAE) were reported ...”
12. Section 3.1: “Due to this representation learning property, the autoencoder has been widely used for novelty detection.” » emphasis on unsupervised learning property, specifically.
13. Section 3.1: “Although this approach has shown a promising result in novelty detection ...” » “Although this approach has shown promising results in novelty detection ...”
14. Section 3.1, last sentence: “... in more details.” » “... in more detail.”
15. Section 3.2: “Those are especially suited for the case of zero-knowledge to interpret identified hidden spaces, which commonly happens when modeling with deep neural networks.” Zero-knowledge case? Reference?
16. In Section 5.1: “Further setups are described in Section 5.1”?
17. Section 5.4.1: “Also, we showed the best score ...” » “Also, we show the best score ...”.
18. Section 5.2: “... maintaining novelty ratios to 35% for the multimodal and 50% for the unimodal normality setups, respectively.” Why use different ratios?
####################
*References*
[1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.
[2] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019.
[3] R. Chalapathy, A. K. Menon, and S. Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360, 2018.
[4] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.
[5] S. Pidhorskyi, R. Almohsen, and G. Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In NeurIPS, pages 6822–6833, 2018.
[6] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
[7] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.