With the rise in employment of deep learning methods in safety-critical scenarios, interpretability is more essential than ever before. Although many different directions regarding interpretability have been explored for visual modalities, timeseries data has been neglected with only a handful of methods tested due to their poor intelligibility. We approach the problem of interpretability in a novel way by proposing TSInsight 1 where we attach an auto-encoder with a sparsity-inducing norm on its output to the classifier and fine-tune it based on the gradients from the classifier and a reconstruction penalty. The auto-encoder learns to preserve features that are important for the prediction by the classifier and suppresses the ones that are irrelevant i.e. serves as a feature attribution method to boost interpretability. In other words, we ask the network to only reconstruct parts which are useful for the classifier i.e. are correlated or causal for the prediction. In contrast to most other attribution frameworks, TSInsight is capable of generating both instance-based and model-based explanations. We evaluated TSInsight along with other commonly used attribution methods on a range of different time-series datasets to validate its efficacy. Furthermore, we analyzed the set of properties that TSInsight achieves out of the box including adversarial robustness and output space contraction. The obtained results advocate that TSInsight can be an effective tool for the interpretability of deep time-series models.
The aim of this work is to improve interpretability in time series prediction. To do so, they propose to use a relatively post-hoc procedure which learns a sparse representation informed by gradients of the prediction objective under a trained model. In particular, given a trained next-step classifier, they propose to train a sparse autoencoder with a combined objective of reconstruction and classification performance (while keeping the classifier fixed), so as to expose which features are useful for time series prediction.  Sparsity, and sparse auto-encoders, have been widely used for the end of interpretability. In this sense, the crux of the approach is very well motivated by the literature.
* Pros
	* The work provides extensive comparison to a battery of other methods for model prediction interpretation. 
	* The method is conceptually simple and is easy to implement. It is also general and can be applied to any prediction model (though this is more property of the sparse auto-encoder).
	* Despite its simplicity and generality, the method is shown to perform well on average, though it sometimes performs significantly worse than simple baselines.
* Cons
	* The method itself is not explained very well. The authors use language such as “attach the auto encoder to the classifier”, which is a bit vague and could mean a number of things. It would be helpful if they provided either a formal definition of the model or a architectural diagram.
	* Though the quantitative evaluation is not entirely flattering, the authors should not be punished for providing experiments on many datasets. That said, if their contribution is then rather one of technical novelty, i.e. a sparse-autoencoder-based framework for time series interpretability, it would be helpful for them to 
		* More formally define their framework / class of solutions
		* Provide a more in depth study of possible variants of the method (this is elaborated on in the “Questions” section)
		* More strongly argue the novelty of their method
	* The authors provide a discussion on automatic hyper-parameter tuning that seems a bit out of place in the main method section, since it is not mentioned much thereafter and is claimed to not bring benefits.
	* The qualitative evaluation made by authors is rather vague:
		* "Alongside the numbers, TSInsight was also able to produce the most plausible explanations”
	
* Additional Remarks
	* Why not train things jointly? Does this have to be done post-hoc? The authors state that they “should expect a drop in performance since the input distribution changes” -> so why not at least try fine-tune and study the effect of training the classifier with sparse representations end-to-end? Exploring whether things can be trained jointly, or in other configurations, might allow the authors to frame their work as more of a general technical contribution.
	* It would be nice to have the simple baseline of a classifier with a sparsity constraint, i.e. 
		* I.e. ablate the reconstruction loss
I’ve given a reject because 1) the explanation of the method is not very precise and could be greatly improved, 2) the quantitative evaluation is not sufficiently convincing, given the lack of technical novelty), and 3) the qualitative evaluation is hand-wavy.