This research paper describes a simplistic architecture named as AANN: Absolute Artificial Neural Network, which can be used to create highly interpretable representations of the input data. These representations are generated by penalizing the learning of the network in such a way that those learned representations correspond to the respective labels present in the labelled dataset used for supervised training; thereby, simultaneously giving the network the ability to classify the input data. The network can be used in the reverse direction to generate data that closely resembles the input by feeding in representation vectors as required. This research paper also explores the use of mathematical abs (absolute valued) functions as activation functions which constitutes the core part of this neural network architecture. Finally the results obtained on the MNIST dataset by using this technique are presented and discussed in brief.
This paper introduces a reversible network with absolute value used as
the activation function.  The network is run in the forward direction
to classify and in the reverse direction to generate.
The key points of the network are the use of the absolute value
activation function and the use of (free) normalization to match
target output. This allows the network to perfectly map inputs to any
point on a vector that goes through the one-hot encoding, allowing for
deterministic generation from different vectors (of different lengths)
with the same normalized output.
I think there are a lot of novel and interesting ideas in this paper
though they have not been fully explored.  The use of the absolute
value transfer function is new to me, though I was able to find a couple of old
references to its use.   In a paper by Gad et al. (2000), it is stated 
" For example, the algorithm presented in Lin and
Unbehauen (1995) < I think they mean Lin and Unbehauen 1990)> 
 is used to train networks with a single hidden layer
employing the absolute value as the activation function of the hidden
neuron. This algorithm was further generalized to multilayer networks
with cascaded structures in Batruni (1991)."   Exploring the properties 
of the abs activation function seems worth exploring.
More details on the training are needed for full clarity in the paper.
(Though it is recognized that some of these could be determined from
links when made active, they should be included in the paper).  How
did you select the training parameters given at the bottom of page 5?
How many layers and units/layer did you use? And how were these
selected?  (The use of the links for providing code and visualizations (when active)
 is a nice feature of this paper).
Also, did you compare to using the leaky ReLU activation function --
That would be interesting as it also doesn't have any areas of zero
slope?  Did you compare the generated digits to those obtained using GANs?
I am also curious, how does accuracy on digit classification differ
when trained only to optimize the forward error?
The MNIST site referenced lists 60,000 training data and test data of
10,000.  How/why did you select 42,000 and then split it to 39900 in
the train set and 2100 in the dev set?
Also, the goal for the paper is presented as creating highly
interpretable representations of the input data.  My interpretation of
interpretable is that the hidden units are "interpretable" and that it
is clear how the combined hidden unit representations allow for
accurate classification.  Towards that end, it would be nice to see
some of the interpretations of the hidden unit representations.  In
the abstract it states " ...These representations are generated by
penalizing the learning of the network in such a way that those
learned representations correspond to the respective labels present in
the labelled dataset used for supervised training".  Does this
statement refer only to the encoding of the representation vector or
also the hidden layers?  If the former, isn't that true for all
supervised algorithms.  If the latter, you should show this.
Batruni, R. (1991). A multilayer neural network with piecewise-linear
structure and backpropagation learning. IEEE Transactions on Neural
Networks, 2, 395–403.
Lin, J.-N., & Unbehauen, R. (1995). Canonical piecewise-linear neural
networks. IEEE Transactions on Neural Networks, 6, 43–50.
Lin, J.-N, & Unbehauen, R. (1990). Adaptive Nonlinear Digital Filter with Canonical Piecewise-Linear Structure,
IEEE Transactions on Circuits and Systems, 37(3) 347-353.
Gad, E.F et al (2000). A new algorithm for learning in piecewise-linear neural networks.
Neural Networks 13,  485-505.