We investigate the methods by which a Reservoir Computing Network (RCN) learns concepts such as 'similar' and 'different' between pairs of images using a small training dataset and generalizes these concepts to previously unseen types of data. Specifically, we show that an RCN trained to identify relationships between image-pairs drawn from a subset of digits from the MNIST database or the depth maps of subset of visual scenes from a moving camera generalizes the learned transformations to images of digits unseen during training or depth maps of different visual scenes. We infer, using Principal Component Analysis, that the high dimensional reservoir states generated from an input image pair with a specific transformation converge over time to a unique relationship. Thus, as opposed to training the entire high dimensional reservoir state, the reservoir only needs to train on these unique relationships, allowing the reservoir to perform well with very few training examples. Thus, generalization of learning to unseen images is interpretable in terms of clustering of the reservoir state onto the attractor corresponding to the transformation in reservoir space. We find that RCNs can identify and generalize linear and non-linear transformations, and combinations of transformations, naturally and be a robust and effective image classifier. Additionally, RCNs perform significantly better than state of the art neural network classification techniques such as deep Siamese Neural Networks (SNNs) in generalization tasks both on the MNIST dataset and more complex depth maps of visual scenes from a moving camera. This work helps bridge the gap between explainable machine learning and biological learning through analogies using small datasets, and points to new directions in the investigation of learning processes.
The paper uses an echo state network to learn to classify image transformations (between pairs of images) into one of fives classes.  The image data is artificially represented as a time series, and the goal is generalization of classification ability to unseen image pairs.  The network dynamics are studied and are claimed to have explanatory power.
The paper is well-written and easy to follow, but I have concerns about the claims it makes relative to how convincing the results are.  The focus is on one simple, and frankly now-overused data set (MNIST).  Further, treating MNIST data as a time series is artificial and clunky.  Why does the series go from left to right rather than right to left or top to bottom or inside out or something else?  How do the results change if the data is "temporalized" in some other way?
For training in Section 2.4, is M the number of columns for a pair of images?  It's not clear how pairs are input in parallel--- one after the other? Concatenated? Interleaved columns?  Something else? What are k, i, j in computing $\delta X_k$?  Later, in Section 3.2, it says, "As in section 2.2, $xl(mn)$ is the differential reservoir state value of the $m$th reservoir node at time $n$ for input image $l$", but nothing like this is discussed in Section 2.2; I'm confused.
The generalization results on this one simple data set seem pretty good.  But, how does this kind of approach do on other kinds of or more complex data?  I'm not sure that RC has historically had very good success scaling up to "real-world" problems to date.
Table 1 doesn't really say anything.  Of course, the diagonals are higher than the off diagonals because these are dot products.  True, they are dot products of averages over different inputs (which is why they are less than 1), but still.  Also, what Table 1 really seems to say is that the off-diagonals really aren't all that different than the diagonals, and that especially the differences between same and different digits is not very different, suggesting that what is learned is pretty fragile and likely won't generalize to harder problems.  I like the idea of using dynamical systems theory to attempt to explain what is going on, but I wonder if it is not being used a bit simplistically or naively.
Why were the five transform classes chosen?  It seems like the "transforms" a (same) and e (different) are qualitatively different than transforms b-d (rotated, scaled, blurred).  This seems like it should talked about.
"Thus, we infer, that the reservoir is in fact, simply training these attractors as opposed to training the entire reservoir space."  What does this mean?  The reservoir isn't trained at all in ESNs (which is also stated explicitly for the model presented here)…
For 3.3, why did were those three classes chosen? Was this experiment tried with other subsets of three classes?  Why are results reported on only the one combination of rotated/blurred vs. rotated?  Were others tried?  If so, what were the results?  If not, why?  How does the network know when to take more than the highest output (so it can say that two transforms have been applied)?  In the case of combination, counting either transform as the correct output kind of seems like cheating a bit—it over states how well the model is doing.  Also, does the order in which the transforms are applied affect their relative representative strength in the reservoir?
The comparison with SNNs is kind of interesting, but I'm not sure that I'm (yet) convinced, as there is little detail on how the experiment was performed and what was done (or not) to try to get the SNN to generalize.  My suspicion is that with the proper approach, an SNN or similar non-dynamical system could generalize well on these tasks.  The need for a dynamical system could be argued to make sense for the camera task, perhaps, as video frames naturally form a time series; however, as already mentioned, for the MNIST data, this is not the case, and the fact that the SNN does not generalize here seems likely due to their under utilization rather than due to an inherent lack of capability.
I don't believe that there is sufficient support for this statement in the conclusion, "[ML/deep networks] do not work as well for generalization of learning. In generalized learning, RCNs outperform them, due to their ability to function as a dynamical system with ‘memory’."  First of all, ML is all about generalization, and there are lots and lots and lots of results showing that many ML systems generalize very well on a wide variety of problems, well beyond just classification, in fact.  And, I don't think the the paper has convincingly shown that a dynamical system 'memory' is doing something especially useful, given that the main task studied, that of character recognition (or classification of transformation or even transformation itself), does not require such a temporal ability.