ABSTRACT k-Nearest Neighbors is one of the most fundamental but effective classification models. In this paper, we propose two families of models built on a sequence to sequence model and a memory network model to mimic the k-Nearest Neighbors model, which generate a sequence of labels, a sequence of out-of-sample feature vectors and a final label for classification, and thus they could also function as oversamplers. We also propose 'out-of-core' versions of our models which assume that only a small portion of data can be loaded into memory. Computational experiments show that our models outperform k-Nearest Neighbors, a feedforward neural network and a memory network, due to the fact that our models must produce additional output and not just the label. As an oversampler on imbalanced datasets, the sequence to sequence kNN model often outperforms Synthetic Minority Over-sampling Technique and Adaptive Synthetic Sampling.
This work uses sequence-to-sequence and memory network neural nets to learn a
network that not only predicts a label, but also predicts nearest neighbors and
their label.  The intuition is that by training on a related but harder task,
the network is forced to learn not just about sampled points but about the
behavior in regions around the actual training examples.
Their work shows that imposing these additional requirements on the model does
result in better performance on unseen data, where only the label of the unseen
data point is required as output.  They show that learning more about the 
global {feature,label} distribution improves F1 scores, and suggest that their
methods can be used as an example generator for datasets with class imbalance.
The writing was clear.  I had only 1 misunderstanding that cause me trouble,
namely the first sentence of "Classification", where I might put the word
'benchmark' up front rather than at the end of this somewhat long sentence.
By the time 'knn benchmark' appeared some paragraphs later, I realized I
had missed something, and had to backtrack.
Figs 1--3 are only viewable on screen with fairly extreme magnification. I
found different viewers varied in legibility at these extreme zoom settings.
However, when expanded so y-axis numbers could be deciphered, it turned out
that actual improvements in F1 score were rather modest.
For 4 datasets, their F1 scores of their best method, V2VSLS, improved by ~ 1.5
to 6%.  They found that their methods were less affected by out-of-core
computation than the benchmark kNN F1 scores.
I did have some questions about the problem formulation.  I did not really
understand why the models wanted to reproduce the exact ordering of the
nearest neighbors, other than this is easy to formulate.  This makes sense
for n.n. label sequences, where further points might be in some neighboring
class.  But for predicting the feature vectors of n.n., it seems that the
exact order is not robust, in the sense very small distance changes can cause
abrupt shifts in the target nearest-neghbor ordering.  Is there some way
for me to understand why this does not pose a problem?  Or is there perhaps
a way to make the loss function a bit less dependent on the precise order
of the generated feature vector sequence?
In the computational experiments, with the choice of datasets, it was often
hard for me to judge how much different aspects of their 4 network structures
were really being excercised.  The main issue is the all problems used only
2--3 classes.  I could not guess what fraction of data points had actual label
changes within the 5 n.n..  For many datasets, "same class" might be a pretty
good predictor of nearest neighbor label.
Alternatively, one can consider the other extreme, of very many classes.
Does it still make sense to try to predict the order of nearest neighbor labels in
such a setting?
The authors provide evidence of some performance improvement, but I would
encourage them to provide some intuition about what the networks are actually
learning.  Some of this can be done with their existing data.
For example, on average, what are the distances from predicted feature vectors
to the actual nn feature vectors?  If the feature sequence is typically badly
predicted, then this might allow the authors to propose that the network is
*actually* learning some simpler features of the the distributions underlying
an actual {feature,label} sequence.  This might allow simplified training losses,
based, on things like direction and distance to same-label cluster center,
direction+distance to average same-class nn.s, direction and distance to
closest differently labeled cluster, etc.  Or does their data suggest that
their models are actually learning the precise nearest neighbor ordering?
Such considerations might be able to improve the OOC training, since
"global" aspects of the distribution features (like "cluster center")
remain approx. valid as training batches changes.
The other question I had was with the oversampling proposition.  The extent
of class imbalance in the datasets is not described.  Perhaps it belongs in
Table 1.  Their approach seems a lot of work for modest gains usually available
with oversampling.  Can the authors provide any guideline for how many members a
minority class should have before using their sequence-to-sequence technique?
Pros: they improve generalization to unseen data.
Cons: their models are considerably more complex, and they do not analyze their
data in enough detail to suggest whether their complexity is necessary, or perhaps
could be reduced.  Figures are too small (many unreadable in printed copy).
Datasets have very few classes and the extent to which nearest neighbors are
of different class not reported.