  Machine learning models have become more and more complex in order to better
approximate complex functions. Although fruitful in many domains, the added
complexity has come at the cost of model interpretability. The once popular
k-nearest neighbors (kNN) approach, which finds and uses the most similar data
for reasoning, has received much less attention in recent decades due to
numerous problems when compared to other techniques. We show that many of these
historical problems with kNN can be overcome, and our contribution has
applications not only in machine learning but also in online learning, data
synthesis, anomaly detection, model compression, and reinforcement learning,
without sacrificing interpretability. We introduce a synthesis between kNN and
information theory that we hope will provide a clear path towards models that
are innately interpretable and auditable. Through this work we hope to gather
interest in combining kNN with information theory as a promising path to fully
auditable machine learning and artificial intelligence.

Machine Learning;Artificial Intelligence;Machine Learning