Artificial neural networks have revolutionized many areas of computer science in recent years, providing solutions to a number of previously unsolved problems. On the other hand, for many problems, classic algorithms exist, which typically exceed the accuracy and stability of neural networks. To combine these two concepts, we present a new kind of neural networks-algorithmic neural networks (AlgoNets). These networks integrate smooth versions of classic algorithms into the topology of neural networks. A forward AlgoNet includes algorithmic layers into existing architectures to enhance performance and explainability while a backward AlgoNet enables solving inverse problems without or with only weak supervision. In addition, we present the algonet package, a PyTorch based library that includes, inter alia, a smoothly evaluated programming language, a smooth 3D mesh renderer, and smooth sorting algorithms.
The paper conceptualizes a neural network architecture that integrates smoothed versions of traditional algorithms into the network topology. The AlgoNet concept employs smoothed versions of algorithms implemented in the WHILE language, with options for different levels of differentiability. The paper outlines a forward version of AlgoNet based on traditional, skip, and residual connections, and a backward version for solving inverse problems in a reconstructive, autoencoder-like fashion. As smoothing introduces a form of domain shift, the paper the reconstructive adversarial network that that employs "domain translators" as well as a discriminator that are trained in an adversarial fashion. The paper concludes by describing versions of AlgoNet for various different algorithms.
The general idea of being able to better control the behavior of neural networks by better leveraging known structure (e.g., algorithms) is appealing. However, the paper does not go beyond conceptualizing how this might be done in a hand wavy manner. It is difficult to see what if anything can be learned from the paper, let alone what practical utility it has, which is important given that the paper claims to propose a new neural network architecture.
The paper would benefit from a discussion of empirical results in the main text, with baseline comparisons. With the exception of a single figure, the details are relegated to the appendices.
The related work discussion is surprisingly short, given the attention that has been paid to designing/optimizing different network topologies. The work that is discussed is very narrow in scope (see below).
The  paper devotes too much discussion to the challenges introduced by non-differentiable layers and the advantages of increasing degrees of differentiability (this is about half of the intro and most of the related work).
Despite the lack of experimental demonstrations, the paper is one page over the suggested 8 page limit.