NEURAL NETWORK GRADIENT-BASED LEARNING OF BLACK-BOX FUNCTION INTERFACES
Deep neural networks work well at approximating complicated functions when provided with data and trained by gradient descent methods. At the same time, there is a vast amount of existing functions that programmatically solve different tasks in a precise manner eliminating the need for training. In many cases, it is possible to decompose a task to a series of functions, of which for some we may prefer to use a neural network to learn the functionality, while for others the preferred method would be to use existing black-box functions. We propose a method for end-to-end training of a base neural network that integrates calls to existing blackbox functions. We do so by approximating the black-box functionality with a differentiable neural network in a way that drives the base network to comply with the black-box function interface during the end-to-end optimization process. At inference time, we replace the differentiable estimator with its external blackbox non-differentiable counterpart such that the base network output matches the input arguments of the black-box function. Using this "Estimate and Replace" paradigm, we train a neural network, end to end, to compute the input to blackbox functionality while eliminating the need for intermediate labels. We show that by leveraging the existing precise black-box function during inference, the integrated model generalizes better than a fully differentiable model, and learns more efficiently compared to RL-based methods.
This paper presents an approach, called EstiNet, to train a hybrid models which uses both neural networks and black-box functions. The key idea is that, during training, a neural network can be used to approximate the functionality of the black-box functions, which makes the whole system end-to-end differentiable. At test time, the true black-box functions are still used. The training objective composes two parts: L_bbf, the loss for approximating the black-box function and L_target, the loss for the end-to-end goal. They tried different variations of when to train the black-box function approximator or not. It is shown to outperform the baselines like end-to-end differentiable model or NALU over 4 synthetic tasks in sample efficiency. There are some analysis about how the entropy loss and label smoothing helps with the gradient flow. 
The proposed model is interesting, and is shown to be effective in the synthetic tasks. The paper is well-written and easy to follow. However, some of the experiment details are missing or scattered in the text, which might make it hard for the readers to reproduce the result. I think it helps to have the experimental details (number of examples, number of offline pretraining steps, size of the neural network, etc) organized in some tables (could be put in the appendix). 
Two main concerns about how generally applicable is the proposed approach: 
1. It helps to show how L_target depends on L_bbf, or how good the approximation of the black-box function has to be to make the approach applicable. For example, some functions, such as sorting, are hard to approximate by neural network in a generalizable way, so in those cases, is it still possible to apply the proposed approach? 
2.The proposed approach can be better justified by discussing some potential real world applications. Two closely related applications I can think of are visual question answering and semantic parsing. However, it is hard to find good black-box functions for VQA and people often learn them as neural networks, and the functions in semantic parsing often need to interact with a database or knowledge graph, which is hard to approximate with a neural network. 
Some minor issues:
Table 3 isn’t very informative since k=2 and k=3 provides very similar results. It would help to show how large k needs to be for the performance to severely degrade. 
Missing references: 
The Related Works section only reviewed some reinforcement learning work on synthetic tasks. However, with some bootstrapping, RL is also shown to achieve state-of-the-art performance on visual question answering and semantic parsing tasks (Johnson et al, 2017; Liang et al, 2018), which might be good to include here. 
Johnson, J., Hariharan, B., van der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. B. (2017, May). Inferring and Executing Programs for Visual Reasoning. In ICCV (pp. 3008-3017).
Liang, C., Norouzi, M., Berant, J., Le, Q., & Lao, N. (2018). Memory augmented policy optimization for program synthesis with generalization. arXiv preprint arXiv:1807.02322.