There are families of neural networks that can learn to compute any function, provided sufficient training data. However, given that in practice training data is scarce for all but a small set of problems, a core question is how to incorporate prior knowledge into a model. Here we consider the case of prior procedural knowledge, such as knowing the overall recursive structure of a sequence transduction program or the fact that a program will likely use arithmetic operations on real numbers to solve a task. To this end we present a differentiable interpreter for the programming language Forth. Through a neural implementation of the dual stack machine that underlies Forth, programmers can write program sketches with slots that can be filled with behaviour trained from program input-output data. As the program interpreter is end-to-end differentiable, we can optimize this behaviour directly through gradient descent techniques on user specified objectives, and also integrate the program into any larger neural computation graph. We show empirically that our interpreter is able to effectively leverage different levels of prior program structure and learn complex transduction tasks such as sequence sorting or addition with substantially less data and better generalisation over problem sizes. In addition, we introduce neural program optimisations based on symbolic computation and parallel branching that lead to significant speed improvements.
1 INTRODUCTION
A central goal of Artificial Intelligence is the creation of machines that learn as effectively from human instruction as they do from data. A recent and important step towards this goal is the invention of neural architectures that can learn to perform algorithms akin to traditional computers, using primitives such as memory access and stack manipulation (Graves et al., 2014; Joulin & Mikolov, 2015; Grefenstette et al., 2015; Kaiser & Sutskever, 2015; Kurach et al., 2015; Graves et al., 2016). These architectures can be trained through standard gradient descent methods, and enable machines to learn complex behavior from input-output pairs or program traces. In this context the role of the human programmer is often limited to providing training data. However, for many tasks training data is scarce. In these cases the programmer may have partial procedural background knowledge: one may know the rough structure of the program, or how to implement several sub-routines that are likely necessary to solve the task. For example, in visual programming, a user often knows a rough sketch of what they want to do, but need to fill in the specific components. In programming by demonstration (Lau et al., 2001) and programming with query languages (Neelakantan et al., 2015a) a user conforms to a larger set of conditions on the data, and needs to settle details. In all these scenarios, the question then becomes how to exploit this type of prior knowledge when learning algorithms.
To address the above question we present an approach that enables programmers to inject their procedural background knowledge into a neural network. In this approach the programmer specifies a program sketch (Solar-Lezama et al., 2005) in a traditional programming language. This sketch defines one part of the neural network behaviour. The other part is learned using training data. The core insight that enables this approach is the fact that most programming languages can be formulated
in terms of an abstract machine that executes the commands of the language. We implement these machines as neural networks, constraining parts of the networks to follow the sketched behaviour. The resulting neural programs are consistent with our prior knowledge and optimised with respect to the training data.
In this paper we focus on the programming language Forth (Brodie, 1980), a simple yet powerful stack-based language that is relatively close to machine code but enables modular programs and facilitates abstraction. Underlying Forth’s semantics is a simple abstract machine. We introduce the Forth Neural Abstract Machine (∂4), an implementation of this machine that is differentiable with respect to the transition it executes at each time step, as well as distributed input representations in the machine buffers. As sketches that users define are also differentiable, any underspecified program content contained within the sketch can be trained through backpropagation.
For two neural programming tasks introduced in previous work (Reed & de Freitas, 2015) we present Forth sketches that capture different degrees of prior knowledge. For example, we define only the general recursive structure of a sorting problem. We show that given only input-output pairs, ∂4 can learn to fill the sketch and generalise well to problems of unseen size. We also use ∂4 to investigate the type and degree of structure necessary when solving tasks, and show how symbolic execution can significantly improve execution time when applicable.
The contribution of our work is fourfold: i) we present a neural implementation of a dual stack machine underlying Forth, ii) we introduce Forth sketches for programming with partial procedural background knowledge, iii) we apply Forth sketches as a procedural prior on learning algorithms from data, and iv) we introduce program code optimisations based on symbolic execution that can speed up neural execution.
2 THE FORTH ABSTRACT MACHINE
Forth is a simple Turing-complete stack-based programming language (ANSI, 1994; Brodie, 1980). Its underlying abstract machine is represented by a state S = (D,R,H, c), which contains two stacks: a data evaluation pushdown stack (data stack) D holds values for manipulation, and a return address pushdown stack (return stack) R assists with return pointers and subroutine calls. These are accompanied by a heap or random memory access buffer H , and a program counter c.
A Forth program P is a flat sequence of Forth words (i.e. commands) P = w1 . . . wn. The role of a word varies, encompassing language keywords, primitives, and user-defined subroutines (e.g. DROP, to discard the top element of the stack, or DUP, to duplicate the top element of the stack).1 Each word wi defines a transition function between machine states, wi : S → S. Therefore, a program P itself defines a transition function by simply applying the word at the current program counter to the current state. Although usually considered as a part of the heap H , we consider Forth programs P separately to ease the analysis.
An example of a Forth program that implements the Bubble sort algorithm, is shown in Listing 1, and a detailed description of how this program is executed by the Forth abstract machine is provided in Appendix B. Notice that while Forth provides common control structures such as looping and branching, these can always be reduced to low-level code that uses jumps and conditional jumps (using the words BRANCH and BRANCH0, respectively). Likewise, we can think of sub-routine definitions as code blocks tagged with a label, and their invocation amounts to jumping to the tagged label.
3 THE DIFFERENTIABLE FORTH ABSTRACT MACHINE
When a programmer writes a Forth program, they define a sequence of Forth words, i.e., a sequence of known state transition functions. In other words, the programmer knows exactly how computation should proceed. To accommodate for cases when the developer’s procedural background knowledge is incomplete, we extend Forth to support the definition of a program sketch. As is the case with Forth programs, sketches are sequences of transition functions. However, a sketch may contain transition functions whose behavior is learned from data.
1In this work, we restrict ourselves to a subset of all Forth words, detailed in Appendix A.
To learn the behaviour of transition functions within a program we would like the machine output to be differentiable with respect to these functions (and possibly representations of inputs to the program). This enables us to choose parameterized transition functions such as neural networks, and efficiently train their parameters through backpropagation and gradient methods. To this end we first provide a continuous representation of the state of a Forth abstract machine. We then present a recurrent neural network (RNN) that models program execution on this machine, parametrised by the transition functions at each time step. Lastly, we discuss optimizations based on symbolic execution and the interpolation of conditional branches.
3.1 MACHINE STATE ENCODING
We map the symbolic machine state S = (D,R,H, c) to a continuous representation S = (D,R,H, c) – into two differentiable stacks (with pointers), the data stack D = (D,d) and the return stack R = (R, r), a heap H, and an attention vector c indicating which word of the sketch Pθ is being executed at the current time step. All three memory structures, the data stack, the return stack and the heap, are based on differentiable flat memory buffers M ∈ {D,R,H}, where D,R,H ∈ Rl×v , for a stack size l and a value size v. Each has a well-defined, differentiable read operation: readM(a) = aTM (1) and write operation:
writeM(x,a) : M←M− (a⊗ 1) M+ x⊗ a (2) akin to the Neural Turing Machine (NTM) memory (Graves et al., 2014), where ⊗ is the outer product, is the Hadamard product, and a is the address pointer.2 In addition to the memory buffers D and R, the data stack and the return stack contain pointers to the current top-of-the-stack (TOS) element d, r ∈ Rl. This allows us to implement pushing as writing a value x into M and incrementing the TOS pointer as follows:
pushM(x) : writeM(x,p) [side-effect:p← inc(p)] (3) where p ∈ {d, r}, inc(p) = pTR+, and dec(p) = pTR−. and R+ and R− are increment and decrement matrices (left and right circular shift matrices).
Likewise, popping is realized by multiplying the TOS pointer and the memory buffer, and decreasing the TOS pointer:
popM( ) = readM(p) [side-effect:p← dec(p)] (4)
Finally, the program counter c ∈ Rp is a vector that, when one-hot, points to a single word in a program of length p, and is equivalent to the c vector of the symbolic state machine.3 We will use S to denote the space of all continuous representations S.
2The equal widths of H and D allow us to directly move vector representations of values between the heap and the stack.
3During training c can become distributed, and is considered as attention over the program code.
0 : BUBBLE ( a1 ... an n-1 -- one pass ) 1 DUP IF >R 2 OVER OVER < IF SWAP THEN 3 R> SWAP >R 1- BUBBLE R> 4 ELSE 5 DROP 6 THEN 7 ; 8 : SORT ( a1 .. an n -- sorted ) 9 1- DUP 0 DO >R R@ BUBBLE R> LOOP DROP
10 ; 11 2 4 2 7 4 SORT \ Example call
Listing 1: BubbleSort in Forth.
0 : BUBBLE ( a1 ... an n-1 -- one pass ) 1 DUP IF >R 2 { observe D0 D-1 -> permute D-1 D0 R0 } 3 1- BUBBLE R> 4 \ ** Alternative sketch ** 5 \ { observe D0 D-1 -> choose NOP SWAP } 6 \ R> SWAP >R 1- BUBBLE R> 7 ELSE 8 DROP 9 THEN
10 ;
Listing 2: BUBBLE sketch with trainable permutation (trainable comparison in comments).
Neural Forth Words It is straightforward to convert Forth words, defined as functions on discrete machine states, to functions operating on the continuous space S. For example, consider the Forth word DUP, which duplicates the TOS element of the data stack. A differentiable version works by first calculating the value e on the TOS address of D, as e = dTD. It then shifts the stack pointer via d ← inc(d), and writes e to D using writeD(e,d). We present the complete description of implemented Forth Words in Appendix A and their differentiable counterparts in Appendix C.
3.2 FORTH SKETCHES
We define a Forth sketch Pθ as a sequence of continuous transition functions P = w1 . . .wn. Here, wi ∈ S → S either corresponds to a neural Forth word, or is a trainable transition function. We will call these trainable functions slots, as they correspond to underspecified “slots” in the program code that need to be filled by learned behaviour.
We allow users to define a slot w by specifying a pair of a state encoder wenc that produces a latent representation h of the current machine state using a multi-layer perceptron, and a decoder wdec that consumes this representation to produce the next machine state. We hence have w = wdec ◦wenc. To use slots within Forth program code we introduce a notation that reflects this decomposition. In particular, slots are defined using the syntax { encoder -> decoder } where encoder and decoder are specifications of the corresponding slot parts as described below.
Encoders We provide the following options for encoders:
• static: produces a static representation, independent of the actual machine state. • observe e1 . . . em: concatenates the elements e1 . . . em of the machine state. An element
can be a stack item Di at relative index i, a return stack item Ri, etc.
Decoders Users can specify the following decoders:
• choose w1 . . . wm: chooses from the Forth words w1 . . . wm. Takes an input vector h of length m to produce a weighted combination of machine states ∑m i hiwi(S). • manipulate e1 . . . em: directly manipulates the machine state elements e1 . . . em by writing the appropriately reshaped output of the encoder over the machine state elements with writeM.
• permute e1 . . . em: permutes the machine state elements e1 . . . em via a linear combination of m! state vectors.
3.3 THE EXECUTION RNN
We model execution using an RNN which produces a state Si+1 conditioned on a previous state Si. It does so by first passing the current state to each function wi in the program, and then weighing each of the produced next states by the component of the program counter vector ci that corresponds to program index i, effectively using c as an attention vector over code. Formally we have:
Si+1 = RNN(Si,Pθ) = ∑ i ciwi(Si) (5)
Clearly this recursion, and its final state, are differentiable with respect to the program code P, and its inputs. Furthermore, for differentiable Forth programs it is easy to show that the final state of this RNN will correspond to the final state of a symbolic execution.
3.3.1 ∂4 EXECUTION OF A BUBBLESORT SKETCH
Listing 2 defines the BUBBLE word as a sketch capturing several types of prior knowledge. In this section we describe the PERMUTE sketch. In it, we assume BUBBLE involves a recursive call, that terminates at length 1, and that the next BUBBLE call takes as input some function of the current length and the top two stack elements.
The input to this sketch are the sequence to be sorted and its length decremented by one, n− 1 (line 0). These inputs are expected on the data stack. After the length (n− 1) is duplicated for further use with DUP, the machine tests whether it is non-zero (using IF, which consumes the TOS during the check). If n− 1 > 0, it is stored on the R stack for future use (line 1). At this point (line 2) the programmer only knows that a decision must be made based on the top two data stack elements D0 and D-1 (comparison elements), and the top return stack, R0 (length decremented by 1). Here the precise nature of this decision is unknown, but is limited to variants of permutation of these elements, the output of which produce the input state to the decrement -1 and the recursive BUBBLE call (line 3). At the culmination of the call, R0, the output of the learned slot behavior, is moved onto the data stack using R>, and execution proceeds to the next step.
Figure 2 illustrates how portions of this sketch are executed on the ∂4 RNN. The program counter initially resides at >R (line 3 in P), as indicated by the vector c, next to program P. Both data and return stacks are partially filled (R has 1 element, D has 4), and we show the content both through horizontal one-hot vectors and their corresponding integer values (color coded). The vectors d and r point to the top of both stacks, and are in a one-hot state as well. In this execution trace the slot at line 4 is already showing optimal behaviour: it remembers the element on the return stack (4) is larger, and executes BUBBLE on the remaining sequence with the counter n subtracted by one, to 1.
3.4 PROGRAM CODE OPTIMIZATIONS
The ∂4 RNN requires one time step per transition. After each time step the program counter is either incremented or decremented by one, or explicitly set or popped from the stack to jump. In turn a new machine state is calculated by executing all words in the program, and then weighting the result states by the activation of the program counter at the given word. This parallel execution of all words
is expensive, and it is therefore advisable to avoid full RNN steps wherever possible. We use two strategies to significantly speed-up ∂4.
Symbolic Execution Whenever we have a sequence of Forth words that contains no branch entry or exit points, we collapse this sequence to a single transition. We do this using symbolic execution (King, 1976): we first fill the stacks and heap of a standard Forth abstract machine with symbols representing arbitrary values (e.g. D = d1 . . . dl and R = r1 . . . rl), and execute the sequence of Forth words on the machine. This results in a new symbolic state. We use this state, and its difference to the original state, to derive the transition function of the complete sequence. For example, the sequence R> SWAP >R that swaps the top the data stack with the top of the return stack yields the symbolic state D = r1d2 . . . dl. and R = d1r2 . . . rl. Compared to the initial state we have only changed the top elements on both stacks, and hence the neural transition will only need to swap the top elements of D and R.
Interpolation of If-Branches When symbolic execution hits a branching point we generally cannot simply continue execution, as the branching behaviour will depend on the current machine state and we cannot symbolically resolve it. However, for branches arising from if-clauses that involve no function calls or loop structures, we can still avoid giving control back to the program counter and evaluating all words. We simply execute both branches in parallel, and then let the resulting state be the sum of the output states of both branches, weighted by the score given to the symbol TRUE expected on top of the data stack.
3.5 TRAINING
Our training procedure assumes input-output pairs of machine start and end states (xi,yi) only. The output yi defines a target memory YDi and a target pointer y d i on the data stack D, Additionally, we may have a mask Ki that indicates which components of the stack should be assessed and which should be ignored. For example, we do not care about values in the stack buffer above the target stack depth, dependent on ydi . We use DT (θ,xi) and dT (θ,xi) to denote the final state of D and d after T steps of execution RNN, when using initial state xi, and define the loss function:
L(θ) = ∑ i Ki (DT (θ,xi)−YDi )2 + ∑ i (dT (θ,xi)− ydi )2 (6)
We can use backpropagation and any variant of SGD to optimise our loss function. Note that it is trivial to also provide supervision of the intermediate states (trace-level), as done by the Neural Program Interpreter Reed & de Freitas (2015).
4 EXPERIMENTS
We test ∂4 on the sorting and addition tasks presented in Reed & de Freitas (2015) with varying levels of program structure. For each problem we introduce two sketches.
The parameters of each sketch are trained using Adam (Kingma & Ba, 2014), with gradient clipping and gradient noise (Neelakantan et al., 2015b). Hyperparameters were tuned via random search on a development variant of each task, for 1000 epochs, repeating each experiment 5 times. During testing we employ memory element discretisation, replacing differentiable stacks and pointers with their discrete counterparts, and effectively allowing the trained model to generalize to any sequence length if the correct sketch behavior has been learned.
To illustrate the generalization ability of this architecture, we compare against a Seq2Seq (Sutskever et al., 2014) baseline. All Seq2Seq models are single-layer, with a hidden size of 50, trained similarly for 1000 epochs using Adam.
4.1 SORTING
Sorting sequences of digits is a hard task for RNNs such as LSTMs, as they fail to generalize to sequences that are marginally longer than the ones they have been trained on (Reed & de Freitas,
2015). We investigate several strong priors based on BubbleSort for this transduction task and present two ∂4 sketches that enable us to learn sorting from only few training examples.
1. PERMUTE. The PERMUTE sketch specifies that three elements (the top two elements of the stack, and the top of the return stack) must be permuted based on the former’s values. Both the value comparison and the permutation behavior must be learned from input-output examples. Code for this sketch is shown in Listing 2.
2. COMPARE. The COMPARE sketch provides additional prior procedural knowledge to the model. In contrast to PERMUTE, only the comparison between the top two elements on the stack must be learned. It is shown in the Listing 2 comments (lines 5 and 6),
In both sketches, the outer loop can be specified in ∂4 (Listing 1, line 9), which repeatedly calls a function BUBBLE. In doign so, it defines sufficient structure so that the behavior of the network is invariant to the input sequence length.
Quantitative Evaluation on BubbleSort A quantitative comparison of our models on the BubbleSort task is provided in Table 1. For a given test sequence length, we vary the training set sizes to illustrate the model’s ability to generalize to sequences longer than those it observed during training. Here ∂4 is shown to quickly learn the correct sketch behavior, and is able to generalize perfectly to sort sequences of 64 elements after observing only two element sequences during training. In comparison, the Seq2Seq baseline falters when attempting similar generalizations, and performs close to chance when tested on the longer sequences. The exception to ∂4’s flawless performance on the BubbleSort task arises from computational difficulties when training from longer sequence lengths (as indicated by the absence of length 4 training set results when using the COMPARE sketch). We discuss this issue further in Section 5.
When measuring the performance of the model as the number of training instances varies, we can observe the benefit of additional prior knowledge to the optimization process. We show this in Figure 3a, on both train and test accuracy, using train sequences of length 3 and test sequences of length 8. When prior knowledge is provided (COMPARE), the model quickly maximizes the training accuracy. Providing less structure (PERMUTE) results in lower training accuracy when only a few training examples have been observed. However, with additional training instances both sketches learn the correct behavior and generalize equally well.
Qualitative Analysis on BubbleSort It is interesting to analyse the program counter traces, depicted in Figure 4. The trace follows a single example from start, to middle, and the end of the training process. In the beginning of training, the program counter starts to deviate from the one-hot representation in the first 20 steps (not observed in the figure due to unobservable changes), and after 2 iterations of SORT, ∂4 fails to correctly determine the next word. After a few training epochs ∂4 learns better permutations which enables the algorithm to take crisp decisions and halt in the correct state.
Program Code Optimizations We measure the runtime of BubbleSort on sequences of varying length with and without the optimizations described in Section 3.4. The results of ten repeated runs are shown in Figure 3b and demonstrate large relative improvements for symbolic execution and interpolation of if-branches compared to non-optimized ∂4 code.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99100101102103104105106107108109110111112113114115116117118119
(a) Program Counter trace in early stages of training.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99100101102103104105106107108109110111112113114115116117118119
(b) Program Counter trace in the middle of training.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99100101102103104105106107108109110111112113114115116117118119
4.2 ADDITION
Next we applied ∂4 to the problem of learning to add two numbers of n digits each. We rely on the standard elementary school addition algorithm, where the goal is to iterate over pairs of the aligned digits, calculating the sum of each to yield the sum of the original numbers. The key complication arises when two digits sum to a two-digit number, requiring that the correct extra digit (a carry) be carried over to the subsequent column.
Here ∂4 will assume aligned pairs of digits as input, with a carry for the least significant digit (potentially 0), and the length of the respective numbers. The sketches define the high-level operations through recursion, leaving the core addition to be learned from data.
The specified high-level behavior includes the recursive call template and the halting condition of the recursion (no remaining digits, line 1-2). The underspecified addition operation must take three digits from the previous call, the two digits to sum and a previous carry, and produce a single digit (the sum) and the resultant carry. We introduce two sketches for inducing this behavior:
1. MANIPULATE. This sketch provides little prior procedural knowledge as it directly manipulates the ∂4 machine state, filling in a carry and the result digits, based on the top three elements on the data stack (two digits and the carry). It is described in Listing 3.
Test len. 8 64 Train len. 2 4 8 2 4 8 Seq2Seq 37.9 57.8 99.8 15.0 13.5 13.3 Choose 100.0 100.0 100.0 100.0 100.0 100.0
Manipulate 100.0 – – – – –
Listing 3: Manipulate sketch for Elementary Addition. Input data is used to fill data stack externally
Listing 4: Choose sketch for Elementary Addition. Comments as in Listing 3 apply.
2. CHOOSE. Incorporating additional prior information, CHOOSE exactly specifies the results of the computation, namely the output of the first slot (line 5) is the carry, and the output of the second one (line 6) is the propagating result digit, both conditioned on the two digits and the carry on the data stack. It is described in Listing 4.
The rest of the sketch code reduces the problem size by one, and returns the solution by popping it from the return stack.
Quantitative Evaluation on Addition In set of experiments analogous to those in our evaluation on BubbleSort, we demonstrate the performance of ∂4 on the addition task by examining test set sequence lengths of 8 and 64 while varying the lengths of the training set instances (Table 2). The Seq2Seq model again fails to generalize to longer sequences than those observed during training. In comparison, for the CHOOSE sketch ∂4 learns the correct sketch behavior and generalizes to all test sequence lengths. However, for the less structured MANIPULATE sketch, the model is unable to converge when training on sequence lengths greater than one. We discuss this result, and similar difficulties which occurred in the sorting experiments, in the following section.
5 DISCUSSION
∂4 bridges the gap between a traditional programming language and a modern machine learning architecture. However, as we have seen in our evaluation experiments, faithfully simulating the underlying abstract machine architecture introduces its own unique set of concerns.
One such concern is the additional complexity of performing even simple tasks when they are viewed in terms of operations on the underlying machine state. As illustrated in Table 1, ∂4 sketches can be effectively trained from the smallest of training sets, and generalize perfectly to sequences of any length. However, and perhaps unintuitively, difficulty arises when training from longer sequences of modest lengths. Even when dealing with relatively short training length sequences, the underlying machine can unroll into a problematically large number states. For problems whose machine execution is quadratic, like the sorting task (which at input sequences of length 4 has 120 machine states), we observe instabilities during training from backpropagating through such long RNN sequences.
While for many configurations we were able to train ∂4 effectively on the sorting problem, we were unable to do so when using the MANIPULATE sketch on the addition problem. This is despite the addition experiments having comparatively shorter underlying execution RNNs, and highlights the
relationship between the computational difficulty of the problem and the degree of prior knowledge which must be provided in the sketch for successful learning. Using sketches with greater degrees of prior procedural knowledge ∂4 is able to learn in difficult problem scenarios, but for sketches with high degrees of freedom we require an easier scenario for effective gradient propagation.
A second issue arises from incongruencies between how algorithmic data structures are typically used in a traditional language, and our newly introduced desire for them to learn behaviors which generalize to unseen data. For instance, it is common in Forth implementations of sequence processing (Sec. 3.3.1) to include the sequence length on the input, and to store it during computation for use in the algorithm. This is a sensible approach when working purely in the traditional programming paradigm, but in the context of learning it introduces information which influences the model’s representations, prevents it from generalizing, and possibly leads to large memory requirements (when encoded with a one-hot vector). This motivates investigation into which traditional language properties are most suitable to this new hybrid paradigm, and which representations can be used to circumvent the problem.
6 RELATED WORK
Program Synthesis The idea of program synthesis is as old as Artificial Intelligence, and has a long history in computer science (Manna & Waldinger, 1971). Whereas a large body of work has focused on using genetic programming (Koza, 1992) to induce programs from the given input-output specification (Nordin, 1997), there are also various Inductive Programming approaches (Kitzelmann, 2009) aimed at inducing programs from incomplete specifications of the code to be implemented (Albarghouthi et al., 2013; Solar-Lezama et al., 2006). We tackle the same problem of sketching, but in our case we fill the sketches with neural networks able to learn the slot behavior.
Probabilistic and Bayesian Programming Our work is closely related to probabilistic programming languages such as Church (Goodman et al., 2008). They allow users to inject random choice primitives into programs as a way to define generative distributions over possible execution traces. In a sense, the random choice primitives in such languages correspond to the slots in our sketches. A core difference lies in the way we train the behaviour of slots: instead of calculating their posteriors using probabilistic inference, we estimate their parameters using backpropagation and gradient descent, similar to TerpreT (Gaunt et al., 2016), who induce code via backpropagation, and Autograd (Maclaurin et al., 2015), who enable automatic gradient computation in Python code. In addition, the underlying programming and probabilistic paradigm in these programming languages is often functional and declarative, whereas our approach focuses on a procedural and discriminative view. By using an end-to-end differentiable architecture, it is easy to seamlessly connect our sketches to further neural input and output modules, such as an LSTM that feeds into the machine heap, or a neural reinforcement learning agent that operates the neural machine. However, we leave connecting ∂4 with neural upstream and downstream models for future work as it is out of the scope of this paper.
Neural approaches Recently, there has been a surge of research in program synthesis, and execution in deep learning, with increasingly elaborate deep models. Many of these models were based on differentiable versions of abstract data structures (Joulin & Mikolov, 2015; Grefenstette et al., 2015; Kurach et al., 2015), and a few abstract machines, such as the NTM (Graves et al., 2014), Differentiable Neural Computers (Graves et al., 2016), and Neural GPUs (Kaiser & Sutskever, 2015). All these models are able to induce algorithmic behavior from training data. Our work differs in that our differentiable abstract machine allows us to seemingly integrate code and neural networks, and train the neural networks specified by slots via backpropagation through code interpretation.
The work in neural approximations to abstract structures and machines naturally leads to more elaborate machinery able to induce and call code or code-like behavior. Neelakantan et al. (2015a) learned SQL-like behavior—querying tables from natural language with simple arithmetic operations. Andreas et al. (2016) learn to compose neural modules to produce a desired behavior for a visual QA task. Neural Programmer-Interpreters (Reed & de Freitas, 2015) learn to represent and execute programs, operating on different modes of environment, and are able to incorporate decisions better captured in a neural network than in many lines of code (e.g. using image as an
input). Users inject prior procedural knowledge by training on program traces and hence require full procedural knowledge. In contrast, we enable users to use their partial knowledge in sketches.
Neural approaches to language compilation have also been researched, from compiling a language into neural networks (Siegelmann, 1994), over building neural compilers (Gruau et al., 1995) to adaptive compilation (Bunel et al., 2016). However, that line of research did not perceive neural interpreters and compilers as a means of injecting procedural knowledge as we did. To the best of our knowledge, ∂4 is the first working neural implementation of an abstract machine for an actual programming language, and this enables us to inject such priors in a straightforward manner.
7 CONCLUSION AND FUTURE WORK
We have presented ∂4, a differentiable abstract machine for the Forth programming language, and showed how it can be used to complement a programmer’s prior knowledge through the learning of unspecified behavior in Forth sketches. The ∂4 RNN successfully learns to sort and add, using only program sketches and program input-output pairs as input. We believe ∂4, and the larger paradigm it helps establish, will be useful for addressing complex problems where low-level representations of the input are necessary, but higher-level reasoning is difficult to learn and potentially easier to specify.
In future work we aim to explore the relationship between backpropagation and programming language semantics. For instance, when performing a single conceptual operation whose implementation requires a series of simple operations on the machine state (e.g. Listing 4, line 7), one may wish to shortcut the flow of gradients around such low-entropy sequences using residual connections (Srivastava et al., 2015). An alternative solution to this problem would involve inducing hierarchies of actions, where a single more abstract action can be substituted for several repetitive actions, reducing the length of the unrolled action sequence. We also plan to apply ∂4 to such problems in the NLP domain, like machine reading and knowledge base inference. In the long-term, we see the integration of non-differentiable transitions (such as those arising when interacting with a real environment), as an exciting future direction which sits at the intersection of reinforcement learning and probabilistic programming. Additionally, connecting ∂4 with other differentiable models upstream and/or downstream is another direction we would like to tackle.
ACKNOWLEDGMENTS
We thank Guillaume Bouchard, Dirk Weissenborn, Danny Tarlow, and the anonymous reviewers for fruitful discussions and helpful comments on previous drafts of this paper. This work was supported by a Microsoft Research PhD Scholarship, an Allen Distinguished Investigator Award, and a Marie Curie Career Integration Award.
A FORTH WORDS
We implemented a small subset of available Forth words in ∂4. The table of these words, together with their descriptions is given in Table 3. The commands are roughly divided into 6 groups. These groups, line-separated in the table, are:
Data stack operations {num}, 1+, 1-, DUP, SWAP, OVER, DROP Heap operations @, !
Comparators >, <, =
Return stack operations >R, R>, @R
Control statements IF..ELSE..THEN, BEGIN..WHILE..REPEAT, DO..LOOP
Subroutine control :, {sub}, ;
B BUBBLE SORT ALGORITHM DESCRIPTION
An example of a Forth program, that implements the Bubble sort algorithm, is shown in Listing 1. Here we provide a description of how the first iteration of this algorithm is executed by the Forth abstract machine:
The program begins at line 11, putting the sequence [2 4 2 7] on the data stack D, followed by the sequence length 44. It then calls the SORT word.
D R c comment 1 [] [] 11 execution start 2 [2 4 2 7 4] [] 8 pushing sequence to D, calling
SORT subroutine puts ASORT to R
For a sequence of length 4, SORT performs a do-loop in line 9 that calls the BUBBLE sub-routine 3 times. It does so by decrementing the top of D with the 1- word to 3. Subsequently, 3 is duplicated on D by using DUP, and 0 is pushed onto D.
3 [2 4 2 7 3] [ASORT] 9 1- 4 [2 4 2 7 3 3] [ASORT] 9 DUP 6 [2 4 2 7 3 3 0] [ASORT] 9 0
DO consumes the top two stack elements 3 and 0 as the limit and starting point of the loop, leaving the stack D to be [2,4,2,7,3]. We use the return stack R as a temporary variable buffer and push 3 onto it using the word >R. This drops 3 from D, which we copy from R with R@
7 [2 4 2 7 3] [AddrSORT] 9 DO 8 [2 4 2 7] [AddrSORT 3] 9 >R 9 [2 4 2 7 3] [AddrSORT 3] 9 @R
Next, we call BUBBLE to perform one iteration of the bubble pass, (calling BUBBLE 3 times internally), and consuming 3. Notice that this call puts the current program counter onto R, to be used for the program counter c when exiting BUBBLE.
Inside the BUBBLE subroutine, DUP duplicates 3 onR. IF consumes the duplicated 3 and interprets is as TRUE. >R puts 3 on R.
10 [2 4 2 7 3] [ASORT 3 ABUBBLE] 0 calling BUBBLE subroutine puts ABUBBLE to R 11 [2 4 2 7 3 3] [ASORT 3 ABUBBLE] 1 DUP 12 [2 4 2 7 3] [ASORT 3 ABUBBLE] 1 IF 13 [2 4 2 7] [ASORT 3 ABUBBLE 3] 1 >R
Calling OVER twice duplicates the top two elements of the stack, to test them with <, which tests whether 2 < 7. IF tests if the result is TRUE (0), which it is, so it executes SWAP.
14 [2 4 2 7 2 7] [ASORT 3 ABUBBLE 3] 2 OVER OVER 15 [2 4 2 7 1] [ASORT 3 ABUBBLE 3] 2 < 16 [2 4 2 7] [ASORT 3 ABUBBLE 3] 2 IF 17 [2 4 7 2] [ASORT 3 ABUBBLE 3] 2 SWAP
To prepare for the next call to BUBBLE we move 3 back from the return stack R to the data stack D via R>, SWAP it with the next element, put it back to R with >R, decrease the TOS with 1- and invoke BUBBLE again. Notice that R will accumulate the analysed part of the sequence, which will be recursively taken back.
18 [2 4 7 2 3] [ASORT 3 ABUBBLE] 3 R > 19 [2 4 7 3 2] [ASORT 3 ABUBBLE] 3 SWAP 20 [2 4 7 3] [ASORT 3 ABUBBLE 2] 3 >R 21 [2 4 7 2] [ASORT 3 ABUBBLE 2] 3 1- 22 [2 4 7 2] [ASORT 3 ABUBBLE 2] 0 ...BUBBLE
4Note that Forth uses Reverse Polish Notation and that the top of the data stack is 4 in this example.
When we reach the loop limit we drop the length of the sequence and exit SORT using the ; word, which takes the return address from R. At the final point, the stack should contain the ordered sequence [7 4 2 2].
C ∂4 WORDS IMPLEMENTATION

This paper develops a differentiable interpreter for the Forth programming
language. This enables writing a program "sketch" (a program with parts left
out), with a hole to be filled in based upon learning from input-output
examples. The main technical development is to start with an abstract machine
for the Forth language, and then to make all of the operations differentiable.
The technique for making operations differentiable is analogous to what is done
in models like Neural Turing Machine and Stack RNN. Special syntax is developed
for specifying holes, which gives the pattern about what data should be read
when filling in the hole, which data should be written, and what the rough
structure of the model that fills the hole should be. Motivation for why one
should want to do this is that it enables composing program sketches with other
differentiable models like standard neural networks, but the experiments focus
on sorting and addition tasks with relatively small degrees of freedom for how
to fill in the holes.
Experimentally, result show that sorting and addition can be learned given
strong sketches.
The aim of this paper is very ambitious: convert a full programming language to
be differentiable, and I admire this ambition. The idea is provocative and I
think will inspire people in the ICLR community.
The main weakness is that the experiments are somewhat trivial and there are no
baselines. I believe that simply enumerating possible values to fill in the
holes would work better, and if that is possible, then it's not clear to me what
is practically gained from this formulation. (The authors argue that the point
is to compose differentiable Forth sketches with neural networks sitting below,
but if the holes can be filled by brute force, then could the underlying neural
network not be separately trained to maximize the probability assigned to any
filling of the hole that produces the correct input-output behavior?)
Related, one thing that is missing, in my opinion, is a more nuanced outlook of
where the authors believe this work is going. Based on the small scale of the
experiments and from reading other related papers in the area, I sense that it
is hard to scale up differentiable forth to large real-world problems. It
would be nice to have more discussion about this, and perhaps even an experiment
that demonstrates a failure case. Is there a problem that is somewhat more
complex than the ones that appear in the paper where the approach does not work?
What has been tried to make it work? What are the failure modes? What are the
challenges that the authors believe need to be overcome to make this work.
Overall, I think this paper deserves consideration for being provocative.
However, I'm hesitant to strongly recommend acceptance because the experiments
are weak.


TIGHTER BOUNDS LEAD TO IMPROVED CLASSIFIERS
The standard approach to supervised classification involves the minimization of a log-loss as an upper bound to the classification error. While this is a tight bound early on in the optimization, it overemphasizes the influence of incorrectly classified examples far from the decision boundary. Updating the upper bound during the optimization leads to improved classification rates while transforming the learning into a sequence of minimization problems. In addition, in the context where the classifier is part of a larger system, this modification makes it possible to link the performance of the classifier to that of the whole system, allowing the seamless introduction of external constraints.
1 INTRODUCTION
Classification aims at mapping inputs X ∈ X to one or several classes y ∈ Y . For instance, in object categorization, X will be the set of images depicting an object, usually represented by the RGB values of each of their pixels, and Y will be a set of object classes, such as “car” or “dog”. We shall assume we are given a training set comprised of N independent and identically distributed labeled pairs (Xi, yi). The standard approach to solve the problem is to define a parameterized class of functions p(y|X, θ) indexed by θ and to find the parameter θ∗ which minimizes the log-loss, i.e.
θ∗ = arg min θ − 1 N ∑ i log p(yi|Xi, θ) (1.1)
= arg min θ Llog(θ) ,
with
Llog(θ) = − 1
N ∑ i log p(yi|Xi, θ) . (1.2)
One justification for minimizing Llog(θ) is that θ∗ is the maximum likelihood estimator, i.e. the parameter which maximizes
θ∗ = arg max θ p(D|θ)
= arg max θ ∏ i p(yi|Xi, θ) .
There is another reason to use Eq. 1.1. Indeed, the goal we are interested in is minimizing the classification error. If we assume that our classifiers are stochastic and outputs a class according to p(yi|Xi, θ), then the expected classification error is the probability of choosing the incorrect classa. This translates to
L(θ) = 1
N ∑ i (1− p(yi|Xi, θ))
= 1− 1 N ∑ i p(yi|Xi, θ) . (1.3)
aIn practice, we choose the class deterministically and output argmaxy p(y|Xi, θ).
This is a highly nonconvex function of θ, which makes its minimization difficult. However, we have
L(θ) = 1− 1 N ∑ i p(yi|Xi, θ)
≤ 1− 1 N ∑ i 1 K (1 + log p(yi|Xi, θ) + logK)
= (K − 1− logK) K + Llog(θ) K ,
where K = |Y| is the number of classes (assumed finite), using the fact that, for every nonnegative t, we have t ≥ 1 + log t. Thus, minimizing Llog(θ) is equivalent to minimizing an upper bound of L(θ). Further, this bound is tight when p(yi|Xi, θ) = 1K for all yi. As a model with randomly initialized parameters will assign probabilities close to 1/K to each class, it makes sense to minimize Llog(θ) rather than L(θ) early on in the optimization.
However, this bound becomes looser as θ moves away from its initial value. In particular, poorly classified examples, for which p(yi|Xi, θ) is close to 0, have a strong influence on the gradient of Llog(θ) despite having very little influence on the gradient of L(θ). The model will thus waste capacity trying to bring these examples closer to the decision boundary rather than correctly classifying those already close to the boundary. This will be especially noticeable when the model has limited capacity, i.e. in the underfitting setting.
Section 2 proposes a tighter bound of the classification error as well as an iterative scheme to easily optimize it. Section 3 experiments this iterative scheme using generalized linear models over a variety of datasets to estimate its impact. Section 4 then proposes a link between supervised learning and reinforcement learning, revisiting common techniques in a new light. Finally, Section 5 concludes and proposes future directions.
2 TIGHTER BOUNDS ON THE CLASSIFICATION ERROR
We now present a general class of upper bounds of the classification error which will prove useful when the model is far from its initialization.
Lemma 1. Let
pν(y|X, θ) = p(y|X, ν) (
1 + log p(y|X, θ) p(y|X, ν)
) (2.1)
with ν any value of the parameters. Then we have
pν(y|X, θ) ≤ p(y|X, θ) . (2.2)
Further, if ν = θ, we have
pθ(y|X, θ) = p(y|X, θ) , (2.3) ∂pν(y|X, θ)
∂θ
∣∣∣∣ ν=θ = ∂p(y|X, θ) ∂θ . (2.4)
Proof.
p(y|X, θ) = p(y|X, ν)p(y|X, θ) p(y|X, ν)
≥ p(y|X, ν) (
1 + log p(y|X, θ) p(y|X, ν) ) = pν(y|X, θ) .
The second line stems from the inequality t ≥ 1 + log t.
pν(y|X, θ) = p(y|X, θ) is immediate when setting θ = ν in Eq. 2.1. Deriving pν(y|X, θ) with respect to θ yields
∂pν(y|X, θ) ∂θ = p(y|X, ν)∂ log p(y|X, θ) ∂θ
= p(y|X, ν) p(y|X, θ) ∂p(y|X, θ) ∂θ .
Taking θ = ν on both sides yields ∂pν(y|X,θ)∂θ ∣∣∣∣ ν=θ = ∂p(y|X,θ)∂θ .
Lemma 1 suggests that, if the current set of parameters is θt, an appropriate upper bound on the probability that an example will be correctly classified is
L(θ) = 1− 1 N ∑ i p(yi|Xi, θ)
≤ 1− 1 N ∑ i p(yi|Xi, θt) ( 1 + log p(yi|Xi, θ) p(yi|Xi, θt) ) = C − 1
N ∑ i p(yi|Xi, θt) log p(yi|Xi, θ) ,
where C is a constant independent of θ. We shall denote
Lθt(θ) = − 1
N ∑ i p(yi|Xi, θt) log p(yi|Xi, θ) . (2.5)
One possibility is to recompute the bound after every gradient step. This is exactly equivalent to directly minimizing L. Such a procedure is brittle. In particular, Eq. 2.5 indicates that, if an example is poorly classified early on, its gradient will be close to 0 and it will difficult to recover from this situation. Thus, we propose using Algorithm 1 for supervised learning: In regularly recomputing
The data: A dataset D comprising of (Xi, yi) pairs, initial parameters θ0 The result: Final parameters θT for t = 0 to T-1 do
θt+1 = arg minθ Lθt = − ∑ i p(yi|Xi, θt) log p(yi|Xi, θ)
end Algorithm 1: Iterative supervised learning
the bound, we ensure that it remains close to the quantity we are interested in and that we do not waste time optimizing a loose bound.
The idea of computing tighter bounds during optimization is not new. In particular, several authors used a CCCP-based (Yuille & Rangarajan, 2003) procedure to achieve tighter bounds for SVMs (Xu et al., 2006; Collobert et al., 2006; Ertekin et al., 2011). Though Collobert et al. (2006) show a small improvement of the test error, the primary goal was to reduce the number of support vectors to keep the testing time manageable. Also, the algorithm proposed by Ertekin et al. (2011) required the setting of an hyperparameter, s, which has a strong influence on the final solution (see Fig. 5 in their paper). Finally, we are not aware of similar ideas in the context of the logistic loss.
Additionally, our idea extends naturally to the case where p is a complicated function of θ and not easily written as a sum of a convex and a concave function. This might lead to nonconvex inner optimizations but we believe that this can still yield lower classification error. A longer study in the case of deep networks is planned.
REGULARIZATION
As this model further optimizes the training classification accuracy, regularization is often needed. The standard optimization procedure minimizes the following regularized objective:
θ∗ = arg min θ − ∑ i log p(yi|Xi, θ) + λΩ(θ)
= arg min θ − ∑ i 1 K log p(yi|Xi, θ) + λ K Ω(θ) .
Thus, we can view this as an upper bound of the following “true” objective: θ∗ = arg min θ − ∑ i p(yi|Xi, θ) + λ K Ω(θ) ,
which can then be optimized using Algorithm 1.
ONLINE LEARNING
Because of its iterative nature, Algorithm 1 is adapted to a batch setting. However, in many cases, we have access to a stream of data and we cannot recompute the importance weights on all the points. A natural way around this problem is to select a parameter vector θ and to use ν = θ for the subsequent examples. One can see this as “crystallizing” the current solution as the value of ν chosen will affect all subsequent gradients.
3 EXPERIMENTS
We experimented the impact of using tighter bounds to the expected misclassification rate on several datasets, which will each be described in their own section. The experimental setup for all datasets was as follows. We first set aside part of the dataset to compose the test set. We then performed k-fold cross-validation, using a generalized linear model, on the remaining datapoints for different values of T , the number of times the importance weights were recomputed, and the `2-regularizer λ. For each value of T , we then selected the set of hyperparameters (λ and the number of iterations) which achieved the lowest validation classification error. We computed the test error for each of the k models (one per fold) with these hyperparameters. This allowed us to get a confidence intervals on the test error, where the random variable is the training set but not the test set.
For a fair comparison, each internal optimization was run for Z updates so that ZT was constant. Each update was computed on a randomly chosen minibatch of 50 datapoints using the SAG algorithm (Le Roux et al., 2012). Since we used a generalized linear model, each internal optimization was convex and thus had no optimization hyperparameter.
Fig. 1 presents the training classification errors on all the datasets.
3.1 COVERTYPE BINARY DATASET
The Covertype binary dataset (Collobert et al., 2002) has 581012 datapoints in dimension 54 and 2 classes. We used the first 90% for the cross-validation and the last 10% for testing. Due to the small dimension of the input, linear models strongly underfit, a regime in which tighter bounds are most beneficial. We see in Fig. 2 that using T > 1 leads to much lower training and validation classification errors. Training and validation curves are presented in Fig. 2 and the test classification error is listed in Table 1.
3.2 ALPHA DATASET
The Alpha dataset is a binary classification dataset used in the Pascal Large-Scale challenge and contains 500000 samples in dimension 500. We used the first 400000 examples for the cross-validation and the last 100000 for testing. A logistic regression trained on this dataset overfits quickly and, as a result, the results for all values of T are equivalent. Training and validation curves are presented in Fig. 3 and the test classification error is listed in Table 2.
3.3 MNIST DATASET
The MNist dataset is a digit recognition dataset with 70000 samples. The first 60000 were used for the cross-validation and the last 10000 for testing. Inputs have dimension 784 but 67 of them are always equal to 0. Despite overfitting occurring quickly, values of T greater than 1 yield significant improvements over the log-loss. Training and validation curves are presented in Fig. 4 and the test classification error is listed in Table 3.
3.4 IJCNN DATASET
The IJCNN dataset is a dataset with 191681 samples. The first 80% of the dataset were used for training and validation (70% for training, 10% for validation, using random splits), and the last 20% were used for testing samples. Inputs have dimension 23, which means we are likely to be in the underfitting regime. Indeed, larger values of T lead to significant improvements over the log-loss. Training and validation curves are presented in Fig. 5 and the test classification error is listed in Table 4.
4 SUPERVISED LEARNING AS POLICY OPTIMIZATION
We now propose an interpretation of supervised learning which closely matches that of direct policy optimization in reinforcement learning. This allows us to naturally address common issues in the literature, such as optimizing ROC curves or allowing a classifier to withhold taking a decision.
A machine learning algorithm is often only one component of a larger system whose role is to make decisions, whether it is choosing which ad to display or deciding if a patient needs a specific treatment. Some of these systems also involve humans. Such systems are complex to optimize and it is often appealing to split them into smaller components which are optimized independently. However, such splits might lead to poor decisions, even when each component is carefully optimized (Bottou). This issue can be alleviated by making each component optimize the full system with respect to its own parameters. Doing so requires taking into account the reaction of the other components in the system to the changes made, which cannot in general be modeled. However, one may cast it as a reinforcement learning problem where the environment is represented by everything outside of our component, including the other components of the system (Bottou et al., 2013).
Pushing the analogy further, we see that in one-step policy learning, we try to find a policy p(y|X, θ) over actions y given the state X b to minimize the expected loss defined as
L̄(θ) = − ∑ i ∑ y R(y,Xi)p(y|Xi, θ) . (4.1)
L̄(θ) is equivalent to L(θ) from Eq. 1.3 where all actions have a reward of 0 except for the action choosing the correct class yi yielding R(yi, Xi) = 1. One major difference between policy learning and supervised learning is that, in policy learning, we only observe the reward for the actions we have taken, while in supervised learning, the reward for all the actions is known.
Casting the classification problem as a specific policy learning problem yields a loss function commensurate with a reward. In particular, it allows us to explicit the rewards associated with each decision, which was difficult with Eq. 1.1. We will now review several possibilities opened by this formulation.
OPTIMIZING THE ROC CURVE
In some scenarios, we might be interested in other performance metrics than the average classification error. In search advertising, for instance, we are often interested in maximizing the precision at a given recall. Mozer et al. (2001) address the problem by emphasizing the training points whose output is within a certain interval. Gasso et al. (2011); Parambath et al. (2014), on the other hand, assign a different cost to type I and type II errors, learning which values lead to the desired false positive rate. Finally, Bach et al. (2006) propose a procedure to find the optimal solution for all costs efficiently in the context of SVMs and showed that the resulting models are not the optimal models in the class.
To test the impact of optimizing the probabilities rather than a surrogate loss, we reproduced the binary problem of Bach et al. (2006). We computed the average training and testing performance over 10 splits. An example of the training set and the results are presented in Fig. 6.
Even though working directly with probabilities solved the non-concavity issue, we still had to explore all possible cost asymmetries to draw this curve. In particular, if we had been asked to maximize the true positive rate for a given false positive rate, we would have needed to draw the whole curve then find the appropriate point.
However, expressing the loss directly as a function of the probabilities of choosing each class allows us to cast this requirement as a constraint and solve the following constrained optimization problem:
θ∗ = arg min θ − 1 N1 ∑ i/yi=1 p(1|xi, θ) such that 1 N0 ∑ i/yi=0 p(1|xi, θ) ≤ cFP ,
bIn standard policy learning, we actually consider full rollouts which include not only actions but also state changes due to these actions.
with N0 (resp. N1) the number of examples belonging to class 0 (resp. class 1). Since p(1|xi, θ) = 1− p(0|xi, θ) , we can solve the following Lagrangian problem
min θ max λ≥0 L(θ, λ) = min θ max λ≥0 − 1 N1 ∑ i/yi=1 p(1|xi, θ) + λ 1− 1 N0 ∑ i/yi=0 p(0|xi, θ)− cFP  . This is an approach proposed by Mozer et al. (2001) who then minimize this function directly. We can however replace L(θ, λ) with the following upper bound:
L(θ, λ) ≤ − 1 N1 ∑ i/yi=1 p(1|xi, ν) ( 1 + log p(1|xi, θ) p(1|xi, ν) )
+ λ 1− 1 N0 ∑ i/yi=0 p(0|xi, ν) ( 1 + log p(0|xi, θ) p(0|xi, ν) ) − cFP  and jointly optimize over θ and λ. Even though the constraint is on the upper bound and thus will not be exactly satisfied during the optimization, the increasing tightness of the bound with the convergence will lead to a satisfied constraint at the end of the optimization. We show in Fig. 7 the obtained false positive rate as a function of the required false positive rate and see that the constraint is close to being perfectly satisfied. One must note, however, that the ROC curve obtained using the constrained optimization problems matches that of T = 1, i.e. is not concave. We do not have an explanation as to why the behaviour is not the same when solving the constrained optimization problem and when optimizing an asymmetric cost for all values of the asymmetry.
ALLOWING UNCERTAINTY IN THE DECISION
Let us consider a cancer detection algorithm which would automatically classify patients in two categories: healthy or ill. In practice, this algorithm will not be completely accurate and, given the high price of a misclassification, we would like to include the possibility for the algorithm to hand over the decision to the practitioner. In other words, it needs to include the possibility of being “Undecided”.
The standard way of handling this situation is to manually set a threshold on the output of the classifier and, should the maximum score across all classes be below that threshold, deem the example too hard to classify. However, it is generally not obvious how to set the value of that threshold nor how it relates to the quantity we care about, even though some authors provided guidelines (?). The difficulty is heightened when the prior probabilities of each class are very different.
Figure 7: Test false positive rate as a function of the desired false positive rate cFP . The dotted line representing the optimal behaviour, we can see that the constraint is close to being satisfied. T = 10 was used.
Eq. 4.1 allows us to naturally include an extra “action”, the “Undecided” action, which has its own reward. This reward should be equal to the reward of choosing the correct class (i.e., 1) minus the cost ch of resorting to external intervention c, which is less than 1 since we would otherwise rather have an error than be undecided. Let us denote by rh = 1− ch the reward obtained when the model chooses the “Undecided” class. Then, the reward obtained when the input is Xi is:
R(yi|Xi) = 1 R(“Undecided′′|Xi) = rh ,
and the average under the policy is p(yi|Xi, θ) + rhp(“Undecided′′|Xi, θ). Learning this model on a training set is equivalent to minimizing the following quantity:
θ∗ = arg min θ − 1 N ∑ i (p(yi|Xi, θ) + rhp(“Undecided”|Xi, θ)) . (4.2)
For each training example, we have added another example with importance weight rh and class “Undecided”. If we were to solve this problem through a minimization of the log-loss, it is wellknown that the optimal solution would be, for each example Xi, to predict yi with probability 1/(1 + rh) and “Undecided” with probability rh/(1 + rh). However, when optimizing the weighted sum of probabilities, the optimal solution is still to predict yi with probability 1. In other words, adding the “Undecided” class does not change the model if it has enough capacity to learn the training set accurately.
5 DISCUSSION AND CONCLUSION
Using a general class of upper bounds of the expected classification error, we showed how a sequence of minimizations could lead to reduced classification error rates. However, there are still a lot of questions to be answered. As using T > 1 increases overfitting, one might wonder whether the standard regularizers are still adapted. Also, current state-of-the-art models, especially in image classification, already use strong regularizers such as dropout. The question remains whether using T > 1 with these models would lead to an improvement.
Additionally, it makes less and less sense to think of machine learning models in isolation. They are increasingly often part of large systems and one must think of the proper way of optimizing them in this setting. The modification proposed here led to an explicit formulation for the true impact of a classifier. This facilitates the optimization of such a classifier in the context of a larger production system where additional costs and constraints may be readily incorporated. We believe this is a critical venue of research to be explored further.
cThis is assuming that the external intervention always leads to the correct decision. Any other setting can easily be used.
ACKNOWLEDGMENTS
We thank Francis Bach, Léon Bottou, Guillaume Obozinski, and Vianney Perchet for helpful discussions.

The paper proposes new bounds on the misclassification error. The bounds lead to training classifiers with an adaptive loss function, and the algorithm operates in successive steps: the parameters are trained by minimizing the log-loss weighted by the probability of the observed class as given by the parameters of the previous steps. The bound improves on standard log-likelihood when outliers/underfitting prevents the learning algorithm to properly optimize the true classification error. Experiments are performed to confirm the therotical intuition and motivation. They show different cases where the new algorithm leads to improved classification error because underfitting occurs when using standard log-loss, and other cases where the new bounds do not lead to any improvement because the log-loss is sufficient to fit the dataset.
The paper also discusses the relationship between the proposed idea and reinforcement learning, as well as with classifiers that have an "uncertain" label.
While the paper is easy to read and well-written overall, in a second read I found it difficult to fully understand because two problems are somewhat mixed together (here considering only binary classification for simplicity):
(a) the optimization of the classification error of a *randomized* classifier, which predicts 1 with probability P(1|x, theta), and
(b) the optimization of the deterministic classifier, which predicts sign(P(1|x, theta) - 0.5), in a way that is robust to outliers/underfitting.
The reason why I am confused is that "The standard approach to supervised classification", as is mentioned in the abstract, is to use deterministic classifiers at test time, and the log-loss (up to constants) is an upper bound on the classification error of the deterministic classifier. However, the bounds discussed in the paper only concern the randomized classifier.
=== question:
In the experiments, what kind of classifier is used? The randomized one (as would the sentence in the first page suggest "Assuming the class is chosen according to p(y|X, θ)"), or the more standard deterministic classifier argmax_y P(y|x, theta) ?
As far as I can see, there are two cases: either (i) the paper deals with learning randomized classifiers, in which case it should compare the performances with the deterministic counterparts that people use in practice, or (ii) the paper makes sense as soon as we accept that the optimization of criterion (a) is a good surrogate for (b). In both cases,  I think the write-up should be made clearer (because in case (ii) the algorithm does not minimize an upper bound on the classification error, and in case (i) what is done does not correspond to what is usually done in binary classification).
=== comments:
- The section "allowing uncertainty in the decision" may be improved by adding some references, e.g. Bartlett & Wegkamp (2008) "Classification with a Reject Option using a Hinge Loss" or Sayedi et al. (2010) "Trading off Mistakes and Don’t Know Predictions".
- there seems to be a "-" sign missing in the P(1|x, theta) in L(theta, lambda) in Section 3.
- The idea presented in the paper is interesting and original. While I give a relatively low score for now, I am willing to increase this score if the clarifications are made.
Final comments:
I think the paper is clear enough in its current form, even though there should still be improvement in the justification of why and to what extent the error of the randomized classifier is a good surrogate for the error of the true classifier. While the "smoothed" version of the 0/1 loss is an acceptable explanation in the standard classification setup, it is less clear in the section dealing with an additional "uncertain" label. I increase my score from 5 to 6.


Unsupervised learning of probabilistic models is a central yet challenging problem in machine learning. Specifically, designing models with tractable learning, sampling, inference and evaluation is crucial in solving this task. We extend the space of such models using real-valued non-volume preserving (real NVP) transformations, a set of powerful, stably invertible, and learnable transformations, resulting in an unsupervised learning algorithm with exact log-likelihood computation, exact and efficient sampling, exact and efficient inference of latent variables, and an interpretable latent space. We demonstrate its ability to model natural images on four datasets through sampling, log-likelihood evaluation, and latent variable manipulations.
1 Introduction
The domain of representation learning has undergone tremendous advances due to improved supervised learning techniques. However, unsupervised learning has the potential to leverage large pools of unlabeled data, and extend these advances to modalities that are otherwise impractical or impossible.
One principled approach to unsupervised learning is generative probabilistic modeling. Not only do generative probabilistic models have the ability to create novel content, they also have a wide range of reconstruction related applications including inpainting [61, 46, 59], denoising [3], colorization [71], and super-resolution [9].
As data of interest are generally high-dimensional and highly structured, the challenge in this domain is building models that are powerful enough to capture its complexity yet still trainable. We address this challenge by introducing real-valued non-volume preserving (real NVP) transformations, a tractable yet expressive approach to modeling high-dimensional data.
This model can perform efficient and exact inference, sampling and log-density estimation of data points. Moreover, the architecture presented in this paper enables exact and efficient reconstruction of input images from the hierarchical features extracted by this model.
2 Related work
Substantial work on probabilistic generative models has focused on training models using maximum likelihood. One class of maximum likelihood models are those described by probabilistic undirected graphs, such as Restricted Boltzmann Machines [58] and Deep Boltzmann Machines [53]. These models are trained by taking advantage of the conditional independence property of their bipartite structure to allow efficient exact or approximate posterior inference on latent variables. However, because of the intractability of the associated marginal distribution over latent variables, their training, evaluation, and sampling procedures necessitate the use of approximations like Mean Field inference and Markov Chain Monte Carlo, whose convergence time for such complex models ∗Work was done when author was at Google Brain.
remains undetermined, often resulting in generation of highly correlated samples. Furthermore, these approximations can often hinder their performance [7].
Directed graphical models are instead defined in terms of an ancestral sampling procedure, which is appealing both for its conceptual and computational simplicity. They lack, however, the conditional independence structure of undirected models, making exact and approximate posterior inference on latent variables cumbersome [56]. Recent advances in stochastic variational inference [27] and amortized inference [13, 43, 35, 49], allowed efficient approximate inference and learning of deep directed graphical models by maximizing a variational lower bound on the log-likelihood [45]. In particular, the variational autoencoder algorithm [35, 49] simultaneously learns a generative network, that maps gaussian latent variables z to samples x, and a matched approximate inference network that maps samples x to a semantically meaningful latent representation z, by exploiting the reparametrization trick [68]. Its success in leveraging recent advances in backpropagation [51, 39] in deep neural networks resulted in its adoption for several applications ranging from speech synthesis [12] to language modeling [8]. Still, the approximation in the inference process limits its ability to learn high dimensional deep representations, motivating recent work in improving approximate inference [42, 48, 55, 63, 10, 59, 34].
Such approximations can be avoided altogether by abstaining from using latent variables. Autoregressive models [18, 6, 37, 20] can implement this strategy while typically retaining a great deal of flexibility. This class of algorithms tractably models the joint distribution by decomposing it into a product of conditionals using the probability chain rule according to a fixed ordering over dimensions, simplifying log-likelihood evaluation and sampling. Recent work in this line of research has taken advantage of recent advances in recurrent networks [51], in particular long-short term memory [26], and residual networks [25, 24] in order to learn state-of-the-art generative image models [61, 46] and language models [32]. The ordering of the dimensions, although often arbitrary, can be critical to the training of the model [66]. The sequential nature of this model limits its computational efficiency. For example, its sampling procedure is sequential and non-parallelizable, which can become cumbersome in applications like speech and music synthesis, or real-time rendering. Additionally, there is no natural latent representation associated with autoregressive models, and they have not yet been shown to be useful for semi-supervised learning.
Generative Adversarial Networks (GANs) [21] on the other hand can train any differentiable generative network by avoiding the maximum likelihood principle altogether. Instead, the generative network is associated with a discriminator network whose task is to distinguish between samples and real data. Rather than using an intractable log-likelihood, this discriminator network provides the training signal in an adversarial fashion. Successfully trained GAN models [21, 15, 47] can consistently generate sharp and realistically looking samples [38]. However, metrics that measure the diversity in the generated samples are currently intractable [62, 22, 30]. Additionally, instability in their training process [47] requires careful hyperparameter tuning to avoid diverging behavior.
Training such a generative network g that maps latent variable z ∼ pZ to a sample x ∼ pX does not in theory require a discriminator network as in GANs, or approximate inference as in variational autoencoders. Indeed, if g is bijective, it can be trained through maximum likelihood using the change of variable formula:
pX(x) = pZ(z) ∣∣∣∣det(∂g(z)∂zT )∣∣∣∣−1 . (1)
This formula has been discussed in several papers including the maximum likelihood formulation of independent components analysis (ICA) [4, 28], gaussianization [14, 11] and deep density models [5, 50, 17, 3]. As the existence proof of nonlinear ICA solutions [29] suggests, auto-regressive models can be seen as tractable instance of maximum likelihood nonlinear ICA, where the residual corresponds to the independent components. However, naive application of the change of variable formula produces models which are computationally expensive and poorly conditioned, and so large scale models of this type have not entered general use.
3 Model definition
In this paper, we will tackle the problem of learning highly nonlinear models in high-dimensional continuous spaces through maximum likelihood. In order to optimize the log-likelihood, we introduce a more flexible class of architectures that enables the computation of log-likelihood on continuous data using the change of variable formula. Building on our previous work in [17], we define a powerful class of bijective functions which enable exact and tractable density evaluation and exact and tractable inference. Moreover, the resulting cost function does not to rely on a fixed form reconstruction cost such as square error [38, 47], and generates sharper samples as a result. Also, this flexibility helps us leverage recent advances in batch normalization [31] and residual networks [24, 25] to define a very deep multi-scale architecture with multiple levels of abstraction.
3.1 Change of variable formula
Given an observed data variable x ∈ X , a simple prior probability distribution pZ on a latent variable z ∈ Z, and a bijection f : X → Z (with g = f−1), the change of variable formula defines a model distribution on X by
pX(x) = pZ ( f(x) ) ∣∣∣∣det(∂f(x)∂xT )∣∣∣∣ (2)
log (pX(x)) = log ( pZ ( f(x) )) + log (∣∣∣∣det(∂f(x)∂xT )∣∣∣∣) , (3)
where ∂f(x) ∂xT is the Jacobian of f at x.
Exact samples from the resulting distribution can be generated by using the inverse transform sampling rule [16]. A sample z ∼ pZ is drawn in the latent space, and its inverse image x = f−1(z) = g(z) generates a sample in the original space. Computing the density on a point x is accomplished by computing the density of its image f(x) and multiplying by the associated Jacobian determinant det ( ∂f(x) ∂xT ) . See also Figure 1. Exact and efficient inference enables the accurate and fast evaluation of the model.
3.2 Coupling layers
Computing the Jacobian of functions with high-dimensional domain and codomain and computing the determinants of large matrices are in general computationally very expensive. This combined with the restriction to bijective functions makes Equation 2 appear impractical for modeling arbitrary distributions.
As shown however in [17], by careful design of the function f , a bijective model can be learned which is both tractable and extremely flexible. As computing the Jacobian determinant of the transformation is crucial to effectively train using this principle, this work exploits the simple observation that the determinant of a triangular matrix can be efficiently computed as the product of its diagonal terms.
We will build a flexible and tractable bijective function by stacking a sequence of simple bijections. In each simple bijection, part of the input vector is updated using a function which is simple to invert, but which depends on the remainder of the input vector in a complex way. We refer to each of these simple bijections as an affine coupling layer. Given a D dimensional input x and d < D, the output y of an affine coupling layer follows the equations
y1:d = x1:d (4) yd+1:D = xd+1:D exp ( s(x1:d) ) + t(x1:d), (5)
where s and t stand for scale and translation, and are functions from Rd 7→ RD−d, and is the Hadamard product or element-wise product (see Figure 2(a)).
3.3 Properties
The Jacobian of this transformation is
∂y
∂xT =
[ Id 0
∂yd+1:D ∂xT1:d
diag ( exp [s (x1:d)] ) ] , (6) where diag ( exp [s (x1:d)] ) is the diagonal matrix whose diagonal elements correspond to the vector exp [s (x1:d)]. Given the observation that this Jacobian is triangular, we can efficiently compute its determinant as exp [∑ j s (x1:d)j ] . Since computing the Jacobian determinant of the coupling layer operation does not involve computing the Jacobian of s or t, those functions can be arbitrarily complex. We will make them deep convolutional neural networks. Note that the hidden layers of s and t can have more features than their input and output layers.
Another interesting property of these coupling layers in the context of defining probabilistic models is their invertibility. Indeed, computing the inverse is no more complex than the forward propagation
(see Figure 2(b)), { y1:d = x1:d yd+1:D = xd+1:D exp ( s(x1:d) ) + t(x1:d)
(7)
⇔ { x1:d = y1:d xd+1:D = ( yd+1:D − t(y1:d) ) exp ( − s(y1:d) ) ,
(8)
meaning that sampling is as efficient as inference for this model. Note again that computing the inverse of the coupling layer does not require computing the inverse of s or t, so these functions can be arbitrarily complex and difficult to invert.
3.4 Masked convolution
Partitioning can be implemented using a binary mask b, and using the functional form for y, y = b x+ (1− b) ( x exp ( s(b x) ) + t(b x) ) . (9)
We use two partitionings that exploit the local correlation structure of images: spatial checkerboard patterns, and channel-wise masking (see Figure 3). The spatial checkerboard pattern mask has value 1 where the sum of spatial coordinates is odd, and 0 otherwise. The channel-wise mask b is 1 for the first half of the channel dimensions and 0 for the second half. For the models presented here, both s(·) and t(·) are rectified convolutional networks.
3.5 Combining coupling layers
Although coupling layers can be powerful, their forward transformation leaves some components unchanged. This difficulty can be overcome by composing coupling layers in an alternating pattern, such that the components that are left unchanged in one coupling layer are updated in the next (see Figure 4(a)).
The Jacobian determinant of the resulting function remains tractable, relying on the fact that
∂(fb ◦ fa) ∂xTa (xa) = ∂fa ∂xTa (xa) · ∂fb ∂xTb
( xb = fa(xa) ) (10)
det(A ·B) = det(A) det(B). (11)
Similarly, its inverse can be computed easily as
(fb ◦ fa)−1 = f−1a ◦ f−1b . (12)
3.6 Multi-scale architecture
We implement a multi-scale architecture using a squeezing operation: for each channel, it divides the image into subsquares of shape 2× 2× c, then reshapes them into subsquares of shape 1× 1× 4c. The squeezing operation transforms an s × s × c tensor into an s2 × s 2 × 4c tensor (see Figure 3), effectively trading spatial size for number of channels.
At each scale, we combine several operations into a sequence: we first apply three coupling layers with alternating checkerboard masks, then perform a squeezing operation, and finally apply three more coupling layers with alternating channel-wise masking. The channel-wise masking is chosen so that the resulting partitioning is not redundant with the previous checkerboard masking (see Figure 3). For the final scale, we only apply four coupling layers with alternating checkerboard masks.
Propagating a D dimensional vector through all the coupling layers would be cumbersome, in terms of computational and memory cost, and in terms of the number of parameters that would need to be trained. For this reason we follow the design choice of [57] and factor out half of the dimensions at regular intervals (see Equation 14). We can define this operation recursively (see Figure 4(b)),
h(0) = x (13)
(z(i+1), h(i+1)) = f (i+1)(h(i)) (14)
z(L) = f (L)(h(L−1)) (15)
z = (z(1), . . . , z(L)). (16)
In our experiments, we use this operation for i < L. The sequence of coupling-squeezing-coupling operations described above is performed per layer when computing f (i) (Equation 14). At each layer, as the spatial resolution is reduced, the number of hidden layer features in s and t is doubled. All variables which have been factored out at different scales are concatenated to obtain the final transformed output (Equation 16).
As a consequence, the model must Gaussianize units which are factored out at a finer scale (in an earlier layer) before those which are factored out at a coarser scale (in a later layer). This results in the definition of intermediary levels of representation [53, 49] corresponding to more local, fine-grained features as shown in Appendix D.
Moreover, Gaussianizing and factoring out units in earlier layers has the practical benefit of distributing the loss function throughout the network, following the philosophy similar to guiding intermediate layers using intermediate classifiers [40]. It also reduces significantly the amount of computation and memory used by the model, allowing us to train larger models.
3.7 Batch normalization
To further improve the propagation of training signal, we use deep residual networks [24, 25] with batch normalization [31] and weight normalization [2, 54] in s and t. As described in Appendix E we introduce and use a novel variant of batch normalization which is based on a running average over recent minibatches, and is thus more robust when training with very small minibatches.
We also apply batch normalization to the whole coupling layer output. The effects of batch normalization are easily included in the Jacobian computation, since it acts as a linear rescaling on each dimension. That is, given the estimated batch statistics µ̃ and σ̃2, the rescaling function
x 7→ x− µ̃√ σ̃2 +
(17)
has a Jacobian determinant (∏ i (σ̃2i + ) )− 12 . (18)
This form of batch normalization can be seen as similar to reward normalization in deep reinforcement learning [44, 65].
We found that the use of this technique not only allowed training with a deeper stack of coupling layers, but also alleviated the instability problem that practitioners often encounter when training conditional distributions with a scale parameter through a gradient-based approach.
4 Experiments
4.1 Procedure
The algorithm described in Equation 2 shows how to learn distributions on unbounded space. In general, the data of interest have bounded magnitude. For examples, the pixel values of an image typically lie in [0, 256]D after application of the recommended jittering procedure [64, 62]. In order to reduce the impact of boundary effects, we instead model the density of logit(α+(1−α) x256 ), where α is picked here as .05. We take into account this transformation when computing log-likelihood and bits per dimension. We also augment the CIFAR-10, CelebA and LSUN datasets during training to also include horizontal flips of the training examples.
We train our model on four natural image datasets: CIFAR-10 [36], Imagenet [52], Large-scale Scene Understanding (LSUN) [70], CelebFaces Attributes (CelebA) [41]. More specifically, we train on the downsampled to 32× 32 and 64× 64 versions of Imagenet [46]. For the LSUN dataset, we train on the bedroom, tower and church outdoor categories. The procedure for LSUN is the same as in [47]: we downsample the image so that the smallest side is 96 pixels and take random crops of 64×64. For CelebA, we use the same procedure as in [38]: we take an approximately central crop of 148× 148 then resize it to 64× 64. We use the multi-scale architecture described in Section 3.6 and use deep convolutional residual networks in the coupling layers with rectifier nonlinearity and skip-connections as suggested by [46]. To compute the scaling functions s, we use a hyperbolic tangent function multiplied by a learned scale, whereas the translation function t has an affine output. Our multi-scale architecture is repeated recursively until the input of the last recursion is a 4× 4× c tensor. For datasets of images of size 32 × 32, we use 4 residual blocks with 32 hidden feature maps for the first coupling layers with checkerboard masking. Only 2 residual blocks are used for images of size 64× 64. We use a batch size of 64. For CIFAR-10, we use 8 residual blocks, 64 feature maps, and downscale only once. We optimize with ADAM [33] with default hyperparameters and use an L2 regularization on the weight scale parameters with coefficient 5 · 10−5. We set the prior pZ to be an isotropic unit norm Gaussian. However, any distribution could be used for pZ , including distributions that are also learned during training, such as from an auto-regressive model, or (with slight modifications to the training objective) a variational autoencoder.
4.2 Results
We show in Table 1 that the number of bits per dimension, while not improving over the Pixel RNN [46] baseline, is competitive with other generative methods. As we notice that our performance increases with the number of parameters, larger models are likely to further improve performance. For CelebA and LSUN, the bits per dimension for the validation set was decreasing throughout training, so little overfitting is expected.
We show in Figure 5 samples generated from the model with training examples from the dataset for comparison. As mentioned in [62, 22], maximum likelihood is a principle that values diversity
over sample quality in a limited capacity setting. As a result, our model outputs sometimes highly improbable samples as we can notice especially on CelebA. As opposed to variational autoencoders, the samples generated from our model look not only globally coherent but also sharp. Our hypothesis is that as opposed to these models, real NVP does not rely on fixed form reconstruction cost like an L2 norm which tends to reward capturing low frequency components more heavily than high frequency components. Unlike autoregressive models, sampling from our model is done very efficiently as it is parallelized over input dimensions. On Imagenet and LSUN, our model seems to have captured well the notion of background/foreground and lighting interactions such as luminosity and consistent light source direction for reflectance and shadows.
We also illustrate the smooth semantically consistent meaning of our latent variables. In the latent space, we define a manifold based on four validation examples z(1), z(2), z(3), z(4), and parametrized by two parameters φ and φ′ by,
z = cos(φ) ( cos(φ′)z(1) + sin(φ ′)z(2) ) + sin(φ) ( cos(φ′)z(3) + sin(φ ′)z(4) ) . (19)
We project the resulting manifold back into the data space by computing g(z). Results are shown Figure 6. We observe that the model seems to have organized the latent space with a notion of meaning that goes well beyond pixel space interpolation. More manifold visualization are shown in the Appendix. To further test whether the latent space has a consistent semantic interpretation, we trained a class-conditional model on CelebA, and found that the learned representation had a consistent semantic meaning across class labels (see Appendix F).
5 Discussion and conclusion
In this paper, we have defined a class of invertible functions with tractable Jacobian determinant, enabling exact and tractable log-likelihood evaluation, inference, and sampling. We have shown that this class of generative model achieves competitive performances, both in terms of sample quality and log-likelihood. Many avenues exist to further improve the functional form of the transformations, for instance by exploiting the latest advances in dilated convolutions [69] and residual networks architectures [60].
This paper presented a technique bridging the gap between auto-regressive models, variational autoencoders, and generative adversarial networks. Like auto-regressive models, it allows tractable and exact log-likelihood evaluation for training. It allows however a much more flexible functional form, similar to that in the generative model of variational autoencoders. This allows for fast and exact sampling from the model distribution. Like GANs, and unlike variational autoencoders, our technique does not require the use of a fixed form reconstruction cost, and instead defines a cost in terms of higher level features, generating sharper images. Finally, unlike both variational
autoencoders and GANs, our technique is able to learn a semantically meaningful latent space which is as high dimensional as the input space. This may make the algorithm particularly well suited to semi-supervised learning tasks, as we hope to explore in future work.
Real NVP generative models can additionally be conditioned on additional variables (for instance class labels) to create a structured output algorithm. More so, as the resulting class of invertible transformations can be treated as a probability distribution in a modular way, it can also be used to improve upon other probabilistic models like auto-regressive models and variational autoencoders. For variational autoencoders, these transformations could be used both to enable a more flexible reconstruction cost [38] and a more flexible stochastic inference distribution [48]. Probabilistic models in general can also benefit from batch normalization techniques as applied in this paper.
The definition of powerful and trainable invertible functions can also benefit domains other than generative unsupervised learning. For example, in reinforcement learning, these invertible functions can help extend the set of functions for which an argmax operation is tractable for continuous Qlearning [23] or find representation where local linear Gaussian approximations are more appropriate [67].
6 Acknowledgments
The authors thank the developers of Tensorflow [1]. We thank Sherry Moore, David Andersen and Jon Shlens for their help in implementing the model. We thank Aäron van den Oord, Yann Dauphin, Kyle Kastner, Chelsea Finn, Maithra Raghu, David Warde-Farley, Daniel Jiwoong Im and Oriol Vinyals for fruitful discussions. Finally, we thank Ben Poole, Rafal Jozefowicz and George Dahl for their input on a draft of the paper.
A Samples
B Manifold
C Extrapolation
Inspired by the texture generation work by [19, 61] and extrapolation test with DCGAN [47], we also evaluate the statistics captured by our model by generating images twice or ten times as large as present in the dataset. As we can observe in the following figures, our model seems to successfully create a “texture” representation of the dataset while maintaining a spatial smoothness through the image. Our convolutional architecture is only aware of the position of considered pixel through edge effects in convolutions, therefore our model is similar to a stationary process. This also explains why these samples are more consistent in LSUN, where the training data was obtained using random crops.
D Latent variables semantic
As in [22], we further try to grasp the semantic of our learned layers latent variables by doing ablation tests. We infer the latent variables and resample the lowest levels of latent variables from a standard gaussian, increasing the highest level affected by this resampling. As we can see in the following figures, the semantic of our latent space seems to be more on a graphic level rather than higher level concept. Although the heavy use of convolution improves learning by exploiting image prior knowledge, it is also likely to be responsible for this limitation.
E Batch normalization
We further experimented with batch normalization by using a weighted average of a moving average of the layer statistics µ̃t, σ̃2t and the current batch batch statistics µ̂t, σ̂2t ,
µ̃t+1 = ρµ̃t + (1− ρ)µ̂t (20) σ̃2t+1 = ρσ̃ 2 t + (1− ρ)σ̂2t , (21)
where ρ is the momentum. When using µ̃t+1, σ̃2t+1, we only propagate gradient through the current batch statistics µ̂t, σ̂2t . We observe that using this lag helps the model train with very small minibatches.
We used batch normalization with a moving average for our results on CIFAR-10.
F Attribute change
Additionally, we exploit the attribute information y in CelebA to build a conditional model, i.e. the invertible function f from image to latent variable uses the labels in y to define its parameters. In order to observe the information stored in the latent variables, we choose to encode a batch of images x with their original attribute y and decode them using a new set of attributes y′, build by shuffling the original attributes inside the batch. We obtain the new images x′ = g ( f(x; y); y′ ) .
We observe that, although the faces are changed as to respect the new attributes, several properties remain unchanged like position and background.

This paper proposes a new generative model that uses real-valued non-volume preserving transformations in order to achieve efficient and exact inference and sampling of data points.
The authors use the change-of-variable technique to obtain a model distribution of the data from a simple prior distribution on a latent variable. By carefully designing the bijective function used in the change-of-variable technique, they obtain a Jacobian that is triangular and allows for efficient computation.
Generative models with tractable inference and efficient sampling are an active research area and this paper definitely contributes to this field.
While not achieving state-of-the-art, they are not far behind. This doesn't change the fact that the proposed method is innovative and worth exploring as it tries to bridge the gap between auto-regressive models, variational autoencoders and generative adversarial networks.
The authors clearly mention the difference and similarities with other types of generative models that are being actively researched.
Compared to autoregressive models, the proposed approach offers fast sampling.
Compared to generative adversarial networks, Real NVP offers a tractable log-likelihood evaluation.
Compared to variational autoencoders, the inference is exact.
Compared to deep Boltzmann machines, the learning of the proposed method is tractable.
It is clear that Real NVP goal is to bridge the gap between existing and popular generative models.
The paper presents a lot of interesting experiments showing the capabilities of the proposed technique. Making the code available online will certainly contribute to the field. Is there any intention of releasing the code?
Typo: (Section 3.7) We also "use apply" batch normalization