LIE-ACCESS NEURAL TURING MACHINES
External neural memory structures have recently become a popular tool for algorithmic deep learning (Graves et al., 2014; Weston et al., 2014). These models generally utilize differentiable versions of traditional discrete memory-access structures (random access, stacks, tapes) to provide the storage necessary for computational tasks. In this work, we argue that these neural memory systems lack specific structure important for relative indexing, and propose an alternative model, Lieaccess memory, that is explicitly designed for the neural setting. In this paradigm, memory is accessed using a continuous head in a key-space manifold. The head is moved via Lie group actions, such as shifts or rotations, generated by a controller, and memory access is performed by linear smoothing in key space. We argue that Lie groups provide a natural generalization of discrete memory structures, such as Turing machines, as they provide inverse and identity operators while maintaining differentiability. To experiment with this approach, we implement a simplified Lie-access neural Turing machine (LANTM) with different Lie groups. We find that this approach is able to perform well on a range of algorithmic tasks.
1 INTRODUCTION
Recent work on neural Turing machines (NTMs) (Graves et al., 2014; 2016) and memory networks (MemNNs) (Weston et al., 2014) has repopularized the use of explicit external memory in neural networks and demonstrated that these networks can be effectively trained in an end-to-end fashion. These methods have been successfully applied to question answering (Weston et al., 2014; Sukhbaatar et al., 2015; Kumar et al., 2015), algorithm learning (Graves et al., 2014; Kalchbrenner et al., 2015; Kaiser & Sutskever, 2015; Kurach et al., 2015; Zaremba & Sutskever, 2015; Grefenstette et al., 2015; Joulin & Mikolov, 2015), machine translation (Kalchbrenner et al., 2015), and other tasks. This methodology has the potential to extend deep networks in a general-purpose way beyond the limitations of fixed-length encodings such as standard recurrent neural networks (RNNs).
A shared theme in many of these works (and earlier exploration of neural memory) is to re-frame traditional memory access paradigms to be continuous and possibly differentiable to allow for backpropagation. In MemNNs, traditional random-access memory is replaced with a ranking approach that finds the most likely memory. In the work of Grefenstette et al. (2015), classical stack-, queue-, and deque-based memories are replaced by soft-differentiable stack, queue, and deque datastructures. In NTMs, sequential local-access memory is simulated by an explicit tape data structure.
This work questions the assumption that neural memory should mimic the structure of traditional discrete memory. We argue that a neural memory should provide the following: (A) differentiability for end-to-end training and (B) robust relative indexing (perhaps in addition to random-access). Surprisingly many neural memory systems fail one of these conditions, either lacking Criterion B, discussed below, or employing extensions like REINFORCE to work around lack of differentiability (Zaremba & Sutskever, 2015).
We propose instead a class of memory access techniques based around Lie groups, i.e. groups with differentiable operations, which provide a natural structure for neural memory access. By definition, their differentiability satisfies the concerns of Criterion A. Additionally the group axioms provide identity, invertibility, and associativity, all of which are desirable properties for a relative indexing scheme (Criterion B), and all of which are satisfied by standard Turing machines. Notably though,
simple group properties like invertibility are not satisfied by neural Turing machines, differentiable neural computers, or even by simple soft-tape machines. In short, in our method, we construct memory systems with keys placed on a manifold, and where relative access operations are provided by Lie groups.
To experiment with this approach, we implement a neural Turing machine with an LSTM controller and several versions of Lie-access memory, which we call Lie-access neural Turing machines (LANTM). The details of these models are exhibited in Section 4.1 Our main experimental results are presented in Section 5. The LANTM model is able to learn non-trivial algorithmic tasks such as copying and permutating sequences with higher accuracy than more traditional memory-based approaches, and significantly better than fixed memory LSTM models. The memory structures and key transformation learned by the model resemble interesting continuous space representations of traditional discrete memory data structures.
2 BACKGROUND: RECURRENT NEURAL NETWORKS WITH MEMORY
This work focuses particularly on recurrent neural network (RNN) controllers of abstract neural memories. Formally, an RNN is a differentiable function RNN : X × H → H, where X is an arbitrary input space and H is the hidden state space. On input (x(1), . . . , x(T )) ∈ X T and with initial state h(0) ∈ H, the RNN produces states h(1), . . . , h(T ) based on the recurrence,
h(t) := RNN(x(t), h(t−1)).
These states can be used for downstream tasks, for example sequence prediction which produces outputs (y(1), . . . , y(T )) based on an additional transformation and prediction layer y(t) = F (h(t)) such as a linear-layer followed by a softmax. RNNs can be trained end-to-end by backpropagationthrough-time (BPTT) (Werbos, 1990). In practice, we use long short-term memory (LSTM) RNNs (Hochreiter & Schmidhuber, 1997). LSTM’s hidden state consists of two variables (c(t), h(t)), where h(t) is also the output to the external world; we however use the above notation for simplicity.
An RNN can also serve as the controller for an external memory system (Graves et al., 2014; Grefenstette et al., 2015; Zaremba & Sutskever, 2015), which enables: (1) the entire system to carry state over time from both the RNN and the external memory, and (2) the RNN controller to collect readings from and compute additional instructions to the external memory. Formally, we extend the recurrence to,
h(t) := RNN([x(t); ρ(t−1)], h(t−1)),
Σ(t), ρ(t) := RW(Σ(t−1), h(t)),
where Σ is the abstract memory state, and ρ(t) is the value read from memory, and h is used as an abstract controller command to a read/write function RW. Writing occurs in the mutation of Σ at each time step. Throughout this work, Σ will take the form of an ordered set {(ki, vi, si)}i where ki ∈ K is an arbitrary key, vi ∈ Rm is a memory value, and si ∈ R+ is a memory strength. In order for the model to be trainable with backpropagation, the memory function RW must also be differentiable. Several forms of differentiable memory have been proposed in the literature. We begin by describing two simple forms: (neural) random-access memory and (neural) tape-based memory. For this section, we focus on the read step and assume Σ is fixed.
Random-Access Memory Random-access memory consists of using a now standard attentionmechanism or MemNN to read a memory (our description follows Miller et al. (2016)). The controller hidden state is used to output a random-access pointer, q′(h) that determines a weighting of memory vectors via dot products with the corresponding keys. This weighting in turn determines the read values via linear smoothing based on a function w,
wi(q,Σ) := si exp 〈q, ki〉∑
j sj exp 〈q, kj〉 ρ := ∑ i wi(q ′(h),Σ)vi.
The final read memory is based on how “close” the read pointer was to each of the keys, where closeness in key space is determined by w.
1Our implementations are available at https://github.com/harvardnlp/lie-access-memory
Tape-Based Memory Neural memories can also be extended to support relative access by maintaining read state. Following notation from Turing machines, we call this state the head, q. In the simplest case the recurrence now has the form,
Σ′, q′, ρ = RW(Σ, q, h),
and this can be extended to support multiple heads.
In the simplest case of soft tape-based memory (a naive version of the much more complicated neural Turing machine), the keys ki indicate one-hot positions along a tape with ki = δi. The head q is a probability distribution over tape positions. It determines the read value by directly specifying the weights. The controller can only “shift” the head by outputting a kernel K(h) = (K−1,K0,K+1) in the probability simplex ∆2 and applying convolution.
q′(q, h) := q ∗K(h), i.e. q′j = qj−1K+1 + qjK0 + qj+1K−1
We can view this as the soft version of a single-step discrete Turing machine where the kernel can softly shift the “head” of the machine one to the left, one to the right, or remain in the same location. The value returned can then be computed with linear smoothing as above,
wi(q,Σ) := si〈q, ki〉∑
j sj〈q, kj〉 ρ := ∑ i wi(q ′(q, h),Σ)vi.
3 LIE GROUPS FOR MEMORY
Let us now take a brief digression and consider the standard (non-neural) Turing machine (TM) and the movement of its head over a tape. A TM has a head q ∈ Z indicating the position on a tape. Between reads, the head can move any number of steps left or right. Moving a + b steps and then c steps eventually puts the head at the same location as moving a steps and then b + c steps — i.e. the head movement is associative. In addition, the machine should be able to reverse a head shift, for example, in a stack simulation algorithm, going from push to pop — i.e. each head movement should also have a corresponding inverse. Finally, the head should also be allowed to stay put, for example, to read a single data item and use it for multiple time points, an identity.
These movements correspond directly to group actions: the possible head movements should be associative, and contain inverse and identity elements. This group acts on the set of possible head locations. In a TM, the set of Z-valued head movement acts on the set of locations on the Z-indexed infinite tape. By our reasoning above, if a Turing machine is to store data contents at points in a general space K (instead of an infinite Z-indexed tape), then its head movements should form a group and act on K via group actions. For a neural memory system, we desire the network to be (almost everywhere) differentiable. The notion of “differentiable” groups is well-studied in mathematics, where they are known as Lie groups, and “differentiable group actions” are correspondingly called Lie group actions. In our case, using Lie group actions as generalized head movements on a general key space (more accurately, manifolds) would most importantly mean that we can take derivatives of these movements and perform the usual backpropagation algorithm.
4 LIE-ACCESS NEURAL TURING MACHINES
These properties motivate us to propose Lie access as an alternative formalism to popular neural memory systems, such as probabilistic tapes, which surprisingly do not satisfy invertibility and often do not provide an identity.2 Our Lie-access memory will consist of a set of points in a manifold K.
2The Markov kernel convolutional soft head shift mechanism proposed in Graves et al. (2014) and sketched in Section 2 does not in general have inverses. Indeed, the authors reported problems with the soft head losing “sharpness” over time, which they dealt with by sharpening coefficients. In the followup work, Graves et al. (2016) utilize a temporal memory link matrix for actions. They note, “the operation Lw smoothly shifts the focus forwards to the locations written ... whereas L>w shifts the focus backwards” but do not enforce this as a true inverse. They also explicitly do not include an identity, noting “Self-links are excluded (the diagonal of the link matrix is always 0)”; however, they could ignore the link matrix with an interpolation gate, which in effect acts as the identity.
We replace the discrete head with a continuous head q ∈ K. The head moves based on a set of Lie group actions a ∈ A generated by the controller. To read memories, we will rely on a distance measure in this space, d : K × K → R≥0.3 Together these properties describe a general class of possible neural memory architectures.
Formally a Lie-access neural Turing machine (LANTM) computes the following function,
Σ′, q′, q′(w), ρ := RW(Σ, q, q(w), h)
where q, q(w) ∈ K are resp. read and write heads, and Σ is the memory itself. We implement Σ, as above, as a weighted dictionary Σ = {(ki, vi, si)}i.
4.1 ADDRESSING PROCEDURE
The LANTM maintains a read head q which at every step is first updated to q′ and then used to read from the memory table. This update occurs by selecting a Lie group action from A which then acts smoothly on the key space K. We parametrize the action transformation, a : H 7→ A by the hidden state to produce the Lie action, a(h) ∈ A. In the simplest case, the head is then updated based on this action (here · denotes group action): q′ := a(h) · q. For instance, consider two possible Lie groups:
(1) A shift group R2 acting additively on R2. This means that A = R2 so that a(h) = (α, β) acts upon a head q = (x, y) by,
a(h) · q = (α, β) + (x, y) = (x+ α, y + β).
(2) A rotation group SO(3) acting on the sphere S2 = {v ∈ R3 : ‖v‖ = 1}. Each rotation can be described by its axis ξ (a unit vector) and angle θ. An action (ξ, θ) · q is just the appropriate rotation of the point q, and is given by Rodrigues’ rotation formula,
a(h) · q = (ξ, θ) · q = q cos θ + (ξ × q) sin θ + ξ〈ξ, q〉(1− cos θ). Here × denotes cross product.
4.2 READING AND WRITING MEMORIES
Recall that memories are stored in Σ, each with a key, ki, memory vector, vi, and strength, si, and that memories are read using linear smoothing over vectors based on a key weighting function w, ρ := ∑ i wi(q
′,Σ)vi . While there are many possible weighting schemes, we use one based on the distance of each memory address from the head in key-space assuming a metric d on K. We consider two different weighting functions (1) inverse-square and (2) softmax. There first uses the polynomial law and the second an annealed softmax of the squared distances:
w (1) i (q,Σ) :=
sid(q, ki) −2∑
j sjd(q, kj) −2 w
(2) i (q,Σ, T ) := si exp(−d(q, ki)2/T )∑ j sj exp(−d(q, kj)2/T ) ,
where we use the convention that it takes the limit value when q → ki and T is a temperature that represents the certainty of its reading, i.e. higher T creates more uniform w.
The writing procedure is similar to reading. The LANTM maintains a separate write head q(w) that moves analogously to the read head, i.e. with action function a(w)(h) and updated value q′(w) . At each call to RW, a new memory is automatically appended to Σ with k = q′(w). The corresponding
3This metric should satisfy a compatibility relation with the Lie group action. When points x, y ∈ X are simultaneously moved by the same Lie group action v, their distance should stay the same (One possible mathematical formalization is thatX should be a Riemannian manifold and the Lie group should be a subgroup of X’s isometry group.): d(vx, vy) = d(x, y). This condition ensures that if the machine writes a sequence of data along a “straight line” at points x, vx, v2x, . . . , vkx, then it can read the same sequence by emitting a read location y close to x and then follow the “v-trail” y, vy, v2y, . . . , vky.
memory v and strength s are created by MLP’s v(h) ∈ Rm and s(h) ∈ [0, 1] taking h as input. After writing, the new memory set is,
Σ′ := Σ ∪ {(q′(w), v(h), s(h))}.
No explicit erase mechanism is provided, but to erase a memory (k, v, s), the controller may in theory write (k,−v, s).
4.3 COMBINING WITH RANDOM ACCESS
Finally we combine this relative addressing procedure with direct random-access to give the model the ability for absolute address access. We do this by outputting an absolute address each step and simply interpolating with our current head. Write t(h) ∈ [0, 1] for the interpolation gate and q̃(h) ∈ K for our proposed random-access layer. For key space manifolds K like Rn, 4 there’s a well defined straight-line interpolation between two points, so we can set
q′ := a · (tq + (1− t)q̃)
where we have omitted the implied dependence on h. For other manifolds like the spheres Sn that have well-behaved projection functions π : Rn → Sn, we can just project the straight-line interpolation to the sphere:
q′ := a · π(tq + (1− t)q̃).
In the case of a sphere Sn, π is just L2-normalization.5
5 EXPERIMENTS
We experiment with Lie-access memory on a variety of algorithmic learning tasks. We are particularly interested in: (a) how Lie-access memory can be trained, (b) whether it can be effectively utilized for algorithmic learning, and (c) what internal structures the model learns compared to systems based directly on soft discrete memory. In particular Lie access is not equipped with an explicit stack or tape, so it would need to learn continuous patterns that capture these properties.
Setup. Our experiments utilize an LSTM controller in a version of the encoder-decoder setup (Sutskever et al., 2014), i.e. an encoding input pass followed by a decoding output pass. The encoder reads and writes memories at each step; the decoder only reads memories. The encoder is given 〈s〉,
4Or in general, manifolds with convex embeddings in Rn. 5Technically, in the sphere case, domπ = Rd − {0}. But in practice one almost never gets 0 from a
straight-line interpolation, so computationally this makes little difference.
followed by an the input sequence, and then 〈/s〉 to terminate input. The decoder is not re-fed its output or the correct symbol, i.e. we do not use teacher forcing, so x(t) is a fixed placeholder input symbol. The decoder must correctly emit an end-of-output symbol 〈/e〉 to terminate. Models and Baselines. We implement three main baseline models including: (a) a standard LSTM encoder-decoder, without explicit external memory, (b) a random access memory network, RAM using the key-value formulation as described in the background, roughly analogous to an attentionbased encoder-decoder, and (c) an interpolation of a RAM/Tape-based memory network as described in the background, i.e. a highly simplified version of a true NTM (Graves et al., 2014) with a sharpening parameter. Our models include four versions of Lie-access memory. The main model, LANTM, has an LSTM controller, with a shift group A = R2 acting additively on key space K = R2. We also consider a model SLANTM with spherical memory, utilizing a rotation group A = SO(3) acting on keys in the sphere K = S2. For both of the models, the distance function d is the Euclidean (L2) distance, and we experiment with smoothing using inverse-square (default) and with an annealed softmax.6
Model Setup. For all tasks, the LSTM baseline has 1 to 4 layers, each with 256 cells. Each of the other models has a single-layer, 50-cell LSTM controller, with memory width (i.e. the size of each memory vector) 20. Other parameters such as learning rate, decay, and intialization are found through grid search. Further hyperparameter details are give in the appendix.
Tasks. Our experiments are on a series of algorithmic tasks shown in Table 1a. The COPY, REVERSE, and BIGRAM FLIP tasks are based on Grefenstette et al. (2015); the DOUBLE and INTERLEAVED ADD tasks are designed in a similar vein. Additionally we also include three harder tasks: ODD FIRST, REPEAT COPY, and PRIORITY SORT. In ODD FIRST, the model must output the oddindexed elements first, followed by the even-indexed elements. In REPEAT COPY, each model must repeat a sequence of length 20, N times. In PRIORITY SORT, each item of the input sequence is given a priority, and the model must output them in priority order.
We train each model in two regimes, one with a small number of samples (16K) and one with a large number of samples (320K). In the former case, the samples are iterated through 20 times, while in the latter, the samples are iterated through only once. Thus in both regimes, the total training times are the same. Training is done by minimizing negative log likelihood with RMSProp.
Prediction is performed via argmax/greedy prediction at each step. To evaluate the performance of the models, we compute the fraction of tokens correctly predicted and the fraction of all answers completely correctly predicted, respectively called fine and coarse scores. We assess the models on 3.2K randomly generated out-of-sample 2x length examples, i.e. with sequence lengths 2k (or repeat number 2N in the case of REPEAT COPY) to test the generalization of the system. More precisely, for all tasks other than repeat copy, during training, the length k is varied in the interval [lk, uk] (as shown in table 1ba). During test time, the length k is varied in the range [uk + 1, 2uk]. For repeat copy, the repetition number N is varied similarly, instead of k.
Results. Main results comparing the different memory systems and read computations on a series of tasks are shown in Table 1b. Consistent with previous work the fixed-memory LSTM system fails consistently when required to generalize to the 2x samples, unable to solve any 2x problem correctly, and only able to predict at most ∼ 50% of the symbols for all tasks except interleaved addition, regardless of training regime. The RAM (attention-based) and the RAM/tape hybrid are much stronger baselines, answering more than 50% of the characters correctly for all but the 6-ODD FIRST task. Perhaps surprisingly, RAM and RAM/tape learned the 7-REPEAT COPY task with almost perfect generalization scores when trained in the large sample regime. In general, it does not seem that the simple tape memory confers much advantage to the RAM model, as the generalization performances of both models are similar for the most part, which motivates more advanced NTM enhancements beyond sharpening.
The last four columns illustrate the performance of the LANTM models. We found the inversesquare LANTM and SLANTM models to be the most effective, achieving > 90% generalization
6Note that the read weight calculation of a SLANTM with softmax is essentially the same as the RAM model: For head q, exp(−d(q, ki)2/T ) = exp(−‖q − ki‖2/T ) = exp(−(2 − 2〈q, ki〉)/T ), where the last equality comes from ‖q‖ = ‖ki‖ = 1 (key-space is on the sphere). Therefore the weights wi =
si exp(−d(q,ki)2/T )∑ j sj exp(−d(q,kj)2/T ) = si exp(−2〈q,ki〉/T )∑ j sj exp(−2〈q,kj〉/T ) , which is the RAM weighting scheme.
Task Input Output Size k |V| 1 - COPY a1a2a3 · · · ak a1a2a3 · · · ak [2, 64] 128 2 - REVERSE a1a2a3 · · · ak akak−1ak−2 · · · a1 [2, 64] 128 3 - BIGRAM FLIP a1a2a3a4 · · · a2k−1a2k a2a1a4a3 · · · a2ka2k−1 [1, 16] 128 4 - DOUBLE a1a2 · · · ak 2× |ak · · · a1| [2, 40] 10 5 - INTERLEAVED ADD a1a2a3a4 · · · a2k−1a2k |a2ka2k−2 · · · a2|+ |a2k−1 · · · a1| [2, 16] 10 6 - ODD FIRST a1a2a3a4 · · · a2k−1a2k a1a3 · · · a2k−1a2a4 · · · a2k [1, 16] 128 7 - REPEAT COPY Na1 · · · a20 a1 · · · a20 · · · a1 · · · a20 (N times) N ∈ [1, 5] 128 8 - PRIORITY SORT 5a52a29a9 · · · a1a2a3 · · · ak [2, 10] 128
(a) Task descriptions and parameters. |ak · · · a1| means the decimal number repesented by decimal digits ak · · · a1. Arithmetic tasks have all numbers formatted with the least significant digits on the left and with zero padding. The DOUBLE task takes an integer x ∈ [0, 10k) padded to k digits and outputs 2x in k + 1 digits, zero padded to k+ 1 digits. The INTERLEAVED ADD task takes two integers x, y ∈ [0, 10k) padded to k digits and interleaved, forming a length 2k input sequence and outputs x + y zero padded to k + 1 digits. The last two tasks use numbers in unary format: N is the shorthand for a length N sequence of a special symbol @, encoding N in unary, e.g. 3 = @@@.
Base Memory Lie LSTM RAM RAM/Tape LANTM LANTM-s SLANTM SLANTM-s
S L S L S L S L S L S L S L
1 16/0 21/0 61/0 61/1 70/2 70/1 ? ? ? ? ? ? ? ? 2 26/0 32/0 58/2 54/2 24/1 43/2 ? ? 97/44 98/88 99/96 ? ? ? 3 30/0 39/0 56/5 54/9 64/8 69/9 ? ? ? 99/94 99/99 97/67 93/60 90/43 4 44/0 47/0 72/8 74/15 70/12 71/6 ? ? ? ? ? ? ? ? 5 60/0 61/0 74/13 76/17 77/23 67/19 99/93 99/93 90/38 94/57 99/91 99/97 98/78 ? 6 29/0 42/0 31/5 46/4 43/8 62/8 99/91 99/95 90/29 50/0 49/7 56/8 74/15 76/16 7 24/0 37/0 98/56 99/98 71/18 99/93 67/0 70/0 17/0 48/0 99/91 99/78 96/41 99/51 8 46/0 53/0 60/5 80/22 78/15 66/9 87/35 98/72 99/95 99/99 ? 99/99 98/79 ?
(b) Main results. Numbers represent the accuracy percentages on the fine/coarse evaluations on the out-ofsample 2× tasks. The S and L columns resp. indicate small and large sample training regimes. Symbol ? indicates exact 100% accuracy (Fine scores above 99.5 are not rounded up). Baselines are described in the body. LANTM and SLANTM use inverse-square while LANTM-s and SLANTM-s use softmax weighting scheme. The best scores, if not 100% (denoted by stars), are bolded for each of the small and large sample regimes.
accuracy on most tasks, and together they solve all of the tasks here with > 90% coarse score. In particular, LANTM is able to solve the 6-ODD FIRST problem when no other model can correctly solve 20% of the 2x instances; SLANTM on the other hand is the only Lie access model able to solve the 7-REPEAT COPY problem.
The best Lie access model trained with the small sample regime beats or is competitive with any of the baseline trained under the large sample regime. In all tasks other than 7-REPEAT COPY, the gap in the coarse score between the best Lie access model in small sample regime and the best baseline in any sample regime is ≥ 70%. However, in most cases, training under the large sample regime does not improve much. For a few tasks, small sample regime actually produces a model with better generalization than large sample regime. We observed in these instances, the generalization error curve under a large sample regime reaches an optimum at around 2/3 to 3/4 of training time, and then increases almost monotonically from there. Thus, the model likely has found an algorithm that works only for the training sizes; in particular, this phenomenon does not seem to be due to lack of training time.
6 DISCUSSION
Qualitative Analysis. We did further visual analysis of the different Lie-access techniques to see how the models were learning the underlying tasks, and to verify that they were using the relative addressing scheme. Figure 2 shows two diagrams of the LANTM model of the tasks of priority sort and repeat copy. Figure 3 shows two diagrams of the SLANTM model for the same two tasks. Fig-
ure 4 shows the memory access pattern of LANTM on 6-ODD FIRST task. Additionally, animations tracing the evolution of the memory access pattern of models over training time can be found at http://nlp.seas.harvard.edu/lantm. They demonstrate that the models indeed learn relative addressing and internally are constructing geometric data structures to solve these algorithmic tasks.
Unbounded storage One possible criticism of the LANTM framework could be that the amount of information stored increases linearly with time, which limits the usefulness of this framework for long timescale tasks. This is indeed the case with our implementations, but need not be the case in general. There can be many ways of limiting physical memory usage. For example, a simple way is to discard the least recently used memory, as in the work of Graves et al. (2016) and Gulcehre et al. (2016). Another way is to approximate with fixed number of bits the read function that takes a head position and returns the read value. For example, noting that this function is a rational function on the head position, keys, and memory vectors, we can approximate the numerators and denominators with a fixed degree polynomial.
Content address Our Lie-access framework is not mutually exclusive from content addressing methods. For example, in each of our implementations, we could have the controllers output both a position in the key space and a content addresser of the same size as memory vectors, and interpolated the read values from Lie-access and the read values from content addressing.
7 CONCLUSION
This paper introduces Lie-access memory as an alternative neural memory access paradigm, and explored several different implementations of this approach. LANTMs follow similar axioms as discrete Turing machines while providing differentiability. Experiments show that simple models can learn algorithmic tasks. Internally these models naturally learn equivalence of standard data structures like stack and cyclic lists. In future work we hope to experiment with more groups and to scale these methods to more difficult reasoning tasks. For instance, we hope to build a general purpose encoder-decoder model for tasks like question answering and machine translation that makes use of differentiable relative-addressing schemes to replace RAM-style attention.
A EXPERIMENTAL DETAILS
We obtain our results by performing a grid search over the hyperparameters specified in Table A.1 and also over seeds 1 to 3, and take the best scores. We bound the norm of the LANTM head shifts by 1, whereas we try both bounding and not bounding the angle of rotation in our grid for SLANTM. We initialize the Lie access models to favor Lie access over random access through the interpolation mechanism discussed in section 4.3.
The RAM model read mechanism is as discussed in section 2, and writing is done by appending new (k, v, s) tuples to the memory Σ. The only additions to this model in RAM/tape is that left and right keys are now computed using shifted convolution with the read weights:
kL := ∑ i wi+1ki
kR := ∑ i wi−1ki
and these keys kL and kR are available (along with the random access key output by the controller) to the controller on the next turn to select from via interpolation. We also considered weight sharpening in the RAM/Tape model according to Graves et al. (2014): the controller can output a sharpening coefficient γ ≥ 1 each turn, so that the final weights are w̃i =
wγi∑ j w γ j
. We included this as a feature to grid search over.
We found that weight sharpening only confers small advantage over vanilla on the COPY, BIGRAM FLIP, and DOUBLE tasks, but deteriorates performance on all other tasks.
B ACTION INTERPOLATION
We also experimented with adding an interpolation between the last action a(t−1) with a candidate action a(h) via a gate r(h) ∈ [0, 1] to produce the final action a(t). Then the final equation of the new head is q′ := a(t) · π(tq + (1− t)q̃). This allows the controller to easily move in “a straight line” by just saturating both t and r.
For example, for the translation group we have straight-line interpolation, a(t) := ra+(1−r)a(t−1). For the rotation group SO(3), each rotation is represented by its axis ξ ∈ S2 and angle θ ∈ (−π, π],
and we just interpolate each separately ξ(t) := π(rξ+(1−r)ξ(t−1)) and θ(t) := rθ+(1−r)θ(t−1). where π is L2-normalization.7
We perform the same experiments, with the same grid as specified in the last section, and with the initial action interpolation gates biased toward the previous action. The results are given in table B.2. Figure B.1 shows action interpolation’s impact on performance. Most notably, interpolation seems to improve performance of most models in the 5-INTERLEAVED ADD task and of the spherical memory models in the 6-ODD FIRST task, but causes failure to learn in many situations, most significantly, the failure of LANTM to learn 6-ODD FIRST.
7There is, in fact, a canonical way to interpolate the most common Lie groups, including all of the groups mentioned above, based on the exponential map and the Baker-Campbell-Hausdorff formula (Lee, 2012), but the details are outside the scope of this paper and the computational cost, while acceptable in control theory settings, is too hefty for us. Interested readers are referred to Shingel (2009) and Marthinsen (1999).

The Neural Turing Machine and related “external memory models” have demonstrated an ability to learn algorithmic solutions by utilizing differentiable analogues of conventional memory structures. In particular, the NTM, DNC and other approaches provide mechanisms for shifting a memory access head to linked memories from the current read position.
The NTM, which is the most relevant to this work, uses a differentiable version of a Turing machine tape. The controller outputs a kernel which “softly” shifts the head, allowing the machine to read and write sequences. Since this soft shift typically “smears” the focus of the head, the controller also outputs a sharpening parameter which compensates by refocusing the distribution.
The premise of this work is to notice that while the NTM emulates a differentiable version of a Turing tape, there is no particular reason that one is constrained to follow the topology of a Turing tape. Instead they propose memory stored at a set of points on a manifold and shift actions which form a Lie group. In this way, memory points can have have different relationships to one another, rather than being constrained to Z.
This is mathematically elegant and here they empirically test models with the shift group R^2 acting on R^2 and the rotation group acting on a sphere.
Overall, the paper is well communicated and a novel idea.
The primary limitation of this paper is its limited impact. While this approach is certainly mathematically elegant, even likely beneficial for some specific problems where the problem structure matches the group structure, it is not clear that this significantly contributes to building models capable of more general program learning. Instead, it is likely to make an already complex and slow model such as the NTM even slower. In general, it would seem memory topology is problem specific and should therefore be learned rather than specified.
The baseline used for comparison is a very simple model, which does not even having the sharpening (the NTM approach to solving the problem of head distributions becoming ‘smeared’). There is also no comparison with the successor to the NTM, the DNC, which provides a more general approach to linking memories based on prior memory accesses.
Minor issues:
Footnote on page 3 is misleading regarding the DNC. While the linkage matrix explicitly excludes the identity, the controller can keep the head in the same position by gating the following of the link matrix.
Figures on page 8 are difficult to follow.


HYPERBAND: BANDIT-BASED CONFIGURATION EVAL-
Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian Optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation. We present HYPERBAND, a novel algorithm for hyperparameter optimization that is simple, flexible, and theoretically sound. HYPERBAND is a principled early-stoppping method that adaptively allocates a predefined resource, e.g., iterations, data samples or number of features, to randomly sampled configurations. We compare HYPERBAND with popular Bayesian Optimization methods on several hyperparameter optimization problems. We observe that HYPERBAND can provide more than an order of magnitude speedups over competitors on a variety of neural network and kernel-based learning problems.
1 INTRODUCTION
The task of hyperparameter optimization is becoming increasingly important as modern data analysis pipelines grow in complexity. The quality of a predictive model critically depends on its hyperparameter configuration, but it is poorly understood how these hyperparameters interact with each other to affect the quality of the resulting model. Consequently, practitioners often default to either hand-tuning or automated brute-force methods like random search and grid search.
In an effort to develop more efficient search methods, the problem of hyperparameter optimization has recently been dominated by Bayesian optimization methods (Snoek et al., 2012; Hutter et al., 2011; Bergstra et al., 2011) that focus on optimizing hyperparameter configuration selection. These methods aim to identify good configurations more quickly than standard baselines like random search by selecting configurations in an adaptive manner; see Figure 1(a). Existing empirical evidence suggests that these methods outperform random search (Thornton et al., 2013; Eggensperger et al., 2013; Snoek et al., 2015). However, these methods tackle a fundamentally challenging problem of simultaneously fitting and optimizing a high-dimensional, non-convex function with unknown smoothness, and possibly noisy evaluations. To overcome these difficulties, some Bayesian optimization methods resort to heuristics, at the expense of consistency guarantees, to model the objective function or speed up resource intensive subroutines.1 Moreover, these adaptive configuration selection methods are intrinsically sequential and thus difficult to parallelize.
An orthogonal approach to hyperparameter optimization focuses on speeding up configuration evaluation; see Figure 1(b). These methods are adaptive in computation, allocating more resources to promising hyperparameter configurations while quickly eliminating poor ones. Resources can take various forms, including size of training set, number of features, or number of iterations for iterative algorithms. By adaptively allocating these resources, these methods aim to examine orders of magnitude more hyperparameter configurations than methods that uniformly train all configurations to completion, thereby quickly identifying good hyperparameters. While there are methods that combine Bayesian optimization with adaptive resource allocation (Swersky et al., 2013; 2014; Domhan et al., 2015), we focus on speeding up random search as it offers a simple, parallelizable, and theoretically principled launching point and is shown to outperform grid search (Bergstra & Bengio, 2012).
1Consistency can be restored by allocating a fraction of resources to performing random search.
Our novel configuration evaluation method, HYPERBAND, relies on a principled early-stopping strategy to allocate resources, allowing it to evaluate orders of magnitude more configurations than uniform allocation strategies. HYPERBAND is a general-purpose technique that makes minimal assumptions, unlike prior configuration evaluation approaches (Swersky et al., 2013; Domhan et al., 2015; Swersky et al., 2014; György & Kocsis, 2011; Agarwal et al., 2011). In this work, we describe HYPERBAND, provide intuition for the algorithm through a detailed example, and present a wide range of empirical results comparing HYPERBAND with well established competitors. We also briefly describe the theoretical underpinnings of HYPERBAND, however a thorough theoretical treatment is beyond the scope of this paper and is deferred to Li et al. (2016).
2 RELATED WORK
Bayesian optimization techniques model the conditional probability p(f |λ) of a configuration’s performance on a metric f given a set of hyperparameters λ. For instance, SMAC uses random forests to model p(f |λ) as a Gaussian distribution (Hutter et al., 2011). TPE is a non-standard Bayesian optimization algorithm based on tree-structured Parzen density estimators (Bergstra et al., 2011). A third popular method, Spearmint, uses Gaussian processes (GP) to model p(f |λ) and performs slice sampling over the GP’s hyperparameters (Snoek et al., 2012).
Adaptive configuration evaluation is not a new idea. Maron & Moore (1993) considered a setting where training time is negligible (e.g., k-nearest-neighbor classification) and evaluation on a large validation set is accelerated by evaluating on an increasing subset of the validation set, stopping early configurations that are performing poorly. Since subsets of the validation set provide unbiased estimates of its expected performance, this is an instance of the stochastic best-arm identification problem for multi-armed bandits (see Jamieson & Nowak (2014) for a brief survey).
In contrast, this paper assumes that evaluation time is negligible and the goal is to early-stop longrunning training procedures by evaluating partially trained models on the validation set. Previous approaches either require strong assumptions or use heuristics to perform adaptive resource allocation. Several works propose methods that make strong assumptions on the convergence behavior of training algorithms, providing theoretical performance guarantees under these assumptions (György & Kocsis, 2011; Agarwal et al., 2011; Swersky et al., 2013; 2014; Domhan et al., 2015; Sabharwal et al., 2016). Unfortunately, these assumptions are often hard to verify, and empirical performance can drastically suffer when they are violated. One recent work of particular interest proposes a heuristic based on sequential analysis to determine stopping times for training configurations on increasing subsets of the data (Krueger et al., 2015). However, it has a few shortcomings: (1) it is designed to speedup multi-fold cross-validation and is not significantly faster than standard holdout, (2) it is not an anytime algorithm and requires the set of configurations to be evaluated as an input, and (3) the theoretical correctness and empirical performance of this method are highly dependent on
a user-defined “safety-zone.”2 Lastly, in an effort avoid heuristics and strong assumptions, Sparks et al. (2015) proposed a halving style algorithm that did not require explicit convergence behavior, and Jamieson & Talwalkar (2015) analyzed a similar algorithm, providing theoretical guarantees and encouraging empirical results. Unfortunately, these halving style algorithms suffer from the n vs B/n issue which we will discuss in Section 3.
Finally, Klein et al. (2016) recently introduced Fabolas, a Bayesian optimization method that combines adaptive selection and evaluation. Similar to Swersky et al. (2013; 2014), it models the conditional validation error as a Gaussian process using a kernel that captures the covariance with downsampling rate to allow for adaptive evaluation. While we intended to compare HYPERBAND with Fabolas, we encountered some technical difficulties when using the package3 and are working with the authors of Klein et al. (2016) to resolve the issues.
3 HYPERBAND ALGORITHM
HYPERBAND extends the SUCCESSIVEHALVING algorithm proposed for hyperparameter optimization in Jamieson & Talwalkar (2015) and calls it as a subroutine. The idea behind SUCCESSIVEHALVING follows directly from its name: uniformly allocate a budget to a set of hyperparameter configurations, evaluate the performance of all configurations, throw out the worst half, and repeat until one configurations remains. The algorithm allocates exponentially more resources to more promising configurations. Unfortunately, SUCCESSIVEHALVING requires the number of configurations n as an input to the algorithm. Given some finite time budget B (e.g. an hour of training time to choose a hyperparameter configuration), B/n resources are allocated on average across the configurations. However, for a fixed B, it is not clear a priori whether we should (a) consider many configurations (large n) with a small average training time; or (b) consider a small number of configurations (small n) with longer average training times.
We use a simple example to better understand this tradeoff. Figure 1(c) shows the validation loss as a function of total resources allocated for two configurations with terminal validation losses ν1 and ν2. The shaded areas bound the maximum deviation from the terminal validation loss and will be referred to as “envelope” functions. It is possible to differentiate between the two configurations when the envelopes diverge. Simple arithmetic shows that this happens when the width of the envelopes is less than ν2 − ν1, i.e. when the intermediate losses are guaranteed to be less than ν2−ν12 away from the terminal losses. There are two takeaways from this observation: more resources are needed to differentiate between the two configurations when either (1) the envelope functions are wider or (2) the terminal losses are closer together.
However, in practice, the optimal allocation strategy is unknown because we do not have knowledge of the envelope functions nor the distribution of terminal losses. Hence, if more resources are required before configurations can differentiate themselves in terms of quality (e.g., if an iterative training method converges very slowly for a given dataset or if randomly selected hyperparameter configurations perform similarly well) then it would be reasonable to work with a small number of configurations. In contrast, if the quality of a configuration is typically revealed using minimal resources (e.g., if iterative training methods converge very quickly for a given dataset or if randomly selected hyperparameter configurations are of low-quality with high probability) then n is the bottleneck and we should choose n to be large.
3.1 HYPERBAND
HYPERBAND, shown in Algorithm 1, addresses this “n versus B/n” problem by considering several possible values of n for a fixed B, in essence performing a grid search over feasible value of n. Associated with each value of n is a minimum resource r that is allocated to all configurations before some are discarded; a larger value of n corresponds to a smaller r and hence more aggressive early stopping. There are two components to HYPERBAND; (1) the inner loop invokes SUCCESSIVEHALVING for fixed values of n and r (lines 3-9) and (2) the outer loop which iterates over different values
2The first two drawbacks prevent a full comparison to HYPERBAND on our selected empirical tasks, however, for completeness, we provide a comparison in Appendix A to Krueger et al. (2015) on some experimental tasks replicated from their paper.
3The package provided by Klein et al. (2016) is available at https://github.com/automl/RoBO.
of n and r (lines 1-2). We will refer to each such run of SUCCESSIVEHALVING within HYPERBAND as a “bracket.” Each bracket is designed to use about B total resources and corresponds to a different tradeoff between n and B/n. A single execution of HYPERBAND takes a finite number of iterations, and in practice can be repeated indefinitely.
HYPERBAND requires two inputs (1) R, the maximum amount of resource that can be allocated to a single configuration, and (2) η, an input that controls the proportion of configurations discarded in each round of SUCCESSIVEHALVING. The two inputs dictate how many different brackets are considered; specifically, smax + 1 different values for n are considered with smax = blogη(R)c. HYPERBAND begins with the most aggressive bracket s = smax, which sets n to maximize exploration, subject to the constraint that at least one configuration is allocated R resources. Each subsequent bracket reduces n by a factor of approximately η until the final bracket, s = 0, in which every configuration is allocated R resources (this bracket simply performs classical random search). Hence, HYPERBAND performs a geometric search in the average budget per configuration to address the “n versus B/n” problem, at the cost of approximately smax+1 times more work than running SUCCESSIVEHALVING for a fixed n. By doing so, HYPERBAND is able to exploit situations in which adaptive allocation works well, while protecting itself in situations where more conservative allocations are required.
Algorithm 1: HYPERBAND algorithm for hyperparameter optimization. input :R, η (default η = 3) initialization :smax = blogη(R)c, B = (smax + 1)R 1 for s ∈ {smax, smax − 1, . . . , 0} do 2 n = dBR ηs (s+1)e, r = Rη −s // begin SUCCESSIVEHALVING with (n, r) inner loop 3 T =get hyperparameter configuration(n) 4 for i ∈ {0, . . . , s} do 5 ni = bnη−ic 6 ri = rη i 7 L = {run then return val loss(t, ri) : t ∈ T} 8 T =top k(T, L, bni/ηc) 9 end
10 end 11 return Configuration with the smallest intermediate loss seen so far.
R represents the maximum amount of resources that can be allocated to any given configuration. In most cases, there is a natural upper bound on the maximum budget per configuration that is often dictated by the resource type (e.g., training set size for dataset downsampling; limitations based on memory constraint for feature downsampling; rule of thumb regarding number of epochs when iteratively training neural networks). R is also the number of configurations evaluated in the bracket that performs the most exploration, i.e s = smax. In practice one may want n ≤ nmax to limit overhead associated with training many configurations on a small budget, i.e. costs associated with initialization, loading a model, and validation. In this case, set smax = blogη(nmax)c. The value of η can be viewed as a knob that can be tuned based on practical user constraints. Larger values of η correspond to a more aggressive elimination schedule and thus fewer rounds of elimination; specifically, each round retains 1/η configurations for a total of blogη(n)c+ 1 rounds of elimination with n configurations. If one wishes to receive a result faster at the cost of a sub-optimal asymptotic constant, one can increase η to reduce the budget per bracket B = (blogη(R)c + 1)R. We stress that results are not very sensitive to the choice of η. In practice we suggest taking η to be equal to 3 or 4.
HYPERBAND requires the following methods to be defined for any given learning problem: get hyperparameter configuration(n) returns a set of n i.i.d. samples from some distribution defined over the hyperparameter configuration space; run then return val loss(t, r) takes a hyperparameter configuration (t) and resource allocation (r) and returns the validation loss after training for the allocated resources; and top k(configs, losses, k) takes a set of configurations as well as their associated losses and returns the top k performing configurations.
3.2 EXAMPLE APPLICATION WITH ITERATIONS AS A RESOURCE: LENET
We next present a simple example to provide intuition. We work with the MNIST dataset and optimize hyperparameters for the LeNet convolutional neural network trained using mini-batch SGD. Our search space includes learning rate, batch size, and number of kernels for the two layers of the network as hyperparameters (details are shown in Table 3 in Appendix A).
We further define the number of iterations as the resource to allocate, with one unit of resource corresponding to one epoch or a full pass over the dataset. We set R to 81 and use the default value of η = 3, resulting in smax = 4 and thus 5 brackets of SUCCESSIVEHALVING with different tradeoffs between n and B/n. The resources allocated within each bracket are displayed in Table 1.
3.3 OVERVIEW OF THEORETICAL RESULTS
Although a detailed theoretical analysis is beyond the scope of this paper, we provide an intuitive, high-level description of theoretical properties of HYPERBAND. Suppose there are n configurations, each with a given terminal validation error νi for i = 1, . . . , n. Without loss of generality, index the configurations by performance so that ν1 corresponds to the best performing configuration, ν2 to the second best, and so on. Now consider the task of identifying the best configuration. The optimal strategy would allocate to each configuration i the minimum resource required to distinguish it from ν1, i.e., enough so that the envelope functions depicted in Figure 1(c) bound the intermediate loss to be less than νi−ν12 away from the terminal value. As shown in Jamieson & Talwalkar (2015) and Li et al. (2016), the budget required by SUCCESSIVEHALVING is in fact only a small factor away from this optimal approach because it capitalizes on configurations that are easy to distinguish from ν1. In contrast, the naive uniform allocation strategy, which allocates B/n to each configuration, has to allocate to every configuration the resource required to distinguish ν2 from ν1.
The relative size of the budget required for uniform allocation and SUCCESSIVEHALVING depends on the envelope functions bounding deviation from terminal losses as well as the distribution from which νi’s are drawn. The budget required for SUCCESSIVEHALVING is smaller when the optimal n versus B/n tradeoff requires fewer resources per configuration. Hence, if the envelope functions tighten quickly as a function of resource allocated, or the average distances between terminal losses is large, then SUCCESSIVEHALVING can be substantially faster than uniform allocation. Of course we do not have knowledge of either function in practice, so we will hedge our aggressiveness with HYPERBAND. Remarkably, despite having no knowledge of the envelope functions or the distribution of νi’s, HYPERBAND requires a budget that is only log factors larger than the optimal for SUCCESSIVEHALVING. See Li et al. (2016) for details.
4 HYPERPARAMETER OPTIMIZATION EXPERIMENTS
In this section, we evaluate the empirical behavior of HYPERBAND with iterations, data subsamples, and features as resources. For all experiments, we compare HYPERBAND with three well known Bayesian optimization algorithms — SMAC, TPE, and Spearmint. Additionally, we show results for SUCCESSIVEHALVING corresponding to repeating the most exploration bracket of HYPERBAND. Finally for all experiments, we benchmark against standard random search and random 2×, which is a variant of random search with twice the budget of other methods.
4.1 EARLY STOPPING ITERATIVE ALGORITHMS FOR NEURAL NETWORKS
We study a convolutional neural network with the same architecture as that used in Snoek et al. (2012) and Domhan et al. (2015) from cuda-convnet. The search spaces used in the two previous works differ, and we used a search space similar to that of Snoek et al. (2012) with 6 hyperparameters for stochastic gradient decent and 2 hyperparameters for the response normalization layers. In line with the two previous works, we used a batch size of 100 for all experiments. For these experiments, we also compare against a variant of SMAC named SMAC early that uses the early termination criterion proposed in Domhan et al. (2015) for neural networks. We view SMAC with early stopping to be a combination of adaptive configuration selection and configuration evaluation. See Appendix A for more details about the experimental setup.
Datasets: We considered three image classification datasets: CIFAR-10 (Krizhevsky, 2009), rotated MNIST with background images (MRBI) (Larochelle et al., 2007), and Street View House Numbers (SVHN) (Netzer et al., 2011). CIFAR-10 and SVHN contain 32 × 32 RGB images while MRBI contains 28× 28 grayscale images. The splits used for each dataset are as follows: (1) CIFAR-10 has 40k, 10k, and 10k instances; (2) MRBI has 10k, 2k, and 50k instances; and (3) SVHN has close to 600k, 6k, and 26k instances for training, validation, and test respectively. For all datasets, the only preprocessing performed on the raw images was demeaning.
HYPERBAND Configuration: For these experiments, one unit of resource corresponds to 100 minibatch iterations. For CIFAR-10 and MRBI, R is set to 300 (or 30k total iterations). For SVHN, R is set to 600 (or 60k total iterations) to accommodate the larger training set. η was set to 4 for all experiments, resulting in 5 SUCCESSIVEHALVING brackets for HYPERBAND.
Results: Ten independent trials were performed for each searcher. For CIFAR-10, the results in Figure 3(a) show that HYPERBAND is more than an order of magnitude faster than its competitors. In Figure 6 of Appendix A, we extend the x-axis for CIFAR-10 out to 100R. The results show that Bayesian optimization methods ultimately converge to similar errors as HYPERBAND. For MRBI, HYPERBAND is more than an order of magnitude faster than standard configuration selection approaches and 5× faster than SMAC with early stopping. For SVHN, while HYPERBAND finds a good configuration faster, Bayesian optimization methods are competitive and SMAC with early stopping outperforms HYPERBAND. This result demonstrates that there is merit to incorporating early stopping with configuration selection approaches.
Across the three datasets, HYPERBAND and SMAC early are the only two methods that consistently outperform random 2×. On these datasets, HYPERBAND is over 20× faster than random search while SMAC early is ≤ 7× faster than random search within the evaluation window. In fact, the first result returned by HYPERBAND after using a budget of 5R is often competitive with results returned by other searchers after using 50R. Additionally, HYPERBAND is less variable than other searchers across trials, which is highly desirable in practice (see Appendix A for plots with error bars).
For computationally expensive problems in high dimensional search spaces, it may make sense to just repeat the most exploratory brackets. Similarly, if meta-data is available about a problem or it is known that the quality of a configuration is evident after allocating a small amount of resource, then one should just repeat the most exploration bracket. Indeed, for these experiments, repeating the most exploratory bracket of HYPERBAND outperforms cycling through all the brackets. In fact, bracket s = 4 vastly outperforms all other methods on CIFAR-10 and MRBI and is nearly tied with SMAC early for first on SVHN.
Finally, CIFAR-10 is a very popular dataset and state-of-the-art models achieve much better accuracy than what is shown in Figure 3. The difference in performance is mainly attributable to higher model complexities and data manipulation (i.e. using reflection or random cropping to artificially increase the dataset size). If we limit the comparison to published results that use the same architecture and exclude data manipulation, the best human expert result for the dataset is 18% error and hyperparameter optimized result is 15.0% for Snoek et al. (2012)4 and 17.2% for Domhan et al. (2015). These results are better than our results on CIFAR-10 because they use 25% more data by including the validation set and also train for more epochs. The best model found by HYPERBAND achieved a test error of 17.0% when trained on the combined training and validation data for 300 epochs.
4.2 DATA DOWNSAMPLING KERNEL REGULARIZED LEAST SQUARES CLASSIFICATION
In this experiment, we use HYPERBAND with data samples as the resource to optimize the hyperparameters of a kernel-based classification task on CIFAR-10. We use the multi-class regularized least squares classification model which is known to have comparable performance to SVMs (Rifkin & Klautau, 2004; Agarwal et al., 2014) but can be trained significantly faster. The hyperparameters considered in the search space include preprocessing method, regularization, kernel type, kernel length scale, and other kernel specific hyperparameters (see Appendix A for more details). HYPERBAND is run with η = 4 and R = 400, with each unit of resource representing 100 datapoints. Similar to previous experiments, these inputs result in a total of 5 brackets. Each hyperparameter optimization algorithm is run for ten trials on Amazon EC2 m4.2xlarge instances; for a given trial, HYPERBAND is allowed to run for two outer loops, bracket s = 4 is repeated 10 times, and all other searchers are run for 12 hours.
Figure 4 shows that HYPERBAND returns a good configuration after just the first SUCCESSIVEHALVING bracket in approximately 20 minutes; other searchers fail to reach this error rate on average even after the entire 12 hours. Notably, HYPERBAND was able to evaluate over 250 configurations in this first bracket of SUCCESSIVEHALVING, while competitors were able to evaluate only three configurations in the same amount of time. Consequently, HYPERBAND is over 30× faster than Bayesian optimization methods and 70× faster than random search. Bracket s = 4 sightly outperforms HYPERBAND but the terminal performance for the two algorithms are the same. Random 2× is competitive with SMAC and TPE.
4.3 FEATURE SUBSAMPLING TO SPEED UP APPROXIMATE KERNEL CLASSIFICATION
We next demonstrate the performance of HYPERBAND when using features as a resource, focusing on random feature approximations for kernel methods. Features are randomly generated using the method described in Rahimi & Recht (2007) to approximate the RBF kernel, and these random features are then used as inputs to a ridge regression classifier. We consider hyperparameters of a random feature kernel approximation classifier trained on CIFAR-10, including preprocessing method, kernel length scale, and l2 penalty. We impose an upper bound of 100k random features for the kernel approximation so that the data will comfortably fit into a machine with 60GB of
4We were unable to reproduce this result even after receiving the optimal hyperparameters from the authors through a personal communication.
memory. Additionally, we set one unit of resource to be 100 features for an R = 1000, which gives 5 different brackets with η = 4. Each searcher is run for 10 trials, with each trial lasting 12 hours on a n1-standard-16 machine from Google Cloud Compute. The results in Figure 5 show that HYPERBAND is around 6x faster than Bayesian methods and random search. HYPERBAND performs similarly to bracket s = 4. Random 2× outperforms Bayesian optimization algorithms.
4.4 EXPERIMENTAL DISCUSSION
For a given R, the most exploratory SUCCESSIVEHALVING round performed by HYPERBAND evaluates ηblogη(R)c configurations using a budget of (blogη(R)c+1)R, which gives an upper bound on the potential speedup over random search. If training time scales linearly with the resource, the maximum speedup offered by HYPERBAND compared to random search is η blogη(R)c
(blogη(R)c+1) . For
the values of η and R used in our experiments, the maximum speedup over random search is approximately 50× given linear training time. However, we observe a range of speedups from 6× to 70× faster than random search. The differences in realized speedup can be explained by two factors: (1) the scaling properties of total evaluation time as a function of the allocated resource and (2) the difficulty of finding a good configuration.
If training time is superlinear as a function of the resource, then HYPERBAND can offer higher speedups. More generally, if training scales like a polynomial of degree p > 1, the maximum speedup of HYPERBAND over random search is approximately η
p−1−1 ηp−1 η blogη(R)c. Hence, higher speedups were observed for the the kernel least square classifier experiment discussed in Section 4.2 because the training time scaled quadratically as a function of the resource.
If 10 randomly sampled configurations is sufficient to find a good hyperparameter setting, then the benefit of evaluating orders of magnitude more configurations is muted. Generally the difficulty of the problem scales with the dimension of the search space since coverage diminishes with dimensionality. For low dimensional problems, the number of configurations evaluated by random search and Bayesian methods is exponential in the number of dimensions so good coverage can be achieved; i.e. if d = 3 as in the features subsampling experiment, then n = O(2d = 8). Hence, HYPERBAND is only 6× faster than random search on the feature subsampling experiment. For the neural network experiments however, we hypothesize that faster speedups are observed for HYPERBAND because the dimension of the search space is higher.
5 FUTURE WORK
We have introduced a novel bandit-based method for adaptive configuration evaluation with demonstrated competitive empirical performance. Future work involves exploring (i) embedding HYPERBAND into parallel and distributed computing environments; (ii) adjusting for training methods with different convergence rates; and (iii) combining HYPERBAND with non-random sampling methods.
A ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present a comparison of HYPERBAND with the CVST algorithm from Krueger et al. (2015) and provide additional details for experiments presented in Section 3 and 4.
A.1 COMPARISON WITH CVST
The CVST algorithm from Krueger et al. (2015) focuses on speeding up standard k-fold crossvalidation. We did not include it as one of the competitors in Section 4 because the experiments we selected were too computational expensive for multi-fold cross-validation and CVST is not an any time algorithm. Nonetheless, the CVST algorithm is an interesting approach and was shown to have promising empirical performance in Krueger et al. (2015). Hence, we performed a small scale comparison modeled after their empirical studies between CVST and HYPERBAND.
We replicated the classification experiments in Krueger et al. (2015) that train a support vector machine on the datasets from the IDA benchmark (Rätsch et al., 2001). All experiments were performed on Google Cloud Compute’s n1-standard-1 instances. Following Krueger et al. (2015), we evaluated HYPERBAND and CVST on the same 2d grid of 610 hyperparameters and recorded the best test error and duration for 50 trials . The only modification we made to their original experimental setup was the data splits; instead of half for test and half for training, we used 1/11th for test and 10/11th for training. HYPERBAND performed holdout evaluation using 1/10th of the training data as the validation set. We set η = 3, and R was set for each dataset so that a minimum resource of 50 datapoints is allocated to each configuration. Table 2 shows that CVST and HYPERBAND achieve comparable test errors (the differences are well within the error bars), while HYPERBAND is significantly faster than CVST on all datasets. More granularly, while CVST on average has slightly lower mean error, HYPERBAND is within 0.2% of CVST on 5 of the 7 datasets. Additionally, for each of the 7 datasets, HYPERBAND does as well as or better than CVST in over half of the trials.
A.2 LENET EXPERIMENT
We trained the LeNet convolutional neural network on MNIST using mini-batch SGD. Code is available for the network at http://deeplearning.net/tutorial/lenet.html. The search space for the LeNet example discussed in Section 3.2 is shown in Table 3.
A.3 EXPERIMENTS USING ALEX KRIZHEVSKY’S CNN ARCHITECTURE
For the experiments discussed in Section 4.1, the exact architecture used is the 18% model provided on cuda-convnet for CIFAR-10.5
Search Space: The search space used for the experiments is shown in Table 4. The learning rate reductions hyperparameter indicates how many times the learning rate was reduced by a factor of 10 over the maximum iteration window. For example, on CIFAR-10, which has a maximum iteration of 30,000, a learning rate reduction of 2 corresponds to reducing the learning every 10,000 iterations, for a total of 2 reductions over the 30,000 iteration window. All hyperparameters with the exception of the learning rate decay reduction overlap with those in Snoek et al. (2012). Two hyperparameters in Snoek et al. (2012) were excluded from our experiments: (1) the width of the response normalization layer was excluded due to limitations of the Caffe framework and (2) the number of epochs was excluded because it is incompatible with dynamic resource allocation.
Datasets: CIFAR-10 and SVHN contain 32×32 RGB images while MRBI contains 28×28 grayscale images. For all datasets, the only preprocessing performed on the raw images was demeaning. For CIFAR-10, the training (40,000 instances) and validation (10,000 instances) sets were sampled from data batches 1-5 with balanced classes. The original test set (10,000 instances) is used for testing. For MRBI, the training (10,000 instances) and validation (2,000 instances) sets were sampled from the original training set with balanced classes. The original test set (50,000 instances) is used for testing. Lastly, for SVHN, the train, validation, and test splits were created using the same procedure as that in Sermanet et al. (2012).
Computational Considerations: The experiments took the equivalent of over 1 year of GPU hours on NVIDIA GRID K520 cards available on Amazon EC2 g2.8xlarge instances. We set a total budget constraint in terms of iterations instead of compute time to make comparisons hardware independent.6 Comparing progress by iterations instead of time ignores overhead costs not associated with training like cost of configuration selection for Bayesian methods and model initialization and validation costs for HYPERBAND. While overhead is hardware dependent, the overhead for HYPERBAND is below 5% on EC2 g2.8xlarge machines, so comparing progress by time passed would not impact results significantly.
Due to the high computational cost of these experiments, we were not able to run all searchers out to convergence. However, we did double the budget for each trial of CIFAR-10 to allow for a comparison of the searchers as they near convergence. Figure 6 shows while Bayesian optimization methods achieve similar performance as HYPERBAND and SUCCESSIVEHALVING, it takes them much longer to achieve a comparable error rate.
Comparison with Early Stopping: Adaptive allocation for hyperparameter optimization can be thought of as a form of early stopping where less promising configurations are halted before completion. Domhan et al. (2015) propose an early stopping method for neural networks and combine it
5The model specification is available at http://code.google.com/p/cuda-convnet/. 6Most trials were run on Amazon EC2 g2.8xlarge instances but a few trials were run on different machines
due to the large computational demand of these experiments.
with SMAC to speed up hyperparameter optimization. Their method stops training a configuration if the probability of the configuration beating the current best is below a specified threshold. This probability is estimated by extrapolating learning curves fit to the intermediate validation error losses of a configuration. If a configuration is terminated early, the predicted terminal value from the estimated learning curves is used as the validation error passed to the hyperparameter optimization algorithm. Hence, if the learning curve fit is poor, it could impact the performance of the configuration selection algorithm. While this approach is heuristic in nature, it does demonstrate promising empirical performance so we included SMAC with early termination as a competitor. We used the conservative termination criterion with default parameters and recorded the validation loss every 400 iterations and evaluated the termination criterion 3 times within the training period (every 8k iterations for CIFAR-10 and MRBI and every 16k iterations for SVHN).7 Comparing performance by the total multiple of R used is conservative because it does not account for the time spent fitting the learning curve in order to check the termination criterion.
A.4 KERNEL CLASSIFICATION EXPERIMENTS
We trained the regularized least-squares classification model using a block coordinate descent solver. Our models take less than 10 minutes to train on CIFAR-10 using an 8 core machine, while the default SVM method in Scikit-learn is single core and takes hours. Table 5 shows the hyperparameters and associated ranges considered in the kernel least squares classification experiment discussed in
7We used the code provided at https://github.com/automl/pylearningcurvepredictor.
Section 4.2. The cost term C is divided by the number of samples so that the tradeoff between the squared error and the l2 penalty would remain constant as the resource increased (squared error is summed across observations and not averaged). The regularization term λ is equal to the inverse of the scaled cost term C. Additionally, the average test error with associated minimum and maximum ranges across 10 trials are show in Figure 7.
Table 6 shows the hyperparameters and associated ranges considered in the random features kernel approximation classification experiment discussed in Section 4.3. The regularization term λ is divided by the number of features so that the tradeoff between the squared error and the l2 penalty would remain constant as the resource increased. Additionally, the average test error with associated minimum and maximum ranges across 10 trials are show in Figure 8.

This paper discusses Hyperband, an extension of successive halving by Jamieson & Talwalkar (AISTATS 2016). Successive halving is a very nice algorithm that starts evaluating many configurations and repeatedly cuts off the current worst half to explore many configuration for a limited budget.
Having read the paper for the question period and just rereading it again, I am now not entirely sure what its contribution is meant to be: the only improvement of Hyperband vs. successive halving is in the theoretical worst case bounds (not more than 5x worse than random search), but you can (a) trivially obtain that bound by using a fifth of your time for running random configurations to completion and (b) the theoretical analysis to show this is said to be beyond the scope of the paper. That makes me wonder whether the theoretical results are the contribution of this paper, or whether they are the subject of a different paper and the current paper is mostly an empirical study of the method?
I hope to get a response by the authors and see this made clearer in an updated version of the paper.
In terms of experiments, the paper fails to show a case where Hyperband actually performs better than the authors' previous algorithm successive halving with its most agressive setting of bracket b=4. Literally, in every figure, bracket b=4 is at least as good (and sometimes substantially better) than Hyperband. That makes me think that in practice I would prefer successive halving with b=4 over Hyperband. (And if I really want Hyperband's guarantee of not being more than 5x worse than random search I can run random search on a fifth of my machines.)
The experiments also compare to some Bayesian optimization methods, but not to the most relevant very closely related Multi-Task Bayesian Optimization methods that have been dominating effective methods for deep learning in that area in the last 3 years: "Multi-Task Bayesian Optimization" by Swersky, Snoek, and Adams (2013) already showed 5x speedups for deep learning by starting with smaller datasets, and there have been several follow-up papers showing even larger speedups.
Given that this prominent work on multitask Bayesian optimization exists, I also think the introduction, which sells Hyperband as a very new approach to hyperparameter optimization is misleading. I would've much preferred a more down-to-earth pitch that says "configuration evaluation" has been becoming a very important feature in hyperparameter optimization, including Bayesian optimization, that sometimes yields very large speedups (this can be quantified by examples from existing papers) and this paper adds some much-needed theoretical understanding to this and demonstrates how important configuration evaluation is even in the simplest case of being used with random search. I think this could be done easily and locally by adding a paragraph to the intro.
As another point regarding novelty, I think the authors should make clear that approaches for adaptively deciding how many resources to use for which evaluation have been studied for (at least) 23 years in the ML community -- see Maron & Moore, NIPS 1993: "Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation" (


GENERATIVE ADVERSARIAL NETWORKS FOR IMAGE STEGANOGRAPHY
Steganography is collection of methods to hide secret information (“payload”) within non-secret information (“container”). Its counterpart, Steganalysis, is the practice of determining if a message contains a hidden payload, and recovering it if possible. Presence of hidden payloads is typically detected by a binary classifier. In the present study, we propose a new model for generating image-like containers based on Deep Convolutional Generative Adversarial Networks (DCGAN). This approach allows to generate more setganalysis-secure message embedding using standard steganography algorithms. Experiment results demonstrate that the new model successfully deceives the steganography analyzer, and for this reason, can be used in steganographic applications.
1 INTRODUCTION
Recently developed Generative Adversarial Networks (GAN, see Goodfellow et al. (2014)) are powerful generative models, the main idea of which is to train a generator and a discriminator network through playing a minimax game. In the image domain, for a dataset generated by some density pdata(x) a generator G attempts to approximate the image generating distribution and to synthesize as realistic image as possible, while a discriminator D strives to distinguish real images from fake ones.
There are several modifications of GAN that can generate realistic images:
• Deep Convolutional Generative Adversarial Networks (DCGAN, see Radford et al. (2015)) — this model is a modification of a GAN, specialized for generation of images; • Conditional GAN — it allows generating objects from a specified class, see Mirza &
Osindero (2014); • Generation of images from textual description, see Reed et al. (2016).
In the present study we apply the DCGAN model to the problem of secure steganography. We construct a special container-image generator, synthetic output of which is less susceptible to successful steganalysis compared to containers, directly derived from original images. In particular, we investigate whether this methodology allows to deceive a given steganography analyzer, represented by a binary classifier detecting presence of hidden messages in an image.
2 STEGANOGRAPHY
Steganography is the practice of concealing a secret message, e.g. a document, an image, or a video, within another non-secret message in the most inconspicuous manner possible. In this paper we consider a text-to-image embedding, with the text given by bit string. More formally, for a message T and an image I , a steganography algorithm is a map S : T × I → Î , where Î is an image, containing the message T , such that Î can not be visually distinguished from I .
The most popular and easy-to-implement algorithm of embedding is the Least Significant Bit (LSB) algorithm. The main idea of LSB is to store the secret message in the least significant bits (last bits) of some color channel of each pixel in the given image container. Since pixels are adjusted independently of each other, the LSB algorithm alters the distribution of the least significant bits, thereby simplifying detection of the payload. A modification of this method, which does not substantially alter the distribution of the least significant bits, is a so-called ±1-embedding (Ker, 2005). This approach randomly adds or subtracts 1 from some color channel pixel so that the last bits would match the ones needed. In this paper we basically consider the ±1-embedding algorithm. There are more sophisticated algorithms for information embedding to raster images: WOW (Holub & Fridrich, 2012), HUGO (Pevny et al., 2010), S-UNIWARD (Holub et al., 2014), and others. They are derived from key ideas of the LSB algorithm, but utilize a more strategic pixel manipulation technique: for the raw image X and its final version with a secret message X̂ the pixels are picked in such a way as to minimize the distortion function
D(X, X̂) = n1∑ i=1 n2∑ j=1 ρ(Xij , X̂ij)|Xij − X̂ij | ,
where ρ(Xij , X̂ij) is the cost of changing pixel of X , specific for each particular steganography algorithm.
For detecting presence of hidden information in the container Steganalysis is usually used. The stage which distinguishes images with some hidden message from empty is usually performed by binary classification. The basic approach to steganalysis is based on feature extractors (such as SPAM (Pevnỳ et al., 2010), SRM (Fridrich & Kodovskỳ, 2012), etc.) combined with traditional machine learning classifiers, such as SVM, decision trees, ensembles etc. With the recent overwhelming success of deep neural networks, newer neural network based approaches to steganalysis are gaining popularity, Qian et al. (2015b). For example, in Pibre et al. (2015) authors propose to use deep convolution neural networks (CNN) for steganalysis and show that classification accuracy can be significantly increased while using CNN instead of usual classifiers.
3 ADVERSARIAL NETWORKS
Generative Adversarial Networks (GAN) is a recent approach to deep unsupervised learning, proposed in 2014 in Goodfellow et al. (2014), which is capable of dynamically representing a sampler from input data distribution and generate new data samples.
The main idea of such approach to learning is that two neural networks are trained simultaneously:
• a generative model (G) that receives noise from the prior distribution pnoise(z) on input and transforms it into a data sample from the distribution pg(x) that approximates pdata(x); • a discriminative model (D) which tries to detect if an object is real or generated by G.
The learning process can be described as a minimax game: the discriminator D maximizes the expected log-likelihood of correctly distinguishing real samples from fake ones, while the generator G maximizes the expected error of the discriminator by trying to synthesize better images. Therefore during the training GAN solve the following optimization problem:
L(D,G) = Ex∼pdata(x) [ logD(x) ] + Ez∼pnoise(z) [ log(1−D(G(z))) ]→ min G max D , (1)
where D(x) represents the probability that x is a real image rather then synthetic, and G(z) is a synthetic image for input noise z.
Coupled optimization problem (1) is solved by alternating the maximization and minimization steps: on each iteration of the mini-batch stochastic gradient optimization we first make a gradient ascent step on D and then a gradient descent step on G. If by θM we denote the parameters of the neural network M , then the update rules are:
• Keeping the G fixed, update the model D by θD ← θD + γD∇DL with
∇DL = ∂
∂θD
{ Ex∼pdata(x) [ logD(x, θD) ]+Ez∼pnoise(z) [ log(1−D(G(z, θG), θD)) ] } ,
(2)
• Keeping D fixed, update G by θG ← θG − γG∇GL where
∇GL = ∂
∂θG Ez∼pnoise(z) [ log(1−D(G(z, θG), θD)) ] . (3)
In Radford et al. (2015) the GAN idea was extended to deep convolutional networks (DCGAN), which are specialized for image generation. The paper discusses the advantages of adversarial training in image recognition and generation, and give recommendations on constructing and training DCGANs. In fig. 1 we depict a sample of synthetic images of a freshly trained DCGAN on the Celebrities dataset (Ziwei Liu & Tang, 2015). The images indeed look realistic, albeit with occasional artifacts.
4 STEGANOGRAPHIC GENERATIVE ADVERSARIAL NETWORKS
In order to apply GAN methodology to steganographic applications, we introduce Steganographic Generative Adversarial Networks model (SGAN), which consists of
• a generator network G, which produces realistic looking images from noise; • a discriminator network D, which classifies whether an image is synthetic or real; • a discriminator network S, the steganalyser, which determines if an image contains a
concealed secret message.
By Stego(x) we denote the result of embedding some hidden message in the container x.
Since we want the generator to produce realistic images that could serve as containers for secure message embedding, we force G to compete against the models D and S simultaneously. If we denote by S(x) the probability that x has some hidden information, then we arrive at the following game:
L = α ( Ex∼pdata(x) [ logD(x) ] + Ez∼pnoise(z) [ log(1−D(G(z))) ] ) +
+(1− α)Ez∼pnoise(z) [ logS(Stego(G(z))) + log(1− S(G(z))) ]→ min G max D max S
. (4)
We use a convex combination of errors of D and S with parameter α ∈ [0, 1], which controls the trade-off between the importance of realism of generated images and their quality as containers
against the steganalysis. Analysis of preliminary experimental results showed that for α ≤ 0.7 the generated images are unrealistic and resemble noise.
The full scheme of SGAN is presented in fig. 2. Each arrows represent output- input data flows.
Stochastic mini-batch Gradient descent update rules for components of SGAN are listed below:
• for D the rule is θD ← θD + γD∇GL with
∇GL = ∂
∂θD
{ Ex∼pdata(x) [ logD(x, θD) ]+Ez∼pnoise(z) [ log(1−D(G(z, θG), θD)) ] } ;
• for S (it is updated similarly to D): θS ← θS + γS∇SL where
∇SL = ∂
∂θS Ez∼pnoise(z) [ logS(Stego(G(z, θG)), θS) + log(1− S(G(z, θG), θS)) ] ;
• for the generator G: θG ← θG − γG∇GL with∇GL given by
∇GL = ∂
∂θG αEz∼pnoise(z) [ log(1−D(G(z, θG), θD)) ]
+ ∂
∂θG (1− α)Ez∼pnoise(z) [ log(S(Stego(G(z, θG), θS))) ]
+ ∂
∂θG (1− α)Ez∼pnoise(z) [ log(1− S(G(z, θG), θS)) ] .
The main distinction from the GAN model is that we update G in order to maximize not only the error of D, but to maximize the error of the linear combination of the classifiers D and S.
5 EXPERIMENTS
5.1 DATA DESCRIPTION
In our experiments 1 we use the Celebrities dataset (Ziwei Liu & Tang, 2015) that contains 200 000 images. All images were cropped to 64× 64 pixels. For steganalysis purposes we consider 10% of data as a test set. We denote the train set by A, the test set by B and steganography algorithms used for hiding information by Stego(x). After embedding some secret information we get the train set A+ Stego(A) and the test set B + Stego(B), We end up with 380 000 images for steganalysis training and 20 000 for testing. For training the SGAN model we used all 200 000 cropped images. After 8 epochs of training our SGAN produces images displayed in fig. 3.
For information embedding we use the ±1-embedding algorithm with a payload size equal to 0.4 bits per pixel for only one channel out of three. As a text for embedding we use randomly selected excerpts from some article from The New York Times.
1Code is available at https://github.com/dvolkhonskiy/adversarial-steganography
5.2 EXPERIMENTAL SETUP
In this section we describe the SGAN model structure. By C2D-BN-LR we denote the following structural block of a convolutional neural network: Conv2d→ Batch Normalization→ Leaky ReLU. The Steganalyser network S and the Image Discriminator network have similar structure: four C2D-BN-LR layers, then a fully connected layer (1 neuron)→ Sigmoid function is used to compute an output. The Image generator network G is (in order) a fully-connected layer (8192 neurons), four C2D-BN-LR with Fractional-Strided convolution, then the Hyperbolic tangent function layer is used to compute normalised output.
The SGAN model is trained to solve (4) using the Adam optimization algorithm (Kingma & Ba, 2014) with the learning rate 2−4 and update parameters β1 = 0.5 and β2 = 0.999. For each mini-batch of images we update weights of D and S once, then we update weights of G twice.
In the following experiments, in addition to the steganalyser S we use an independent steganalyser S∗. We define a filter F (0) that is special for steganalysis applications (see Pibre et al. (2015), Qian et al. (2015a), Tan & Li (2014)), as follows
F (0) = 1
12  −1 2 −2 2 −1 2 −6 8 −6 2 −2 8 −12 8 −2 2 −6 8 −6 2 −1 2 −2 2 −1  .
The structure of the individual steganalyser S∗ has the form: 2D convolution with F (0) filter → Conv2D→ Conv2D→Max Pooling→ Conv2D→ Conv2D→Max Pooling→ Fully connected layer (1024 neurons) → Fully connected layer (1 neuron) → Sigmoid function for output. This structure provides state-of-the-art steganalysis accuracy, Pibre et al. (2015), and the filter F (0) allows to increase convergence speed of the steganalyser S∗ training.
For training of this steganalyser we use the Adam optimization algorithm on the loss (4) with the learning rate equal to 5−6, β1 = 0.9, β2 = 0.999. As a loss function we use a binary cross-entropy.
The setup of experiments can be described as follows:
• We train and use the SGAN and/or DCGAN model to generate images to be used as containers;
• We train the independent steganalyser S∗ using either real images (sec. 5.3) or generated images (sec. 5.4);
• We measure the accuracy of the steganalyser S∗.
5.3 TRAINING/TESTING ON REAL IMAGES
In this set of experiments we train the independent steganalyser S∗ on real images. Results are provided in tab. 1. From the results we conclude that even the usual DCGAN generate synthetic
container images, that can easily deceive a steganalyser. Although containers generated by on the SGAN model, which is specially designed to deceive the steganalyser, improve on the result of DCGAN, the improvement is not significant. These results also suggest that S∗ may have been overfit to the data distribution of the real images, and failed to generalize to the GAN approximation of the data distribution in the synthetic image domain.
Another interesting result follows from an experiment with HUGO steganography algorithm and SGAN generated containers:
• We use the ±1-embedding algorithm for training the SGAN model; • We test the HUGO steganography analyzer on real images and on images, generated by the
SGAN model.
Containers generated by the SGAN model decrease HUGO steganalysis accuracy from 0.624 to 0.499, which demonstrates that the Steganographic Generative Adversarial Networks can potentially be used as a universal tool for generating Steganography containers tuned to deceive any specific steganalysis algorithm.
5.4 TRAINING/TESTING ON GENERATED IMAGES
In this set of experiments we train the steganalyser S∗ on images, generated using the DCGAN model from the prior noise distribution pnoise(z) for some fixed seed value. In all experiments the size of the train set is fixed at ≈ 160 000. After training we test the analyser S∗ on images generated with the DCGAN model according to the following experimental setups:
C1. We use the same seed value;
C2. We use some randomly selected seed value;
C3. We use the same seed value, as in C2, and we additionally tune the DCGAN model for several epochs.
The experiment results in tab. 2 indicate that using different seed values when generating containers most likely affects the distribution of pixel value bits in such a way as make it easier to deceive
the steganalyser, fitted to another distribution in the train sample. Additional tuning of the image generator G make this effect even more pronounced.
In the next set of experiments we train and test the steganalyser S∗ on images, generated according to the following experimental conditions:
C4. We generate a train set for the steganalyser S∗ using several different randomly selected seed values, and when generating the test set we use another fixed seed value;
C5. We generate the train set and the test set using a number of different randomly selected seed values;
C6. We use the same train and test sets, as in C5, and we additionally train the DCGAN model for several epochs.
According to tab. 3 the accuracy in case C5 is lower than in the C4 case, which can be explained by the test set of C5 having more variability, being generated with different randomly selected seed values. Similarly, the accuracy in the C4 case is higher than in C2, since in C4 the train set was generated with several different randomly selected seed values, and thus is more representative. These observations confirm out initial conclusions, drawn from tab. 2.
We also conduct an experiment with classification of generated images without steganographic embeddings. For this purposes we train a DCGAN conditional model on the MNIST dataset, and train a separate classifier for the MNIST classification task. The trained classifier achieved almost perfect accuracy both on the held-out real MNIST dataset, and on synthetic images produced by the DCGAN. This provides evidence that it is possible to train an image classifier that shows acceptable accuracy both on real and synthetic images. However it is the artificial generation of image containers that breaks the usual approaches to steganalysis.
6 CONCLUSIONS AND FUTURE WORK
In this work
1. We open a new field for applications of Generative Adversarial Networks, namely, container generation for steganography applications;
2. We consider the ±1-embedding algorithm and test novel approaches to more steganalysissecure information embedding:
a) we demonstrate that both SGAN and DCGAN models are capable of decreasing the detection accuracy of a steganalysis method almost to that of a random classifier;
b) if we initialize a generator of containers with different random seed values, we can even further decrease the steganography detection accuracy.
In future, We plan to test our approach on more advanced steganographic algorithms, e.g. WOW (Holub & Fridrich, 2012), HUGO (Pevny et al., 2010) and S-UNIWARD (Holub et al., 2014).
ACKNOWLEDGMENTS
The research was supported solely by the Russian Science Foundation grant (project 14-50-00150). The authors would like to thank I. Nazarov for his assistance in preparation of this paper.

I found this paper very original and thought-provoking, but also a bit difficult to understand. It is very exciting to see a practical use case for image-generating GANs, with potentially meaningful benchmarks aside from subjective realism.
I found eq. 4 interesting because it introduces a potentially non-differentiable black-box function Stego(...) into the training of (S, G). Do you in fact backprop through the Stego function?
- For the train/test split, why is the SGAN trained on all 200k images? Would it not be cleaner to use the same splits for training SGAN as for "steganalysis purposes"? Could this account for the sensitivity to random seed shown in table 2?
- Sec. 5.3: "Steganographic Generative Adversarial Networks can potentially be used as a universal tool for generating Steganography containers tuned to deceive any specific steganalysis algorithm.". This experiment showed that SGAN can fool HUGO, but I do not see how it was "tuned" to deceive HUGO, or how it could be tuned in general for a particular steganalyzer.
Although S* seems to be fooled by the proposed method, in general for image generation the discriminator D is almost never fooled. I.e. contemporary GANs never converge to actually fooling the discriminator, even if they produce samples that sometimes fool humans. What if I created an additional steganalyzer S**(x) = S*(x) * D(x)? This I think would be extremely difficult to fool reliably because it requires realistic image generation.
After reading the paper several times, it is still a bit unclear to me how or why precisely one would use a trained SGAN. I think the paper could be greatly improved by detailing, step by step, the workflow of how a hypothetical user would use a trained SGAN. This description should be aimed at a reader who knows nothing or very little about steganography (e.g. most of ICLR attendees).