In this paper, we present multiple approaches for improving the performance of gradient descent when utilizing mutiple compute resources. The proposed approaches span a solution space ranging from equivalence to running on a single compute device to delaying gradient updates a fixed number of times. We present a new approach, asynchronous layer-wise gradient descent that maximizes overlap of layer-wise backpropagation (computation) with gradient synchronization (communication). This approach provides maximal theoretical equivalence to the de facto gradient descent algorithm, requires limited asynchronicity across multiple iterations of gradient descent, theoretically improves overall speedup, while minimizing the additional space requirements for asynchronicity. We implement all of our proposed approaches using Caffe – a high performance Deep Learning library – and evaluate it on both an Intel Sandy Bridge cluster connected with InfiniBand as well as an NVIDIA DGX-1 connected with NVLink. The evaluations are performed on a set of well known workloads including AlexNet and GoogleNet on the ImageNet dataset. Our evaluation of these neural network topologies indicates asynchronous gradient descent has a speedup of up to 1.7x compared to synchronous.
1 INTRODUCTION
Deep Learning (DL) algorithms are a class of Machine Learning and Data Mining (MLDM) algorithms, which use an inter-connection of neurons and synapses to emulate the computational structure of a mammalian brain. DL algorithms have demonstrated resounding success in many computer vision tasks and science domains such as high energy physics, computational chemistry and high performance computing use-cases. Several DL implementations such as TensorFlow, Caffe, Theano, and Torch have become available. These implementations are primarily geared towards compute nodes that may contain multi-core architecture (such as Intel Xeon/KNC/KNL) and or many-core architectures (GPUs).
DL algorithms are under-going a tremendous revolution of their own. Widely used DL algorithms such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are computationally expensive. Their computational requirements are further worsened by: 1) Very deep neural networks such as recently proposed 1000-layer complex Residual Networks (ResNet), 2) Increasing volume of data produced by simulations, experiments and handheld devices. An important solution to these problems is the design and implementation of DL algorithms that are capable of execution on distributed memory large scale cluster/cloud computing systems. A few distributed DL implementations such as CaffeonSpark, Distributed TensorFlow, CNTK, Machine Learning Toolkit on Extreme Scale (MaTEx), and FireCaffe have become available. Implementations such as CNTK, FireCaffe and MaTEx use MPI (Gropp et al., 1996; Geist et al., 1996) – which makes them a natural fit for high-end systems.
DL algorithms primarily use gradient descent – an iterative technique in which the weights of synapes are updated using the difference between the ground truth (actual value) and the predicted value (using the current state of the neural network). The larger the difference, the steeper the de-
scent to a minima (a low value of minima generates the solution). An important type of gradient descent is batch gradient descent – where a random subset of samples are used for iterative feedforward (calculation of predicted value) and back-propagation (update of synaptic weights). A small batch is prone to severe pertubations to the descent, while a large batch results in slow convergence. Hence, a data scientist tends to use a fairly average batch – which finds the balance between these two conflicting metrics.
A large scale parallelization of gradient descent must maximize the equivalence to the default algorithm, such that the convergence property is maintained. Consider a scenario where a batch (b) in the original algorithm is split across multiple compute nodes (n) – an example of data parallelism. To provide equivalence to the default algorithm, the batch must be split equally to bn , although the communication which would require an all-to-all reduction would increase as Θ(log n). Naturally, as n is increased and b is held constant (strong scaling), this becomes prohibitive, whereas keeping the batch size per node b/n constant (weak scaling) increases the convergence time.
Several researchers have proposed methods to alleviate the communication requirements of distributed gradient descent. Parameter-server based approaches use a server to hold the latest version of the model while clients send computed gradients and request the latest model. This approach has been proposed and extended by several researchers. While theoretically this provides O(1) timecomplexity since all batch updates can be computed simultaneously, this approach fails to scale beyond a few compute nodes when considering the time to convergence relative to having run the computation on a single device. Others have proven divergence from the original algorithm. Remote Direct Memory Access (RDMA) based approaches have been proposed, but they also diverge from the original algorithm. Several other implementations are primarily geared towards shared memory systems, and address the thread contention issue for gradient descent.
Our objective is to design a non-parameter-server based technique, which maximizes the equivalence to the default algorithm, while leveraging high performance architectures – including computational units such as GPUs and high performance interconnects such as InfiniBand, Intel Omni-path architectures by using MPI.
1.1 CONTRIBUTIONS
Specifically, we make the following contributions in this paper:
• We design a baseline asynchronous gradient descent, which delays the gradient updates of the entire model by one or more iterations adaptively on the basis of available overlap and user-defined input.
• We propose a layer-wise gradient descent method, which overlaps weight updates of a layer with inter-node synchronization of other layers. The proposed method is exactly equiavalent to the default sequential algorithm.
• We implement our approaches and other baseline techniques using the Machine Learning Toolkit for Extreme Scale (MaTEx), which consists of a distributed memory implementation of Caffe using MPI (Gropp et al., 1996; Geist et al., 1996).
• We evaluate our approaches and other baseline implementations on a large scale CPU-based InfiniBand cluster as well as on NVIDIA’s DGX-1 multi-GPU system. We use several well studied datasets and DNN topologies such as ImageNet (1.3M images, 250GB dataset) with AlexNet and GoogleNet DNNs.
Our evaluation indicates the efficacy of the proposed approach. Specifically, the best asynchronous approach is up to 1.7x faster than the synchronous approach while achieving up to 82% parallel efficiency.
The rest of the paper is organized as follows: In section 2, we present related work of our proposed research. We present the background in section 3, followed by an in-depth solution space in section 4. In section 6, we present a detailed performance evaluation of asynchronous gradient descent, and conclusions with future directions in section 7.
2 RELATED WORK
Batch gradient descent is the most widely used algorithm for training Deep Learning models. This algorithm has been implemented several times for sequential, multi-core and many-core systems such as GPUs. The most widely used implementations are Caffe (Jia et al., 2014) (CPUs/GPUs), Warp-CTC (GPUs), Theano (Bastien et al., 2012; Bergstra et al., 2010) (CPUs/GPUs), Torch (Collobert et al., 2002) (CPUs/GPUs), CNTK (Agarwal et al., 2014) (GPUs and Distributed Memory using MPI) and Google TensorFlow (Abadi et al., 2015) which use nVIDIA CUDA Deep Neural Network (cuDNN).
Caffe is one of the leading software tools for training and deploying deep learning algorithms, and it can be used to develop novel extensions to these algorithms such as the ones described below. Caffe supports execution on a single node (connected with several GPUs) and a version has been implemented that takes full advantage of Intel systems. While the research described below was performed using Caffe, the extensions can be applied to Tensorflow as well.
Caffe (and other deep learning software) is also equipped with several optimizations designed to avoid significant problems in training deep networks. The vanishing gradient problem (Bianchini & Scarselli, 2014) causes deep networks to fail to learn much at all in the early layers, and was solved in (Hinton & Osindero, 2006) and (Bengio et al., 2007) where it was shown that a network could be trained one layer at a time with autoencoders (Hinton & Salakhutdinov, 2006), and then put together to form a single network (Vincent et al., 2010). Another optimization that helps to solve this problem is switching from sigmoidal neurons to rectified linear neurons.
The problem of accelerating gradient descent, especially disctributed across compute resources, is of interest to many researchers. Approaches generally fall into two categories, whether or not they are equivalent to having run using a single compute device; utilizing a single compute device necessarily computes gradient updates and applies them immediately to the model. Further, the gradient updates can be classified as either synchronous or asynchronous depending on whether the communication of the gradients can be overlapped with any computation of the gradients. For example, the DistBelief parameter server approach (Dean et al., 2012) computes gradient updates asynchronously based on an out-of-date copy of the model and applies them to the latest model. Though this is not equivalent to having run on a single device, it is able to process samples much faster.
Chen et al. (2016) revisit asynchronous gradient descent and propose a few synchronous variants in order to impove time to convergence. Notably, they show that waiting for all workers to complete, aggregating the gradients, and applying the gradients to the same common model (thereby each worker has a copy of the latest model) provides a good time to convergence while also leveraging multiple compute devices. Their approach is where this paper begins while additionally proposing approaches ranging from synchronous to parameter server variants.
3 FUNDAMENTALS
3.1 NEURAL NETWORKS
Machine Learning algorithms designed to emulate the computational structure of the brain to model data are called “Neural Networks.” The basic unit of a neural network is the neuron and neurons are connected to one another via synapses.
3.1.1 BACKPROPAGATION
Neural networks are trained through an algorithm called backpropagation. This is a means of computing gradients layer by layer to implement the gradient descent algorithm’s update rule of
w′ = w + λ∇wC (1) b′ = b + λ∇bC (2)
where w are the weights, b the biases, λ the learning rate, and C is a cost function to be optimized, usually square error or cross-entropy. This rule is often replaced by a slightly more complex rule, such as Adaptive Gradient Descent (AdaGrad) (Duchi et al., 2011) or Momentum (Qian, 1999).
To compute the gradients, we set W (`), b(`) the weights and biases for each layer, z(`+1) = W (`)a(`) + b(`) and a(`) = σ(z(`)), where σ is the activation function. Let n` represent the number of layers. Then, we use Algorithm 1.
Algorithm 1 Back Propagation
1: input: Data X ∈ Rn×p and labels Y ∈ Rn×` 2: for i from 1 to n do 3: Compute all z(`) and a(`). 4: δ(n`) = −(y − an`) σ(z(n`)) 5: for ` from n` − 1 to 2 do 6: δ(`) = W `δ(`+1) σ′(z(`)) 7: end for 8: ∇W (`)C = δ(`+1)a(`) T
9: ∇b(`)C = δ(`+1) 10: end for
Although there are several nonlinear activation functions in common use, the networks examined in this paper only include rectified linear units (ReLU) where ReLU(x) = max(0, x).
3.2 CAFFE
Caffe (Jia et al., 2014) is one of the leading software packages for building and training neural networks. It provides abstractions for a wide range of topologies and for training them with many different types of optimizers. Caffe provides abstractions for operations on multi-dimensional arrays (tensors) which are essential for implementing Deep Learning algorithms. From an input tensor, an output tensor, and tensors for each hidden layer, Caffe constructs a computational graph that manages these tensors and their updates as a single object. Caffe is particularly useful for researchers, because it is heavily optimized and can be modified through an open source C++ backend.
As Caffe’s runtime is implemented in C++, it can extract native performance from the computation environment it is run on. Furthermore, Caffe abstracts GPU computations, leveraging nVIDIA CUDA Deep Neural Network Library (cuDNN) for the task. We have modified this code for distributed memory computation on large scale systems using MPI to natively use network hardware for optimal performance. The base, synchronous implementation is similar to FireCaffe (Iandola et al., 2015), another distributed memory implementation of Caffe. Further modifications are described in Section 4.
There are three phases of computation within Caffe that pass over the enumerated layers of the network. First, the forward pass computes the output result given the samples from the input batch, starting at the first layer. Next, starting at the last (output) layer, based on the difference between the output result and the ground truth, the backward pass uses the backpropagation technique to compute the gradients for each layer. Lastly, one final pass is made over the network to apply the gradients to the weights and biases before starting the process over again with the next batch.
4 SOLUTION SPACE
The goal of improving gradient descent is to accelerate the time to solution without sacrificing the accuracy of the model. The base case to consider is then computing and applying gradients one batch at a time on a single compute device. One way to accelerate the computation while also maintaining equivalence to the sequential is to use data parallelism. Data parallelism is where the traditional batch is further subdivided into equally-sized mini-batches, each mini-batch is computed on separate devices, then the gradients resulting from each mini-batch is averaged together. Since each gradient update is itself an average, taking the average of the mini-gradients results in an update that is effectively the same as having computed the original batch size. This is called the effective batch size. Data parallelism is the approach we explore in this paper, attempting many ways of hiding the latency of the gradient communication that occurs between compute devices. We use MPI to communicate the gradients.
Caffe provides callback methods in its C++ interface that interject user-defined functionality into key phases of the computation (see 3.2). Specifically, one user-defined function is executed immediately before the foward pass when the batch computation begins. The other user-defined function executes after the backward pass finishes, but before the application of the gradients to the weights and biases. Additional callback functions were added to support finer-grained control over the three phases of computation. One of the additional callbacks executes after each gradient is computed during the backward phase, once per set of learnable parameters, such as the weights or biases of a given layer. Another callback function that was added is called once per learnable parameter during the apply phase, just before the gradients are applied. Lastly, a callback function was added that turns the gradient application into a task queue, requesting additional tasks in an unspecified order until all gradients have been applied.
A critical implementation detail for any of our proposed approaches is to make sure the individual network models maintained by each compute device start from the same random initial conditions for the weights and biases. Before the first batch is computed, the weights and biases from the master process are copied (broadcast) to the other processes. That way any gradients that are computed, when averaged together, are based on the same initial conditions.
4.1 SYNCHRONOUS GRADIENT DESCENT
Similar to what Chen et al. (2016) proposes and what is implemented in FireCaffe (Iandola et al., 2015), synchronous gradient descent averages the gradients from each mini-batch together before applying them, forming one complete batch at a time. The way this is implemented in Caffe is to use the callback function that executes when all gradients are ready to be applied. During this callback, MPI Allreduce is used to sum the gradients, placing the same resulting sum on each compute device. This function is blocking, meaning it returns control back to Caffe only after the sum is computed across all devices. Since the result is a sum and not the intended average, it is then scaled down based on the number of compute devices in use. It is important to note that the reduction operation can be performed in-place, meaning it can use the memory location directly holding the gradient without performing any costly memory copies, especially for networks with a large number of parameters such as AlexNet. This approach also has the important quality that the gradients are averaged after they have been used by each layer of the backpropagation, preserving the importance of any activations within the network against the mini-batch instead of against the effective batch.
4.2 LAYER-WISE GRADIENT DESCENT
Chen et al. (2016) proposes the pipelining of gradient computation and application. For example, the gradients of upper layers can be concurrently applied while computing the gradients of lower layers. This approach must be done carefully to maintain equivalence with the sequential base case. We make the observation that gradients can be averaged as soon as they are computed during the backward phase, instead of waiting for all gradients to be computed. However, adjacent layers will use and/or update the gradients of layers that have otherwise finished computing their gradients. This implies the averaging of the gradients must be performed on a copy of the gradients rather than in-place. Further, the averaging of the copied gradients must finish before they can be applied.
We utilize a background thread of computation in order to perform the gradient averaging concurrent with the remaining gradient computation. This provides maximal overlap of the communication latency with useful computation. There are a few options when to apply the averaged gradients. Waiting for all communication to finish before applying all gradients is straightfoward and similar to the synchronous approach described previously, though perhaps at least some of the communication latency would be overlapped. Another approach is to wait, one layer at a time, for the gradients for a particular layer to finish averaging and then apply the gradients. It is intuitive to perform the waiting in the same order in which backpropagation was performed, from the last layer to the first layer. Lastly, since all gradient updates are independent, we can perform them in an arbitrary order. This takes advantage of the observation that not all layers have the same number of parameters, and further, the gradients for the weights and the gradients for the biases can be averaged separately; the size of the weight gradients are typically larger than the bias gradients, implying that the bias gradients will complete their communication more quickly. Since the communcation of the various parameters can finish somewhat arbitrarily based on when the communication was initiated and the
size of the communication, we can apply the gradients as soon as they complete their averaging. We evaluate these strategies in 6.
4.3 ASYNCHRONOUS GRADIENT DESCENT
As stated in (Chen et al., 2016), parameter server implementations suffer from poor convergence since gradient updates are calculated based on out-of-date networks. Continuing with our data parallel approach, there is a lower limit to the size of the mini-batches and therefore the number of compute devices that can be utilized. As the amount of work per compute device decreases proportional to the decreasing size of the mini-batches, there is less computation available to mask the latency of the gradient averaging across the devices. Initiating the averaging layer-wise as described above may not be enough to mitigate this problem.
We propose delaying the application of the gradients by a fixed number of iterations much smaller than the number of compute devices as would have been done in a parameter server approach. The gradients are delayed by using a concurrent communication thread and applying the gradient one, two, or three iterations later thus giving the averaging enough time to complete as needed. If the gradient needs to be delayed by one iteration, this requires one communication thread and one additional buffer to hold the gradient; delaying by two iterations requires two communication threads and two additional buffers and so on. This approach is somewhere between a parameter server (Dean et al., 2012) and the various approaches that maintain equivalency with a sequential computation.
5 IMPLEMENTATION DETAILS
The implementations evaluated in this paper focus on data parallelism and the averaging of gradients across compute devices. This is achieved using MPI and parallel I/O.
5.1 HANDLING I/O
The data parallelism is achieved by distributing datasets across compute devices, partitioning them based on the number of devices utilized; each device receives a disjoint subset of the dataset and no samples are shuffled or exchanged between compute devices outside of the gradient averaging. Caffe frequently uses a database in LMDB format for its datasets, however this format cannot be used on remote (network) filesystems or even between processes on the same host. Caffe mitigates this issue when using more than one GPU on the same host by using a single I/O reading thread and a round-robin deal of the samples to device-specific queues. Our implementations mitigate this issue by first converting an LMDB database into a netCDF file (Rew & Davis, 1990). netCDF files can be read and partitioned using parallel MPI-IO via the parallel netCDF library (Li et al., 2003).
5.2 DISTRIBUTED MEMORY IMPLEMENTATION USING MPI
For single-node GPU computation, using one or more GPU devices in a single host, Caffe provides a means of allocating one contiguous buffer to hold the data for the weights and biases and a second buffer to hold the gradients for each. We extended this approach for CPU hosts. A single contigous buffer allows the non-layer-wise, i.e., network-wise gradient averages to be performed using a single MPI reduction operation. The layer-wise implementations require one MPI reduction operation per network parameter. There is a fixed cost to start a communication primitive regardless of how much data is communicated. It is sometimes beneficial to aggregate otherwise many small communication requests into a larger one.
Although Caffe provides a way of utilizing all GPUs within the host, it does not currently leverage NVIDIA’s NCCL package (NVIDIA Corporation, 2015) for optimized, high-bandwidth collective communication routines. We used the NCCL equivalent to the MPI all reduction to sum gradients across GPU devices on the DGX-1 platform.
6 EXPERIMENTAL EVALUATION
In this section, we present an experimental evaluation and analysis of the heuristics described in section 4.
6.1 HARDWARE ARCHITECTURES
We evaluate using a CPU cluster as well as NVIDIA’s speialized DGX-1 multi-GPU host system. Each node of the multi-node cluster consists of a multi-core Intel Sandybridge CPU connected via InfiniBand. We use Intel MPI 5.1.2 for performance evaluation. The heuristics are implemented in Caffe (Jia et al., 2014), specifically the intelcaffe branch designed to optimize performance on Intel CPUs.
The DGX-1 system contains 8 Pascal GPUs connected using the high-speed NVlink interconnect. For the DGX-1 evaluations, the latest version of Berkley’s Caffe was modified to use the NCCL communicaiton primitives in addition to our algorithmic changes.
6.2 IMAGENET AND NETWORK ARCHITECTURES
We evaluate on two distinct network architectures trained on the ImageNet dataset. ImageNet refers specifically to the ILSVRC2015 (Russakovsky et al., 2015) dataset. This dataset consists of a training set of just under 1.3 million images of various sizes (as jpg files) divided among 1000 classes, along with a validation set consisting of 50000 images of the same type and classes. Additionally, for the competition, there is a testing set, but it is held separately and not available publicly. It is established as one of the benchmark dataset for machine learning with large datasets, and among the famous architectures that achieved record top 1 and top 5 accuracies on it are AlexNet (Krizhevsky et al., 2012) and GoogLeNet (Szegedy et al., 2015).
We evaluate on AlexNet and GoogLeNet because they are now well-established models with known training regimes and loss curves. They also demonstrate two different regimes for parallelization: AlexNet has approximately 60 million parameters that need to be communicated, whereas GoogLeNet has approximately 4 million. In contrast to the smaller amount of communication for GoogLeNet, it requires roughly twice the amount of time to process a each image than AlexNet does when communication is ignored.
6.3 EVALUATION
Figure 1 compares the implemented approaches relative to a communication-less baseline “no comm”. The effective batch sizes were 256 and 32 for AleNet and GoogLeNet, respectively. For example, using 8 compute devices for GoogLeNet uses a mini-batch size of 32/8 = 4. The evaluation on DGX-1 were limited to 8 compute devices whereas the CPU cluster evaluation eventually hit the strong scaling limit for data parallelism.
These results show that delaying the gradient updates by one or more iterations is the most effective means of hiding the communication latency. The layer-wise approaches did not perform as well as expected. These trends were consistent across both hardware platforms.
The layer-wise approaches, though promising as equivalent to a sequential computation, were not able to complete their gradient averages quickly enough. Compared to the delayed gradient approach, this is perhaps intuitive. The delayed gradient approach is able to hide the communication latency across all three complete phases of the computation whereas the layer-wise approaches only have as long as it takes to complete the backpropagation phase. This is not enough time to complete the communication, especially as the mini-batch sizes decrease and therefore provide less work to mask the communication.
In addition to looking at the time per batch above, the rates of convergence of these heuristics must be evaluated. All of the heuristics completed training AlexNet to the standard top-1 accuracy of ≈ 54% using the default AlexNet settings that come with Caffe. However, it is worth noting that at the beginning of training, they showed different loss curves showing that there is a tradeoff between number of batches per second and accuracy at a given batch as shown in Table 1.
We also evaluated whether these approaches converged in addition to just improving the number of iterations per second. All approaches evaluated managed to converge within the exepcted number of iterations. Notably, AlexNet on DGX-1 reached convergence in 11 hours using the delayed gradient approach and two communication threads using the standard AlexNet network from Caffe.
7 CONCLUSIONS
There is a tradeoff between maintaining equivalence to sequential methods versus leveraging the vast computational resources available for gradient descent. We find that asynchronous methods can give a 1.7x speedup while not sacrificing accuracy at the end of an otherwise identical training regime. This improvement was achieved without the need for a warm start, contrary to previously published results using parameter servers.

This paper is relatively difficult to parse. Much of the exposition of the proposed algorithm could be better presented using pseudo-code describing the compute flow, or a diagram describing exactly how the updates take place. As it stands, I'm not sure I understand everything. I would also have liked to see exactly described what the various labels in Fig 1 correspond to ("SGD task-wise, 1 comm"? Did you mean layer-wise?).
There are a couple of major issues with the evaluation: first, no comparison is reported against baseline async methods such as using a parameter server. Second, using AlexNet as a benchmark is not informative at all. AlexNet looks very different from any SOTA image recognition model, and in particular it has many fewer layers, which is especially relevant to the discussion in 6.3. It also uses lots of fully-connected layers which affect the compute/communication ratios in ways that are not relevant to most interesting architectures today.


VARIATIONAL RECURRENT ADVERSARIAL DEEP DOMAIN ADAPTATION
We study the problem of learning domain invariant representations for time series data while transferring the complex temporal latent dependencies between domains. Our model termed as Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) is built atop a variational recurrent neural network (VRNN) and trains adversarially to capture complex temporal relationships that are domain-invariant. This is (as far as we know) the first to capture and transfer temporal latent dependencies of multivariate time-series data. Through experiments on real-world multivariate healthcare time-series datasets, we empirically demonstrate that learning temporal dependencies helps our model’s ability to create domain-invariant representations, allowing our model to outperform current state-of-the-art deep domain adaptation approaches.
1 INTRODUCTION
Many real-world applications require effective machine learning algorithms that can learn invariant representations across related time-series datasets. For example, precision medicine for patients of various age groups, mobile application recommendation for users based on locations, and so on. In these examples, while the domains (i.e. age group and location) may vary, there exist common predictive patterns that can aid in inferring knowledge from one domain to another. More often than not, some domains have a significantly larger number of observations than others (e.g., respiratory failure in adults vs. children). Therefore effective domain adaption of time-series data is in great demand.
The general approach to tackling domain adaptation has been explored under many facets which include reducing the domain discrepancy between the source and target domains(Ben-David et al. (2007)), instance re-weighting (Jiang & Zhai (2007)), subspace alignment (Fernando et al. (2013)), and deep learning (Tzeng et al. (2015); Ganin & Lempitsky (2014)). Many of these approaches work very well for non-sequential data but are not suitable for multivariate time-series data as they do not usually capture the temporal dependencies present in the data. For sequential data, earlier work has successfully used dynamic Bayesian Networks(Huang & Yates (2009)) and Recurrent Neural Networks (Socher et al. (2011)) to learn latent feature representations which were domaininvariant. Unfortunately, these works were not flexible enough to model non-linear dynamics or did not explicitly capture and transfer the complex latent dependencies needed to perform domain adaptation of time-series data.
In this paper, we address this problem with a model that learns temporal latent dependencies (i.e. dependencies between the latent variables across timesteps) that can be transferred across domains that experience different distributions in their features. We draw inspiration from the Variational Recurrent Neural Network (Chung et al. (2016)) and use variational methods to produce a latent representation that captures underlying temporal latent dependencies. Motivated by the theory of domain adaptation (Ben-David et al. (2010)), we perform adversarial training on this representation
*: Co-first authors
t-SNE projections for the latent representations of DNN, R-DANN, and our VRADA model. We show adaption from Adult-AHRF to Child-AHRF data. Source data is represented with red circles and target data with blue circles. From left to right, one can see that domain adaptation results in mixing the source and target domain
data distributions. We can also see a story of how encoding more temporal dependency into the latent representation induces more domain-invariant representations. As models capture more underlying factors of
variation, post domain adaptation representations gradually smoothen and become evenly dispersed, indicating that temporal dependency acts synergestically with domain adaptation.
similarly to the Domain Adversarial Neural Network (DANN) (Ganin et al. (2016)) to make the representations invariant across domains. We call our model the Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) model. As far as we know, this is the first model capable of accomplishing unsupervised domain adaptation while transferring temporal latent dependencies for complex multivariate time-series data. Figure 1 shows an example of the domain invariant representations learned by different deep learning models including our VRADA model. From this figure, we can see that our model (VRADA) shows better mixing of the domain distributions than the competing models indicating that it learns better domain invariant representations.
In order to prove the efficacy of our model, we perform domain adaptation using real-world healthcare time-series data. We choose healthcare data for two primary reasons. (1) Currently, a standard protocol in healthcare is to build, evaluate, and deploy machine learning models for particular datasets that may perform poorly on unseen datasets with different distributions. For example, models built around patient data from particular age groups perform poorly on other age groups because the features used to train the models have different distributions across the groups (Alemayehu & Warner (2004); Lao et al. (2004); Seshamani & Gray (2004)). Knowledge learned from one group is not transferrable to the other group. Domain adaptation seems like a natural solution to this problem as knowledge needs to be transferred across domains which share features that exhibit different distributions. (2) Healthcare data has multiple attributes recorded per patient visit, and it is longitudinal and episodic in nature. Thus, healthcare data is a suitable platform on which to study a model which seeks to capture complex temporal representations and transfer this knowledge across domains.
The rest of the paper is structured as follows. In the following section, we briefly discuss the current state-of-the-art deep domain adaptation approaches. Afterwards, we present our model mathematically, detailing how it simultaneously learns to capture temporal latent dependencies and create domain-invariant representations. In Section 4, we compare and contrast the performance of proposed approach with other approaches on two real-world health care datasets, and provide analysis on our domain-invariant representations.
2 RELATED WORK
Domain adaptation is a specific instance of transfer learning in which the feature spaces are shared but their marginal distributions are different. A good survey on the two has been done in several previous works (Pan & Yang (2009); Jiang (2008); Patel et al. (2015)). Domain adaptation has been thoroughly studied in computer vision(Saenko et al. (2010); Gong et al. (2012); Fernando et al. (2013)) and natural language processing (NLP) (Blitzer (2007); Foster et al. (2010)) applications. Recently, the deep learning paradigm has become popular in domain adaptation (Chen et al. (2012); Tzeng et al. (2015); Yang & Eisenstein; Long & Wang (2015)) due to its ability to learn rich, flexible, non-linear domain-invariant representations. Here, we briefly discuss two deep domain adaptation approaches which are closely related to our proposed model. Domain Adversarial Neural Networks (DANN)
(Ganin et al. (2016)) is a deep domain adaptation model which uses two core components to create domain-invariant representations, a feature extractor that produces the data’s latent representation, and an adversarial domain labeler that attempts to classify that data’s domain to help the feature extractor produce latent representations which are domain-invariant. In Louizos et al. (2015), the authors propose Variational Fair AutoEncoder, which uses Variational Autoencoding architecture (Kingma & Welling (2013)) to learn latent representations where most of the information about certain known factors of variation are purged from the representation while still retaining as much information about the data as possible. While, these deep learning approaches learn domain-invariant representations, they fail to capture and transfer the underlying complex temporal latent relationships from one domain to another as they use convolutional or feed forward neural networks which we claim are not suitable for multivariate time-series data.
Other works such as Huang & Yates (2009); Xiao & Guo (2013) have used distributed representations for domain adaptation in NLP sequence labeling tasks. However, they either induce hidden states as latent features using dynamic Bayesian networks (DBNs) or learn generalizable distributed representations of words using Recurrent Neural Networks (RNN) (Socher et al. (2011)) to enable domain adaptation. These works either model the highly non-linear dynamics, as one can with RNN, or capture the complex latent dependencies present in sequential data, as one can with DBNs, but not both. To overcome the challenges of DBNs and RNNs, Variational Recurrent Neural Network (VRNN)( Chung et al. (2016)) was proposed recently to capture the complex relationship between the underlying hidden factors of variation and the output variables at different time-steps. The VRNN uses Variational Autoencoders (VAEs)( Kingma & Welling (2013); Goodfellow et al. (2016)) at each time-step to learn a complex relationship between the latent hidden factors across time-steps. Like the VAE, its latent variable is parametric. Combined, these things make it well-suited for multimodal sequential data such as multivariate time-series. In the following section, we discuss our approach, Variational Adversarial Deep Domain Adaptation (VRADA), which uses a VRNN to model and transfer complex domain-invariant temporal latent relationships for unsupervised domain adaptation of multivariate time-series.
3 VARIATIONAL RECURRENT ADVERSARIAL DEEP DOMAIN ADAPTATION
In this section, we present our Variational Recurrent Adversarial Deep Domain Adaptation (VRADA) model for the purpose of capturing and transferring temporal latent dependencies across domains via domain-invariant representations. First, we introduce the notations used in this paper and then discuss our VRADA model in detail.
3.1 NOTATIONS
Let us denote a multivariate variable-length time series with N data samples as {xi = (xit)T i t=1}Ni=1, where xit ∈ RD. (Note: in our experiments, for all data samples T i = τ , but for generality we maintain T i). We denote {xiS}ni=1 as source domain data and {xiT }Ni=n+1 as target domain data. We assume that each source domain data sample xiS comes with L labels yi ∈ {0, 1}L (for example, these labels may correspond to a clinical outcome such as mortality or ICD9 diagnosis codes), while
target domain has no labeled data samples. We assign a domain label di ∈ {0, 1} to each data sample to indicate if it comes from the source or target domain. di will be used for adversarial training.
3.2 VRADA
The block diagram of our VRADA model is shown in Figure 2. To explicitly model the dependencies between the latent random variable across time steps, the VRADA model utilizes Variational Recurrent Neural Networks (VRNN) (Chung et al. (2016)). The VRNN effectively contains a Variational AutoEncoders (Kingma & Welling (2013)) at every time step, all of which are conditioned on previous auto-encoders via the hidden state ht−1 of an RNN, such as an LSTM (Hochreiter & Schmidhuber (1997)). Therefore, for each time-step of xit, we infer a latent random variable z i t via
zit|xit ∼ N (µz,t, diag(σz,t)), where [µz,t, σz,t] = ϕencτ (ϕxτ (xit), ht−1)
with prior zit ∼ N (µ0,t, diag(σ0,t)), where [µ0,t, σ0,t] = ϕpriorτ (ht−1) where µ∗,t, σ∗,t denote parameters of a generating distribution, and ϕ∗τ can be any highly flexible function such as deep neural networks. For each zit, x i t is generated via
xit|zit ∼ N (µx,t, diag(σx,t)), where [µx,t, σx,t] = ϕdecτ (ϕzτ (zit), ht−1)
and learned by optimizing the VRNN objective function: Lr(xit; θe, θg) = Eqθe (zi≤Ti |xi≤Ti )[ T i∑ t=1 (−D(qθe(zit|xi≤t, zi<t)||p(zit|xi<t, zi<t))+log pθg (xit|zi≤t, xi<t))])
where qθe(z i t|xi≤t, zi<t) is the inference model, p(zit|xi<t, zi<t) is the prior, pθg (xit|zi≤t, xi<t) is the generative model, θe is the parameters of the VRNN’s encoder, θg the parameters of the VRNN’s decoder, and D(·||·) refers to KL-Divergence. Note: z≤T refers to the set of all zt such that t ≤ T , likewise for z<T . For each xi, we use z̃i ∼ qθe(ziT i |x i ≤T i , z i <T i) as our feature representation for source domain classification task since it captures temporal latent dependencies across the time-steps. Training the VRNN for the source domain classification involves solving the following optimization:
min θe,θg,θy
1
n n∑ i=1 1 T i Lr(xi; θe, θg) + 1 n n∑ i=1 Ly(xi; θy, θe) + λR(θe) (1)
whereR(θe) is a regularizer for the parameters of VRNN encoder (which is also the feature extractor of VRADA) with a tuning hyperparameter λ.
As we are interested in achieving domain adaptation via the latent representation z̃i (i.e. to make z̃i domain-invariant), we can adversarially train the above objective function (equation 1) by employing the domain adaptation idea proposed in Ganin et al. (2016). Let Gy(z̃i; θy) and Gd(z̃i; θd) represent the source label classifier (to predict source labels yi) and domain label classifier (to predict domain labels di) respectively with parameters θy and θd for a given input z̃i. Here, Gy(.) and Gd(.) can be deep neural networks. Let us denote their loss functions respectively as
Ly(xi; θy, θe) = LB(Gy(Ve(xi; θe); θy), yi); Ld(xi; θd, θe) = LB(Gd(Ve(xi; θe); θd), di)
where LB is the classification loss such as a binary or categorical cross-entropy loss function and Ve(x i; θe) is the VRNN encoder that maps input xi to z̃i.
Now, for adversarial training, we consider the following domain adaptation term as the regularizer of equation 1.
R(θe) = max θd [ − 1 n n∑ i=1 Ld(xi; θd, θe)− 1 n′ N∑ i=n+1 Ld(xi; θd, θe) ]
(2)
where n′ is the number of target domain samples. As shown in Ganin et al. (2016),R is the domain regularizer and it is derived from the empiricalH−divergence between the source domain and target domain samples( Ben-David et al. (2010)).
Combining the joint optimization problem of equations 1 and 2 leads to our VRADA model, where we minimize the source classification risk and at the same time achieve domain adaptation. Mathematically, we optimize the following complete objective function:
E(θe, θg, θy, θd) = 1
N N∑ i=1 1 T i Lr(xi; θe, θg)+ 1 n n∑ i=1 Ly(xi; θy)−λ( 1 n n∑ i=1 Ld(xi; θd)+ 1 n′ N∑ i=n+1 Ld(xi; θd)))
(3)
where λ is a trade-off between optimizing on making domain-invariant representations and optimizing source classification accuracy. Our optimization involves minimization with respect to some parameters, and maximization with respect to the others, i.e., we iteratively solve the following:
(θ̂g, θ̂y, θ̂e) = arg min θg,θy,θe E(θe, θg, θy, θ̂d)
θ̂d = argmax θd E(θ̂e, θ̂g, θ̂y, θd)
with the gradient updates calculated as:
θe ← θe − η(∂Lr∂θe + ∂Ly ∂θy − λ∂Ld∂θd ) (4)
θg ← θg − η ∂Lr∂θg (5) θd ← θd − η ∂Ld∂θd (6) θy ← θy − ηλ∂Ly∂θy (7)
where η is the learning rate. We can use stochastic gradient descent (SGD) to solve the equations (5-7). To solve equation (4), we can use SGD and the gradient reversal layer (GRL)(Ganin et al. (2016)). The role of GRL is to reverse the gradient sign while performing backpropagation. This ensures that the domain classification loss is maximized which makes the feature representations domain-invariant.
Thus, VRADA results in learning feature representations which are domain-invariant (due to domain regressorR) and which capture the temporal latent dependencies (due to optimizing VRNN objective function Lr). These things combine to allow the VRADAs’ discriminative power on the source domain to transfer to the target domain.
4 EXPERIMENTS
We conduct experiments on two real-world health care datasets to answer the following questions: (a) How does our VRADA model perform when compared to the state-of-the-art domain adaptation and non-adaptation approaches? (b) How different are the domain-invariant representations learned by various domain adaptation methods? (c) How do we show that the temporal latent dependencies are transferred between domains? In the remainder of this section, we will describe the datasets, methods, empirical results, and show visualizations to answer the above questions.
4.1 DATASET DESCRIPTION
We conduct experiments on two health care datasets, including the MIMIC-III dataset and a Pediatric ICU (PICU) dataset from Children’s Hospital Los Angeles.
MIMIC-III( Johnson et al. (2016)) is a public dataset with deidentified clinical care data collected at Beth Israel Deaconess Medical Center from 2001 to 2012. It contains over 58,000 hospital admission records of 38,645 adults and 7,875 neonates. For our experiments, we extracted the following two datasets:
• Adult-AHRF dataset: To study domain adaptation for adult patients with acute hypoxemic respiratory failure (AHRF), we extracted 20 time series features (such as Base excess, blood pH value, Mean Air Pressure, PaO2, etc.) from 5527 admission records based on Khemani
et al. (2009). We grouped the patients into 4 groups/cohorts based on their age[1] - Group 2: working-age adult (20 to 45 yrs, 508 patients); Group 3: old working-age adult (46 to 65 yrs, 1888 patients); Group 4: elderly (66 to 85 yrs, 2394 patients); Group 5: old elderly (85 yrs and up, 437 patients). We treated each group as a separate domain with which we could perform domain adaptation. For each patient, we used the first 4 day after admission (with each day serving as a single time-step) as time series data for training and testing our models. • ICD9 dataset: For this dataset we extracted 99 time series features from 19714 admission
records from 4 modalities including input-events (fluids into patient, e.g., insulin), outputevents (fluids out of the patient, e.g., urine), lab-events (lab test results, e.g., blood pH values, platelet count, etc.) and prescription-events (drugs prescribed by doctors, e.g., aspirin, potassium chloride, etc.). These modalities are known to be extremely useful for monitoring ICU patients. All the time series are of more than 48 hours of duration, and only the first 24 hours (after admission) 2-hourly sampled time series data is used for training and testing our models. We use this dataset to predict the ICD9 Diagnosis code categories for each patient’s admission record.
Child-AHRF dataset: This is a PICU dataset which contains health records of 398 children patient with acute hypoxemic respiratory failure in the intensive care unit at Children’s Hospital Los Angeles (CHLA)(Khemani et al. (2009)). Similar to Adult-AHRF, this dataset has 20 time series features collected for 4 days after ICU admission. This dataset is considered as one group (Group 1: children, age 0 to 19 yrs) and represents one domain.
4.1.1 PREDICTION AND DOMAIN ADAPTATION TASKS
Mortality Prediction: For Adult-AHRF and Child-AHRF datasets, we are interested in predicting mortality, i.e. whether a patient dies from AHRF during their hospital stay. 20.10% of all the patients in Child-AHRF and 13.84% of all patients in Adult-AHRF have a positive mortality label (i.e. the patients who die in hospital).
ICD9 Code Prediction: Each admission record in MIMIC-III dataset has multiple ICD-9 diagnosis codes. We group all the occurrences of the ICD-9 codes into 20 diagnosis groups[2] . For the ICD9 dataset, we are interested in predicting these 20 ICD-9 Diagnosis Categories for each admission record. We treat this as a multi-task prediction problem.
Domain Adaptation Tasks: We study unsupervised domain adaptation (i.e. target domain labels are unavailable during training and validation) task with-in age groups of Adult-AHRF dataset, ICD9 dataset and across Adult and Child-AHRF datasets. For Adult-AHRF and ICD9 datasets, we created 12 source-target domain pairs using the age groups, pairing up each domain Di with another domain Dj 6=i, for example, the source-target pair 2-5 was used for adapting from group 2 (working-age adult) to group 5 (old elderly). We also created 4 source-target pairs for performing domain adaptation from 4 adult age-groups to 1 child age-group.
4.2 METHODS AND IMPLEMENTATION DETAILS
We categorize the methods used in our main experiments into the following groups:
• Non-adaptive baseline methods: Logistic Regression (LR), Adaboost with decision regressors (Adaboost), and feed forward deep neural networks (DNN)
• Deep Domain adaptation methods: Domain Adversarial Neural Networks (DANN) (Ganin et al. (2016)); DANN with a RNN (LSTM) as feature extractor (R-DANN); Variational Fair Autocoder (VFAE)(Louizos et al. (2015))
• Our method: Variational Recurrent Adversarial Deep Domain Adaptation (VRADA)[3] .
[1]:https://www.cms.gov/Research-Statistics-Data-and-Systems/ Statistics-Trends-and-Reports/NationalHealthExpendData/
[2]: http://tdrdata.com/ipd/ipd_SearchForICD9CodesAndDescriptions.aspx. “Conditions Originating in the Perinatal Period” is not present in the preprocessed dataset.
[3]: Codes will be publicly released soon
In all our experiments, we conducted unsupervised domain adaptation where target domain labels are unavailable during training and validation. For R-DANN, we used LSTM(Hochreiter & Schmidhuber (1997)) as the feature extractor network instead of the feed-forward neural networks used in DANN. For VFAE, DANN and all the non-domain adaptive approaches we flattened the time series along time axis and treat it as the input to the model. For fairness, the classifier and feature extractors of the VRADA and R-DANN were equivalent in depth and both had the same model capacity. We also ensure that the size of latent feature representation z̃i are similar for VRADA and DANN models. The model capacity of VFAE was chosen to be similar to VRADA. All the deep domain adaptation models including ours had depth of size 8 (including output classifier layers). We used the Adam optimizer ( Kingma & Ba (2014)) and ran all models for 500 epochs with a learning rate of 3e−4. We set an early stopping criteria that the model does not experience a decrease in the validation loss for 20 epochs. Source domain data was split into train/validation subsets with a 70/30 ratio and target domain data into train/validation/test subsets with a 70/15/15 ratio. In order to compare all the methods, we report AUC scores on the entire target domain set, and the test subset for each target domain data of a source-target pair.
4.3 QUANTITATIVE RESULTS
In Table 1, we compare non domain adaptation and domain adaptation models’ performance on the target domain test subset for the AHRF mortality prediction task. It is immediately clear that domain adaptation methods consistently outperform non domain adaptation methods. We see that generally the VRADA outperforms both variants of the DANN with it consistently seeing scores ∼ 4% higher. While the standard deviation for the VRADA was about 1%, it was about 2% for the R-DANN, further showing our models efficacy as it converges to more stable local optima. Our model VRADA beats state-of-the-art DANN(Ganin et al. (2016)) and VFAE(Louizos et al. (2015)) on all the source-pair domain adaptation tasks for Adult-AHRF dataset. For the domain adaptation from Adult-AHRF to Child-AHRF dataset, we observe that VRADA mostly outperforms all the competing models. This shows that our model can perform well even for smaller target domain datasets.
In the above table, we test classification without adaptation using Logistic Regression (LR), Adaboost with decision tree classifiers and Feed forward Deep Neural Networks (DNN); and with adaptation using Deep Domain Adversarial Neural Networks (DANN), a DANN with an LSTM in its feature extractor (R-DANN), Variational Fair Autoencoder (VFAE) and our Variational Adversarial Domain Adaptation Model (VRADA). All results are reported on the target domain test subset dataset.
As the AHRF mortality prediction task made it clear that domain adaptation is necessary for intergroup adaptation, for the ICD9 multi-task prediction task that involved data with time-steps of length 12, we focused strictly on domain adaptive models (i.e. the DANN, R-DANN, and VRADA). Table 2 shows the aggregated AUC scores on the entire target domain dataset and test data of the target domain for the 20 tasks of the ICD9 Code Prediction task. Here, we clearly see that VRADA and
Here, we compare results for the ICD9 Diagnosis Code Prediction task on the ICD9 dataset. For each model, the top row corresponds to the performance on the entire target domain dataset and the bottom row corresponds to performance on the test subset (15%) of the target domain dataset.
R-DANN models outperform DANN Ganin et al. (2016) by significant margins. We also observe that VRADA outperforms R-DANN by 1.5 ∼ 2% when averaged over all the source-target domain pairs.
4.4 DISCUSSION
Figure 3 shows the temporal latent dependencies captured by our VRADA as compared to the R-DANN for 3-4 source-target pair. While both models learn temporal latent dependencies fairly well, the VRADA outperforms the R-DANN in two ways. First, the VRADA’s neurons learned stronger predictions of whether features are relevant towards modeling the data. If we look at the VRADA row, for both AHRF and ICD9 we see that the neural activation patterns are more consistent across time-steps than for R-DANN. Figure 4 shows the unrolled memory cell states (in the form Examples× (Time ∗ Neurons)) for all the source and target domain data points. We see a consistent activation firing patterns across all these data points for VRADA but not for R-DANN. Together with the stronger performance on 3-4 for AHRF and 2-5 for ICD9, this potentially indicates that VRADA is better learning the temporal dependencies.
Second, nuanced values are consistent across time-steps for the VRADA, exhibiting a gradual transition towards stronger activation with time, whereas the temporal activation pattern of the RDANN seems somewhat sporadic. While activation gradients across time are consistent for both the R-DANN and VRADA, more consistent inhibitory and excitatory neuron firing patterns indicate that the VRADA better transfers knowledge. Another indication of domain adaptation was shown in Figure 1c. Looking at the t-SNE projections of feature representations of DNN, R-DANN, and VRADA we can see that the addition of temporal latent dependencies might help in better mixing of the domain distributions since we observe that the data is more evenly spread out. Figure 1c and Figure 3 together indicate that the VRADA’s temporal latent dependency capturing power and ability to create domain-invariant representations act synergistically. For plots of activation patterns without domain adaptation, please see appendix section 6.2.3.
5 SUMMARY
Because of its diverse range of patients and its episodic and longitudal nature, healthcare data provides a good platform to test domain adaptation techniques for temporal data. With it as our example, we showcase the Variational Recurrent Adversarial Domain Adaptation (VRADA) model’s ability to learn temporal latent representations that are domain-invariant. By comparing our model’s latent representations to others’, we show its ability to use variational methods to capture hidden factors of variation and produce more robust domain-invariant representations. We hope this work serves as a bedrock for future work capturing and adapting temporal latent representations across domains.
ACKNOWLEDGMENTS
This material is based upon work supported by the NSF research grants IIS-1134990, IIS-1254206, Samsung GRO Grant and the NSF Graduate Research Fellowship Program under Grant No. DGE1418060. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies. We also acknowledge Thailand’s Development and Promotion of Science and Technology Talents Project for financial support. We thank Dr. Robinder Khemani for sharing the Child-AHRF dataset.
6 APPENDIX
6.1 TRAINING VARIATIONS
We tested 3 variations of training VRADA: (a) training VRADA regularly as discussed in Section 3 (denoted by I), (b) loading a pretrained VRNN encoder and optimizing strictly off the classification errors, i.e.
E(θe, θy, θd) = 1
n n∑ i=1 Ly(xi; θy)− λ( 1 n n∑ i=1 Ld(xi; θd) + 1 n′ N∑ i=n+1 Ld(xi; θd))) (8)
and (c) loading a pretrained VRNN encoder and using the objective as presented in equation 3 (denoted by III). Key to note is that in method II, we do not apply variational methods towards learning the shared latent representation. This was done to test whether they were helpful or harmful towards the learned latent representation used for classification. In method III, we train VRADA as normal but load a pretrained encoder. We pretrain the encoder by training the VRNN on all source and target domain samples for a desired source-target adaptation pair. In order to choose how many samples would be used for training, we looked at which domain had more examples and chose the larger of the two. For example, if the source domain was group 2 with 508 patients and the target domain was group 5 with 437 patients, the VRNN would see 508 samples of each domain, with group 5 being sampled with replacement after seeing all its samples. As the encoder was used for learning latent representations, we thought it worth investigating whether if pretrained it better captured the latent representations that were being used by the domain classifier for adversarial training. We thought beginning domain classification at a better initialization point might help VRADA avoid local minima. For each method, we fed one source domain sample to Gy and either a source or target domain sample to Gd. (For this training and all training samples, order was randomized.) We only calculated the loss Lr once for the Gd samples so as to not bias the optimization of the VRNN. Table 3 shows the results of AHRF Mortality Prediction task for different types of VRADA training. From these experiments, we found that jointly training VRADA (i.e method I) usually performed better than the other pretrained training approaches.
6.2 MODEL VARIATIONS
6.2.1 ADVERSARIAL TRAINING AT EVERY TIME-STEP
A natural question is whether adversarial training at every time-step is more effective than adversarial training at the last time-step of a latent representation. If done at every time-step, the network learns
to create domain-invariant representations of subsets of your input x≤T . Do these domain-invariant representations help the network find more optimal domain-invariant representations of x? We empirically tested this scenario (Table 4) and found the results to be sub-optimal when compared to only performing adversarial training at the last time-step (Table 1). Below are results for the R-DANN and VRADA models for adversarial training at every time-step.
6.2.2 EFFECT OF RECONSTRUCTION LOSS
Table 5 shows the effect of reconstruction loss for our VRADA model. We observe that reconstructing the original data (i.e. using the decoder for reconstructing the data) helps in the overall performance improvement of our VRADA model.
6.2.3 IMPACT OF ADVERSARIAL TRAINING
In figures 5 and 6 we show the cell state activations for the VRADA and R-DANN without domain adaptation (i.e. no adversarial training). From these figures, we see that the dependencies between source and target domains are not transferred correctly since we do not perform adversarial training. On the otherhand, as discussed in section 4.4, figure 3 shows that adversarial training helps in transferring the dependencies between source and target domains efficiently.
6.3 R-DANN MODEL INFORMATION
Here we provide more details on the network architectures of the R-DANN and DANN. Please refer to Figure 7 for a diagram of the R-DANN model showing the dimensions of each layer and the connections between layers. The R-DANN and DANN were essentially identical except that, for the DANN, the first layer used a fully-connected layer instead of an RNN and took input flattened over the time-dimension. Thus the input dimensions corresponded to f and t× f for the R-DANN and DANN, respectively, where f is the number of features and t is the length of the time-dimension.

The work combines variational recurrent neural networks, and adversarial neural networks to handle domain adaptation for time series data. The proposed method, along with several competing algorithms are compared on two healthcare datasets constructed from MIMIC-III in domain adaptation settings.
The new contribution of the work is relatively small. It extends VRNN with adversarial training for learning domain agnostic representations. From the experimental results, the proposed method clearly out-performs competing algorithms. However, it is not clear where the advantage is coming from. The only difference between the proposed method and R-DANN is using variational RNN vs RNN. Little insights were provided on how this could bring such a big difference in terms of performance and the drastic difference in the temporal dependencies captured by these two methods in Figure 4.
Detailed comments:
1. Please provide more details on what is plotted in Figure 1. Is 1 (b) is the t-sne projection of representations learned by DANN or R-DANN? The text in section 4.4 suggests it’s the later case. It is surprising to see such a regular plot for VRADA.  What do you think are the two dominant latent factors encoded in figure 1 (c)?
2. In Table 2, the two baselines have quite significant difference in performance testing on the entire target (including validation set) vs on the test set only. VRADA, on the other hand, performs almost identical in these two settings. Could you please offer some explanation on this?
3. Please explain figure 3 and 4 in more details. how to interpret the x-axis of figure 3, and the x and y axes of figure 4. Again the right two plots in figure 4 are extremely regular comparing to the ones on the left.


PROGRAM SYNTHESIS FOR CHARACTER LEVEL LANGUAGE MODELING
We propose a statistical model applicable to character level language modeling and show that it is a good fit for both, program source code and English text. The model is parameterized by a program from a domain-specific language (DSL) that allows expressing non-trivial data dependencies. Learning is done in two phases: (i) we synthesize a program from the DSL, essentially learning a good representation for the data, and (ii) we learn parameters from the training data – the process is done via counting, as in simple language models such as n-gram. Our experiments show that the precision of our model is comparable to that of neural networks while sharing a number of advantages with n-gram models such as fast query time and the capability to quickly add and remove training data samples. Further, the model is parameterized by a program that can be manually inspected, understood and updated, addressing a major problem of neural networks.
1 INTRODUCTION
Recent years have shown increased interest in learning from large datasets in order to make accurate predictions on important tasks. A significant catalyst for this movement has been the ground breaking precision improvements on a number of cognitive tasks achieved via deep neural networks. Deep neural networks have made substantial inroads in areas such as image recognition (Krizhevsky et al., 2012) and natural language processing (Józefowicz et al., 2016) thanks to large datasets, deeper networks (He et al., 2016) and substantial investments in computational power Oh & Jung (2004).
While neural networks remain a practical choice for many applications, they have been less effective when used for more structured tasks such as those concerning predictions about programs (Allamanis et al., 2016; Raychev et al., 2014). Initially targeting the programming languages domain, a new method for synthesizing probabilistic models proposed by Bielik et al. (2016), without a neural network, has shown to be effective for modeling source code, and has gained traction.
In this work, we investigate the applicability of this new method to tasks which have so far been addressed with recurrent neural networks and n-gram language models. The probabilistic models we propose are defined by a program from a domain-specific language (DSL). A program in this DSL describes a probabilistic model such as n-gram language models or a variant of it - e.g. trained on subsets of the training data, queried only when certain conditions are met and specialized in making specific classes of predictions. These programs can also be combined to produce one large program that queries different specialized submodels depending on the context of the query.
For example, consider predicting the characters in an English text. Typically, the first character of a word is much more difficult to predict than other characters and thus we would like to predict it differently. Let f be a function that takes a prediction position t in a text x and returns a list of characters to make prediction on. For example, let f be defined as follows:
f(t, x) = if xt−1 is whitespace then xs else xt−1xt−2
where xs is the first character of the word preceding the predicted word at position t. Now, consider a model that predicts a character by estimating each character xt from a distribution P (xt|f(t, x)). For positions that do not follow a whitespace, this distribution uses the two characters preceding xt and thus it simply encodes a trigram language model (in this example, without backoff, but we
consider backoff separately). However, P (xt|f(t, x)) combines the trigram model with another interesting model for the samples in the beginning of the words where trigram models typically fail.
More generally, f is a function that describes a probabilistic model. By simply varying f , we can define trivial models such as n-gram models, or much deeper and interesting models and combinations. In this work, we draw f from a domain-specific language (DSL) that resembles a standard programming language: it includes if statements, limited use of variables and one iterator over the text, but overall that language can be further extended to handle specific tasks depending on the nature of the data. The learning process now includes finding f from the DSL such that the model P (xt|f(t, x)) performs best on a validation set and we show that we can effectively learn such functions using Markov chain Monte Carlo (MCMC) search techniques combined with decision tree learning.
Advantages An advantage of having a function f drawn from a DSL is that f becomes humanly readable, in contrast to neural networks that generally provide non-human readable matrices (Li et al., 2016). Further, the training procedure is two-fold: first, we synthesize f from the DSL, and then for a given f , we estimate probabilities for P (xt|f(t, x)) by counting in the training data. This gives us additional advantages such as the ability to synthesize f and learn the probability distribution P on different datasets: e.g., we can easily add and remove samples from the dataset used for computing the probability estimate P . Finally, because the final model is based on counting, estimating probabilities P (xt|f(t, x)) is efficient: applying f and looking up in a hashtable to determine how frequently in the training data, xt appears in the resulting context of f(t, x).
Before we continue, we note an important point about DSL-based models. In contrast to deep neural networks that can theoretically encode all continuous functions (Hornik, 1991), a DSL by definition targets a particular application domain, and thus comes with restricted expressiveness. Increasing the expressibility of the DSL (e.g., by adding new instructions) can in theory make the synthesis problem intractable or even undecidable. Overall, this means that a DSL should balance between expressibility and efficiency of synthesizing functions in it (Gulwani, 2010).
Contributions This work makes two main contributions:
• We define a DSL which is useful in expressing (character level) language models. This DSL can express n-gram language models with backoff, with caching and can compose them using if statements in a decision-tree-like fashion. We also provide efficient synthesis procedures for functions in this DSL.
• We experimentally compare our DSL-based probabilistic model with state-of-the-art neural network models on two popular datasets: the Linux Kernel dataset (Karpathy et al., 2015) and the Hutter Prize Wikipedia dataset (Hutter, 2012).
2 A DSL FOR CHARACTER LEVEL LANGUAGE MODELING
We now provide a definition of our domain specific language (DSL) called TChar for learning character level language models. At a high level, executing a program p ∈ TChar at a position t ∈ N in the input sequence of characters x ∈ X returns an LMProgram that specifies language model to be used at position t. That is, p ∈ TChar : N×X → LMProgram. The best model to use at position t is selected depending on the current program state (updated after processing each character) and conditioning on the dynamically computed context for position t of the input text x. This allows us to train specialized models suited for various types of prediction such as for comments vs. source code, newlines, indentation, opening/closing brackets, first character of a word and many more. Despite of the fact that the approach uses many specialized models, we still obtain a valid probability distribution as all of these models operate on disjoint data and are valid probability distributions. Subsequently, the selected program f ∈ LMProgram determines the language model that estimates the probability of character xt by building a probability distribution P (xt|f(t, x)). That is, the probability distribution is conditioned on the context obtained by executing the program f .
The syntax of TChar is shown in Fig. 1 and is designed such that it contains general purpose instructions and statements that operate over a sequence of characters. One of the advantages of our approach is that this language can be further refined by adding more instructions that are specialized for a given domain at hand (e.g., in future versions, we can easily include set of instructions specific
Program State state ∈ N Accumulated Context v ∈ N∗ Backoff Threshold d ∈ 〈0, 1〉 TChar ::= SwitchProgram | StateProgram | return LMProgram
StateProgram ::= StateUpdate; StateSwitch
to modeling C/C++ source code). We now informally describe the general TChar language of this work. We provide a formal definition and semantics of the TChar language in the appendix.
2.1 SIMPLEPROGRAMS
The SimpleProgram is a basic building block of the TChar language. It describes a loop-free and branch-free program that accumulates context with values from the input by means of navigating within the input (using Move instructions) and writing the observed values (using Write instructions). The result of executing a SimpleProgram is the accumulated context which is used either to condition the prediction, to update the program state or to determine which program to execute next.
Move Instructions We define four basic types of Move instructions – LEFT and RIGHT that move to the previous and next character respectively, PREV CHAR that moves to the most recent position in the input with the same value as the current character xt and PREV POS which works as PREV CHAR but only considers positions in the input that are partitioned into the same language model. Additionally, for each character c in the input vocabulary we generate instruction PREV CHAR(c) that navigates to the most recent position of character c. We note that all Move instructions are allowed to navigate only to the left of the character xt that is to be predicted.
Write Instructions We define three Write instructions – WRITE CHAR that writes the value of the character at current position, WRITE HASH that writes a hash of all the values seen between the current position and the position of last Write instruction, and WRITE DIST that writes a distance (i.e., number of characters) between current position and the position of last Write instruction. In our implementation we truncate WRITE HASH and WRITE DIST to a maximum size of 16.
Example With Write and Move instructions, we can express various programs that extract useful context for a given position t in text x. For example, we can encode the context used in trigram language model with a program LEFT WRITE CHAR LEFT WRITE CHAR. We can also express programs such as LEFT PREV CHAR RIGHT WRITE CHAR that finds the previous occurrence of the character on the left of the current position and records the character following it in the context.
2.2 SWITCHPROGRAMS
A problem of using only one SimpleProgram is that the context it generates may not work well for the entire dataset, although combining several such programs can generate suitable contexts for the different types of predictions. To handle these cases, we introduce SwitchProgram with switch statements that can conditionally select appropriate subprograms to use depending on the context of the prediction. The checked conditions of switch are themselves program pguard ∈ SimpleProgram that accumulate values that are used to select the appropriate branch that should be executed next. During the learning the goal is then to synthesize best program pguard to be used as a guard, the values v1, · · · , vn used as branch conditions as well as the programs to be used in each branch. We note that we support disjunction of values within branch conditions. As a result, even if the same program is to be used for two different contexts v1 and v2, the synthesis procedure can decide whether a single model should be trained for both (by synthesizing a single branch with a disjunction of v1 and v2) or a separate models should be trained for each (by synthesizing two branches).
Example We now briefly discuss some of the BranchProgram synthesized for the Linux Kernel dataset in our experiments described in Section 3. By inspecting the synthesized program we identified interesting SimplePrograms building blocks such as PREV CHAR( ) RIGHT WRITE CHAR that conditions on the first character of the current word, PREV CHAR(\n) WRITE DIST that conditions on the distance from the beginning of the line or PREV CHAR( ) LEFT WRITE CHAR that checks the preceding character of a previous underscore (useful for predicting variable names). These are examples of more specialized programs that are typically found in the branches of nested switches of a large TChar program. The top level switch of the synthesized program used the character before the predicted position (i.e. switch LEFT WRITE CHAR) and handles separately cases such as newline, tabs, special characters (e.g., !#@.∗), upper-case characters and the rest.
2.3 STATEPROGRAMS
A common difficulty in building statistical language models is capturing long range dependencies in the given dataset. Our TChar language partially addresses this issue by using Move instructions that can jump to various positions in the data using PREV CHAR and PREV POS instructions. However, we can further improve by explicitly introducing a state to our programs using StateProgram. The StateProgram consists of two sequential operations – updating the current state and determining which program to execute next based on the value of the current state. For both we reuse the switch construct defined previously for SwitchProgram. In our work we consider integer valued state that can by either incremented, decremented or left unmodified after processing each input character. We note that other definitions of the state, such as stack based state, are possible.
Example As an example of a StateProgram consider the question of detecting whether the current character is inside a comment or is a source code. These denote very different types of data that we might want to model separately if it leads to improvement in our cost metric. This can be achieved by using a simple state program with condition LEFT WRITE CHAR LEFT WRITE CHAR that increments the state on ’/*’, decrements on ’*/’ and leaves the state unchanged otherwise.
2.4 LMPROGRAMS
The LMProgram describes a probabilistic model trained and queried on a subset of the data as defined by the branches taken in the SwitchPrograms and StatePrograms. The LMProgram in TChar is instantiated with a language model described by a SimpleProgram plus backoff. That is, the prediction is conditioned on the sequence of values returned by executing the program, i.e. P (xt|f(t, x)). Recall that given a position t in text x, executing a SimpleProgram returns context f(t, x). For example, executing LEFT WRITE CHAR LEFT WRITE CHAR returns the two characters xt−1xt−2 preceding xt. In this example P (xt|f(t, x)) is a trigram model. To be effective in practice, however, such models should support smoothing or backoff to lower order models. We provide backoff in two ways. First, because the accumulated context by the SimpleProgram is a sequence we simply backoff to a model that uses a shorter sequences by using Witten-Bell backoff (Witten & Bell, 1991). Second, in the LMProgram we explicitly allow to backoff to other models specified by a TChar pro-
gram if the probability of the most likely character from vocabulary V according to the P (xt|f(t, x)) model is less than a constant d. Additionally, for some of our experiments we also consider backoff to a cache model (Kuhn & De Mori, 1990).
Predicting Out-of-Vocabulary Labels Finally, we incorporate a feature of the language models as proposed by Raychev et al. (2016a) that enables us not only to predict characters directly but instead to predict a character which is equal to some other character in the text. This is achieved by synthesising a pair of (SimpleProgram, SimpleProgram). The first program is called equality program and it navigates over the text to return characters that may be equal to the character that we are trying to predict. Then, the second program f describes P (xt|f(t, x)) as described before, except that a possible output is equality to one of the characters returned by the equality program.
2.5 SYNTHESISING TChar PROGRAMS
The goal of the synthesizer is given a set of training and validation samples D, to find a program:
pbest = arg min p∈TChar cost(D, p)
where cost(D, p) = −logprob(D, p) + λ · Ω(p). Here logprob(D, p) is the log-probability of the trained models on the dataset D and Ω(p) is a regularization that penalizes complex functions to avoid over-fitting to the data. In our implementation, Ω(p) returns the number of instructions in p.
The language TChar essentially consists of two fragments: branches and straight-line SimplePrograms. To synthesize branches, we essentially need a decision tree learning algorithm that we instantiate with the ID3+ algorithm as described in Raychev et al. (2016a). To synthesize SimplePrograms we use a combination of brute-force search for very short programs (up to 5 instructions), genetic programming-based search and Markov chain Monte Carlo-based search. These procedures are computationally feasible, because each SimpleProgram consists of only a small number of moves and writes. We provide more details about this procedure in Appendix B.1.
3 EXPERIMENTS
Datasets For our experiments we use two diverse datasets: a natural language one and a structured text (source code) one. Both were previously used to evaluate character-level language models – the Linux Kernel dataset Karpathy et al. (2015) and Hutter Prize Wikipedia dataset Hutter (2012). The Linux Kernel dataset contains header and source files in the C language shuffled randomly, and consists of 6, 206, 996 characters in total with vocabulary size 101. The Hutter Prize Wikipedia dataset contains the contents of Wikipedia articles annotated with meta-data using special mark-up (e.g., XML or hyperlinks) and consists of 100, 000, 000 characters and vocabulary size 205. For both datasets we use the first 80% for training, next 10% for validation and final 10% as a test set.
Evaluation Metrics To evaluate the performance of various probabilistic language models we use two metrics. Firstly, we use the bits-per-character (BPC) metric which corresponds to the negative log likelihood of a given prediction E[− log2 p(xt | x<t)], where xt is character being predicted and x<t denotes characters preceding xt. Further, we use error rate which corresponds to the ratio of mistakes the model makes. This is a practical metric that directly quantifies how useful is the model in a concrete task (e.g., completion). As we will see, having two different evaluation metrics is beneficial as better (lower) BPC does not always correspond to better (lower) error rate.
3.1 LANGUAGE MODELS
We compare the performance of our trained DSL model, instantiated with the TChar language described in Section 2, against two widely used language models – n-gram model and recurrent neural networks. For all models we consider character level modeling of the dataset at hand. That is, the models are trained by feeding the input data character by character, without any knowledge of higher level word boundaries and dataset structure.
N-gram We use the n-gram model as a baseline model as it has been traditionally the most widely used language model due to it simplicity, efficient training and fast sampling. We note that ngram can be trivially expressed in the TChar language as a program containing a sequence of LEFT and WRITE instructions. To deal with data sparseness we have experimented with various smoothing techniques including Witten-Bell interpolation smoothing Witten & Bell (1991) and modified Kneser-Ney smoothing Kneser & Ney (1995); Chen & Goodman (1998).
Recurrent Neural Networks To evaluate the effectiveness of the DSL model we compare to a recurrent network language model shown to produce state-of-the-art performance in various natural language processing tasks. In particular, for the Linux Kernel dataset we compare against a variant of recurrent neural networks with Long Short-Term Memory (LSTM) proposed by Hochreiter & Schmidhuber (1997). To train our models we follow the experimental set-up and use the implementation of Karpathy et al. (2015). We initialize all parameters uniformly in range [−0.08, 0.08], use mini-batch stochastic gradient descent with batch size 50 and RMSProp Dauphin et al. (2015) perparameter adaptive update with base learning rate 2 × 10−3 and decay 0.95. Further, the network is unrolled 100 time steps and we do not use dropout. Finally, the network is trained for 50 epochs (with early stopping based on a validation set) and the learning rate is decayed after 10 epochs by multiplying it with a factor of 0.95 each additional epoch. For the Hutter Prize Wikipedia dataset we compared to various other, more sophisticated models as reported by Chung et al. (2017).
DSL model To better understand various features of the TChar language we include experiments that disable some of the language features. Concretely, we evaluate the effect of including cache and backoff. In our experiments we backoff the learned program to a 7-gram and 3-gram models and we use cache size of 800 characters. The backoff thresholds d are selected by evaluating the model performance on the validation set. Finally, for the Linux Kernel dataset we manually include a StateProgram as a root that distinguishes between comments and code (illustrated in Section 2.3). The program learned for the Linux Kernel dataset contains ≈ 700 BranchPrograms and ≈ 2200 SimplePrograms and has over 8600 Move and Write instructions in total. We provide an interactive visualization of the program and it’s performance on the Linux Kernel dataset online at:
www.srl.inf.ethz.ch/charmodel.html
3.2 MODEL PERFORMANCE
We compare the performance of the DSL model, n-gram and neural networks for the tasks of learning character level language models by discussing a number of relevant metrics shown in Table 1 and Table 2.
Precision In terms of model precision, we can see that as expected, the n-gram model performs worse in both BPC and error rate metrics. However, even though the best BPC is achieved for a 7-gram model, the error rate decreases up to 15-gram. This suggests that none of the smoothing techniques we tried can properly adjust to the data sparsity inherent in the higher order n-gram models. It is however possible that more advanced smoothing techniques such as one based on PitmanYor Processes (Teh, 2006) might address this issue. As the DSL model uses the same smoothing technique as n-grams, any improvement to smoothing is directly applicable to it.
As reported by Karpathy et al. (2015), the LSTM model trained on the Linux Kernel dataset improves BPC over the n-gram. However, in our experiments this improvement did not translate to lower error rate. In contrast, our model is superior to n-gram and LSTM in all configurations, beating the best other model significantly in both evaluation metrics – decreasing BPC by over 0.5 and improving error rate by more than 12%.
For the Hutter Prize Wikipedia dataset, even though the dataset consists of natural language text and is much less structured than the Linux Kernel, our model is competitive with several neural network models. Similar to the results achieved on Linux Kernel, we expect the error rate of the DSL model for Hutter Prize Wikipedia dataset, which is 30.5%, to be comparable to the error rate achieved by other models. However, this experiment shows that our model is less suitable to unstructured text such as the one found on Wikipedia.
Training Time Training time is dominated by the LSTM model that takes several days for the network with the highest number of parameters. On the other hand, training n-gram models is extremely fast since the model is trained simply by counting in a single pass over the training data. The DSL model sits between these two approaches and takes ≈ 8 hours to train. The majority of the DSL training time is spent in the synthesis of SwitchPrograms where one needs to consider a massive search space of possible programs from which the synthesis algorithm aims to find one that is approximately the best (e.g., for Linux Dataset the number of basic instructions is 108 which means that naive enumeration of programs up to size 3 already leads to 1083 different candidate programs). All of our experiments were performed on a machine with Intel(R) Xeon(R) CPU E52690 with 14 cores. All training times are reported for parallel training on CPU. Using GPUs for training of the neural networks might provide additional improvement in training time.
Query (Sampling) Time Sampling the n-gram is extremely fast, answering≈ 46, 000 queries per second, as each query corresponds to only a several hash look-ups. The query time for the DSL model is similarly fast, in fact can be even faster, by answering ≈ 62, 000 queries per second. This is because it consists of two steps: (i) executing the TChar program that selects a suitable language model (which is very fast once the program has been learned), and (ii) querying the language model. The reason why the model can be faster is that there are fewer hash lookups which are also faster due to the fact that the specialized language models are much smaller compared to the n-gram (e.g.,
≈ 22% of the models simply compute unconditioned probabilities). Adding backoff and cache to the DSL model decreases sampling speed, which is partially because our implementation is currently not optimized for querying and fully evaluates all of the models even though that is not necessary.
Model Size Finally, we include the size of all the trained models measured by the size in MB of the model parameters. The models have roughly the same size except for the n-gram models with high order for which the size increases significantly. We note that the reason why both the n-gram and DSL models are relatively small is that we use hash-based implementation for storing the prediction context. That is, in a 7-gram model the previous 6 characters are hashed into a single number. This decreases the model size significantly at the expense of some hash collisions.
4 RELATED WORK
Program Synthesis Program synthesis is a well studied research field in which the goal is to automatically discover a program that satisfies a given specification that can be expressed in various forms including input/output examples, logical formulas, set of constraints or even natural language description.In addition to the techniques that typically scale only for smaller programs and attempt to satisfy the specification completely (Alur et al., 2013; Solar-Lezama et al., 2006; Solar-Lezama, 2013; Jha et al., 2010) a recent line of work considers a different setting consisting of large (e.g., millions of examples) and noisy (i.e., no program can satisfy all examples perfectly) datasets. This is the setting that needs to be considered when synthesizing programs in our setting and in a language such as TChar. Here, the work of Raychev et al. (2016b) showed how to efficiently synthesize straight-line programs and how to handle noise, then Raychev et al. (2016a) showed how to synthesize branches, and thework of Heule et al. (2015) proposed a way to synthesize loops.
In our work we take advantage of these existing synthesis algorithms and use them to efficiently synthesize a program in TChar. We propose a simple extension that uses MCMC to sample from a large amount of instructions included in TChar (many more than prior work). Apart from this, no other modifications were required in order to use existing techniques for the setting of character level language modeling we consider here.
Recurrent Neural Networks Recent years have seen an emerging interest in building a neural language models over words (Bengio et al., 2003), characters (Karpathy et al., 2015; Sutskever et al., 2011; Wu et al., 2016) as well as combination of both (Chung et al., 2017; Kim et al., 2015; Mikolov et al., 2012). Such models have been shown to achieve state-of-the-art performance in several domains and there is a significant research effort aimed at improving such models further.
In contrast, we take a very different approach, one that aims to explain the data by means of synthesizing a program from a domain specific language. Synthesising such programs efficiently while achieving competitive performance to the carefully tuned neural networks creates a valuable resource that be used as a standalone model, combined with existing neural language models or even used for their training. For example, the context on which the predictions are conditioned is similar to the attention mechanism (Sukhbaatar et al., 2015; Bahdanau et al., 2014) and might be incorporated into that training in the future.
5 CONCLUSION
In this paper we proposed and evaluated a new approach for building character level statistical language models based on a program that parameterizes the model. We design a language TChar for character level language modeling and synthesize a program in this language. We show that our model works especially well for structured data and is significantly more precise that prior work. We also demonstrate competitive results in the less structured task of modeling English text.
Expressing the language model as a program results in several advantages including easier interpretability, debugging and extensibility with deep semantic domain knowledge by simply incorporating a new instruction in the DSL. We believe that this development is an interesting result in bridging synthesis and machine learning with much potential for future research.

This paper proposes an approach to character language modeling (CLMs) based on developing a domain specific language to represent CLMs. The experiments show mixed performance versus neural CLM approaches to modeling linux kernel data and wikipedia text, however the proposed DSL models are slightly more compact and fast to query as compared with neural CLMs. The proposed approach is difficult to understand overall and perhaps is aimed towards the sub-community already working on this sort of approach but lacks sufficient explanation for the ICLR audience. Critically the paper glosses over the major issues of demonstrating the proposed DSL is a valid probabilistic model and how training is performed to fit the model to data (there is clearly not a gradient-based training approach used). FInally the experiments feel incomplete without showing samples drawn from the generative model or analyzing the learned model to determine what it has learned. Overall I feel this paper does not describe the approach in enough depth for readers to understand or re-implement it.
Almost all of the model section is devoted to exposition of the DSL without specifying how probabilities are computed using this model and how training is performed. How are probabilities actually encoded? The DSL description seems to have only discrete decisions rather than probabilities.
Training is perhaps covered in previous papers but there needs to be some discussion of how it works here. Section 2.5 does not do enough to explain how training works or how any measure of optimality is achieved.
Given this model is quite a different hypothesis space from neural models or n-grams, looking and samples drawn from the model seems critical. The current experiments show it can score utterances relatively well but it would be very interesting if the model can sample more structured samples than neural approaches (for example long-range syntax constraints like brackets)