SAMPLE IMPORTANCE IN TRAINING DEEP NEURAL NETWORKS
The contribution of each sample during model training varies across training iterations and the model’s parameters. We define the concept of sample importance as the change in parameters induced by a sample. In this paper, we explored the sample importance in training deep neural networks using stochastic gradient descent. We found that “easy” samples – samples that are correctly and confidently classified at the end of the training – shape parameters closer to the output, while the “hard” samples impact parameters closer to the input to the network. Further, “easy” samples are relevant in the early training stages, and “hard” in the late training stage. Further, we show that constructing batches which contain samples of comparable difficulties tends to be a poor strategy compared to maintaining a mix of both hard and easy samples in all of the batches. Interestingly, this contradicts some of the results on curriculum learning which suggest that ordering training examples in terms of difficulty can lead to better performance.
The contribution of each sample during model training varies across training iterations and the model’s parameters. We define the concept of sample importance as the change in parameters induced by a sample. In this paper, we explored the sample importance in training deep neural networks using stochastic gradient descent. We found that “easy” samples – samples that are correctly and confidently classified at the end of the training – shape parameters closer to the output, while the “hard” samples impact parameters closer to the input to the network. Further, “easy” samples are relevant in the early training stages, and “hard” in the late training stage. Further, we show that constructing batches which contain samples of comparable difficulties tends to be a poor strategy compared to maintaining a mix of both hard and easy samples in all of the batches. Interestingly, this contradicts some of the results on curriculum learning which suggest that ordering training examples in terms of difficulty can lead to better performance.
1 INTRODUCTION
Sample importance is the sample’s contribution to the parameter change during training. In statistics, the concept “leverage” of a point is used (St Laurent & Cook (1992)) to measure the impact of a sample on the training of a model. In the context of SVM, the most important samples are the support vectors as they define the separating hyperplane. Understanding the importance of the samples can help us interpret trained models and structure training to speed up convergence and improve prediction accuracy. For example, Curriculum learning (CL) from Bengio et al. (2009) shows that training with easy samples first, then gradually transitioning to difficult samples can improve the learning. In CL, the “easiness” of a sample is predefined either manually or using an evaluation model. Self-paced learning (SPL) (Kumar et al. (2010)) shows that it is possible to learn from samples in order of easiness. In this framework, easiness is related to the prediction error and can be estimated from the model. However, easiness of a sample may not be sufficient to decide when it should be introduced to a learner. Maintaining diversity among the training samples can have a substantial effect on the training (Jiang et al. (2014)).
In this work, we explore the sample importance in deep neural networks. Deep learning methods have been successfully applied in many tasks and routinely achieve better generalization error than classical shallow methods (LeCun et al. (2015)). One of the key characteristics of a deep network is its capacity to construct progressively more complex features throughout its layers (Lee et al. (2011)). An intuitive question arises: which samples contribute the most to the training of the different layer’s parameters? From literature Saxe et al. (2011), we know that even randomly generated filters can compute features that lead to good performance – presumably on easy samples. However, to learn hard samples correctly, the model may need to construct complex features, which require both more training time and refined filters from bottom layers. Hence, we hypothesized that the hard samples shape the bottom layers – closer to the input – and easy samples shape the top layers – closer to the output.
Motivated by the above hypothesis, we analyzed the sample importance in a 3 layer ReLU network on two standard datasets. The results reveal several interesting facts about the sample importance in easy and hard samples:
1. Easy and hard samples impact the parameters in different training stages. The biggest impact of easy samples on parameters are mostly during the early training stage, while the impact of hard samples become large in the late training stage.
2. Easy and hard samples impact the parameters in different layers. Easy samples impact have larger impact on top layer parameters, while hard samples shape the bottom layer parameters.
3. Mixing hard samples with easy samples in each batch helps training. We conducted batches with homogeneous or mixed “easiness”. We found that use of homogeneous batches hinders the training. Hence, it is preferable for network to see both easy and hard samples during all stages of training.
Next, we are going to give the definition of sample importance in Section 2. The empirical analysis for sample importance in the deep neural network in two real datasets is discussed in Section 3. Extension about sample importance is showed in Section 4.
2 SAMPLE IMPORTANCE
In this section, we are going to introduce the terminology and provide a quantitative measurement of sample importance for a training procedure.
2.1 SAMPLE WEIGHT
In supervised learning, a model is trained by optimizing an objective over a set of observed training samples (xi, yi). Let f(xi,θ) be the output of a model for parameter θ. The training objective can be written as:
n∑ i=1 L(yi, f(xi,θ)) +R(θ), (1)
where L(yi, f(xi,θ)) is the loss on sample i, and R(θ) is the regularization on the parameters. In order to highlight contribution of each sample, we can introduce sample specific weights vi ∈ [0, 1] which scale sample’s contribution to the loss. Hence, the objective in (1) can be rewritten as:
n∑ i=1 viL(yi, f(xi,θ)) +R(θ), (2)
We define the weight vi as the sample weight. Similar definitions on vi has been proposed in Self-paced learning (SPL) Kumar et al. (2010).
In Stochastic Gradient descend (SGD) methods, parameters θ are updated with a certain step size η in each iteration with regard to a set of training samples. If we allow different sample weights in different iterations, a single update can be written as:
θt+1 = θt − η n∑
i=1
vtig t i − ηrt,
where θt is the parameter vector at epoch t, gti = ∂ ∂θt Li(yi, f(xi,θ t)), rt = ∂ ∂θt R(θt), and vti is the weight of ith sample at iteration t.
2.2 SAMPLE IMPORTANCE
If we change the weight of a sample i at iteration t, how would such change impact the parameter training in that iteration? We can answer this question by calculating the first order derivative of parameter change ∆θt = θt+1 − θt with regard to sample weight vti :
φti = ∂
∂vti ∆θt = −ηgti .
We call φti the parameter affectibility by ith sample at iteration t. φ t i is a vector consists of parameter affectibility from all parameters in the network. Specifically, φti,j is the parameter affectibility for jth parameter in the network. φti reflects the relationship between parameter change and different samples.
Typical deep networks contains millions of parameters. Hence, we are going to focus on groups of parameters of interests. We define ith sample’s importance for parameters of dth layer of as:
βti,d = ∑ j∈Qd (φti,j) 2,
where Qd is a set consists of the indexes of all parameters in layer d. Hence, sample’s importance for all the parameters in the model is:
αti = ∑ j (φti,j) 2,
Obviously, we have αti = ∑D d=1 β t i,d.
The sum of sample’s importance across all iterations is defined as overall importance of a sample: τi = ∑ t αti
In general, for each sample i, computing βti,d allows us to decompose its influence in the model’s training across training stages and different layers.
We note that the sample importance is a high-level measurement of the samples influence on parameters at each iteration of the update. This quantity is not an accurate measurement of the relationship between a sample and final trained model. Refinements of this concept are discussed in Section 4.
3 EMPIRICAL ANALYSIS OF SAMPLE IMPORTANCE
We are going to explore the samples’ importance for different layers at different epoch through a series of empirical experiments on two standard datasets.
3.1 EXPERIMENT SETUP
Dataset All the analysis are performed on two standard datasets: MNIST 1 (LeCun et al. (1998)), a benchmark dataset that contains handwritten digit images. Each sample is a 28 × 28 image from 10 classes. We used 50000 samples for training and 10000 samples for testing. CIFAR-10 2 (Krizhevsky & Hinton (2009)), a dataset contains 32× 32 tiny color images from 10 classes. Each sample has 3072 features. We used 50000 samples for training and 10000 samples for testing.
Architecture We used a multilayer feed forward neural network with 3 hidden layers of 512 hidden nodes with rectified linear units (ReLU) activation function, a linear output layer, and a softmax layer on top for classification. The weights in each hidden layer are initialized according to Glorot & Bengio (2010). For hyper-parameters, we used learning rate of 0.1, batch size of 100, 50 total epochs, and weight-decay of 1e − 5. No momentum or learning decay was used. All the code are based on a common deep learning package Theano from Bergstra et al. (2010); Bastien et al. (2012).
3.2 SAMPLE IMPORTANCE IS STABLE WITH RESPECT TO DIFFERENT INITIALIZATIONS
Firstly, we want to explore whether the sample importance is stable under different initializations. We used three different random seeds to initialize the network parameters and calculated the sample importance every five epochs. We computed the Spearman’s rank correlation between sample importance to the model, αti, in each pair of initializations. This correlation remains high in all epochs,
1http://yann.lecun.com/expdb/mnist/ 2https://www.cs.toronto.edu/˜kriz/cifar.html
above 0.9, as shown in Figure 1. This indicates that the sample importance is relatively stable to initialization of the network. Hence, all the following analysis are based on the results from initialization seed 1. (Details of training and test error for the chosen model can be viewed in Appendix Figure 9).
3.3 DECOMPOSITION OF SAMPLE IMPORTANCE
To better understand and visualize the sample importance, we firstly calculate the overall sample importance at each epoch as At = ∑n i=1 α t i. Similarly, the overall sample importance to layer d
is Btd = ∑n i=1 β t i,d. We show the overall sample importance and its decomposition in layers for two datasets in Figure 2. Firstly, we found that even with a fixed learning rate, the overall sample importance is different under different epochs. Output layer always has the largest average sample importance per parameter, and its contribution reaches the maximum in the early training stage and then drops. Each layer contributes differently to the total sample importance. In both MNIST and CIFAR-10, the 2nd layer contributes more than the 3rd layer. In CIFAR-10, the 1st layer’s provides largest contribution the total sample importance, as it contains much more parameters than other layers. Interestingly, all classes do not provide the same amount of sample importance.
We found that most samples have small sample importance (Appendix Figure 10). To visualize the contribution of different samples, we split the samples based on their total importance into three groups: 10%, top 10% – top 20% most important samples, and other 80% samples. We show the decomposition of importance contribution in each layer in Figure 3. In MNIST, the top 10% samples contribute almost all the sample importance. In CIFAR-10, most important samples contribute more in lower layers rather than output layer. This result indicates that top 20 % most important samples contribute to the majority of the sample importance.
3.4 SAMPLE IMPORTANCE AND NEGATIVE LOG-LIKELIHOOD
Negative log likelihood (NLL) is the loss metric we used for training objective. It has been used to measure the “easiness” of a sample in Curriculum learning Bengio et al. (2009) and Self-paced learning Kumar et al. (2010). Intuitively, the samples with large NLL should also have large sample importance (SI). However, in our experiment, we found that this is not always the case. In Figure 4, we found that 1) NLL and SI become more correlated as training goes on. However, 2) NLL is not predictive of the SI. There are many points with high NLL but small SI, and otherwise.
3.5 CLUSTERING SAMPLES BASED ON SAMPLE IMPORTANCE
To better visualize the importance of different samples, we provide three representative clusters of samples for each dataset. In MNIST, we clustered all digit “5” samples into 20 clusters based on their epoch-specific, layer-specific sample importance. In CIFAR-10, we clustered all “horse” samples into 30 clusters using the same features as MNIST. Kmeans algorithm is used for clustering.
MNIST In Figure 5, we showed 3 example clusters on digit “5”. In the cluster of easy samples, where NLL converges very fast, most of the sample importance is concentrated in the first few epochs in output layer parameters. The cluster of medium samples has a slow NLL convergence compared to the easy cluster. The biggest impact is in middle training stage and more towards bottom layer parameters. Hard samples hardly converge even during the late stage of training. As training goes on, the sample importance for the bottom layer parameters become larger.
CIFAR-10 In Figure 6, we showed 3 examples clusters on class “horse”. We observed very similar sample importance changing pattern as for the MNIST examples for easy, medium and hard clusters. Comparing to MNIST, all three clusters in CIFAR-10 have a very large impact on the parameters in the bottom layer. We note that the CIFAR-10 has almost 4 times larger number of parameters (3075× 512 ≈ 1574k) in the first layer than MNIST (785× 512 ≈ 401k).
3.6 BATCH ORDER AND SAMPLE IMPORTANCE
With the observations from empirical analysis on sample importance, we know that time – iteration – and place – layer – of sample’s impact varies according to its “easiness” . We wanted to know whether constructing batches based on the sample importance or negative log likelihood would make a difference in training. Hence, we designed an experiment to explore how the different construction of batches influence the training. We note that the information used to structure the batches (negative log-likelihood and sample importance) – was obtained from a full training run.
We split all 50, 000 samples into b = 500 batch subsets {B1,B2, . . . ,Bb}. Each batch has batch size |Bi| = 100. In our experiment, each training sample must be in exactly one batch. There is no intersection between batches.
During training, in each epoch, we update the parameters with each batch in order 1, 2, . . . , b iteratively.
We used seven different batch construction methods in this experiment:
1. Rand: Randomly constructed batch. All 50k samples are randomly split into b batches before training. The batches and orders stay fixed during training.
2. NLO: Negative Log-likelihood Order. We sort all the samples based on their final NLL from low to high. The batches are constructed based on the sorted samples. First 100 samples with least NLL are in B1, 101 to 200 samples are in B2, and so on. Hence, during training, the batches with small NLL will be trained first.
3. RNLO Reverse-Negative Log-likelihood Order. We construct the batches same as NLO. During training, we update the batches in reverse order Bb,Bb−1, . . . ,B1. Hence, the batches with large NLL will be trained first.
4. NLM Negative Log-likelihood Mixed. We sort all the samples based on their final NLL from low to high. Next, for each sample i in the sorted sequence, we put that sample into batch j = (i mod b) + 1. This ordering constructs batches out of samples with diverse levels of NLL.
5. SIO: Sample Importance Order. Similar to NLO, except that we sort all the samples based on their sum sample importance over all epochs from low to high. Hence, batches with small sample importance will be trained first.
6. RSIO Reverse-Sample Importance Order. We construct the batches same as SIO. During training, we update the batches in reverse order Bb,Bb−1, . . . ,B1. Hence, during training, the batches with large sample importance will be trained first.
7. SIM Sample Important Mixed. Similar to NLM, but we sort the samples based on overall sample importance. Thus, batches contain samples with divers sample importance.
We performed five different runs (with different random initializations) on MNIST and CIFAR-10. The result is shown in Figure 7. From the result, we found that: 1) In both MNIST and CIFAR-10, Rand, SIS, and NLS have the lowest test error compared to all other methods. This indicates that diverse batches are helpful for training. 2) NLO and SIO got the worst performance in CIFAR10. Their training error even goes up after the early stage. RNLO and RSIO have same batch constructions as NLO and SIO, but their performances are drastically different. This indicates that the order of batches during training is important. Further, training on easy samples first and hard later seems to be counter-productive.
To better understand the impact of different batch construction, we performed the principle component analysis on the learned parameters in each epoch (Figure 8). In MNIST, the impact of batch construction is not very significant. In CIFAR-10, batch construction and even the order of batch training do have a large impact on the training.
Our experiment result shows a different conclusion to Curriculum Learning and Self-paced learning, where easy samples are trained on before introducing hard samples. We found that constructing and ordering the batches – hard to easy and easy to hard – seems to hinder the performance of learning. Having hard samples mixed in with the easy ones in each batch helps the training.
Also, the results show that we want to learn from the hard samples in early epochs and “see” hard samples more frequently, even if their major impact on parameters is during the late stage. As hard
examples are few compared to easy samples and hard examples need a longer time to train, we do want to mix the hard samples into each batch to start learning from those samples early and learn longer.
4 EXTENSIONS OF SAMPLE IMPORTANCE
We calculated the sample importance in each iteration in Stochastic Gradient Descent. However, such quantity only reflects the impact on the change in parameters within each iteration. The influence of a sample at a particular iteration can be accumulated through updates and impact the final model. Here, we are going to derive the exact calculation of the sample’s influence to the model. We rewrite the Objective (2) in Section 2 here:
min θ n∑ i=1 viL(yi, f(xi,θ)) +R(θ), (3)
Here, we deem the sample weight vi is fixed across all iterations. The update rule for stochastic gradient descent in each iteration is:
θt+1 = θt − η ∑ i vig t i − ηrti
The derivative of θt+1 with respect to sample weight vi is:
∂
∂vi θt+1 =
∂
∂vi θt − ηgti − ηH(θ
t) ∂
∂vi θt
∂
∂vi θ1 = −ηg0i ,
where H(θt) is the Hessian matrix of the objective in (2) with regard to all parameters in iteration t. If we iterate the updates until convergence, then we can assume that θT is a fix-point, θ∗ = θT+1 = θT , and we obtain:
∂ ∂vi θT+1 − ∂ ∂vi θT = −ηgti − ηH(θ ∗) ∂ ∂vi θ∗
Hence, the derivative of parameters in the final model with regard to a sample weight is:
∂
∂vi θ∗ = −H(θ∗)−1gTi (4)
Equation (4) indicates that we can calculate the sample specific impact on final trained model by using the parameters learned at the convergence point. In deep learning methods, due to early stopping, fix point might not be achieved, and Equation (4) might not be an accurate.
For any target quantity T (θ∗) that depends on the final trained parameter θ∗, we can calculate the impact of a particular sample on that target as:
∂
∂vi T (θ∗) =
∂ ∂θ∗ T (θ∗) ∂ ∂vi θ∗ (5)
For example, if we are interested in the sum of predictions on a set of samples T (θ∗) =∑ i∈Sc f(xi,θ t), we can use Equation (5) to calculate the derivative:
∂
∂vi T (θ∗) = ∑ i∈Sc ∂ ∂θ∗ f(xi,θ t) ∂ ∂vi θ∗
We note evaluating the exact impact of a sample, as shown above, is computationally cumbersome for all but the simplest models.
5 DISCUSSION
Samples’ impact on the deep network’s parameters vary across stages of training and network’s layers. In our work, we found that easy samples predominantly shape parameters the top layers at the early training stages, while hard samples predominantly shape the parameters of the bottom layers at the late training stage. Our experiments show that it is important to mix hard samples into different batches rather than keep them together in the same batch and away from other examples.
There are many future extensions to the current work. Firstly, we want to expand our sample importance analysis to different deep learning structures, like Convolution Neural Network and Recurrent Neural Networks. Secondly, we want to use the sample importance as a guidance to extract a minimal subset of samples that are sufficient to achieve performance comparable to a network trained on the full dataset.

(paper summary) The authors introduce the notion of “sample importance”, meant to measure the influence of a particular training example on the training of a deep neural network. This quantity is closely related to the squared L2 norm of the gradient, where the summation is performed over (i) parameters of a given layer or (ii) across all parameters. Summing this quantity across time gives the “overall importance”, used to tease apart easy from hard examples. From this quantity, the authors illustrate the impact of [easy,hard] example during training, and their impact on layer depth.
(detailed review)
I have several objections to this paper. First and foremost, I am not convinced of the “sample importance” as a meaningful metric. As previously mentioned, the magnitude of gradients will change significantly during learning, and I am not sure what conclusions one can draw from \sum_t g_i^t vs \sum_t g_j^t. For example, gradients tend to have higher norms early in training than at convergence, in which case weighting each gradient equally seems problematic. I tried illustrating the above with a small thought experiment during the question period: “if” the learning rate were too high, training may not even converge in which case sample importance would be ill-defined.  Having a measure which depends on the learning rate seems problematic to me, as does the use of the L2 norm. The “input Fisher” norm, \mathbb{E} \frac{\partial \log p} {\partial x} (for a given time-step) may be better suited, as it speaks directly to the sensitivity of the classifier to the input x (and is insensitive to changes in the mean gradient norm). But again summing Fisher norms across time may not be meaningful.
The experimental analysis also seems problematic. The authors claim from Fig. 2 that output layers are primarily learnt in the early stage of training. However, this is definitely not the case for CIFAR-10 and is debatable for MNIST: sample importance remains high for all layers during training, despite a small spike early on the output layer. Fig 2. (lower, middle) and Fig. 6 also seems to highlight an issue with the SI measure: the SI is dominated by the input layer which has the most parameters, and can thus more readily impact the gradient norm. Different model architectures may have yielded different conclusions. Had the authors managed to use the SI to craft a better curriculum, this would have given significant weight to the measure. Unfortunately, these results are negative.
PROS:
+ extensive experiments
CONS:
- sample importance is a heuristic, not entirely well justified
- SI yields limited insight into training of neural nets
- SI does not inform curriculum learning


THE NEURAL NOISY CHANNEL
We formulate sequence to sequence transduction as a noisy channel decoding problem and use recurrent neural networks to parameterise the source and channel models. Unlike direct models which can suffer from explaining-away effects during training, noisy channel models must produce outputs that explain their inputs, and their component models can be trained with not only paired training samples but also unpaired samples from the marginal output distribution. Using a latent variable to control how much of the conditioning sequence the channel model needs to read in order to generate a subsequent symbol, we obtain a tractable and effective beam search decoder. Experimental results on abstractive sentence summarisation, morphological inflection, and machine translation show that noisy channel models outperform direct models, and that they significantly benefit from increased amounts of unpaired output data that direct models cannot easily use.
1 INTRODUCTION
Recurrent neural network sequence to sequence models (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015) are excellent models of p(output sequence y | input sequence x), provided sufficient input–output (x,y) pairs are available for estimating their parameters. However, in many domains, vastly more unpaired output examples are available than input–output pairs (e.g., transcribed speech is relatively rare although non-spoken texts are abundant; Swahili–English translations are rare although English texts are abundant; etc.). A classic strategy for exploiting both kinds of data is to use Bayes’ rule to rewrite p(y | x) as p(x | y)p(y)/p(x), a factorisation which is called a noisy channel model (Shannon, 1948). A noisy channel model thus consists of two component models: the conditional channel model, p(x | y), which characterizes the reverse transduction problem and whose parameters are estimated from the paired (x,y) samples, and the unconditional source model, p(y), whose parameters are estimated from both the paired and (usually much more numerous) unpaired samples.1
Beyond their data omnivorousness, noisy channel models have other benefits. First, the two component models mean that two different aspects of the transduction problem can be addressed independently. For example, in many applications, source models are language models and innovations in these can be leveraged to obtain improvements in any system that uses them as a component. Second, the component models can have complementary strengths, since inference is carried out in the product space; this simplifies design because a single model does not have to get everything perfectly right. Third, the noisy channel operates by selecting outputs that both are a priori likely and that explain the input well. This addresses a failure mode that can occur in conditional models in which inputs are “explained away” by highly predictive output prefixes, resulting in poor training (Klein & Manning, 2001). Since the noisy channel formulation requires its outputs to explain the observed input, this problem is avoided.
In principle, the noisy channel decomposition is straightforward; however, in practice, decoding (i.e., computing argmaxy p(x | y)p(y)) is a significant computational challenge, and tractability concerns impose restrictions on the form the component models can take. To illustrate, an appealing parameterization would be to use an attentional seq2seq network (Bahdanau et al., 2015) to model ∗Work completed at DeepMind. 1We do not model p(x) since, in general, we will be interested in finding argmaxy p(y | x), and argmaxy p(y | x) = argmaxy p(x|y)p(y) p(x) = argmaxy p(x | y)p(y).
the channel probability p(x | y). However, seq2seq models are designed under the assumption that the complete conditioning sequence is available before any prefix probabilities of the output sequence can be computed. This assumption is problematic for channel models since it means that a complete output sequence must be constructed before the channel model can be evaluated (since the channel model conditions on the output). Therefore, to be practical, the channel probability must decompose in terms of prefixes of the conditioning variable, y. While the chain rule justifies decomposing output variable probabilities in terms of successive extensions of a partial prefix, no such convenience exists for conditioning variables, and approximations must be introduced.
In this work, we use a variant of the newly proposed online seq2seq model of Yu et al. (2016) which uses a latent alignment variable to enable its probabilities to factorize in terms of prefixes of both the input and output, making it an appropriate channel model (§2). Using this channel model, the decoding problem then becomes similar to the problem faced when decoding with direct models (§3). Experiments on abstractive summarization, machine translation, and morphological inflection show that the noisy channel can significantly improve performance and exploit unpaired output training samples and that models that combine the direct model and a noisy channel model offer further improvements still (§4).
2 BACKGROUND: SEGMENT TO SEGMENT NEURAL TRANSDUCTION
Our model is based on the Segment to Segment Neural Transduction model (SSNT) of Yu et al., 2016. At a high level, the model alternates between encoding more of the input sequence and decoding output tokens from the encoded representation. This presentation deviates from the Yu et al.’s presentation so as to emphasize the incremental construction of the conditioning context that is enabled by the latent variable.
2.1 MODEL DESCRIPTION
Similar to other neural sequence to sequence models, SSNT models the conditional probability p(y | x) of a output sequence y given a input sequence x. To avoid having to observe the complete input sequence x before making a prediction of the beginning of the output sequence, we introduce a latent alignment variable z which indicates when each token of the output sequence is to be generated as the input sequence is being read. Since we assume that the input is read just once from left to right, we restrict z to be a monotonically increasing alignment (i.e., zj+1 ≥ zj is true with probability 1), where zj = i denotes that the output token at position j (yj) is generated when the input sequence up through position i has been read. The SSNT model is:
p(y | x) = ∑ z p(y, z | x)
p(y, z | x) ≈ |y|∏ j=1 p(zj | zj−1,x zj 1 ,y j−1 1 )︸ ︷︷ ︸
alignment probability
p(yj | x zj 1 ,y j−1 1 )︸ ︷︷ ︸
word probability
. (1)
We explain the model in terms of its two components, starting with the word generation term. In the SSNT, the input and output sequences x, y are encoded with two separate LSTMs (Hochreiter & Schmidhuber, 1997), resulting in sequences of hidden states representing prefixes of these sequences. In Yu et al.’s formulation, the input sequence encoder (i.e., the conditioning context encoder) can either be a unidirectional or bidirectional LSTM, but here we assume that it is a unidirectional LSTM, which ensures that it will function well as a channel model that can compute probabilities with incomplete conditioning contexts (this is necessary since, at decoding time, we will be constructing the conditioning context incrementally). Let hi represent the input sequence encoding for the prefix xi1. Since the final action at timestep j will be to predict yj , it is convenient to let sj denote the hidden state that excludes yj , i.e., the encoding of the prefix y j−1 1 .
The probability of the next token yj is calculated by concatenating the aligned hidden state vectors sj and hzj followed by a softmax layer,
p(yj | x zj 1 ,y j−1 1 ) ∝ exp(Ww[hzj ; sj ] + bw).
The model thus depends on the current alignment position zj , which determines how far into x it has read.
We now discuss how the sequence of zj’s are generated. First, we remark that modelling this distribution requires some care so as to avoid conditioning on the entire input sequence. To illustrate why one might induce a dependency on the entire input sequence in this model, it is useful to compare to a standard attention model. Attention models operate by computing a score using a representation of alignment candidate (in our case, the candidates would be every unread token remaining in the input). If we followed this strategy, it would be necessary to observe the full input sequence when making the first alignment decision.
We instead model the alignment transition from timestep j to j+1 by decomposing it into a sequence of conditionally independent SHIFT and EMIT operations that progressively decide whether to read another token or stop reading. That is, at input position i, the model decides to EMIT, i.e., to set zj = i and predict the next output token yj from the word model, or it decides to SHIFT, i.e., to read one more input token and increment the input position i ← i + 1. The probability p(ai,j = EMIT | xi1,y j−1 1 ) is calculated using the encoder and decoder states defined above as:
p(ai,j = EMIT | xi1,y j−1 1 ) = σ(MLP(Wt[hi; sj ] + bt)).
The probability of SHIFT is simply 1 − p(ai,j = EMIT). In this formulation, the probabilities of aligning zj to each alignment candidate i can be computed by reading just xi1 (rather than the entire sequence). The probabilities are also independent of the contents of the suffix x|x|i+1.
Using the probabilities of the auxiliary ai,j variables, the alignment probabilities needed in Eq. 1 are computed as:
p(zj = i | zj−1,yj−11 ,xi1) =  0 if i < zj−1 p(ai,j = EMIT) if i = zj−1(∏i−1
i′=zj−1 p(ai′,j = SHIFT)
) p(ai,j = EMIT) if i > zj−1
2.2 INFERENCE ALGORITHMS
In SSNT, the probability of generating each yj depends only on the current output position’s alignment (zj), the current output prefix (y j−1 1 ), the input prefix up to the current alignment (x zj 1 ). It does not depend on the history of the alignment decisions. Likewise, the alignment decisions at each position are also conditionally independent of the history of alignment decisions. Because of these independence assumptions, z can be marginalised using a O(|x|2 · |y|) time dynamic programming algorithm where each fill in a chart with computing the following marginal probabilities:
α(i, j) = p(zj = i,y j 1 | x zj 1 ) = i∑ i′=1 α(i′, j − 1) p(zj | zj−1,x zj 1 ,y j−1 1 )︸ ︷︷ ︸
alignment probability
p(yj | x zj 1 ,y j−1 1 )︸ ︷︷ ︸
word probability
.
The model is trained to minimize the negative log likelihood of the parallel corpus S: L(θ) = − ∑
(x,y)∈S
log p(y | x;θ)
= − ∑
(x,y)∈S
logα(|x|, |y|). (2)
The gradients of this objective with respect to the component probability models can be computed using automatic differentiation or using a secondary dynamic program that computes ‘backward’ probabilities. We refer the reader to Section 3.1 of Yu et al. (2016) for details.
In this paper, we use a slightly different objective from the one described in Yu et al. (2016). Rather than marginalizing over the paths that end in any possible input positions ∑I i=1 α(i, |y|), we require that the full input be consumed when the final output symbol is generated. This constraint biases away from predicting outputs without explaining them using the input sequence.
3 DECODING
We now turn to the problem of decoding, that is, of computing
ŷ = argmax y
p(x | y)p(y),
where we are using the SSNT model described in the previous section as the channel model and a language model that delivers prior probabilities of the output sequence in left-to-right order, i.e., p(yi | yi−1). Marginalizing the latent variable during search is computationally hard (Sima’an, 1996), and so we approximate the search problem as
ŷ = argmax y max z
p(x, z | y)p(y).
However, even with this simplification, the search problem remains nontrivial. On one hand, we must search over the space of all possible outputs with a model that makes no Markovian assumptions. This is similar to the decoding problem faced in standard seq2seq transducers. On the other hand, our model computes the probability of the given input conditional on the predicted output hypothesis. Therefore, instead of just relying on a single softmax to provide a probability for every output word type (as we conveniently can in the direct model), we must loop over each output word type, and run a softmax over the input vocabulary—a computational expense that is quadratic in the size of the vocabulary!
To reduce this computational effort, we make use of an auxiliary direct model q(y, z | x) to explore probable extensions of partial hypotheses, rather than trying to perform an exhaustive search over the vocabulary each time we extend an item on the beam.
Algorithm 1, in Appendix A, describes the decoding algorithm based on a formulation by Tillmann et al. (1997). The idea is to create a matrix Q of partial hypotheses. Each hypothesis in cell (i, j) covers the first i words of the input (xi1) and corresponds to an output hypothesis prefix of length j (yj1). The hypothesis is associated with a model score. For each cell (i, j), the direct proposal model first calculates the scores of possible extensions of previous cells that could then reach (i, j) by considering every token in the output vocabulary, from all previous candidate cells (i− 1,≤ j). That gives the top K1 partial output sequences. These partial output sequences are subsequently rescored by the noisy channel model, and the K2 best candidates are kept in the beam and used for further extension. The beam size K1 and K2 are hyperparameters to be tuned in the experiments.
3.1 MODEL COMBINATION
The decoder we have just described makes use of an auxiliary decoding model. This means that, as a generalisation, it is capable of decoding under an objective that is a linear combination of the direct model, channel model, language model and a bias for the output length2,
Oxi1,y j 1 = λ1 log p(y
j 1 | xi1) + λ2 log p(xi1 | y j 1) + λ3 log p(y j 1) + λ4|y j 1|. (3)
The bias is used to penalize the noisy channel model for generating too-short (or long) sequences. The λ’s are hyperparameters to be tuned using on a small amount of held-out development data.
4 EXPERIMENTS
We evaluate our model on three natural language processing tasks, abstractive sentence summarisation, machine translation and morphological inflection generation. For each task, we compare the performance of the direct model, noisy channel model, and the interpolation of the two models.
2In the experiments, we did not marginalize the probability of the direct model when calculating the general search objective. We found that marginalizing the probability does not give better performance and makes decoding extremely slow.
4.1 ABSTRACTIVE SENTENCE SUMMARISATION
Sentence summarisation is the problem of constructing a shortened version of a sentence while preserving the majority of its meaning. In contrast to extractive summarisation, which can only copy words from the original sentence, abstractive summarisation permits arbitrary rewording of the sentence. The dataset (Rush et al., 2015) that we use is constructed by pairing the first sentence and the headline of each article from the annotated Gigaword corpus (Graff et al., 2003; Napoles et al., 2012). There are 3.8m, 190k and 381k sentence pairs in the training, validation and test sets, respectively. Yu et al. (2016) filtered the dataset by restricting the lengths of the input and output sentences to be no greater than 50 and 25 tokens, respectively. From the filtered data, they further sampled 1 million sentence pairs for training. We experimented on training the direct model and channel model with both the sampled 1 million and the full 3.8 million parallel data. The language model is trained on the target side of the parallel data, i.e. the headlines. We evaluated the generated summaries of 2000 randomly sampled sentence pairs using full length ROUGE F1. This setup is in line with the previous work on this task (Rush et al., 2015; Chopra et al., 2016; Gülçehre et al., 2016; Yu et al., 2016).
The same configuration is used to train the direct model and the channel model. The loss (Equation 2) is optimized by Adam (Kingma & Ba, 2015), with initial learning rate of 0.001. We use LSTMs with 1 layer for both the encoder and decoders, with hidden units of 256. The mini-batch size is 32, and dropout of 0.2 is applied to the input and output of LSTMs. For the language model, we use a 2-layer LSTM with 1024 hidden units and 0.5 dropout. The learning rate is 0.0001. All the hyperparameters are optimised via grid search on the perplexity of the validation set. During decoding, beam search is employed with the number of proposals generated by the direct model K1 = 20, and the number of best candidates selected by the noisy channel model K2 = 10.
Table 1 presents the ROUGE-F1 scores of the test set from the direct model, noisy channel model (channel + LM + bias), the interpolation of the direct model and the noisy channel model (direct + channel + LM + bias), and the interpolation of the direct model and language model (direct + LM + bias) trained on different sizes of data. The noisy channel model with the language model trained on the target side of the 1 million parallel data outperforms the direct model by approximately 1 point. Such improvement indicates that the language model helps improve the quality of the output sequence when no extra unlabelled data is available. Training the language model with all the headlines in the dataset, i.e. 3.8 million sentences, gives a further boost to the ROUGE score. This is in line with our expectation that the model benefits from adding large amounts of unlabelled data. The interpolation of the direct model, channel model, language model and bias of the output length achieves the best results — the ROUGE score is close to the direct model trained on all the parallel
data. Although there is still improvement, when the direct model is trained with more data, the gap between the direct model and the noisy channel model is smaller. No gains is observed if the language model is combined with the direct model. We find that as we increase the weight of the language model, the result is getting worse.
Table 2 surveys published results on this task, and places our best models in the context of the current state-of-the-art results. ABS+ (Rush et al., 2015), RAS-LSTM and RAS-Elman (Chopra et al., 2016) are different variations of the attentive models. Pointing the unkown words uses pointer networks (Vinyals et al., 2015) to select the output token from the input sequence in order to avoid generating unknown tokens. ASC + FSC (Miao & Blunsom, 2016) is a semi-supervised model based on a variational autoencoder. Trained on 1m paired samples and 3.8m unpaired samples, the noisy channel achieves comparable or better results than (direct) models trained with 3.8m paired samples. Compared to Miao & Blunsom (2016), whose ASC + FSC models is an alternative strategy for using unpaired data, the noisy channel is significantly more effective — 33.21 versus 31.09 in ROUGE-1.
Finally, motivated by the qualitative observation that noisy channel model outputs were quite fluent and often used reformulations of the input rather than a strict compression (which would be poorly scored by ROUGE), we carried out a human preference evaluation whose results are summarised in Table 3. This confirms that noisy channel summaries are strongly preferred over those of the direct model.
4.2 MACHINE TRANSLATION
We next evaluate our models on a Chinese–English machine translation task. We used parallel data with 184k sentence pairs (from the FBIS corpus, LDC2003E14) and monolingual data with 4.3 million of English sentences (selected from the English Gigaword). The training data is preprocessed by lowercasing the English sentences, replacing digits with ‘#’ token, and replacing tokens appearing
less than 5 times with an UNK token. This results in vocabulary sizes of 30k and 20k for Chinese sentences and English sentences, respectively.
The models are trained using Adam (Kingma & Ba, 2015) with initial learning rate of 0.001 for the direct model and the channel model, and 0.0001 for the language model. The LSTMs for the direct and channel models have 512 hidden units and 1 layer, and 2 layers with 1024 hidden units per layer for the language model. Dropout of 0.5 on the input and output of LSTMs is set for all the model training. The noisy channel decoding uses K1 = 20 and K2 = 10 as the beam sizes.
Table 4 lists the translation performance of different models in BLEU scores. To set benchmarks, we train the vanilla and attentional sequence to sequence models (Sutskever et al., 2014; Bahdanau et al., 2015) using the same parallel data. For direct models, we leverage bidirectional LSTMs as the encoder for this task. We can see that the vanilla sequence to sequence model behaves poorly due to the small amounts of parallel data. By contrast, the direct model (SSNT) and the attentional model work relatively well, with the attentional model outperforming the SSNT direct model. Although these models both directly model p(y | x), this result is unsurprising because the SSNT direct model is most effective when the alignment between sequences is largely monotonic, and Chinese–English translation word orders diverge considerably. However, despite this limitation, the noisy channel model is approximately 3 points higher in BLEU than the direct model, and the combination of noisy channel and direct model gives extra boost. Confirming the empirical findings of prior work (and in line with theoretical predictions), the interpolation of the direct model and language model is not effective.
4.3 MORPHOLOGICAL INFLECTION GENERATION
Morphological inflection is the task of generating a target (inflected form) word from a source word (base form), given a morphological attribute, e.g. number, tense, and person etc.. It is useful for reducing data sparsity issues in translating morphologically rich languages. The transformation from the base form to the inflected form is usually to add prefix or suffix, or to do character replacement. The dataset (Durrett & DeNero, 2013) that we use in the experiments is created from Wiktionary, including inflections for German nouns, German verbs, Spanish Verbs, Finnish noun and adjective, and Finnish verbs. We only experimented on German nouns and German verbs, as German nouns is the most difficult task3, and the direct model does not perform as well as other state-of-theart systems on German verbs. The train/dev/test split for German nouns is 2364/200/200, and for German verbs is 1617/200/200. There are 8 and 27 inflection types in German nouns and German verbs, respectively. Following previous work, we learn a separate model for each type of inflection independent of the other inflections. We report results on the average accuracy across different inflections. Our language models were trained on word types extracted by running a morphological analysis tool on the WMT 2016 monolingual data and extracting examples of appropriately inflected word forms.4 After annotation the number of instances for training the language model ranged from 300k to 3.8m for different inflection types in German nouns, and from 200 to 54k in German verbs.
The experimental setup that we use on this task is K1 = 60, K2 = 30,
• direct and channel model: 1 layer LSTM with 128 hidden, η = 0.001, dropout = 0.5. 3While state-of-the-art systems can achieve 99% accuracies on Spanish verbs and Finnish verbs, they can
only get 89% accuracy on German nouns. 4http://www.statmt.org/wmt16/translation-task.html
• language model: 2 layer LSTM with 512 hidden, η = 0.0001, dropout = 0.5.
Table 1 summarises the results from our models. On both datasets, the noisy channel model (channel + LM + bias) does not perform as well as the direct model, but the interpolation of the direct model and noisy channel model (direct + channel + LM + bias) significantly outperforms the direct model. The interpolation of the direct model and language model (direct + LM + bias) achieves better results than the direct model and the noisy channel model on German nouns, but not on German verbs. For further comparison, we also included the state-of-the-art results as benchmarks. NCK15 (Nicolai et al., 2015) tackles the task based on the three-stage approach: (1) align the source and target word, (2) extract inflection rules, (3) apply the rule to new examples. FTND16 (Faruqui et al., 2016) is based on neural sequence to sequence models. Both models (NCK15+ and FTND16+) rerank the candidate outputs by the scores predicted from n-gram language models, together with other features.
5 ANALYSIS
By observing the output generated by the direct model and noisy channel model, we find (in line with theoretical critiques of conditional models) that the direct model may leave out key information. By contrast, the noisy channel model does seem to avoid this issue. To illustrate, in Example 1 (see Appendix B) in Table 5, the direct model ignores the key phrase ‘coping with’, resulting in incomplete meaning, but the noisy channel model covers it. Similarly, in Example 6, the direct model does not translate the Chinese word corresponding to ‘investigation’. We also observe that while the direct model mostly copies words from the source sentence, the noisy channel model prefers generating paraphrases. For instance, in Example 2, while the direct model copies the word ‘accelerate’ in the generated output, the noisy channel model generate ‘speed up’ instead. While one might argue that copying is a preferable compression technique than paraphrasing (as long as it produces grammatical outputs), it does show the power of these models.
6 RELATED WORK
Noisy channel decompositions have been successfully used in a variety of problems, including speech recognition (Jelinek, 1998), machine translation (Brown et al., 1993), spelling correction (Brill & Moore, 2000), and question answering (Echihabi & Marcu, 2003). The idea of adding language models and monolingual data in machine translation has been explored in earlier work. Gülçehre et al. (2015) propose two strategies of combining a language model with a neural sequence to sequence model. In shallow fusion, during decoding the sequence to sequence model
(direct model) proposes candidate outputs and these candidates are reranked based on the scores calculated by a weighted sum of the probability of the translation model and that of the language model. In deep fusion, the language model is integrated into the decoder of the sequence to sequence model by concatenating their hidden state at each time step. Sennrich et al. (2016) incorporate target language unpaired training data by doing back-translation to create synthetic parallel training data. While this technique is quite effective, its practicality seems limited to problems where the inputs and outputs contain roughly the same information (such as translation). Cheng et al. (2016) leverages the abundant monolingual data by doing multitask learning with an autoencoding objective.
A number of papers have remarked on the tendency for content to get dropped (or repeated) in translation. Liu et al. (2016) propose translating in both a left-to-right and a left-to-right direction and seeking a consensus. Tu et al. (2016) propose augmenting a direct model’s decoding objective with a reverse translation model (similar to our channel model except it conditions on the direct model’s output RNN’s hidden states rather than the words); however, that work just reranks complete translation hypotheses rather than developing a model that permits an incremental search.
Another trend of work that is related to our model is the investigation of making online prediction for machine translation (Gu et al., 2016; Grissom II et al., 2014; Sankaran et al., 2010) and speech recognition (Hwang & Sung, 2016; Jaitly et al., 2016).
Our direct model (and channel model) shares the idea of introducing stochastic latent variables to neural networks with several papers and marginalising these during training. Examples include connectionist temporal classification (CTC) (Graves et al., 2006) and the more recent segmental recurrent neural networks (SRNN) (Kong et al., 2016). Compared to these models, our direct model has the advantage of capturing unbounded dependencies of output words. The direct model is closely related to the sequence transduction model (Graves, 2012) in the way of modeling the probability of predicting output tokens and marginalizing latent variables using dynamic programming. However, rather than modeling the joint distribution over outputs and alignments by inserting null symbols into the output sequence, our direct model defines a separate latent alignment variable, with alignment distribution defined with neural networks. Similar to our work, the model in (Alkhouli et al., 2016) is decomposed into the alignment model and the model of word predictions. The two models are trained separately and combined during decoding, with subsequent refinements using a Viterbi-EM approximation. By contrast, in our direct and channel models, the latent and observed components of the models are trained jointly using a dynamic program to exactly marginalise the unobserved variables.
7 CONCLUSION
We have presented and empirically validated a noisy channel transduction model that uses component models based on recurrent neural networks. This formulation lets us use unpaired outputs to estimate the parameters of the source model and input-output pairs to train the channel model. Despite the channel model’s ability to condition on long sequences, we are able to maintain tractable decoding by using a latent segmentation variable that breaks the conditioning context up into a series of monotonically growing segments. Our experiments show that this model makes excellent use of unpaired training data.
A ALGORITHM
Algorithm 1 Noisy Channel Decoding Notation: Q is the Viterbi matrix, bp is the backpointer, W stores the predicted tokens, V refers to the vocabulary, I = |x|, and Jmax denotes the maximum number of output tokens that can be predicted. Input: source sequence x Output: best output sequence y∗ Initialisation: Q ∈ RI×Jmax×K1 , bp ∈ NI×Jmax×K1 , W ∈ NI×Jmax×K1 ,
Qtemp ∈ RK1 , bptemp ∈ NK1 , Wtemp ∈ NK1 for i ∈ [1, I] do
Qtemp ← topk(K1)y∈Vq(z1 = i) q(y | START, z1,xz11 ) . Candidates generated by q(y | x). bptemp ← 0 Wtemp ← arg topk(K1)y∈Vq(z1 = i) q(y | START, z1,xz11 ) Q[i, 1]← topk(K2)y∈WtempOxi1,y . Rerank the candidates by objective (O). W [i, 1]← arg topk(K2)y∈WtempOxi1,y
end for for j ∈ [2, Jmax] do
for i ∈ [1, I] do Qtemp ← topk(K1)y∈V,k∈[1,i]Q[k, j − 1]· q(zj = i | zj−1 = k)q(y | yj−11 , zj ,x zj 1 )
bptemp,Wtemp ← arg topk(K1)y∈V,k∈[1,i] Q[k, j − 1]q(zj = i | zj−1 = k)· q(y | yj−11 , zj ,x) Y ← getCandidateOutputs(bptemp,Wtemp) . Get partial candidate yj1. Q[i, j]← topk(K2)yj∈YOxi1,yj1 bp[i, j],W [i, j]← arg topk(K2)yj1∈Y Oxi1,yj1 end for end for return a sequence of words stored in W by following backpointers starting from (I, argmaxj Q[I, j]).
B EXAMPLE OUTPUTS
Summarisation Example 1: source: the european commission on health and consumers protection −lrb− unk −rrb−
has offered cooperation to indonesia in coping with the spread of avian influenza in the country , official news agency antara said wednesday .
reference: eu offers indonesia cooperation in avian flu eradication direct: eu offers cooperation to indonesia in avian flu nc: eu offers cooperation to indonesia in coping with bird flu
Example 2: source: vietnam will accelerate the export of industrial goods mainly by developing auxil-
iary industries , and helping enterprises sharpen competitive edges , according to the ministry of industry on thursday .
reference: vietnam to boost industrial goods export direct: vietnam to accelerate export of industrial goods nc: vietnam to speed up export of industrial goods
Example 3: source: japan ’s toyota team europe were banned from the world rally championship for one
year here on friday in a crushing ruling by the world council of the international automobile federation -lrb- fia -
reference: toyota are banned for a year direct: toyota banned from world rally championship nc: toyota europe banned from world rally championship for one year Example 4: source: oil prices roared higher towards ## dollars on monday as equity markets surged on government action aimed at tackling a severe economic downturn . reference: oil prices soar towards ## dollars direct: oil prices jump towards ## dollars nc: oil prices climb towards ## dollars
Translation Example 5: source: 欧盟和美国都表示可以接受这一妥协方案。 reference: both the eu and the us indicated that they can accept this plan for a compromise . direct: the eu and the united states indicated that it can accept this compromise . nc: the european union and the united states have said that they can accept such a com-
promise plan .
Example 6: source: 那么这些这个方面呢是现在警方调查重点。 reference: well , this is the current focus of police investigation . direct: these are present at the current police . nc: then these are the key to the current police investigation .
Example 7: source: 双方有可能就此问题在下周进行磋商。 reference: the two sides may conduct negotiations on this issue next week . direct: the two sides may hold consultations on next week . nc: the two sides are likely to hold consultations on this issue next week . Example 8: source: 那么在这个问题上 ,伊朗现在态度比较强硬 ,而美国的态度更为强硬。 reference: well , iran ’s attitude is now quite firm on this issue , while the us takes an even firmer attitude . direct: on this issue , iran ’s attitude is quite hard and the attitude of the united states is still tougher . nc: then , on this issue , iran has now taken a tougher attitude toward it . however , the
attitude of the united states is even harder .
Table 5: Example outputs on the test set from the direct model and noisy channel model for the summarisation task and machine translation.

The paper proposes an online variant of segment to segment transducers, which allows to circumvent the necessity of observing whole sentence, before making target predictions. Authors mostly build on their previous work, allowing additionally to leverage independent priors on the target hypotheses, like the language grammar or sentence length.
Strong points:
- well written, interesting idea of combining various sources of information in a Bayesian framework for seq2seq models
Handling something in an online manner typically makes things more difficult, and this is what the authors are trying to do here - which is definitely of interest to the community
- strong experimental section, with some strong results (though not complete: see weak points)
Weak points:
- Authors do not improve on computational complexity (w.r.t Tillmann proposal), hence the algorithms may be found difficult to apply in scenarios where inputs may be long (this already takes into account a rather constrained model of alignment latent variables)
- What about the baseline where you only combine direct, LM and bias contributions (no channel)? Was there any (non-obvious) algorithmic constraint why - this has not been included?
Some other (minor) comments:
- Related to the first weak point: can you elaborate more on how the clue of your work is conceptually different from the work of Tillmann et al. (1997) (except, of course, the fact you use connectionist discriminative models to derive particular conditional probabilities).
- How sensitive is the model to different choices of hyper-parameters in eq (3). Do you naively search through the search space of those, or do something more clever?
- Some more comments on details of the auxiliary direct model would be definitely of interest.
- How crucial is the correct choice of the pruning variables (K1 and K2)?
- Sec. 2: makes no Markovian assumptions -> no first-order Markovian assumption?
Typos:
Table 1: chanel -> channel (one before last row)
Apologies for late review.