As for knowledge-based question answering, a fundamental problem is to relax the assumption of answerable questions from simple questions to compound questions. Traditional approaches firstly detect topic entity mentioned in questions, then traverse the knowledge graph to find relations as a multi-hop path to answers, while we propose a novel approach to leverage simple-question answerers to answer compound questions. Our model consists of two parts: (i) a novel learning-to-decompose agent that learns a policy to decompose a compound question into simple questions and (ii) three independent simple-question answerers that classify the corresponding relations for each simple question. Experiments demonstrate that our model learns complex rules of compositionality as stochastic policy, which benefits simple neural networks to achieve state-of-the-art results on WebQuestions and MetaQA. We analyze the interpretable decomposition process as well as generated partitions.

Summary: the paper is interested in parsing compound questions for querying on knowledge graph, e.g. MetaQA by Zhang et al. (2017). The paper proposes to have two modules, one that segments the question into partitions (up to three) and the other that looks at each segment to get the relation. The relations are merged to obtain a single KG path, which is queried to obtain the answer. Since the segmentation is a non-differentiable process, the paper uses reinforcement learning to propagate gradient to the segmentation model. The segmentation is a process of classifying each word for which partition it should be tied to. Answering is a process of classifying the partition into one of the possible relation edges. The model shows expected results in a synthetic arithmetic dataset, and obtains the state of the art in MetaQA, improving nearly 5% over the baseline. The model especially does much better on 3-hop questions, with nearly 20% improvement.
Strengths: the paper is well-written. The model is simple yet effective and is a novel contribution to compound question answering on KG. Especially, the improvement on 3-hop category is nearly 20%, which is substantial and quite impressive.
Weaknesses: My biggest concern is the lack of discussions on its relevance to  (Iyyer et al., 2016), which also proposed to decompose question into simpler ones for WIkiTableQuestions. Also, I think it would be good to mention Semantic Role Labeling as related literature, which is about tagging each word with its role in the sentence. The partition index can be somewhat considered as a “role” in the sentence.
Questions:
1. How do you obtain x^(k)? Is it the last state of the LSTM?
2. Why did you have to augment “NO_OP” relation in the MetaQA dataset?
3. Why +1 reward has lower variance than probabilistic reward? Explanation or citation would be needed.
4. What if two partitions need to share a word? The current setup necessitates that a word participates in only one partition. Wouldn’t this be problematic?
5. I am a bit confused about how the simple question answering module is trained. Is it directly trained by the gold relation label?
Typos and Suggestions:
- Second paragraph of 2.1: in stead -> instead
- Third paragraph of 2.1: research. -> research
- c_t + h_t: would be good to explicitly mention that the circled plus sign is concatenation.
- Last paragraph on page 4: “leave to be”?
- Second last paragraph of 4.1: he -> The
- Second paragraph of 4.2: “if exists a proper meaning”?
- First paragraph of page 7: be either assume -> either assume
- Last paragraph of Section 5: generalizing -> generalize
- I think you should not put acknowledgment in a double-blind submission.
M Iyyer, W Yih, MW Chang. Answering complicated question intents expressed in decomposed question sequences. 2016 (https://arxiv.org/abs/1611.01242)


In this article, we show how we applied a simple approach coming from deep learning networks for object detection to the task of optical character recognition in order to build image features taylored for documents. In contrast to scene text reading in natural images using networks pretrained on ImageNet, our document reading is performed with small networks inspired by MNIST digit recognition challenge, at a small computational budget and a small stride. The object detection modern frameworks allow a direct end-to-end training, with no other algorithm than the deep learning and the non-max-suppression algorithm to filter the duplicate predictions. The trained weights can be used for higher level models, such as, for example, document classification, or document segmentation.

The authors experiment with building an SSD-like object detection network for text detection in documents, by replacing the usual VGG or ResNet base architecture with a light weight model inspired by the original digits classification CNN from [LeCun et al 1999].
This paper is a pure technical report with no novel contribution: all the authors do is replace the "body" network in the well-known SSD architecture with a simpler model (taken from existing literature) and evaluate it on two synthetic benchmarks of their creation.
The idea of employing object detection CNNs for OCR is not novel either, as pointed out in the related works section.
Beside the absence of novelty, the paper also suffers from several other serious flaws:
1) One of the main motivations provided by the authors for this work is that existing "classification [...] detection [...] or segmentation networks, cannot be applied directly, even with finetuning".
However, no experimental results are reported to justify this claim.
In fact, in the experimental section the proposed network is not compared against any existing baseline.
2) The text has serious clarity and formatting issues, in particular:
- most tables and figures have no caption, and the few that have one are not numbered
- the text exceed both the 8 pages limit and the extended 10 pages limit allowed in the case of big figures
- the experimental section is very confusing, in particular the way the authors refer to the various network variants using long code names makes it really hard for the reader to follow the ablation studies
- given the absence of proper captions and numbering, it is quite hard to understand which table refers to which experiment
- most of the graphs seem to be in the form of low-resolution bitmaps, which are quite hard to read even on screen
- many entries in the References section are either missing the venue, or point to an arXiv link even when a proper conference / journal reference would be available
3) Some important details about the network are missing, in particular the authors do not mention how labels are assigned to the network outputs, and only give a vague indication about the losses being used.
Similarly, there's no mention about the use of NMS, which is also an important component of the two architectures (SSD and YOLO) that inspire this work.
Assigning labels and performing NMS are actually some of the most crucial components in the training of object proposal / object detection networks, often requiring numerous meta-parameters to be properly configured and tuned, as testified by the meticulous descriptions given in previous works (e.g. YOLO and Fast / Faster / Mask r-CNN).
4) The experimental section is very poorly organized and formatted (as mentioned in (2) above), and completely lacks any comparison with other state of the art approaches.
A lot of space is devoted to presenting a detailed ablation study which, in my opinion, doesn't contribute much to the overall paper and actually reads more like a report on meta-parameter tuning.
Finally, starting from Section 5.3.1 the text seems to be copy-pasted without a second read from some differently formatted document, as entire phrases or possibly tables / figures seems to be missing.
In conclusion, in my opinion this paper does not meet the conference's minimum quality standards and should definitely be rejected.


An unintended consequence of feature sharing is the model fitting to correlated tasks within the dataset, termed negative transfer. In this paper, we revisit the problem of negative transfer in multitask setting and find that its corrosive effects are applicable to a wide range of linear and non-linear models, including neural networks. We first study the effects of negative transfer in a principled way and show that previously proposed counter-measures are insufficient, particularly for trainable features. We propose a adversarial training approach to mitigate the effects of negative transfer by viewing the problem in a domain adaptation setting. Finally, empirical results on attribute prediction multi-task on AWA and CUB datasets further validate the need for correcting negative sharing in an end-to-end manner.

- The authors study the problem of negative transfer in representation learning, and propose to use the formulation proposed by Ganin & Lempitsky '15 for domain adaptation to reduce negative transfer. Instead of defining the domain classification as the adversarial task to learn a domain-independent representation, they collect a set of classification problems irrelevant to the main task as the adversarial tasks, and aim to learn a representation that focuses only on the primary task. There are very little changes compared to the proposal by Ganin & Lempitsky '15, but the application to solve the problem of negative transfer is interesting.
- My main concern on the whole argument of the paper is whether the benefits we see in the experiments come from the elimination of negative transfer, or just come from having more training labels from different tasks available. In the main formulation of the approach (equation 7), the authors try to learn a feature representation that works well for the primary task but works poorly for the auxiliary(irrelevant) tasks. If we switch the sign for lambda, then it becomes very similar to traditional multi-task learning. I wonder how the multi-task formulation would compare against the adversarial formulation proposed by the authors. There are reasons to suspect the multi-task formulation will also work better than the logistic regression baseline, since more labels from different tasks are available to learn a better joint representation. It is not clear whether the improvements come from modeling the auxiliary tasks using negative transfer (where the adversarial approach should beat the baseline and multi-task approach), or just come from having more information (where both the adversarial approach and the multi-task approach beat the baseline, but have similar performance).
- From a practical point of view, it is not easy to decide what prediction tasks are irrelevant. For example, in the birds dataset, I would expect the color and patterns in the body parts to have some correlations (primary_color, upperparts_color, underparts_color, wing_color, etc). In the case of occlusion of the relevant body parts, I could make a guess on the color based on the colors on other parts of the bird. In the ideal case for the current method, I would expect the adversarial approach proposed to learn a representation that mask out all the irrelevant parts of the animal or irrelevant contextual information. Apart from showing improved prediction performance, have the authors perform analysis on the image activation patterns similar to the motivation example in Figure 1 to see if the new approach actually focus on the relevant body parts of the animals?
- The definition of auxiliary tasks are described in the second last paragraph of 3.3, but it would be clearer if it is also mentioned how they are defined in the experiments section. I went through the whole experiments section having trouble interpreting the results because I could not find the definition of adversarial tasks.
- Overall I like this paper since it attempts to solve an interesting problem in computer vision, but I would like to see the above question on comparison with multi-task learning answered, or some image activation pattern analysis to provide a more solid argument that the improvements come from elimination of negative transfer.


Deep neural networks, which gain great success in a wide spectrum of applications, are often time, compute and storage hungry. Curriculum learning proposed to boost training of network by a syllabus from easy to hard. However, the relationship between data complexity and network training is unclear: why hard example harm the performance at beginning but helps at end. In this paper, we aim to investigate on this problem. Similar to internal covariate shift in network forward pass, the distribution changes in weight of top layers also affects training of preceding layers during the backward pass. We call this phenomenon inverse "internal covariate shift". Training hard examples aggravates the distribution shifting and damages the training. To address this problem, we introduce a curriculum loss that consists of two parts: a) an adaptive weight that mitigates large early punishment; b) an additional representation loss for low weighted samples. The intuition of the loss is very simple. We train top layers on "good" samples to reduce large shifting, and encourage "bad" samples to learn from "good" sample. In detail, the adaptive weight assigns small values to hard examples, reducing the influence of noisy gradients. On the other hand, the less-weighted hard sample receives the proposed representation loss. Low-weighted data gets nearly no training signal and can stuck in embedding space for a long time. The proposed representation loss aims to encourage their training. This is done by letting them learn a better representation from its superior neighbours but not participate in learning of top layers. In this way, the fluctuation of top layers is reduced and hard samples also received signals for training. We found in this paper that curriculum learning needs random sampling between tasks for better training. Our curriculum loss is easy to combine with existing stochastic algorithms like SGD. Experimental result shows an consistent improvement over several benchmark datasets.

This paper suggests a source of slowness when training a two-layer neural networks: improperly trained output layer (classifier) may hamper learning of the hidden layer (feature). The authors call this “inverse” internal covariate shift (as opposed to the usual one where the feature distribution shifts and trips the classifier). They identify “hard” samples, those with large loss, as being the impediment. They then propose a curriculum, where such hard samples are identified at early epochs, their loss attenuated and replaced with a requirement that their features be close to neighboring (in feature space) samples that are similarly classified, but with a more comfortable margin (thus “easy”.) The authors claim that this allows those samples to contribute through their features at first, without slowing the training down, then in later epochs fully contribute. Some experiments are offered as evidence that this indeed helps speedup.
The paper is extremely unclear and was hard to read. The narrative is too casual, a lot of handwaving is made. The notation is very informal and inconsistent. I had to second guess multiple times until deciphering what could have possibly been said. Based on this only, I do not deem this work ready for sharing. Furthermore, there are some general issues with the concepts. Here are some specific remarks.
-	The intuition of the inverse internal covariate shift is perhaps the main merit of the paper, but I’m not sure if this was not mostly appreciated already.
-	The paper offers some experimental poking and probing to find the source of the issue. But that part of the paper (section 3) is disconnected from what follows, mainly because hardness there is not a single point’s notion, but rather that of regions of space with a heterogeneous presence of classes. This is quite intuitive in fact. Later, in section 4, hard simply means high loss. This isn’t quite the same, since the former notion means rather being near the decision boundary, which is not captured by just having high loss. (Also, the loss is not specified.)
-	Some issues with Section 3: the notions of “task” needs a more formal definition, and then subtasks, and union of tasks, priors on tasks, etc. it’s all too vague. The term “non-computable” has very specific meaning, best to avoid. Figure 2 is very badly explained (I believe the green curve is the number of classes represented by one element or more, while the red curve is the number of classes represented by 5 elements or more, but I had to figure it out on my own). The whole paragraph preceding Figure 3 is hard to follow. I sort of can make up what is going, especially with the hindsight of Section 4, since it’s basically a variant of the proposed schedule (easy to hard making sure all clusters, as proxy to classes, are represented) without the feature loss, but it needs a rewriting.
-	It is important to emphasize that the notion of “easy” and “hard” can change along the training, because they are relative to what the weights are at the hidden layer. Features of some samples may be not very separable at some stage, but they may become very separable later. The suggested algorithm does this reevaluation, but this is not made clear early on.
-	In Section 4, the sentence where S_t(x) is mentioned is unclear. I assume “surpass” means achieving a better loss. Also later M_t (a margin) is used, when I think what is meant is S_t (a set). The whole notation (e.g. “topk”, indexing that is not subscripted, non-math mode math) is bad.
-	If L_t is indeed a loss (and not a “performance” like it’s sometimes referred to, as in minus loss), then I assume larger losses means that the weight on the feature loss in equation (3) should be larger. So I think a minus sign is missing in the exponent of equation (2), and also in the algorithm.
-	I’m not sure if the experiments actually show a speedup, in the sense of what the authors started out motivating. A speedup, for me, would look like the training progress curves are basically compressed: everything happens sooner, in terms of epochs. Instead, what we have is basically the same shape curve but with a slight boost in performance (Figure 4.) It’s totally disingenuous to say “this is a great boost in speed” (end of Section 5.2) by saying it took 30 epochs for the non-curriculum version to get to its performance, when within 4 epochs (just like the curriculum version) it was at its final performance basically.
-	So the real conclusion here is that this curriculum may not have sped up the training in the way we expect it at all. However, the gradual introduction of badly classified samples in later epochs, while essentially replacing their features with similarly classified samples for earlier epochs, has somehow regularized the training. The authors do not discuss this at all, and I think draw the wrong conclusion from the results.


We propose a new generic type of stochastic neurons, called q-neurons, that considers activation functions based on Jackson's q-derivatives, with stochastic parameters q. Our generalization of neural network architectures with q-neurons is shown to be both scalable and very easy to implement. We demonstrate experimentally consistently improved performances over state-of-the-art standard activation functions, both on training and testing loss functions.

############ Updated Review #################
I have read the author(s)' rebuttal. My decision stays unchanged. In my opinion, this first step is not significant enough, and the presentation is clearly below the acceptance threshold for ICLR. Additionally, the author(s) did not update their submission to reflect the changes. I thereby recommending rejection to this submission.
##########################################
This work proposes to replace the regular deterministic activation functions used in artificial neural nets with stochastic variants. In particular, the author(s) considered the q-derivatives of standard activation functions.
The author(s) claimed that ``By Proposition 2, the p-derivative of the q-activation g_q(x) agrees with the original activation function f.'' I have trouble understanding this. I assume by original activation function the author(s) meant f(x), then how can Eqn (4) agree with f(x)?
At the bottom of pp. 3, the author(s) wrote: ``q-neuron ... combines stochasticity and some second-order information in an easy-to-compute way.'' I definitely can not agree with this point. Basically, q-neuron is the ``derivative'' of the original activation function, so there is no surprise that its derivative links to the second derivative of f(x). I can always use the high order derivative of some function as activation and claim now we are combining even higher order information into the neural network, but does that help? I don't think so.
It really annoys me to see that four out of the eight pages are occupied by gigantic figures, which should be placed in supplementary material in my opinion. A simple table could do the job equally well in the text. We are not interested in nitty-gritty details on how the training evolves. Let alone the datasets tested are all small-scale image classification tasks. At least the author(s) should diversify their test beds (e.g., NLP tasks and ImageNet scale experiments) and model architectures (e.g., RNN, ResNet).
What's also missing from their experiments is a fair comparison with the real counterparts. I do not see comparisons with dropouts, and to more direct activation function randomization schemes (additive noise to regular activation functions).
To summarize, I can not approve this paper as it falls well below the acceptance level of an ICLR. In its current form, it's more like a sketchy note rather than a serious academic paper. I would encourage the authors to significantly enrich the content of this writing before considering resubmitting to another venue.


The application of multi-modal generative models by means of a Variational Auto Encoder (VAE) is an upcoming research topic for sensor fusion and bi-directional modality exchange. This contribution gives insights into the learned joint latent representation and shows that expressiveness and coherence are decisive properties for multi-modal datasets. Furthermore, we propose a multi-modal VAE derived from the full joint marginal log-likelihood that is able to learn the most meaningful representation for ambiguous observations. Since the properties of multi-modal sensor setups are essential for our approach but hardly available, we also propose a technique to generate correlated datasets from uni-modal ones.

This paper proposes an objective, M^2VAE, for multi-modal VAEs, which is supposed to learn a more meaningful latent space representation. To summarize my understanding of the proposed objective, in the bi-modal case, it combines both objectives of TELBO [1] and JMVAE-kl [2] with some hyperparameters to learn the uni-modal encoders. The terms of Eqns 7,8, and 9 are equivalent to TELBO and Eqns 9 and 10 are JMVAE-kl. It would be very beneficial for the readers if you could more clearly contrast your objective with the related work given how similar they are.
Given these similarities between objectives, its unclear why JMVAE-Zero was chosen over JMVAE-kl as a baseline. Furthermore, the reasoning for the improvement of the ELBO of M^2VAE over the baselines in Section 5.3 is unclear, given the similarities between the objectives.
The qualitative figures throughout the paper are hard to interpret. By looking at Fig 4., I cannot tell which latent space is best.
“one can see from Fig. 4 that the most coherent latent space distribution was learned by the proposed M^2VAE”
What is meant by ‘coherent latent space’?
This paper was hard to follow and there are a number of typos throughout the paper. For instance, the labels within Fig 4 and the caption contradict themselves. If the clarity and quality of the writing could be improved then perhaps the contributions may become more evident.
[1] R. Vedantam, I. Fischer, J. Huang, and K. Murphy. Generative Models of Visually GroundedImagination. ArXiv e-prints, May 2017.
[2] M. Suzuki, K. Nakayama, and Y. Matsuo. Improving Bi-directional Generation betweenDifferent Modalities with Variational Autoencoders. ArXiv e-prints, January 2018


Sequential learning of multiple tasks in artificial neural networks using gradient descent leads to catastrophic forgetting, whereby previously learned knowledge is erased during learning of new, disjoint knowledge. Here, we propose a new approach to sequential learning which leverages the recent discovery of adversarial examples. We use adversarial subspaces from previous tasks to enable learning of new tasks with less interference. We apply our method to sequentially learning to classify digits 0, 1, 2 (task 1), 4, 5, 6, (task 2), and 7, 8, 9 (task 3) in MNIST (disjoint MNIST task). We compare and combine our Adversarial Direction (AD) method with the recently proposed Elastic Weight Consolidation (EWC) method for sequential learning. We train each task for 20 epochs, which yields good initial performance (99.24% correct task 1 performance). After training task 2, and then task 3, both plain gradient descent (PGD) and EWC largely forget task 1 (task 1 accuracy 32.95% for PGD and 41.02% for EWC), while our combined approach (AD+EWC) still achieves 94.53% correct on task 1. We obtain similar results with a much more difficult disjoint CIFAR10 task (70.10% initial task 1 performance, 67.73% after learning tasks 2 and 3 for AD+EWC, while PGD and EWC both fall to chance level). We confirm qualitatively similar results for EMNIST with 5 tasks and under 3 variants of our approach. Our results suggest that AD+EWC can provide better sequential learning performance than either PGD or EWC.

The paper is about a new method for training neural networks in the continual learning setting, where tasks are presented in a sequential manner (and data from the previous task cannot be revisited). The method proposes a new architecture that adds task-parameters parameters to prevent catastrophic forgetting.
To my understanding, the paper proposes a modification to EWC in which the capacity of the network is augmented after a new task is added. Unlike similar methods (like Progressive networks, see bellow), this augmentation is input agnostic. It acts as a correction of the model parameters such that the new task can be easier to train while still maintaining the 'normal parameters' close to the ones of the initial task (as in EWC). I find this idea interesting and certainly worth publishing. In my view, the paper cannot be published in its current state. With the current presentation it is very difficult to understand what is being proposed. Please correct me if I misunderstood the work.
The writing of the manuscript needs significant improvement. I read it carefully several times and I am still not sure of how exactly the model is trained. I had to read the paper by Elsayed et al 2018, to have an idea of what could have been proposed here. As I mentioned, the paper has novel and interesting ideas, but it would be greatly improved with some important re-writing. Please find bellow some questions.
- In the second to last paragraph of page two, the authors say that: instead of adding a perturbation that would force the network to perform a misclassification, tune it using "the input's own correct class to assist correct classification". If the gradients are computed with respect to the correct class of a given input, why is this called an adversarial perturbation?
- Elsayed maintain the parameters of the first task fixed and train a fixed input-agnostic correction that can be added to the input such that a second task can be trained (with a re-mapping the outputs). Applying Elsayed et al 2018 to the continual learning setting, the model should only learn correction for task 2 (and 3). How do the authors compute the corrections for task 1? Computing a correction requires having access to the training data.
- The authors use the FGSM method to compute "adversarial perturbations". This method was proposed as a proxy for performing gradient descent to minimize the computational load required for finding adversarial examples. In this application, unlike the case of adversarial perturbations, the memories don't need to be constrained to be smaller than a given epsilon. What is the motivation of using this method? How do you explain the difference in the results.
- Having mentioned this, both W_task and M_task are updated by minimizing the same loss function (ignoring the difference of using FGSD or not). In that case, why is it needed to have a factorized form W_task * M_task instead of a single bias?
- Throughout the paper the authors say that the long term memory lies on the "intersection of adversarial subspaces". It is not clear at all why this should be the case. The authors do not explain adversarial subspaces corresponding to which model.
- The authors should cite the Progressive networks as this is a very related work. Unlike progressive networks, this work proposes and augmentation that is input agnostic which is interesting. https://arxiv.org/abs/1606.04671
- With EWC, once the model is trained, one does not need to know the task being evaluated at test time. This is not the case in the proposed model. This should be clarified. Also, when having many tasks mapping to the same input, the fair way of comparing to EWC would to have a different head per task. This baselines should be included.
- What are the task specific functions g_taskA and g_taskB?
- Adding an explicit algorithm, the exact loss functions used should help clarifying the proposed method.
- The paper would be stronger if more complex tasks would be added.


LARGE SCALE GAN TRAINING FOR HIGH FIDELITY NATURAL IMAGE SYNTHESIS
Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128×128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Fréchet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.65.

Summary:
The authors present a empirical investigation of methods for scaling GANs to complex datasets, such as ImageNet, for class-conditioned image generation. They first build and describe a strong baseline based on recently proposed techniques for GANs and push the performance on large datasets with several modifications presented sequentially, to obtain strong state-of-the-art IS/FID scores, as well as impressive visual results. The authors propose a simple truncation trick to control the fidelity/variance which is interesting on its own but cannot always scale with the architecture. The authors further propose a orthogonalization-based regularization to mitigate this problem. An investigation of training collapse at large scale is also performed; the authors investigate some regularization schemes based on gathered empirical evidence. As a result, they explore and discard Spectral Normalization of the generator as a way to prevent collapse and show that a severe tradeoff between stability and quality can be controlled when using zero-centered gradient penalties in the Discriminator. In the end, no solution that can ensure quality and stability is found, except having prohibitively large amounts of data (~300M images). Models are evaluated on the ImageNet and on this internal, bigger dataset.
Pros:
- This investigation gives a significant amount of insights on GAN stability and performance at large scales, which should be useful for anyone working with GANs on complex datasets (and that have access to great computational resources).
- Even though commonly used evaluations metrics for GANs are still not fully adequate, the authors obtain quantitative performance significantly beyond previous work, which seems indeed correlated with remarkable visual results.
- The baseline and added modifications are well presented and clearly explained. The Appendices also have great value in that regard.
Cons:
- Discussions sometimes lack depth or are absent.
For example, it is unclear to me why some larger models are not amenable to truncation. Besides visible artifacts, what does it mean? Why does a smoother G reduces those artifacts? Were samples from those networks better without using truncation? Why would this be?
Authors report how wider networks perform best, and how deeper networks degrade performance. Again, discussions are lacking, and it doesn’t seem the authors tried to understand why such behaviors were shown.
Even though this is mostly an empirical investigation, I think some more efforts should be put in understanding and explaining why some of those behaviors are shown, as I think it can bootstrap future work more easily.
- In Section 3.1 : “Across runs in Table 1, we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation compared to 60% with Orthogonal Regularization.” For me, this is not particularly clear. Is this something the reader should understand from Table 1?
- I question the choice of sections chosen to be in the main paper/appendices. I greatly appreciated the negative results reported in the main text as well as in the appendices and this has significant value. However, as this is to me mostly a detailed empirical investigation and presentation of high-performance GANs on large scales, I would be likely to share this with colleagues who want to tackle similar problems. In this case, if future readers limit themselves to the main text, I think it can have more value to present some content form Appendix B and C than to have more than a full page on stability investigations and attempted tricks that turned out not to be used to reach maximal performance. However I do not want to discourage publishing of negative results, and I definitely wish to see this investigation in the paper, but I merely question the positioning of such information. With regard to my first negative point above about the lack of discussions, it seems the analysis of Section 4 is disproportionate compared to other places.
Suggestions/Comments:
- Regarding the diversity/fidelity tradeoff using different truncation thresholds, I think constraining the norm of the sampled noise vectors to the exact threshold value (by projecting the samples on the 0-centered hyper-sphere of radius = threshold) could yield even more interesting or more informative Figures, as obtained scores or samples on the edge of that hyper-sphere might provide information on the ‘guaranteed’ (not proven) quality/fidelity of samples mapped from inside that hyper-sphere.
- In Appendix D, the Figures could be slightly clarified by using a colored heatmap to color the curve, with colors corresponding to the threshold values. Similar curves could also be produced with the hyper-sphere projection proposed above to have a slightly clearer idea of the behavior on the limit of that hyper-sphere.
- In Section 4.2, in the second paragraph, you refer to Appendix F and describe “sharp upward jump at collapse” in D’s loss. However, it seems the only Figure showing D’s loss when unconstrained is Figure 26, in which it is hard to notice any significant jump in the loss.
- In Appendix F, Figure 20 d), the title seems wrong. It seems to report sigma^2 values, but the title says “losses”.
This investigation of GAN scalability is successful results-wise even though the inability to stabilize training without sacrificing great performance on ImageNet is disappointing. The improvement over previous SOTA is definitely significant. This work thus shows a modern GAN architecture for complex datasets that could be a strong basis for future work. However, I think the paper could and should be improved with some more detailed analysis and discussions of exhibited behaviors in order to further guide and encourage future work. It could also be clarified on some aspects, and potentially re-structured a bit to be better align with its probable impact directions.  I would also be curious to see the proposed techniques applied on simpler datasets. Can this be useful for someone having less compute power and working on something similar to CelebA?


Modern learning models are characterized by large hyperparameter spaces. In order to adequately explore these large spaces, we must evaluate a large number of configurations, typically orders of magnitude more configurations than available parallel workers. Given the growing costs of model training, we would ideally like to perform this search in roughly the same wall-clock time needed to train a single model. In this work, we tackle this challenge by introducing ASHA, a simple and robust hyperparameter tuning algorithm with solid theoretical underpinnings that exploits parallelism and aggressive early-stopping. Our extensive empirical results show that ASHA outperforms state-of-the-art hyperparameter tuning methods; scales linearly with the number of workers in distributed settings; converges to a high quality configuration in half the time taken by Vizier (Google's internal hyperparameter tuning service) in an experiment with 500 workers; and beats the published result for a near state-of-the-art LSTM architecture in under 2× the time to train a single model.

In this paper, the authors propose an extension of the Successive Halving Algorithm (SHA) allowing its deployment in a distributed setting, a so-called massively parallel setting. The proposed algorithm is relatively straightforward and the main contribution appears to be in the experimental validation of the method.
Quality: The work appears to be of a good quality. The experiments could benefit from some more rigorous statistical analysis, but that would most likely require a higher number of repetitions, which is understandably hard to do in a large-scale setting.
Clarity: In general, the paper is well written. However, the presentation of Algorithm 3.2 was confusing. More comments on this below.
Originality: The contribution is incremental, but important.
Justification: The problem is an important one. There is room for more research exploring the optimization of hyperparameters for large architectures such as deep networks.
Overall I think the paper is good enough for acceptation, but I found some elements that deserve attention. The experimental section is a bit perplexing, mostly in the first experiments on the sequential approaches. The final experiment on the large-scale setting is disappointing because it only compares ASHA with an underpowered version of Vizier, so the demonstration is not as impressive as it could be. Furthermore, PBT was discarded based on results from a small-scale benchmark, which were not very convincing either (possibly significantly better on one of two versions of the CIFAR-10 benchmark). If the authors have other reasons as to why PBT was not a good candidate for comparison, they should bring them forth.
Find below some more comments and suggestions:
Regarding Algorithm 3.2, promotability (line 12) is never defined explicitly, so this can lead to confusion (it did in my case). I think promotability also requires that at least eta jobs are finished in the rung k, am I correct? If so it is missing from the definition. Perhaps a subroutine should be defined?
It feels odd to me that the first experiment mentioned in the paper is tucked away in Appendix 1. I find it breaks the flow of the paper. The fact that SHA outperforms Fabolas I believe is one of the important claims of the paper. Hence, the result should probably not be in an Appendix. I would suggest putting Figure 3 in the Appendix instead, or removing/condensing Figure 2, which is nice but wastes a lot of space. I also fail to grasp the difference between the CIFAR-10 benchmarks in Appendix 1 and those in Section 4.2. It seems they could be joined in one single experiment comprising SHA v Hyperband v Fabolas v PBT (perhaps removing a variant of Hyperband to reduce clutter).
I also do not think the comparison between SHA and ASHA in a sequential case is relevant. The behavior of ASHA in the distributed case will be different than in the sequential case, so the comparison between the two variants of SHA does not bring any useful information. If I followed the method correctly, in the 9 worker example with eta=3, the first round of jobs would be all jobs at the lowest bracket (s=0), which would be followed by a round of jobs at the next bracket (3 jobs at s=1), and so on. Hence, the scheduling behavior would be exactly the same as SHA (albeit distributed instead of sequential). Am I correct in my assessment? If so, perhaps ASHA should just be removed from the sequential experiments.
As a point of sale, it might be interesting to provide the performance of models with manually tuned hyperparameters as a reference (i.e., the recommended hyperparameters in the reference implementations of those works that were cited).
Appendix A.2 serves no purpose and should probably be removed.
Section 4.3 & 4.3.1: In both cases, what is the running time R for a single model?


Stochastic Gradient Descent (SGD) methods using randomly selected batches are widely-used to train neural network (NN) models. Performing design exploration to find the best NN for a particular task often requires extensive training with different models on a large dataset, which is very computationally expensive. The most straightforward method to accelerate this computation is to distribute the batch of SGD over multiple processors. To keep the distributed processors fully utilized requires commensurately growing the batch size; however, large batch training often times leads to degradation in accuracy, poor generalization, and even poor robustness to adversarial attacks. Existing solutions for large batch training either significantly degrade accuracy or require massive hyper-parameter tuning. To address this issue, we propose a novel large batch training method which combines recent results in adversarial training (to regularize against "sharp minima") and second order optimization (to use curvature information to change batch size adaptively during training). We extensively evaluate our method on Cifar-10/100, SVHN, TinyImageNet, and ImageNet datasets, using multiple NNs, including residual networks as well as smaller networks for mobile applications such as SqueezeNext. Our new approach exceeds the performance of the existing solutions in terms of both accuracy and the number of SGD iterations (up to 1% and 5×, respectively). We emphasize that this is achieved without any additional hyper-parameter tuning to tailor our proposed method in any of these experiments.

Based on my understanding, this paper describes a novel approach for addressing the large batch training problem. The authors propose increasing the batch size based on reductions in the largest eigenvalue of the Hessian. This is combined with adversarial training using the fast gradient sign method to reduce the total number of iterations required for training and improve generalization performance. Unfortunately, although the numerical results seem quite promising, the algorithm and its explanation and details are not described clearly in the paper, which makes me lean towards rejection. I describe this more fully below:
1. Description of the Algorithm
The description of the algorithm in Section 3.1 is simply not clear, and lacks clear exposition motivating why the algorithm ought to work. To add to this confusion, there appear to be some inconsistencies between the (brief) description of the method and the description given in the Main Contributions and Limitations section in the Introduction.
As an example, in Section 3.1, the approach for computing the eigenvalue of the Hessian is not described. Which eigenvalue is computed? How is this done? What is the batch size used in this computation? Is it computed over the full training set? The Limitations section briefly describes this (power iteration to tolerance <= 10^-2), but this should be elaborated on in Section 3.1. In fact, the limitations should not be discussed until a clear description of the algorithm is given.
The introduction makes this even more confusing by claiming the second order information is computed by “backpropagating the Hessian operator”. This seems to imply that the 3rd derivative information is computed for second-order information. Later in the Introduction, the authors claim to use Hessian matvecs to perform the power iteration. I believe that the authors mean that the Hessian-vector product is obtained by differentiating the product g’v (a scalar quantity).
In addition, it was not described how the learning rate is changed in the algorithm. Later in the experiments, none of the additional hyperparameters in the procedure are given, such as the duration factor, kappa, the hyperparameters in the adversarial training, and more. This all ought to be included for completeness.
2. Questions about Details of the Algorithm
If it is indeed the case that the authors are using power iteration to compute the largest eigenvalue, why not use Lanczos method as it typically works better for symmetric matrices? In addition, if the intention was to compute the largest eigenvalue of the Hessian, one must be wary that the power iteration/Lanczos method computes the eigenvalue with largest magnitude (the absolute value of lambda), which may mean that it’s possible that the algorithm is utilizing negative curvature information rather than positive curvature information (particularly in the earlier epochs), which may contradict their intuition based on flat minima. This needs to be addressed.
Secondly, there is no explanation as to why increasing the batch size would lead to consistent decrease in the eigenvalues of the Hessian. This is certainly not true for all optimization problems. Even if the flat minima/sharp minima hypothesis is assumed, is it possible for the iterates after increasing the batch size to still tend towards sharper minimizers after being in a flat region? This intuition and explanation needs to be expanded on (and argued for) in order for the algorithm to make any conceptual sense.
Lastly, why is the duration factor needed to increase the batch size if the eigenvalue condition fails? if the duration factor is removed, how does the batch size evolve? Is it necessary? How is the duration factor tuned?
3. Inconsequential Theoretical Results
The authors also prove a theorem bounding the expected optimality gap with adaptive batch sizes. On closer look, this is a simple adaptation of the result by Bottou, Curtis, and Nocedal [2] and does not utilize any of the algorithmic mechanisms described in the paper. Hence, the theoretical result is not novel, does not provide any additional insight on the algorithm, and could be applied to any adaptive/changing batch size SG algorithm. In my opinion, this ought to be removed. (Assumption 2 is also mentioned in the main paper, but is only described in the Appendix.)
4. Additional Considerations
The paper is missing much work done by Nocedal’s group on increasing batch sizes (some of which utilize the L-BFGS approximation to the Hessian); see [1, 3].
Other relevant work by Sagun, Bengio, and others on large batch training, flat minima, and the Hessian in deep learning ought to be included as well; see [4-7].
Lastly, the algorithm demonstrates some significant improvements on the number of iterations. However, efficiency with respect to epochs is not discussed. It may make sense to plot test loss/error against epochs and batch size against iterations for clarity.
Typos/Grammatical Errors:
- Page 2: Should not state “(We refer to this method as ABS)”, easier to include by including (ABS) after Adaptive Batch Size in the beginning of the bullet point.
- Page 6: Section 4: “information” not “informatino”
- Page 6: Section 4: “the” not “teh”
- Page 7: Section 4.1: “confirms” not “confirming”
- Page 7: Section 4.1: no “a” in “a very consistent performance”
Summary:
Overall, although the paper presents some promising numerical results, it lacks a detailed description and explanation of the algorithm to be worthy of publication. It leaves many aspects of the algorithm open to the reader’s interpretation, and I do not believe I could reproduce the results with the information provided. The manuscript needs significant changes to the detail, structure, and writing before it can be considered for publication.
References:
[1] Bollapragada, Raghu, et al. "A progressive batching L-BFGS method for machine learning." arXiv preprint arXiv:1802.05374(2018).
[2] Bottou, Léon, Frank E. Curtis, and Jorge Nocedal. "Optimization methods for large-scale machine learning." SIAM Review 60.2 (2018): 223-311.
[3] Byrd, Richard H., et al. "Sample size selection in optimization methods for machine learning." Mathematical programming134.1 (2012): 127-155.
[4] Chaudhari, Pratik, et al. "Entropy-sgd: Biasing gradient descent into wide valleys." arXiv preprint arXiv:1611.01838(2016).
[5] Jastrzębski, Stanisław, et al. "DNN's Sharpest Directions Along the SGD Trajectory." arXiv preprint arXiv:1807.05031(2018).
[6] Sagun, Levent, et al. "Empirical Analysis of the Hessian of Over-Parametrized Neural Networks." arXiv preprint arXiv:1706.04454 (2017).
[7] Zhu, Zhanxing, et al. "The Regularization Effects of Anisotropic Noise in Stochastic Gradient Descent." arXiv preprint arXiv:1803.00195 (2018).


Learning causal relations from observational time series with nonlinear interactions and complex causal structures is a key component of human intelligence, and has a wide range of applications. Although neural nets have demonstrated their effectiveness in a variety of fields, their application in learning causal relations has been scarce. This is due to both a lack of theoretical results connecting risk minimization and causality (enabling function approximators like neural nets to apply), and a lack of scalability in prior causal measures to allow for expressive function approximators like neural nets to apply. In this work, we propose a novel causal measure and algorithm using risk minimization to infer causal relations from time series. We demonstrate the effectiveness and scalability of our algorithms to learn nonlinear causal models in synthetic datasets as comparing to other methods, and its effectiveness in inferring causal relations in a video game environment and real-world heart-rate vs. breath-rate and rat brain EEG datasets.

This paper aims to estimate time-delayed, nonlinear causal influences from time series under the causal sufficiency assumption. It is easy to follow and contains a lot of empirical results. Thanks for the results, but I have several questions.
First, In Theorem 2, which seems to be a main result of the paper, the authors were concerned with the condition when W_{ji} >0, but there is not conclusion if W_{ji} =0. In order to correctly estimate causal relations from data, both cases must be considered.
Second, the conclusion of Theorem 2 seems to be flawed. Let me try to make it clear with the following example. Suppose x^1_{t-2} directly causes x^2_{t-1} and that x^2_{t-1} directly causes x^3_{t}, without a direct influence from x^1_{t-2}  to x^3_{t}. Then when minimizing (2), we have the following results step by step:
1) The noise standard deviation in x^2_{t-1}, denoted by \eta_2, may be non-zero. This is because we minimize a tradeoff of the prediction error (the first term in (2)) and a function of the reciprocal of the noise standard deviation \eta_2 (the second term in (2)), not only the prediction error.
2) If \eta_2 is non-zero, then x^1_{t-2} will be useful for the purpose of predicting x^3_{t}. (Note that if \eta_2 is zero, then x^1_{t-2} is not useful for predicting x^3_{t).) From the d-separation perspective, this is because x^1_{t-2} and x^3_{t} are not d-separated by x^2_{t-1} + \eta_2 \cdot \epsilon_2, although they are d-separated by x^2_{t-1}. Then the causal Markov condition tells use that x^1_{t-2} and x^3_{t} are not independent conditional on x^2_{t-1} + \eta_2 \cdot \epsilon_2, which means that x^1_{t-2} is useful for predicting x^3_{t}.
3) Given that x^1_{t-2} is useful for predicting x^3_{t}, when (2) is minimized, \eta_1 will not go to infinity, resulting in a non-zero W_{13), which *mistakenly* tells us that X^{1}_{t-1} directly structurally causes x^{(3)}_t.
This illustrates that the conclusion of Theorem 2 may be wrong.  I believe this is because the proof of Theorem 2 is flawed in lines 5-6 on Page 16. It does not seem sensible to drop X^{j}_{t-1} + \eta_X \cdot \epsilon_X and attain a smaller value of the cost function at the same time. Please carefully check it, especially the argument given in lines 10-13.
Third, it is rather surprising that the authors didn't mention anything about the traditional causal discovery methods based on conditional independence relations in the data, known as constraint-based methods, such as the PC algorithm (Spirtes et al., 1993), IC algorithm (Pearl, 2000), and FCI (Spirtes et al., 1993). Such methods are directly applicable to time-delayed causal relations by further considering the constraint that effects temporally follow the causes.
Fourth, please make it clear that the proposed method aims to estimate "causality-in-mean" because of the formulation in terms of regression. For instance, if x^j_{t-1} influences only the variance of x^i_{t}, but not its mean, then the proposed method may not detect such a causal influence, although the constraint-based methods can.
Any response would be highly appreciated.


We study the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information I(X; T ) between the input X and internal representations T decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true I(X; T ) over these networks is provably either constant (discrete X) or infinite (continuous X). This work explains the discrepancy between theory and experiments, and clarifies what was actually measured by these past works. To this end, we introduce an auxiliary (noisy) DNN framework for which I(X; T ) is a meaningful quantity that depends on the network's parameters. This noisy framework is shown to be a good proxy for the original (deterministic) DNN both in terms of performance and the learned representations. We then develop a rigorous estimator for I(X; T ) in noisy DNNs and observe compression in various models. By relating I(X; T ) in the noisy DNN to an information-theoretic communication problem, we show that compression is driven by the progressive clustering of hidden representations of inputs from the same class. Several methods to directly monitor clustering of hidden representations, both in noisy and deterministic DNNs, are used to show that meaningful clusters form in the T space. Finally, we return to the estimator of I(X; T ) employed in past works, and demonstrate that while it fails to capture the true (vacuous) mutual information, it does serve as a measure for clustering. This clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest.

Response to author comments:
I would like to thank the authors for answering my questions and addressing the issues in their paper. I believe the edits and newly added comments improve the paper.
I found the response regarding the use of your convergence bound very clear. It is a very reasonable use of the bound and now I see how you take advantage of it in your experimental work. However, I believe the description in the paper, in particular, the last two sentences of Remark 1, could still be improved and better explain how a reasonable and computationally feasible n was chosen.
To clarify one of my questions, you correctly assumed that I meant to write the true label, and not the output of the network.
***********
The paper revises the techniques used in Tishby’s and Saxe et al. work to measure mutual information between the data and a hidden layer of a neural network. The authors point out that these previous papers’ measures of mutual information are not meaningful due to lack of clear theoretical assumptions on the randomness that arises in DNNs.
The authors propose to study a perturbed version of a neural network to turn it into a noisy channel making the mutual information estimation meaningful. The perturbed network has isotropic Gaussian noise added to each layer nodes. The authors then propose a method to estimate the mutual information of interest. They suggest that the mutual information describes how distinguishable the hidden representation values are after a Gaussian perturbation (which is equivalent to estimating the means of a mixture of Gaussians). Data clustering per class is identified as the source of compression.
In addition to proposing a way to estimate a mutual information of a stochastic network, the authors analyze the compression that occurs in stochastic neural networks.
It seems that the contribution is empirical, rather than theoretical, as the theoretical result cited is going to appear in a different article. After reading that the authors “develop sample propagation (SP) estimator”, I expected to see a novel approach/algorithm. However, unless I missed something, the proposed method for estimating MI for this Gaussian channel is just doing MC estimation (and no guarantees are established in this paper). The convergence bounds for the SP estimator are presented(Theorem 1), however, the result is cited from another article of the authors, so it is not a contribution of this submission.
Since the authors have this convergence  bound stated in Theorem 1, it would be great to see it being used - how many samples are needed/being used in the experiments? What should the error bars be around mutual information estimates in the experiments? If the bound is too loose for a reasonable number of samples, then what’s the use of it?
The authors perform two types of experiments on MNIST. The first experiment demonstrates that no compression is observed per layer and the mutual information only increases during training (as measured by the binning approach, which is supposed to track the mutual information of the stochastic version of the network). The second experiments demonstrates that deeper layers perform more clustering.
Regarding the first experiment, could the authors clarify how per unit and per entire layer compression estimation differs?
Also, in my opinion, more clustered representations seem to indicate that the mutual information with the output increases. Could the authors comment on how the noise levels in this particular version of a stochastic network affects the mutual information with the output and the clustering? Do more clustered representations lead to increased mutual information of the layer with the output?
I found it fairly difficult to summarize the experimental contribution after the first read. I think the presentation and summary after each experiment could be improved and made more reader friendly. For example, the authors could include a short section before the experiments stating their hypothesis and pointing to the experiment/figure number supporting their hypothesis.


Deep neural networks and decision trees operate on largely separate paradigms; typically, the former performs representation learning with pre-specified architectures, while the latter is characterised by learning hierarchies over pre-specified features with data-driven architectures. We unite the two via adaptive neural trees (ANTs), a model that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers). ANTs allow increased interpretability via hierarchical clustering, e.g., learning meaningful class associations, such as separating natural vs. man-made objects. We demonstrate this on classification and regression tasks, achieving over 99% and 90% accuracy on the MNIST and CIFAR-10 datasets, and outperforming standard neural networks, random forests and gradient boosted trees on the SARCOS dataset. Furthermore, ANT optimisation naturally adapts the architecture to the size and complexity of the training data.

The paper is written rather well, however I find the experiments incomplete and have some reservations about the
method. My main points of critique are:
1.  Combining DT & NN
I have doubts that the way you combine DT &NN  you get the "Best of both world". In some ways your architecture also
shares disadvantages of both:
1.1 Interpretability
Because each node in the tree can a neural network (with arbritrary complexity), this approach looses one central advantage of DT, that is the interpretability of the result.    Each node in the tree can perform arbritrary complex (and hierarchical)
computations. The authors only show one particular example (Fig. 2a), where the model has learned is a reasonable
structure.
1.2 Complexity:
The whole architecture is much more complex than either a neural network or a decision tree. I expect that therefore training these is not easy, and expert knowledge in either DT  or NN may not be enough to use this model.
2. Limited experiments
2.1 The authors only consider 2 experiments from vision (MNIST & CIFAR 10) while proposing a universal method.  To show universality the authors should use data sets from different domains (e.g UCI data sets)
2.2 The authors argue that a  strength of the method   is  that it uses a low number of parameters on average for a forward path (compared to the total parameter size).  I don't find this argument to be convincing. In the limit this would imply a high memorization of the  data.  Also, a similar case can be made for standard CNN, when a particular filter is mostly inactive for some data points.
2.3 The interpretability of DT compared to NN I mentioned earlier.  To make the argument that their method learns the
hierarchical structure of the data , the authors should have added experiments to support this, where  such a hierarchical structure is clearly present and can be evaluated empirically.
--
In light of the extended experiments w.r.t. to 2.1 I increased my score from 5 to 6.  Overall, I still have doubts about the interpretability and complexity of the proposed method.
Complexity:  "but all the intuitions needed would come solely from training NN".    I disagree with this response.   The architecture is a mix between a tree (hard, decision-tree like error surface,  non-local) and neural network (smooth, mostly convex error surface). This also implies that the training process and its behavior will possess patterns and challenges of both approaches.
Interpretability:  I think the method misses "priors" that enforce credit assignment.  Partitioning the problem in subp-roblems should be done via the tree components, whereas processing (such as image filtering) should be done in the network nodes. However,  the method does not enforce, or encourage this behavior, for instance
via constraints:   also nodes can do partitioning (because neural networks can approximate decision trees)  and edges can do processing (e.g. decisions-trees can be used for mnist).
So I still believe this to be a borderline paper, however, the experiments support a more general applicability.


We train a feed forward neural network with increased robustness against adversarial attacks compared to conventional training approaches. This is achieved using a novel pre-trained building block based on a mean field description of a Boltzmann machine. On the MNIST dataset the method achieves strong adversarial resistance without data augmentation or adversarial training. We show that the increased adversarial resistance is correlated with the generative performance of the underlying Boltzmann machine.

The recent work of Schott et al (which the authors compare results to) proposed the use of Bayes rule inversion as a more robust mechanism for classification under different types of adversarial attacks. The probabilities are approximated with variational autoencoders. During training the inference network is used, but during testing optimization based inference is carried out to compute loglikelihoods.
This paper focuses on the second part, with a different model. Specifically, it proposes a specific Boltzmann machine to be used as a first layer of neural networks for MNIST classification. This Boltzmann machine is pre-trained in two-stages using mean field inference of the binary latent variables and gradient estimation of the parameters.  This pre-trained model is then incorporated into the neural net for MNIST classification.  The existence of couplings J_h among the hidden units means that we have to carry out mean field inference over several iterations to compute the output activations of the model. This is basically analogous to the optimization-based inference proposed by Schott et al. (As a detail, this optimization can be expressed as computation over several layers of a neural net.)
The authors compare to the work of Schott for one type of attack. It would be nice to see more detailed experiments as done in Schott.
Questions:
1- Why not use a single Boltzmann machine with 128 fully connected latent variables? Could you add this experiment please.
2- Why is a two-stage pre-training (Figure 2) process needed? Why not just a single stage?
3- Is the key that you used only 679 patches containing 98% of occurrences in the dataset as the first stage? What if we vary this percentage? How sensitive are the results? Such experiments could be useful to understand better why your method appears to work well.
4- Could you please add the found J_h's to the appendix. This architecture reminds me of the good old MRFs for image denoising. Could it be that what we are seeing is the attack being denoised?
I am puzzled and looking forward to answers to the above questions. I don't yet understand what is the thing that makes this approach appear to work, or why you were able to drop the Bayes inference inversion altogether as done by Schott.
Thanks in advance. I will re-adjust the review rating following your reply.


BAYESIAN PREDICTION OF FUTURE STREET SCENES USING SYNTHETIC LIKELIHOODS
For autonomous agents to successfully operate in the real world, the ability to anticipate future scene states is a key competence. In real-world scenarios, future states become increasingly uncertain and multi-modal, particularly on long time horizons. Dropout based Bayesian inference provides a computationally tractable, theoretically well grounded approach to learn likely hypotheses/models to deal with uncertain futures and make predictions that correspond well to observationsare well calibrated. However, it turns out that such approaches fall short to capture complex real-world scenes, even falling behind in accuracy when compared to the plain deterministic approaches. This is because the used log-likelihood estimate discourages diversity. In this work, we propose a novel Bayesian formulation for anticipating future scene states which leverages synthetic likelihoods that encourage the learning of diverse models to accurately capture the multi-modal nature of future scene states. We show that our approach achieves accurate state-of-the-art predictions and calibrated probabilities through extensive experiments for scene anticipation on Cityscapes dataset. Moreover, we show that our approach generalizes across diverse tasks such as digit generation and precipitation forecasting.

The work proposes a Bayesian neural network model that is a hybrid between autoencoders and GANs, although it is not presented like that. Specifically, the paper starts from a Bayesian Neural Network model, as presented in Gal and Ghahramani, 2016 and makes two modifications.
First, it proposes to define one Bernoulli variational distribution per weight kernel, instead  of per patch (in the original work there was one Bernoulli distribution per patch kernel). As the paper claims, this reduces the complexity to be exponential to the number of weights, instead of the number of patches, which leads to a much smaller number of possible models. Also, because of this modification the same variational distributions are shared between locations, being closer to the convolutional nature of the model.
The second modification is the introduction of synthetic likelihoods. Specifically, in the original network the variational distributions are designed such that the KL-divergence of the true posterior p(ω|X, y) and the approximate posterior q(ω) is minimiezd. This leads to the optimizer encouraging the final model to be close to the mean, thus resulting in less diversity. By re-formulating the KL-divergence, the final objective can be written such that it depends on the likelihood ratio between generated/"fake" samples and "true" data samples. This ratio can then be approximated by a GAN-like discriminator. As the optimizer now is forced to care about the ratio instead of individual samples, the model is more diverse.
Both modifications present some interesting ideas. Specifically, the number of variational parameters is reduced, thus the final models could be much better scaleable. Also, using synthetic likelihoods in a Bayesian context is novel, to the best of my knowledge, and does seem to be somewhat empirically justified.
The negative points of the paper are the following.
- The precise novelty of the first modification is not clearly explained. Indeed, the number of possible models with the proposed approach is reduced. However, what is the degree to which it is reduced. With some rough calculations, for an input image of resolution 224x224, with a kernel size of 3x3 and stride 1, there should be about 90x90 patches. That is roughly a complexity of O(N^2) ~ 8K (N is the number of patches). Consider the proposed variational distributions with 512 outputting channels, this amount to 3x3x512 ~ 4.5K. So, is the advantage mostly when the spatial resolution of the image is very high? What about intermediate layers, where the resolution is typically smaller?
- Although seemingly ok, the experimental validation has some unclarities.
  + First, it is not clear whether it is fair in the MNIST experiment to report results only from the best sampled model, especially considering that the difference from the CVAE baseline is only 0.5%. The standard deviation should also be reported.
  + In Table 2 it is not clear what is compared against what. There are three different variants of the proposed model. The WD-SL does exactly on par with the Bayes-Standard (although for some reason the boldface font is used only for the proposed method. The improvement appears to come from the synthetic likelihoods. Then, there is another "fine-tuned" variant for which only a single time step is reported, namely +0.54 sec. Why not report numbers for all three future time steps? Then, the fine-tuned version (WD-SL-ft) is clearly better than the best baselines of Luc et al., however, the segmentation networks are also quite different (about 7% difference in mIoU), so it is not clear if the improvement really comes from the synhetic likelihoods or from the better segmentation network. In short, the only configuration that appears to be convincing as is is for the 0.06 sec. I would ask the authors to fill in the blank X spots and repeat fair experiments with the baseline.
- Generally, although the paper is ok written, there are several unclarities.
  + Z_K in eq. (4) is not defined, although I guess it's the matrix of the z^{i, j}_{k, k'}
  + In eq (6) is the z x σ a matrix or a scalar operation? Is z a matrix or a scalar?
  + The whole section 3.4 is confusing and it feels as if it is there to fill up space. There is a rather intricate architecture, but it is not clear where it is used. In the first experment a simple fully connected network is used. In the second experiment a ResNet is used. So, where the section 3.4 model used?
  + In the first experiment a fully connected network is used, although the first novelty is about convolutions. I suppose the convolutions are not used here? If not, is that a fair experiment to outline the contributions of the method?
  + It is not clear why considering the mean of the best 5% predictions helps with evaluating the predicted uncertainty? I understand that this follows by the citation, but still an explanation is needed.
All in all, there are some interesting ideas, however, clarifications are required before considering acceptance.


Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks. In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called PBNet. By embracing stochasticity during training, we circumvent the need to approximate the gradient of functions for which the derivative is zero almost always, such as sign(·), while still obtaining a fully Binary Neural Network at test time. Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution. Since all operations in a layer of the PBNet operate on random variables, we introduce stochastic versions of Batch Normalization and max pooling, which transfer well to a deterministic network at test time. We evaluate two related training methods for the PBNet: one in which activation distributions are propagated throughout the network, and one in which binary activations are sampled in each layer. Our experiments indicate that sampling the binary activations is an important element for stochastic training of binary Neural Networks.

## Summary
This work presents a probabilistic training method for binary Neural Network with stochastic versions of Batch Normalization and max pooling. By sampling from the weight distribution an ensemble of Binary Neural Networks could further improve the performance. In the experimental section, the authors compare proposed PBNet with Binarized NN (Hubara et al., 2016) in two image datasets (MNIST and CIFAR10).
In general, the paper was written in poor quality and without enough details. The idea behind the paper is not novel. Stochastic binarization and the (local) reparametrization trick were used to training binary (quantized) neural networks in previous works. The empirical results are not significant.
## Detail comments
Issues with the training algorithm of stochastic neural network
The authors did not give details of the training method and vaguely mentioned that the variational optimization framework (Staines & Barber, 2012). I do not understand equation 1. Since B is binary, the left part of equation 2 is a combination optimization problem. If B is sampled during the training, the gradient would suffer from high variance.
Issues with propagating distributions throughout the network
Equation 3 is based on the assumption of that the activations are random variables from Bernoulli distribution. In equation 4, the activations of the current layer become random variables from Gaussian distribution. How the activations to further propagate?
Issues with ternary Neural Networks in section 2.4
For a ternary NN, the weight will be from a multinomial distribution, I think it will break the assumption used by equation 3.
Issues with empirical evidences
Since the activations are sampled in PBNET-S, a more appropriate baseline should be BNN with stochastic binarization (Hubara et al., 2016) which achieved 89.85% accuracy on CIFAR-10. It means that the proposed methods did not show any significant improvements. By the way BNN with stochastic binarization (Hubara et al., 2016) can also allow for ensemble predictions to improve performance.


We study discrete time dynamical systems governed by the state equation h t+1 = φ(Ah t + Bu t ). Here A, B are weight matrices, φ is an activation function, and u t is the input data. This relation is the backbone of recurrent neural networks (e.g. LSTMs) which have broad applications in sequential learning tasks. We utilize stochastic gradient descent to learn the weight matrices from a finite input/state trajectory {u t , h t } N t=0 . We prove that SGD estimate linearly converges to the ground truth weights while using near-optimal sample size. Our results apply to increasing activations whose derivatives are bounded away from zero. The analysis is based on i) a novel SGD convergence result with nonlinear activations and ii) careful statistical characterization of the state vector. Numerical experiments verify the fast convergence of SGD on ReLU and leaky ReLU in consistence with our theory.

This paper studies the ability of SGD to learn dynamics of a linear system + non-linear activation. That is, in the standard LTI setting, the dynamics of a system evolve according to
h_{t+1} = Ah_t + Bu_t,
on input u_t.
In addition, this paper considers the setting where the evolution is:
h_{t+1} = \phi(Ah_t + Bu_t)
for \phi a non-linear activation function.
This is a difficult problem. Though system identification was for many decades a large and active area in the control community, the understanding of system identification from a modern statistical perspective (understanding sample complexity and computational complexity simultaneously) is surprisingly lacking. This is evidenced by the fact that the first results along these lines for the simplest possible (SISO, LTI) system, came only recently (Hardt, Ma, Recht ’16).
This paper attacks a more general setting, due to the presence of the nonlinearity.
However, the present setting is significantly limited in another sense: the authors assume that the state is observed directly. This is in contrast to the typical situation where we observe only a projection of the state, or possibly even a noisy such projection. Indeed, this is one of the critical complications in the work of Hardt, Ma and Recht. Without it, i.e., under the assumption that the entire state trajectory can be directly observed, much more is possible, and indeed much more has been done. For example, work by Bento, Ibrahimi and Montanari ’10, solves a more difficult problem in that they estimate sparse dynamics (in appropriate sample complexity). Jalali and Sanghavi ’11 generalized the work of Bento et al., to the setting where some of the components of the state are not all observed, but rather some are latent.
The motivating application for this work is estimating RNNs. In this case, the state variable represents the critical information that is carried from one time to the next in the RNN. Presumably the setting here is to show that if indeed data are generated by an RNN, then we can compute this using SGD and backprop. Towards this, the assumption of having access to the internal state is a difficult one. On the one hand, this is a hard and important problem. On the other, we really won’t have access to such an internal state. There are of course other problematic aspects, such as robustness, the inability to use ReLU (Defn 3.1). But the observation model seems important. Again, I believe this is especially so, because the considerable complications present in Hardt, Ma, Rect ’16 specifically seemed to be a consequence of the observation model being partial.
The inability to use ReLU at first look does not seem like a great limitation. But then one problematic aspect here seems that the proof concept and direction critically rely on this, as they basically reduce to the setting of linear activations — something which, presumably, is impossible for something like ReLU. So it is not only the results, but also the developed machinery, that seem to be inherently limited.
This is, overall, an interesting paper, attacking an important and also very challenging area. As with all papers in this vein, we are left with having to make a judgement call on whether this simplified scenario is indeed a good first step towards solving the problems we are hoping to solve. Is it developing the right insight, right tools, etc. While I find there is a lot of interesting and good work in this paper, I am not completely convinced about this last point.


AlphaGo Zero (AGZ) (Silver et al., 2017b) introduced a new tabula rasa reinforcement learning algorithm that has achieved superhuman performance in the games of Go, Chess, and Shogi with no prior knowledge other than the rules of the game. This success naturally begs the question whether it is possible to develop similar high-performance reinforcement learning algorithms for generic sequential decision-making problems (beyond two-player games), using only the constraints of the environment as the "rules." To address this challenge, we start by taking steps towards developing a formal understanding of AGZ. AGZ includes two key innovations: (1) it learns a policy (represented as a neural network) using supervised learning with cross-entropy loss from samples generated via Monte-Carlo Tree Search (MCTS); (2) it uses self-play to learn without training data. We argue that the self-play in AGZ corresponds to learning a Nash equilibrium for the two-player game; and the supervised learning with MCTS is attempting to learn the policy corresponding to the Nash equilibrium, by establishing a novel bound on the difference between the expected return achieved by two policies in terms of the expected KL divergence (cross-entropy) of their induced distributions. To extend AGZ to generic sequential decision-making problems, we introduce a robust MDP framework, in which the agent and nature effectively play a zero-sum game: the agent aims to take actions to maximize reward while nature seeks state transitions, subject to the constraints of that environment, that minimize the agent's reward. For a challenging network scheduling domain, we find that AGZ within the robust MDP framework provides near-optimal performance, matching one of the best known scheduling policies that has taken the networking community three decades of intensive research to develop.

This paper seeks to understand the AlphaGo Zero (AGZ) algorithm and extend the algorithm to regular sequential decision-making problems. Specifically, the paper answers three questions regarding AGZ: (i) What is the optimal policy that AGZ is trying to learn? (ii) Why is cross-entropy the right objective? (iii) How does AGZ extend to generic sequential decision-making problems? This paper shows that AGZ’s optimal policy is a Nash equilibrium, the KL divergence bounds distance to optimal reward, and the two-player zero-sum game could be applied to sequential decision making by introducing the concept of robust MDP. Overall the paper is well written. However, there are several concerns about this paper.
In fact, the key results obtained in this paper is that minimizing the KL-divergence between the parametric policy and the optimal policy (Nash equilibrium) (using SGD) will converge to the optimal policy. It is based on a bound (2), which states that when the KL-divergence between a policy and the optimal policy goes to zero then the return for the policy will approach that of the optimal policy. This bound is not so surprising because as long as certain regularity condition holds, the policies being close should lead to the returns being close. Therefore, it is an overclaim that the KL-divergence bound (2) provides an immediate justification for AGZ’s core learning algorithm. As mentioned earlier, the actual conclusion in Section 4.2 is that minimizing the KL-divergence between the parametric policy and the optimal policy by SGD will converge to the optimal policy, which is straightforward and is not what AlphaGo Zero does. This is because there is an important gap: the MCTS policy is not the same as the optimal policy. The effect of the imperfection in the target policy is not taken into account in the paper. A more interesting question to study is how this gap affect the iterative algorithm, and whether/how the error in the imperfect target policy accumulates/diminishes so that iteratively minimizing KL-divergence with imperfect \pi* (by MCTS) could still lead to optimal policy (Nash equilibrium).
Furthermore, the robust MDP view of the AGZ in sequential decision-making problems is not so impressive either. It is more or less like a reformulation of the AGZ setting in the MDP problem. And it is commonly known that two-player zero-sum game is closely related to minimax robust control. Therefore, it cannot be called as “generalizing AlphaGo Zero” as stated in the title of the paper.


LEARNING ACTIONABLE REPRESENTATIONS WITH GOAL-CONDITIONED POLICIES
Representation learning is a central challenge across a range of machine learning areas. In reinforcement learning, effective and functional representations have the potential to tremendously accelerate learning progress and solve more challenging problems. Most prior work on representation learning has focused on generative approaches, learning representations that capture all the underlying factors of variation in the observation space in a more disentangled or well-ordered manner. In this paper, we instead aim to learn functionally salient representations: representations that are not necessarily complete in terms of capturing all factors of variation in the observation space, but rather aim to capture those factors of variation that are important for decision making -that are "actionable." These representations are aware of the dynamics of the environment, and capture only the elements of the observation that are necessary for decision making rather than all factors of variation, eliminating the need for explicit reconstruction. We show how these learned representations can be useful to improve exploration for sparse reward problems, to enable long horizon hierarchical reinforcement learning, and as a state representation for learning policies for downstream tasks. We evaluate our method on a number of simulated environments, and compare it to prior methods for representation learning, exploration, and hierarchical reinforcement learning.

In this paper, the authors propose a new approach to representation learning in the context of reinforcement learning.
The main idea is that two states should be distinguished *functionally* in terms of the actions that are needed to reach them,
in contrast with generative methods which try to capture all aspects of the state dynamics, even those which are not relevant for the task at hand.
The method of the authors assumes that a goal-conditioned policy is already learned, and they use a Kullback-Leibler-based distance
between policies conditioned by these two states as the loss that the representation learning algorithm should minimize.
The experimental study is based on 6 simulated environments and outlines various properties of the framework.
Overall, the idea is interesting, but the paper suffers from many weaknesses both in the framework description and in the experimental study that make me consider that it is not ready for publication at a good conference like ICLR.
The first weakness of the approach is that it assumes that a learned goal-conditioned policy is already available, and that the representation extracted from it can only be useful for learning "downstream tasks" in a second step. But learning the goal-conditioned policy from the raw input representation in the first place might be the most difficult task. In that respect, wouldn't it be possible to *simultaneously* learn a goal-conditioned policy and the representation it is based on? This is partly suggested when the authors mention that the representation could be learned from only a partial goal-conditioned policy, but this idea definitely needs to be investigated further.
A second point is about unsufficiently clear thoughts about the way to intuitively advocate for the approach. The authors first claim that two states are functionally different if they are reached from different actions. Thinking further about what "functionally" means, I would rather have said that two states are functionally different if different goals can be reached from them. But when looking at the framework, this is close to what the authors do in practice: they use a distance between two *goal*-conditioned policies, not *state*-conditioned policies. To me, the authors have established their framework thinking of the case where the state space and the goal space are identical (as they can condition the goal-conditioned policy by any state=goal). But thinking further to the case where goals and states are different (or at least goals are only a subset of states), probably they would end-up with a different intuitive presentation of their framework. Shouldn't finally D_{act} be a distance between goals rather than between states?
Section 4 lists the properties that can be expected from the framework. To me, the last paragraph of Section 4 should be a subsection 4.4 with a title such as "state abstraction (or clustering?) from actionable representation". And the corresponding properties should come with their own questions and subsection in the experimental study (more about this below).
About the related work, a few remarks:
- The authors do not refer to papers about using auxiliary tasks. Though the purpose of these works is often to supply for additional reward signals in the sparse reward context, then are often concerned with learning efficient representations such as predictive ones.
- The authors refer to Pathak et al. (2017), but not to the more recent Burda et al. (2018) (Large-scale study of curiosity-driven learning) which insists on the idea of inverse dynamical features which is exactly the approach the authors may want to contrast theirs with. To me, they must read it.
- The authors should also read Laversanne-Finot et al. (2018, CoRL) who learn goal space representations and show an ability to extract independently controllable features from that.
A positive side of the experimental study is that the 6 simulated environments are well-chosen, as they illustrate various aspects of what it means to learn an adequate representation. Also, the results described in Fig. 5 are interesting. A side note is that the authors address in this Figure a problem pointed in Penedones et al (2018) about "The Leakage Propagation problem" and that their solution seems more convincing than in the original paper, maybe they should have a look.
But there are also several weaknesses:
- for all experiments, the way to obtain a goal-conditioned policy in the first place is not described. This definitely hampers reproducibility of the work. A study of the effect of various optimization effort on these goal-conditioned policies might also be of interest.
- most importantly, in Section 6.4, 6.5 and 6.6, much too few details are given. Particularly in 6.6, the task is hardly described with a few words. The message a reader can get from this section is not much more than "we are doing something that works, believe us!". So the authors should choose between two options:
* either giving less experimental results, but describing them accurately enough so that other people can try to reproduce them, and analyzing them so that people can extract something more interesting than "with their tuning (which is not described), the framework of the authors outperforms other systems whose tuning is not described either".
* or add a huge appendix with all the missing details.
I'm clearly in favor of the first option.
Some more detailed points or questions about the experimental section:
- not so important, Section 6.2 could be grouped with Section 6.1, or the various competing methods could be described directly in the sections where they are used.
- in Fig. 5, in the four room environment, ARC gets 4 separated clusters. How can the system know that transitions between these clusters are possible?
- in Section 6.3, about the pushing experiment, I would like to argue against the fact that the block position is the important factor and the end-effector position is secundary. Indeed, the end-effector must be correctly positioned so that the block can move. Does ARC capture this important constraint?
- Globally, although it is interesting, Fig.6 only conveys a quite indirect message about the quality of the learned representation.
- Still in Fig. 6, what is described as "blue" appears as violet in the figures and pink in the caption, this does not help when reading for the first time.
- In Section 6.4, Fig.7 a, ARC happens to do better than the oracle. The authors should describe the oracle in more details and discuss why it does not provide a "perfect" representation.
- Still in Section 6.4, the authors insist that ARC outperforms VIME, but from Fig.7, VIME is not among the best performing methods. Why insist on this one? And a deeper discussion of the performance of each method would be much more valuable than just showing these curves.
- Section 6.5 is so short that I do not find it useful at all.
- Section 6.6 should be split into the HRL question and the clustering question, as mentioned above. But this only makes sense if the experiments are properly described, as is it is not useful.
Finally, the discussion is rather empty, and would be much more interesting if the experiments had been analyzed in more details.
typos:
p1: that can knows => know
p7: euclidean => Euclidean