In many machine learning problems, loss functions are weighted sums of several terms. A typical approach to dealing with these is to train multiple separate models with different selections of weights and then either choose the best one according to some criterion or keep multiple models if it is desirable to maintain a diverse set of solutions. This is inefficient both at training and at inference time. We propose a method that allows replacing multiple models trained on one loss function each by a single model trained on a distribution of losses. At test time a model trained this way can be conditioned to generate outputs corresponding to any loss from the training distribution of losses. We demonstrate this approach on three tasks with parametrized losses: β-VAE, learned image compression, and fast style transfer.
Due to the late rebuttal, I was not able to respond during discussion time.
Q1, Q2) sound and look convincing.
Q3) I still cannot find details on this in the paper. How the validation set was chosen? What was the size? The authors need to make all the experiments fully reproducible.
Q4, Q5) ok
My concerns were sufficiently addressed in the current revision, and I will increase my score. However, the paper still feels close to a borderline, but probably, tending to "accept". 
Also, I agree with Review #4, that also wondering about the application of the proposed method to hyperparameter search (Suggestion 1 in my review). Even if this would not be a sota in hyperparameter search, it feels like missing the opportunity to make the paper much stronger, by adding one more nice property to the proposed model.
-----
Generative models often use a loss function that is a weighted sum of different terms, e.g., data-term and regularizer. Let's denote these weights as λ. The paper proposes a method for learning a single model that approximates the result produced by a generative model for a range of loss-term weights λ. The method uses the following mechanisms i) λ-conditioned layers ii) training with a stochastic loss function, that is induced by a (log-uniform) distribution over λ. The performance of the model is demonstrated on the following problems learning β-VAE, image compression, and style transfer. The models clearly demonstrate an ability to approximate problem solutions for a range of coefficients. The paper is clearly written. The experiments, however, need future discussion.
1) The beta-VAE experiments (sec. 4.1)
Q1. While models demonstrate a reasonable behavior on Shapes3d dataset. The samples and reconstructions on CIFAR10 (Figure 8) indicate that all models are not trained well. If this is the case, conclusions might be misleading, since the approximating output of undertrained models might be much simple comparing to well-trained ones. Authors may want to provide a comparison of the trained models with conventional VAEs (with β=1), the reference figures for CIFAR10 are provided, for example, in https://arxiv.org/abs/1606.04934.
Q2. Wider YOTO seems to help a lot, but, what happens to the baseline models of increased size?
"We select the fixed β so that it minimizes the average loss over all β values."
Q3. Was it done directly on a test set, or were validation-data used?
2) Image compression (sec. 4.3)
"Finally, a wider model trained with a larger batch size (“YOTO wider batch16”) closely follows the fixed weight models in the high compression regime and outperforms them in the high quality regime." (Figure 5)
Q4. How is this compared to the baseline with batch16?
Q5. Authors also may want to provide std for provided metrics. The difference does not look statistically significant. 
Suggestion 1: It might also be interesting to see if we can use this technique to perform a hyperparameter search. Train the model, select one the best performing set of hyperparameters, and then train models with this best value.
Overall, the paper proposes an interesting technique, that surprisingly, can work for a range of hyperparameters, and potentially have a high practical impact. However, the empirical evaluation is half-baked, specifically has certain methodological drawbacks e.g., perhaps undertrained beta-VAE model, absence of standard deviations while comparing (close) numerical results, and comparing models with different optimization parameters -- the performance difference might be due to optimization. 
I recommend to reject the paper, however, I will appreciate discussions with authors and other reviewers, and will consider changing my score in case of reasonable argumentation.