Hyperparameter optimization is both a practical issue and an interesting theoretical problem in training of deep architectures. Despite many recent advances the most commonly used methods almost universally involve training multiple and decoupled copies of the model, in effect sampling the hyperparameter space. We show that at a negligible additional computational cost, results can be improved by sampling nonlocal paths instead of points in hyperparameter space. To this end we interpret hyperparameters as controlling the level of correlated noise in training, which can be mapped to an effective temperature. The usually independent instances of the model are coupled and allowed to exchange their hyperparameters throughout the training using the well established parallel tempering technique of statistical physics. Each simulation corresponds then to a unique path, or history, in the joint hyperparameter/model-parameter space. We provide empirical tests of our method, in particular for dropout and learning rate optimization. We observed faster training and improved resistance to overfitting and showed a systematic decrease in the absolute validation error, improving over benchmark results.
REVIEW 2
The paper proposes a new paradigm to perform hyperparameter search by proposing a way to jointly optimize over the hyperparameter space and the parameter space as opposed to the traditional way to performing these in separation (with the hyperparameter search invoking the parameter optimization) The paper’s main claim is that this allows the seach to follow non-local paths in the joint space. The main methodology proposed in the paper is inspired by the idea of parallel tempering from physics. The paper proposes to view parameter learning under a certain hyperparameter as running a langevin chain at a particular temperature. This is motivated by considering common hyperparameters as batch size, dropout rate or learning rate as inducing a specific level of noise to the training process, the variance of which is analogous to the inverse temperature in Langevin diffusion. 
Under the above analogy, the paper proposes to run an analogue of parallel tempering in the following way. Consider running parallel threads of parameter optimization for various fixed choices of hyperparameter values (each representing a temperature). After running certain number of steps of SGD in each configuration, the paper proposes to exchange hyperparameters within a pair of threads, with the chance of exchange depending upon a criteria motivated by metropolis hastings.  
While the paper proposes an interesting idea I have many reservations about the paper and applicability of the idea in practice prompting my score which I outline below. 
1. The first and major shortcoming of the paper is an extreme lack of detail and rigour. The paper seems to be written in a very hand wavy fashion. While that is okay in working towards the proposal of the algorithm, which of course is merely claimed to be inspired by parallel tempering. I believe the paper needs to do an important job of specifying exactly how one selects the notion of temperature for each choice of the hyperparameter. The mapping of the choice to the temperature value is essential for the algorithm to succeed and as described in the paper, there is no hope of replicating the algorithm as the authors are extremely vague about how to figure out this mapping. A second instance of this is diffusion curves in the introduction. The paper does not define what a numerical value of diffusion is. It only relies on a hand-waved understanding of diffusion to even understand presented curves. Note that the paper does not have an appendix and there is no link to code either. 
2. Lack of intuition - for this being a good procedure in terms of selecting a good path eventually. Note that hyperparameter optimization is an optimization procedure and it is very unclear to me why sampling from the distribution which will look like the product distribution over the replicas is even a good idea. If it is the case this needs to be specified carefully.
3. The experiments performed seem to follow a common trend. A single hyperparameter is selected (learning rate/dropout rate), a small grid of fixed values for the hyperparameter are selected and the procedure is performed. In light of this, what a “good” path through the algorithm represents is a good schedule for adjusting these hyperparameters. This is a well known intuition that for parameters like learning rate one needs to adjust them periodically to achieve good training dynamics. Since the comparison is to fixed learning rates, the improvement therefore looks predictable. 
4. Lack of applicability of the algorithm for hyperparameter search in practice - There are a couple of things which make the algorithm restrictive - 
This seems to be useful for hyperparameters that can be in a sense equated to parameters that change the training dynamics. The examples considered in the paper - batch size, learning rate, dropout rate. I think the authors should make this distinction and state it upfront because there are many hyperparameters that we optimize over which change the target distribution and not just the noise. 
Secondly, large scale hyperparameter search takes the idea of separating hyper parameter and parameter search as then it can run in a parallel asynchronous fashion. The algorithm is highly synchronous way of doing this search. While it can be argued that we should move to this paradigm if the results are significant but unfortunately no comparisons with existing hyperparameter search algorithms are provided.