Implicit Maximum Likelihood Estimation
Implicit probabilistic models are models defined naturally in terms of a sampling procedure and often induces a likelihood function that cannot be expressed explicitly. We develop a simple method for estimating parameters in implicit models that does not require knowledge of the form of the likelihood function or any derived quantities, but can be shown to be equivalent to maximizing likelihood under some conditions. Our result holds in the non-asymptotic parametric setting, where both the capacity of the model and the number of data examples are finite. We also demonstrate encouraging experimental results.
 The paper proposes a new algorithm for implicit maximum likelihood estimation based on a fast Nearest Neighbor search. The algorithm can be used to implicitly maximize the likelihood of models for which the former quantity is not intractable but for which sampling is easy which is typically the case for implicit models. The paper shows that under some conditions the optimal solution of the algorithm corresponds to the MLE solution and provides some experimental evidence that the method leads to a higher likelihood. However, The paper lacks clarity and the experiments are not really convincing. Here are some remarks: Experiments: - The estimated likelihood was reported on table 1 using parson window which is known to have bad scaling behavior with the dimension of data. In the end, the table compares methods that maximize different objectives and are evaluated with an unreliable metric. Here are two possible experiments that could be more informative: - Consider toy examples for which the likelihood can be evaluated and the MLE obtained easily and then compare with the proposed method. This would already give a good sense of how well the algorithm behaves in simple cases. - Another possibility is to use generative models like Real-NVP for which the likelihood can also be computed in closed form. This would allow comparing the proposed algorithm to direct likelihood maximization on more complicated datasets as done in [1]. It seems like having experiments of this nature is far more convincing than a long justification for why the results are not necessarily state-of-the-art. - There are way too many samples on the figures so it is very hard to perform any visual assessment. Theory: - More discussions of the assumptions are needed, concrete examples for which these assumptions hold or not would be very useful. - Lemma 2 is a direct consequence of the following result: if p is continuous at x_0 then x_0 is a Lebesgue point. General remarks on the paper: - What complexity is the nearest neighbor algorithm? Since it is crucial for the proposed method to be scalable it is worth presenting this algorithm at a high level in the main paper. - The discussion in section 3 could be much more concise if concrete examples and figures were provided. Most of the facts discussed in that section are generally well understood, so conciseness is very appreciated in this case. - « A secondary issue that is more easily solvable is that samples presented in papers are sometimes cherry-picked; as a result, they capture the maximum sample quality, but not necessarily the mean sample quality. » Could you please provide an example of such paper? I would be very interested in having a closer look. - In the last paragraph of section 5, it is said that although the samples may not be state of the art in terms of precision, other methods which achieve better precision « may » have less recall. It would be good to have empirical evidence to back this claim. Revision: Although this paper presents an interesting idea, there is a serious lack of evidence to support the claims in the paper: - Missing experimental evidence for the efficiency of the NN search algorithm. - Experiments are using Parzen window for estimating likelihood which are known to be unreliable in high dimensions. - None of the suggested experiments were considered. In my opinion these experiments could improve the quality of this work. - Moreover, as mentioned by reviewer 1, Grover et al., 2017 provides evidence contrary to what the authors claim but this was never addressed so far in the paper. - Theorem 1 makes rather strong assumptions: as pointed out by reviewer 1, assumption 3 is unlikely to hold for the distributions used in practice For these reasons I recommend a clear reject. [1] I. Danihelka, B. Lakshminarayanan, B. Uria, D. Wierstra, and P. Dayan. Comparison of Maximum Likelihood and GAN-based training of Real NVPs.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Implicit Maximum Likelihood Estimation
Implicit probabilistic models are models defined naturally in terms of a sampling procedure and often induces a likelihood function that cannot be expressed explicitly. We develop a simple method for estimating parameters in implicit models that does not require knowledge of the form of the likelihood function or any derived quantities, but can be shown to be equivalent to maximizing likelihood under some conditions. Our result holds in the non-asymptotic parametric setting, where both the capacity of the model and the number of data examples are finite. We also demonstrate encouraging experimental results.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper proposes a new algorithm for implicit maximum likelihood estimation based on a fast Nearest Neighbor search. The algorithm can be used to implicitly maximize the likelihood of models for which the former quantity is not intractable but for which sampling is easy which is typically the case for implicit models. The paper shows that under some conditions the optimal solution of the algorithm corresponds to the MLE solution and provides some experimental evidence that the method leads to a higher likelihood. However, The paper lacks clarity and the experiments are not really convincing. Here are some remarks: Experiments: - The estimated likelihood was reported on table 1 using parson window which is known to have bad scaling behavior with the dimension of data. In the end, the table compares methods that maximize different objectives and are evaluated with an unreliable metric. Here are two possible experiments that could be more informative: - Consider toy examples for which the likelihood can be evaluated and the MLE obtained easily and then compare with the proposed method. This would already give a good sense of how well the algorithm behaves in simple cases. - Another possibility is to use generative models like Real-NVP for which the likelihood can also be computed in closed form. This would allow comparing the proposed algorithm to direct likelihood maximization on more complicated datasets as done in [1]. It seems like having experiments of this nature is far more convincing than a long justification for why the results are not necessarily state-of-the-art. - There are way too many samples on the figures so it is very hard to perform any visual assessment. Theory: - More discussions of the assumptions are needed, concrete examples for which these assumptions hold or not would be very useful. - Lemma 2 is a direct consequence of the following result: if p is continuous at x_0 then x_0 is a Lebesgue point. General remarks on the paper: - What complexity is the nearest neighbor algorithm? Since it is crucial for the proposed method to be scalable it is worth presenting this algorithm at a high level in the main paper. - The discussion in section 3 could be much more concise if concrete examples and figures were provided. Most of the facts discussed in that section are generally well understood, so conciseness is very appreciated in this case. - « A secondary issue that is more easily solvable is that samples presented in papers are sometimes cherry-picked; as a result, they capture the maximum sample quality, but not necessarily the mean sample quality. » Could you please provide an example of such paper? I would be very interested in having a closer look. - In the last paragraph of section 5, it is said that although the samples may not be state of the art in terms of precision, other methods which achieve better precision « may » have less recall. It would be good to have empirical evidence to back this claim. Revision: Although this paper presents an interesting idea, there is a serious lack of evidence to support the claims in the paper: - Missing experimental evidence for the efficiency of the NN search algorithm. - Experiments are using Parzen window for estimating likelihood which are known to be unreliable in high dimensions. - None of the suggested experiments were considered. In my opinion these experiments could improve the quality of this work. - Moreover, as mentioned by reviewer 1, Grover et al., 2017 provides evidence contrary to what the authors claim but this was never addressed so far in the paper. - Theorem 1 makes rather strong assumptions: as pointed out by reviewer 1, assumption 3 is unlikely to hold for the distributions used in practice For these reasons I recommend a clear reject. [1] I. Danihelka, B. Lakshminarayanan, B. Uria, D. Wierstra, and P. Dayan. Comparison of Maximum Likelihood and GAN-based training of Real NVPs.
assistant
 The paper proposes a new algorithm for implicit maximum likelihood estimation based on a fast Nearest Neighbor search. The algorithm can be used to implicitly maximize the likelihood of models for which the former quantity is not intractable but for which sampling is easy which is typically the case for implicit models. The paper shows that under some conditions the optimal solution of the algorithm corresponds to the MLE solution and provides some experimental evidence that the method leads to a higher likelihood. However, The paper lacks clarity and the experiments are not really convincing. Here are some remarks: Experiments: - The estimated likelihood was reported on table 1 using parson window which is known to have bad scaling behavior with the dimension of data. In the end, the table compares methods that maximize different objectives and are evaluated with an unreliable metric. Here are two possible experiments that could be more informative: - Consider toy examples for which the likelihood can be evaluated and the MLE obtained easily and then compare with the proposed method. This would already give a good sense of how well the algorithm behaves in simple cases. - Another possibility is to use generative models like Real-NVP for which the likelihood can also be computed in closed form. This would allow comparing the proposed algorithm to direct likelihood maximization on more complicated datasets as done in [1]. It seems like having experiments of this nature is far more convincing than a long justification for why the results are not necessarily state-of-the-art. - There are way too many samples on the figures so it is very hard to perform any visual assessment. Theory: - More discussions of the assumptions are needed, concrete examples for which these assumptions hold or not would be very useful. - Lemma 2 is a direct consequence of the following result: if p is continuous at x_0 then x_0 is a Lebesgue point. General remarks on the paper: - What complexity is the nearest neighbor algorithm? Since it is crucial for the proposed method to be scalable it is worth presenting this algorithm at a high level in the main paper. - The discussion in section 3 could be much more concise if concrete examples and figures were provided. Most of the facts discussed in that section are generally well understood, so conciseness is very appreciated in this case. - « A secondary issue that is more easily solvable is that samples presented in papers are sometimes cherry-picked; as a result, they capture the maximum sample quality, but not necessarily the mean sample quality. » Could you please provide an example of such paper? I would be very interested in having a closer look. - In the last paragraph of section 5, it is said that although the samples may not be state of the art in terms of precision, other methods which achieve better precision « may » have less recall. It would be good to have empirical evidence to back this claim. Revision: Although this paper presents an interesting idea, there is a serious lack of evidence to support the claims in the paper: - Missing experimental evidence for the efficiency of the NN search algorithm. - Experiments are using Parzen window for estimating likelihood which are known to be unreliable in high dimensions. - None of the suggested experiments were considered. In my opinion these experiments could improve the quality of this work. - Moreover, as mentioned by reviewer 1, Grover et al., 2017 provides evidence contrary to what the authors claim but this was never addressed so far in the paper. - Theorem 1 makes rather strong assumptions: as pointed out by reviewer 1, assumption 3 is unlikely to hold for the distributions used in practice For these reasons I recommend a clear reject. [1] I. Danihelka, B. Lakshminarayanan, B. Uria, D. Wierstra, and P. Dayan. Comparison of Maximum Likelihood and GAN-based training of Real NVPs.