Making decisions requires information relevant to the task at hand. Many real-life decision making situations allow acquiring further relevant information at a specific cost. For example, in assessing the health status of a patient we may decide to take additional measurements such as diagnostic tests or imaging scans before making a final assessment. More information that is relevant allows for better decisions but it may be costly to acquire all of this information. How can we trade off the desire to make good decisions with the option to acquire further information at a cost? To this end, we propose a principled framework, named EDDI (Efficient Dynamic Discovery of high-value Information), based on the theory of Bayesian experimental design. In EDDI we propose a novel partial variational autoencoder (Partial VAE), to efficiently handle missing data over varying subsets of known information. EDDI combines this Partial VAE with an acquisition function that maximizes expected information gain on a set of target variables. EDDI is efficient and demonstrates that dynamic discovery of high-value information is possible; we show cost reduction at the same decision quality and improved decision quality at the same cost in benchmarks and in two health-care applications. We believe there is great potential for realizing these gains in real-world decision support systems.
The authors present an information discovery approach based on (partial) variational autoencoders and an information theoretic acquisition function that seeks to maximize the expected information gain over a set of unobserved variables. Results are presented on image inpainting, UCI datasets and health data, namely ICU and NHANES.
It is not clear why multiple recurrent steps improve perfromance. This is not conceptually justified and empirically (see Figure 8), it is also unclear whether PNP5 significantly outperforms PNP1. Further, results seem to support that PNP is always better than PN, so why introduce the methodology around PN or even present it at all. Note that the authors do not offer an explanation about the perfromance differences between PN and PNP.
In the inpainting regions section, the authors write about well-calibrated uncertainties without any context. What do they mean by calibration, well-calibrated and how can they support their claim about it?
In Figure 3 it is not clear that PNP+Ours outperforms PNP+SING. For Boston hosing seems to be marginally better but the error bars (which I assume are standard deviations, not stated) make difficult to ascertain whether the differences are significant. Although I understand the value of having "personalized" decisions, one wonders whether this personalization comes with any generalizable measurable gains given the results.
The results in Table 2 need to be clarified and further explained. 1) what are the error bars, considering multiple runs and datasets? 2) How can EDDI be so much better than SING when individual AUICs in Tables 6-11, the only significant difference (accounting for error bars) is on Boston data? 3) according to Tables 6-11, PNP is only the best in 1 of 5 datasets, so how come is the overall beast by a large margin? This being said, the results in Table 2 are at best misleading.
In Table 4, how can PNP-EDDI be so much better than PNP-SING, when in Figure 6 error bars overlap almost everywhere?
I enjoyed reading the paper, the motivation is clear and the problem is important. The approach is modestly novel compared to existing approaches and in general well explained despite the fact that the need for multiple recurrent steps is not well justified and the differences between PN and PNP, advantages/disadvantages and when to use each are not described or explored in the experiments.