This work proposes a closed loop, uncertainty-driven adaptive sampling framework (CLUDAS) for accelerating magnetic resonance imaging (MRI) via deep Bayesian inversion. By closed loop, we mean that our samples adapt in real-time to the incoming data. To our knowledge, we demonstrate the first generative adversarial network (GAN) based framework for posterior estimation over a continuum sampling rates of an inverse problem. We use this estimator to drive the sampling for accelerated MRI. Our numerical evidence demonstrates that the variance estimate strongly correlates with the expected mean squared error (MSE) improvement for different acceleration rates even with few posterior samples. Moreover, the resulting masks bring improvements to the state-of-the-art fixed and active mask designing approaches across MSE, posterior variance and structural similarity metric on real undersampled MRI scans.
The paper describes a method for accelerating MRI scans by proposing lines in k-space to acquire next. The proposals are based on posterior uncertainty estimates obtained from GAN-based reconstructions from parts of the k-space acquired thus far. The authors address an interesting and important problem of speeding up MRI scans and thus improving the subject's experience. The proposed method achieves better posterior uncertainty and SSIM scores than competing methods.
While the paper considers an important problem and takes a novel approach to solve it (using a GAN generative model to estimate uncertainty), I found that it may be particularly inaccessible to non-experts in the field of MRI image processing. Furthermore, several important methodology-related questions remain unanswered in the paper; and the experiments offered in the paper are insufficient for convincingly arguing the author’s claims.
Specifically, it is unclear whether GANs with their mode dropping behaviour are the right model choice for proposing reconstructions - they are likely to drop modes and by extension - yield overconfident uncertainty estimates. This can be expected to be particularly problematic for scans of images that differ from the training distribution, and is exacerbated by the fact that authors train the model only on scans from healthy subjects. Furthermore, because of the the GANs propensity to drop modes, it is also unclear whether the posterior variance numbers reported in the paper are directly comparable between the methods.
The method used for obtaining the uncertainty estimates from GAN samples implicitly makes the assumption that reconstructions follow a Guassian distribution with a diagonal covariance. This assumption is also made in a competing method of Zhang et. al (2019) that the authors do not compare against, and claim to improve upon methodologically (i.e. the authors state that the method of Zhang et. al (2019) cannot be used to produce uncertainty estimates in Fourier space). I am not convinced that the authors claims about the method differences are sufficiently substantiated (see more under major comments). And because the methods bear significant similarity to each other, an experimental comparison - which is currently missing - should be carried out (on open datasets that ideally include non-healthy subjects).
Finally, the paper teases fast(er) MRI in the title, but doesn’t touch on this topic in the text. This aspect should of the authors contribution be discussed at length, in particular comparing the Cartesian sampling strategy adopted by the authors to other strategies, as well as evaluating the feasibility of implementing the adaptive sampling strategy in an actual scanner (e.g. can the network be ran fast enough)?
==================
Major comments:
==================
1) In the proposed method the authors employ a procedure in which the currently sampled parts y of the k-space are fed to a generator network to obtain n_s reconstructions. These n_s sampled reconstructions are then averaged to obtain the empirical mean and variance, with the latter being used for estimating uncertainty. This procedure is potentially problematic for several reasons:
  a) First, taking the empirical mean and variance of the samples is in fact equivalent to assuming that the reconstructed image follows a Guassian distribution with a diagonal covariance. This is the same assumption the authors argue is not realistic when discussing the work of Zhang et al. (2019) in the end of Section 1. 
  b) In case of GANs, which can model multi-modal distributions this uncertainty estimation is even more problematic in cases when the samples originate from different modes. What do the mean and variance represent then?
  c) As the authors highlight in the discussion section of the paper in the paper, GANs are prone to mode collapse. This is also potentially problematic for their estimator - in case of mode collapse their method would underestimate uncertainty. The fact that authors are able to use only two sampled reconstructions to estimate the mean and variance with acceptable accuracy is consistent with the occurrence of mode collapse in their generator. Furthermore, because mode collapse may occur in the author’s model, it is unsurprising that their method yields the smallest posterior variances in Table 1. The authors should provide evidence that mode collapse either does not occur or would not affect these numbers.
  d) Finally, the authors use the empirical mean and sampled reconstructions to obtain the empirical mean and variance in Fourier (k-) space. Specifically, they argue that “This feature is specific to generative models, as getting samples from P_{X|y_\omega} allows to transform these to a different domain [...] this is not possible with methods that only provide point-wise estimates of the mean and the variance in image space, such as the one used by Zhang et. al (2019)”. I don’t think this is true - Fourier transform is a linear transformation, thus given a mean and a variance in image space it is possible to deduce analytically what the mean and covariance in Fourier space would be. This should be elaborated in the text, and the comparison to Zhang et. al (2019) should thus be extended further.
2) The proposed method, CLUDAS, was evaluated against existing methods on a single proprietary dataset consisting of only 100 images from healthy individuals. This is potentially problematic, for several reasons:
  a) Using a proprietary dataset doesn’t allow follow-up works to compare against the author’s method; or for comparing CLUDAS to 
 existing methods not considered in the paper; the methods should additionally be compared on a public dataset  and
  b) Applying the method only to a single (small) dataset does not allow for reasoning on how the method behaves in different data regimes. I strongly encourage the authors to apply their method on public datasets, for example on data used in Zhang et. al (2019) - this would then allow for comparing the two methods despite not having access to an implementation of Zhang et. al (2019).
  c) Since the data is used to train the GAN model is obtained from healthy individuals, it’s unclear whether it can be used to acquire data from subject that may potentially have aberrations in their scans - the GAN model would be expected to produce low uncertainty estimates for regions where these aberrations would lie and not propose acquiring parts of the k-space that could be used to resolve these aberrations. For similar reasons using a GAN that potentially drops modes (e.g. scans of unhealthy individuals, for example because they are less common in the training data) is also problematic. The authors should consider evaluating their method on a dataset that contains non-healthy subjects, and investigate the performance of the method when it’s evaluated on healthy subjects, but tested on unhealthy ones.
3) The title of the paper (“[...] for fast MRI”) suggests that the authors aim to accelerate the MRI data acquisition process.
  a) Yet they chose the work with the Carterian sampling (i.e. sampling lines in the k-space parallel to the x-axis), which arguably requires larger sections of the k-space to be sampled before a high-quality reconstruction can be obtained (e.g. see http://mriquestions.com/k-space-trajectories.html). Providing information on how this choice influences the speed of data acquisition (and thus subject’s comfort) is important in order to assess the applicability of the author’s method to real world scenarios. This information should be provided.
  b) The paper does not actually describe how MRI is sped up. Given that the acquisition budget (number of lines acquired in k-space) is fixed, and assuming that time per line is constant, it is unclear where the speed up comes in. More generally, the speed claims / aspect of the proposed method should be discussed in more detail - how is speed measured and evaluated? How does it compare to non-Cartesian sampling approaches.
  c) Finally, it’s unclear whether neural network inference can be made fast enough to allow for a real world application of the adaptive CLUDAS sampling - can the data be transferred fast enough from the scanner to do inference and propose the next line to scan without causing delays in the scanning process? This should be discussed in the paper.
4) Currently, the paper places a strong expectation of knowing about MRI and being familiar with MRI-specific terminology on the reader. This makes the work substantially less accessible to a wide audience with machine learning expertise as the common denominator. The authors should take steps towards making the text more accessible to non-(MRI) expert audience, for example by introducing some of the basic knowledge (e.g. the data acquisition and reconstruction processes in MRI) early on, departing from MRI-specific jargon (e.g. k-space, lines in k-space) in favour of ML terminology whenever possible, and taking care to define and possibly illustrate (strongly encouraged) the MRI-specific concepts (e.g. the k-space, lines in k-space, sampling masks). Some specific examples the authors should address follow.
  a) The k-space is defined only in passing as being the frequency domain. It’s unclear whether these lines (which correspond to sampling masks) in this domain are parallel to the x-axis. The math (e.g. Equation 1) and Figure 2 suggest that, but it’s not obvious.
x is referred to both as "model parameter" in Section 2 (and Equation 1) as well as the "ground truth image" later in the same section. This is confusing, especially because in case of GAN model parameter would typically refer parameters of the generator and discriminator.
  b) SSIM is not defined anywhere, but already used in the abstract.
  c) It could be made more clearly why undersampling is required in case of MRI. E.g. how/why does it correlation with patient comfort.
  d) “K-space sampling” is used already in the introduction, but not really defined.
  e) It’s unclear whether sampling, subsampling and undersampling all refer to the same concept or not.
  f) The term “sampling mask” is already used in the introduction, but not clear what it refers to.
  g) Use of the term “innovation” to refer v_t, which appears to be a one-hot vector marking the newly added line in k-space.
  h) The use of term “sampling decision” to refer to v_i.
  i) Unclear what eta in “Z ~ eta“ in Equation 5 refers to - it is not defined. Later in Section 2.2 it is stated that “z_i are independent samples form Z”. Is this the same as “z_i ~ eta”? If so, why the second layer of notation?
  k) In Introduction “[...] which is not feasible on a real problem without the ground truth available”. Unclear what the ground truth refers to, I assume it’s the ground truth image.
  l) In Section 2.1 data refers to x_i, with i=1,...,m; which I assume are images and thus contain real numbers. Yet Section 3 states “As our data are complex [...]”. This is confusing - is that different data?
  m) Sampling is used ambiguously in the paper - to refer to sampling in the k-space (e.g. sampling masks, sampling decisions v_1,...,v_n) and to refer to sampling reconstructions from the generator. This should be resolved to improve readability of the paper.
  n) Not entirely clear what k-space pixel-wise variances are. And what is the difference between spatial and pixels-wise variances is (Section 2.2).
==================
Minor comments:
==================
1. In the Introduction the authors argue that metrics such as MSE and SSIM “[..] do not align with what clinicians see as valuable.”, yet use these throughout the paper for evaluating and comparing methods. This decision should be explained.
2. In Section 2 (and beyond) images x are described as belonging to subspace C^p of complex numbers. While this is technically true, them to actually belong to the space of R^p. If so, this should be reflected in the text for the benefit of the readers. Furthermore, I don’t think dimensionality p is actually defined anywhere.
3. Compressive / compressed sensing abbreviations CS is defined twice in Section 1.
4. It’s not entirely clear why “The CS-inspired methods shift the burden from acquisition to reconstruction [...]”
5. Incorrect double quotes are used throughout the text (both are right quotes).
6.  In Introduction “[...] yielding an estimator which can be used to drive back the whole sampling process in a closed-loop fashion.” 
it is unclear what it means to “drive back a sampling process”. Perhaps “back” should not be there?
7. In some cases it is unclear what the use of double quotes conveys, e.g.
  a) In Section 2.1 “[...] being the “full” mask [...]”
  b) In Section 2 “[...] ground truth “complete” images [...]”
  c) In Section 5 “[...] these inverse problems “depend” from each other [...]”
  d) Multiple places in Appendix B.
8. The convention of using small letters x, y for data samples / instances and capital letters X, Y for random variables could be made explicit.
9. Section mostly provides background (rather than theory) and could be named accordingly.
10. In Section 2.1, t refers to “time”. It may be clear if it were instead referred to as the step of the sampling process or something similar - the use of “time” to refer to some discrete set of actions can be a little confusing.
11. Section 2.1 refers to “[...] the online reconstruction speed of DL [...]”. This should be explained further - why are deep learning based approaches to reconstruction faster? Does this depend only on having the right hardware accelerators? Also, I don’t think the abbreviation “DL” was introduced.
12. Equation 4 is referred to both as an “Equation” and as a “Problem”.
13. In Equation 6 there is a summation over j from v_i. It is my understanding that v_i is a one-hot vector and it’s unclear what this summation means. Presumably it is the summation over pixels covered by line v_i, but the notation doesn’t convey this. It could be nice to also explain the “1D” superscript in this equation.
14. In Section 2.2 “[...] this is why the approach of (Adler & Oktem, 2018) minimizes over distance for observation in Equation 4 [...]”. I couldn’t follow the part about minimizing over the distance for an observation. Please consider making this more clear.
15. In Equation 8, what does index i run over?
16. Minor typos / textual issues:
  a) In Introduction “[...] of the our estimator [...]”
  b) In Introduction “[...] and show that even using a few samples [...]” -> even when using?
  c) In Section 2.2, after Equation 5 “[...] where t After finding the optimal”.
  d) Inconsistent use of “closed-loop” and “closed loop”.
  e) In multiple places throughout the paper a double space appears to be used instead of a single one.
  f) Section 4.1 “as can be in Figure 1”
  g) Section 4.3 “samples art random”
  h) In Section 5 “[...] “depend” from each other” should be depend on each other?
  i) In Section 5 “[...] we we found that [...]”
17. Unclear what’s meant by “[...] using the aggregated variance as a loss function” in Section 2.2
18. Section to refers to i* (integer scalar) as a line. Previously it was v_i (vector).
19. In Section 2.2 the authors state “Once the generator has been trained until convergence [...]”. The authors optimize a generator in an adversarial fashion. To my knowledge, this training procedure is not guaranteed to converge and would typically oscillate around a stable solution. Could the authors please comment on what they mean by convergence in this case and how they guarantee that the generators they train converge.
20. For the posterior variance results in Table 1 it should be discussed whether all the methods obtain / compute the variances the same way.
21. The “data consistency layer” (Section 3) should be explained briefly. How does it enforce perfect consistency and what’s meant by consistency here?
22. It should be made clear what MSE is calculated between in Figure 1 and Table 1. I assume it’s between ground truth (all frequencies sampled) and reconstructed (only some frequencies samples) images.
23. It’s not entirely clear what’s meant by consistency in Section 4.1
24. What do yellow arrows signify in Figure 2?
25. I don’t think the abbreviation UQ was defined in Figure 2.
26. The masks in Figure 2 are such that is a certain line was chosen at lower sampling rate (left on the x-axis), it would also be chosen at higher sampling rate (right on the x-axis). This is somewhat unexpected since the budget of lines v_i to be acquired differs between sampling rates. Why is there such consistency?
27. In Table 1, in the leftmost column, the number in brackets (number of posterior samples?) should be defined.
28. In Table 1 and Section 4.3: LBC-M and LBC-U - the -M and -U suffixes should be explained. What are the differences between the methods?
29. In Section 4.3: what are the FE methods?
30. Appendix A.3 is mostly a copy-paste from the main text. Unnecessary duplication?
31. Axes in Figure 4 (Appendix A.2) are not labeled or described.
32. Loss in A.4 (Equation) does not match Equation 5. In the former the discriminator takes three arguments instead of two, and the arguments z1 and z2 are not described.