Before we saw worldwide collaborative efforts in training machine-learning models or widespread deployments of prediction-as-a-service, we need to devise an efficient privacy-preserving mechanism which guarantees the privacy of all stakeholders (data contributors, model owner, and queriers). Slalom (ICLR '19) preserves privacy only for prediction by leveraging both trusted environment (e.g., Intel SGX) and untrusted GPU. The challenges for enabling private training are explicitly left open -its pre-computation technique does not hide the model weights and fails to support dynamic quantization corresponding to the large changes in weight magnitudes during training. Moreover, it is not a truly outsourcing solution since (offline) pre-computation for a job takes as much time as computing the job locally by SGX, i.e., it only works before all pre-computations are exhausted. We propose Goten, a privacy-preserving framework supporting both training and prediction. We tackle all the above challenges by proposing a secure outsourcing protocol which 1) supports dynamic quantization, 2) hides the model weight from GPU, and 3) performs better than a pure-SGX solution even if we perform the precomputation online. Our solution leverages a non-colluding assumption which is often employed by cryptographic solutions aiming for practical efficiency (IEEE  SP '13, Usenix Security '17, PoPETs '19). We use three servers, which can be reduced to two if the pre-computation is done offline. Furthermore, we implement our tailor-made memory-aware measures for minimizing the overhead when the SGX memory limit is exceeded (cf., EuroSys '17, Usenix ATC '19). Compared to a pure-SGX solution, our experiments show that Goten can speed up linear-layer computations in VGG up to 40×, and overall speed up by 8.64× on VGG11.
Summary
========
This paper proposes a framework for privacy-preserving training of neural networks, by leveraging trusted execution environments and untrusted GPU accelerators.
The system builds heavily on the prior Slalom system, and uses standard MPC techniques (three non-colluding servers, multiplication triplets) to extend Slalom's inference-only protocol to privacy-preserving training.
This is a valuable and hard to reach goal. Unfortunately, the paper's evaluation fails to deliver on its strong promises, by ignoring the high network communication between the non-colluding servers.
Specifically, all experiments were conducted with three servers co-located in a public cloud's LAN. In this setting, it is hard to argue that non-collusion is a valid security assumption as the cloud provider controls all servers (alternatively, if the cloud provider is trusted, then there is no need for any trusted execution or cryptography). If the same experiments were conducted on a WAN, the communication costs would alleviate any savings in computation time.
For these reasons, I lean strongly towards rejection of this paper. 
Detailed comments
=================
Extending the ideas in Slalom to support privacy-preserving training is a good research question, and Tramer and Boneh had discussed some of the challenges and limitations towards this in their original paper.
Getting rid of the pre-processing stage for blinding factors by leveraging non-colluding servers is a well-known trick from the MPC literature, but it does not seem easily applicable here. 
The problem is that the servers need to communicate an amount of data proportional to the size of each internal layer of the network, for each forward and backward pass. If the servers communicate over a standard WAN, the communication time will be much too high to be competitive with the CaffeScone baseline.
In a LAN, as in this paper's experiments, the network latency is low enough for the network costs to be dominated by computation. But this begs the question of whether servers running in a same LAN (e.g., hosted by a single cloud provider) can really be considered non-colluding. In the considered setup, the cloud provider (Google in this case), could just observe the communication between all servers, thereby breaking privacy.
Another security issue with proposed scheme is the lack of computation integrity. This corresponds to the so-called "honest-but-curious" threat model which often appears in the MPC literature, and this should be acknowledged and motivated.
On the experimental side, the considered baseline, CaffeScone, seems pretty weak. In particular, any optimizations that the authors implement for Goten (e.g., better paging) should also be added to their baseline for a fair comparison.
The numbers in Figure 3 show that the baseline could be optimized a lot further.  A gap between hardware/simulation modes of ~6x seems indicative of sub-optimal paging. Even the single-core, simulation mode throughput numbers seem low for CIFAR10.
The experimental setup is quite confusing. Running the baseline and Goten in different environments (e.g., different CPUs) and then re-normalizing throughputs is somewhat convoluted and prone to mistakes. Why not run all experiments on the same setup?
Similarly, converting between results in SGX's hardware and simulation modes is also not very indicative. The authors note (p. 8) that in SGX's simulation mode "code compilation is almost the same as hardware mode except that the program is not protected by SGX, which is fine for our purpose since the DNN training and prediction algorithms are publicly known". This is fundamentally incorrect!
SGX's simulation mode provides absolutely no security guarantees. It simply compiles the code using the SGX libraries and ensures that the enclaved code performs no untrusted operations, but it does not provide any hardware protections whatsoever. In particular, code running in simulation mode will not be affected by the overhead of SGX's paging, as the memory is never encrypted.
As a result, performance results in simulation mode are usually not indicative of performance in hardware mode. Trying to convert runtimes from simulation mode to hardware mode by comparing times of specific layers is also prone to many approximation errors. 
Finally, I had some trouble understanding the way in which Goten quantization works. Section 3.3. mentions that values are treated as floats, but then mentions the use of 53 bits of precision. Did you mean double-precision floats here? But then, aren't modern GPU optimized mainly for single-precision float operations? Section 3.3. also says that the quantization ensures that there are nearly no overflows. What happens when an overflow occurs? I guess that because of the randomized blinding, a single overflow would result in a completely random output. How do you deal with this during training?
Minor
=====
- Typo in abstract: Slaom -> Slalom
- I don't understand the purpose of footnote 3 in Appendix B.2. First, the bibliographic entry for (Volos et al. 2018) explicitly says that the paper was published in OSDI 2018, a top-tier peer-reviewed conference. Regardless, claiming a date for a first unpublished draft of your paper is a little unusual and somewhat meaningless. I'm sure Volos et al. had a draft of their paper ready in late 2017 or even earlier if they submitted to OSDI in XXX 2018. If you want to timestamp your paper, post in to arXiv or elsewhere online.