When deep learning is applied to sensitive data sets, many privacy-related implementation issues arise. These issues are especially evident in the healthcare, finance, law and government industries. Homomorphic encryption could allow a server to make inferences on inputs encrypted by a client, but to our best knowledge, there has been no complete implementation of common deep learning operations, for arbitrary model depths, using homomorphic encryption. This paper demonstrates a novel approach, efficiently implementing many deep learning functions with bootstrapped homomorphic encryption. As part of our implementation, we demonstrate Single and Multi-Layer Neural Networks, for the Wisconsin Breast Cancer dataset, as well as a Convolutional Neural Network for MNIST. Our results give promising directions for privacy-preserving representation learning, and the return of data control to users.
The paper presents a means of evaluating a neural network securely using homomorphic encryption. A neural network is already trained, and its weights are public. The network is to be evaluated over a private input, so that only the final outcome of the computation-and nothing but that-is finally learned.
The authors take a binary-circuit approach: they represent numbers via a fixed point binary representation, and construct circuits of secure adders and multipliers, based on homomorphic encryption as a building block for secure gates. This allows them to perform the vector products needed per layer; two's complement representation also allows for an "easy" implementation of the ReLU activation function, by "checking" (multiplying by) the complement of the sign bit. The fact that multiplication often involves public weights is used to speed up computations, wherever appropriate. A rudimentary  experimental evaluation with small networks is provided.
All of this is somewhat straightforward; a penalty is paid by representing numbers via fixed point arithmetic, which is used to deal with ReLU mostly. This is somewhat odd: it is not clear why, e.g., garbled circuits where not used for something like this, as it would have been considerably faster than FHE.
There is also a work in this area that the authors do not cite or contrast to, bringing the novelty into question; please see the following papers and references therein:
GILAD-BACHRACH, R., DOWLIN, N., LAINE, K., LAUTER, K., NAEHRIG, M., AND WERNSING, J. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of The 33rd International Conference on Machine Learning (2016), pp. 201–210.
SecureML: A System for Scalable Privacy-Preserving Machine Learning
Payman Mohassel and Yupeng Zhang
SHOKRI, R., AND SHMATIKOV, V. Privacy-preserving deep learning. In
Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (2015), ACM, pp. 1310–1321.
The first paper is the most related, also using homomorphic encryption, and seems to cover a superset of the functionalities presented here (more activation functions, a more extensive analysis, and faster decryption times). The second paper uses arithmetic circuits rather than HE, but actually implements training an entire neural network securely.
 Minor details:
The problem scenario states that the model/weights is private, but later on it ceases to be so (weights are not encrypted).
"Both deep learning and FHE are relatively recent paradigms". Deep learning is certainly not recent, while Gentry's paper is now 7 years old.
"In theory, this system alone could be used to compute anything securely." This is informal and incorrect. Can it solve the halting problem?
"However in practice the operations were incredibly slow, taking up to 30 minutes in some cases." It is unclear what operations are referred to here.