Models that output a vector of responses given some inputs, in the form of a conditional mean vector, are at the core of machine learning. This includes neural networks such as the multilayer perceptron (MLP). However, models that output a symmetric positive definite (SPD) matrix of responses given inputs, in the form of a conditional covariance function, are far less studied, especially within the context of neural networks. Here, we introduce a new variant of the MLP, referred to as the matrix MLP, that is specialized at learning SPD matrices. Our construction not only respects the SPD constraint, but also makes explicit use of it. This translates into a model which effectively performs the task of SPD matrix learning even in scenarios where data are scarce. We present an application of the model in heteroscedastic multivariate regression, including convincing performance on six real-world datasets.
This paper explores the problem of deep heteroskedastic multivariate regression where the goal is to regress over symmetric positive definite matrices; that is, the deep learning model should take as input data points, and produce a conditional covariance matrix as the output. The key challenge in this setting is how to ensure the predicted matrix is positive definite (and thus follows the non-linear geometry of these matrices), how the neural network can be trained for this task, and what loss function can be used for effective training. The paper proposes a neural network with bilinear layers in this regard, and uses the von Neumann divergence as the loss function to regress the predicted covariance against a ground truth SPD matrix. The gradients of the von Neumann divergence are provided for learning via backpropagation. Experiments on several synthetic datasets and small scale datasets are provided, showcasing some benefits. 
Pros: 
1. The use of von Neumann divergence as a loss for this task is perhaps novel.
2. The use of \alpha-derivatives, while computationally demanding, is perhaps novel in this context as well. 
Cons:
1. I do not think the problem setting or the proposed framework is entirely new or is the best choice of its ingredients. Specifically, the idea of using second-order neural networks have been attempted in several prior papers, including the ones the paper cite (such as Ionescu et al. ,2015). Several other papers in this regard are listed below. 
[a] Second-order Temporal pooling for action recognition, Cherian and Gould, IJCV, 2018
[b] Deep manifold-to-manifold transforming network, Zhang et al, ICIP, 2018
[c] Statistically motivated second-order pooling, Yu and Salzmann, ECCV, 2018
In comparison to these methods, it is not clear how the proposed setup is novel, or in what way is method better. There are no comparisons to these methods, and thus it is difficult to judge the benefits even empirically. 
2. There are also models that predict the mean and covariance matrices directly from the model, such as the works below. The paper should also include and perhaps compare to their datasets.
[d] Deep Inference for Covariance Estimation: Learning Gaussian Noise Models for State Estimation, Liu et al, ICRA, 2018
3. I do not think the use of von Neumann divergence as a loss is the best choice one could have, esp. for a deep neural network learning setting. This divergence includes the matrix logarithm, which is perhaps computationally expensive. This is a problem when using other popular loss/similarity functions on SPD matrices (such as the log-Euclidean metric). Perhaps a better option is to use the Jensen-Bregman log-det divergence, as suggested in [a] above; this divergence is symmetric and also has computationally efficient gradients. It is unclear why the paper decided to use von Neumann. 
4. The experiments are not compelling, there are no comparisons to alternative models and the datasets used are small scale. Thus, it is unclear if the design choices in the paper have any strong bearing in the empirical performances.
Overall, the paper makes an attempt at designing neural networks for learning SPD matrices. While, there are some components in the model that are perhaps new, the paper lacks any justifications for their choices, and as such these choices seem inferior to alternatives that have been proposed earlier. Also, the experimental results are not convincing against prior works.