QM9 atomization energies

I am doing experiments with the QM9 dataset from


* <http://quantum-machine.org/datasets/>


and began with some simple checks on results from the literature. Consistent with what the authors of the paper


* C.R. Collins, G.J. Gordon, O.A. Von Lilienfeld and D.J. Yaron, Constant size descriptors for accurate machine learning models of molecular properties. J. Chem. Phys. 148 (2018), 241718.


report, I find 3993 small QM9 molecules with 7 or fewer heavy atoms, which serves as their training set.
But I cannot reproduce the Null row of their Table II, probably because I don't understand well enough what they actually do there. In Section V.A they say:



> 
> A “null” model is used to provide a baseline measure of the difficulty of the prediction task. The null model always predicts the mean value of the training data.
> 
> 
> 


They don't specify precisely which quantity they predict, but on p.6 they report values in kcal/mol, and in Table II they talk about atomization energies, so I suppose the target is one of the energies provided by QM9. But there energies are in Hartree and hence must be converted to kcal/mol.


Using the atomization energy (QM9 property 13), I get 319.48 Ha for the mean atomization energy on the training set. With this mean as constant null predictor I obtain an MAE of 30.84 Ha = 19,353 kcal/mol on the training set, and of 31,319, 48,632, and 58,000 kcal/mol, on the first 20,000, 50,000, and 133,000 QM9 molecules, respectively. (The data used are in <https://arnold-neumaier.at/nullQM9.txt>. Calculating the mean and the MAEs for the mean is a standard triviality.) But these numbers are over 100 times larger than the numbers stated in the Null row of their Table II. (The table is also on p.10 of the arXiv preprint <https://arxiv.org/abs/1701.06649> - Section V.A is there called 5.1.)


Without conversion, the numbers would be far too small. The other properties from QM9 don't fare better.


Perhaps I misunderstood what they meant by the Null model, or there is an undocumented preprocessing step for getting the target energies for prediction?


Could anyone please resolve this discrepancy?




First off, let me stress that it's important to have a null model or baseline model for comparison in machine learning. In other words, does the ML model presented in the paper do better than a simple alternative.


Atomization energies are calculated from the electronic structure calculation of the molecule and the relevant atoms. In other words, how much energy does it take to dissociate the molecule into the constituent atoms (e.g., 6 carbon and 6 hydrogen atoms for benzene). **Consequently, atomization energies depend on the number and type of atoms in the molecule.** The atomization energy for benzene is higher than that of methane, because you're breaking more bonds.


My reading of this paper, was that the null model consisted of the average energies for each atom, e.g. $E\_{C\_6H\_6} = 6E\_C + 6E\_H$ and $E\_{C\_5NH\_5} = 5E\_C + E\_N + 5E\_H$ with the various $E\_C$ for each element fit from least squares. (Certainly that's what I'd consider as a null model for this, because standard atomization energies use the isolated atoms, so least-squares fit would at least adjust for atoms in a molecule.)


Your comment suggests that least-squares null model performs better than the reported null model in the paper.


In that case, I don't know, but if/when I see David Yaron next, I'll ask.


 2