Deciding which is the best 1H NMR predictor for organic compounds using statistical tools

1H NMR chemical shifts for 30 organic compounds (396 data points) were predicted using four NMR predictor software and compared with the experimental data from SDBS. The NMR predictors involved were MestReNova, ChemDraw, NMRShiftDB and ACD Workbook Suite. Root mean square deviation (RMSD) and mean absolute percentage error (MAPE) were calculated from the data obtained. One-way analysis of variance (ANOVA), Tukey’s honestly significant difference (HSD) and t-test were carried out to analyse the statistical significance of the differences between the predictors. The results from the statistical analysis were used to predict chemical shifts of three organic compounds.


Introduction
Molecular structure elucidation is the determination of chemical structures which can be achieved by using NMR spectroscopy techniques. The molecular environment around a nucleus determines its chemical shift, and its interpretation is important to elucidate the structure of organic molecules. Accurate predictions of NMR chemical shifts with respect to experimental values are highly valuable for structural elucidations. NMR predictions' software * Corresponding author. primarily rely on coding and computational methods in producing NMR predictions, different programs can use different methods in their internal calculations for predictions. The focus of this study is to evaluate the accuracy of 1 H NMR predictions' software using statistical tools namely on MestReNova, ACD Workbook Suite, ChemDraw and NMRShiftDB. MestReNova and ACD Workbook suite were selected based on the similarity of what they both possess i.e., having the conventional hierarchically ordered spherical description of environment (HOSE) as one of their predictors but the latter having neural network (NN) as an additional tool that can interpolate within its trained database. These predictor's ability was compared with ChemDraw, a predictor that only uses a single prediction method: linear additivity rules. Another predictor's ability was compared, i.e. NMRShiftDB that only uses NNs in predicting NMR shifts.
MestReNova prediction is based on two predictors: MestreLab Predictor and Modgraph Predictor [1]. When an NMR prediction is made, each of these predictors will predict chemical shifts with a certain confidence interval based on different methods, for example with HOSE, along with substituent chemical shifts approach for Modgraph predictor, and CHARGE for MestreLab Predictor. These predicted chemical shifts will be combined using a Bayesian method to produce the final chemical shift, which has higher reliability and accuracy than the individual chemical shifts. The concept utilized is known as ensemble learning, which allows multiple machine-learning algorithms to be combined. The concept of ensemble learning is advantageous as the deficiency of one prediction method can be compensated by another prediction method, improving the overall prediction accuracy while reducing the number of outliers. For example, HOSE prediction method, which relies heavily on the size and quality of database, is highly inaccurate when predicting novel compounds, as very few database compounds possess a structure similar to the novel compound [2]. This deficiency is compensated by the substituent chemical shifts approach, which predicts by identifying substructures and substituents in a molecule and assigning base and increment values respectively based on their identity. Utilizing ensemble learning method, a database containing approximately 1000,000 shift values and 3000 available parameters, MestReNova can produce accurate 1 H NMR chemical shifts by combining the strengths of each prediction method used while minimizing their respective weaknesses. ACD Workbook Suite, on the other hand utilizes multiple prediction methods which are additivity rules, HOSE, and NN [3]. HOSE can be compromised when there is error present in database values, as these errors will be reproduced in the final predicted chemical shifts [4]. NN is well-known for its ability to generalize data, it can interpolate within its trained database, consequently less dependent on database values, and thus, reducing the possibility of reproducing errors within database in the final predicted value [4,5]. To ensure robustness, generality, and accuracy in NN, various types of descriptors, namely topological, physicochemical, and geometric descriptors (to account for stereochemistry and 3D effects) were used to better match the properties of H atoms to be predicted to those available in the database [6]. A database of approximately 1400,000 experimental 1 H chemical shifts ensure robustness and accuracy of ACD Workbook Suite [7]. ChemDraw's predictor, ChemNMR, relies on a single prediction method: linear additivity rules. Linear additivity rules are similar in concept to the substituent chemical shifts approach used in MestReNova and the additivity rules used in ACD Workbook Suite. However, it was found that linear additivity rules are inefficient in chemical shift prediction of molecules that exhibit non-linearity. Even though correction methods are implemented to improve its prediction power, it is still lacking when resonance effects are present in the molecule. Even so, ChemDraw is still reliable because it considers stereoisomerism and is backed by a large database of approximately 700 base values and 2000 increment values based on 4000 parameters [8]. NMRShiftDB utilizes deep NN in predicting NMR shifts, which achieved better precision than the HOSE code approach. When a molecule is predicted, its molecular weight, atom, and proton count will be compared to database molecules. If there are ten or more chemical shifts available, the smallest and largest chemical shift will be taken as confidence limit of the 6-sphere HOSE prediction. However, if values obtained are insufficient for 6-sphere HOSE or confidence limit estimation, the number of spheres used in HOSE will be reduced, leading to significantly decreased HOSE prediction accuracy as it relies heavily on number of spheres [9].
The diversity and variation of prediction methods among the available NMR predictor software in use are quite high and warrant further examination on the reliability and accuracy of these methods in predicting NMR chemical shifts, as well as comparisons and discussions on which method or program produces predictions that most closely matches realworld experimental data. Thirty organic compounds with 396 data points were used. Statistical comparison methods applied were mean absolute percentage error (MAPE), root mean square deviation (RMSD), one-way analysis of variance (ANOVA), Tukey's hon-estly significant difference (HSD) and t -test. From the data analysis, the best NMR predictor software was determined. The predictor software was then used to predict 1 H NMR of recent compounds found in literature.

Methods and analysis
The 1 H NMR chemical shifts were obtained by drawing and inserting the molecules into the NMR predictor software. The 30 organic compounds are shown in the supporting documents. The predictor's accuracy was evaluated in aliphatic, aromatic, cyclic and polycylic molecules. For aliphatic molecules, additional functional groups in the molecules add to the complexity in the prediction e.g. ester (7), ester and dicarbonyl (9), diols and ether (10) and ester, diols, alkene (11, 12 and 13). Molecules 1, 2, 4 and 13 were selected due to their cyclic conformations. It would be interesting to know whether the selected predictors are able to predict the chemical shifts of the hydrogens on the ring in these selected conformations. Molecules 1, 2 and 4 have ester functionality in the ring or near the ring, which acts as an electron withdrawing group. The ring in molecule 13 has additional complexity as it is linked to a nitrogen atom adjacent to a sulfone. Molecules 22 and 24 have cyclic moieties but whether it is in chair or boat conformation is unknown. It would be interesting to evaluate the predictor's capability for these kinds of molecules. Again, we are curious to know the accuracy of the predictor when the cyclic compound is bonded to ester group in 22 and aromatics functionality in molecule 24. We had also selected a range of other aromatic compounds as seen in molecules 5, 6, 8, 14-21, 25, 26, 29 and 30. Various other functional groups are linked to the aromatic groups from carbonyl and thiol (5), ketone and alcohol (6), ethers and alcohol (8), sulfone (14), phosphate (15), phosphate where one oxygen atom is replaced by sulfur (16), fluorine (17-19), amide (20 and 21), thiol, carboxylic acid and amide (25), polycyclic (26) and heteropolycyclic with imine (29 and 30). The predictor's capability was further evaluated, in the case of fused aromatic rings. The complexity was further heightened with presence of double bonds (3), polycyclic and dicarbonyl (27) and heteropolycyclic and esters (28).
The details of each software and its link are provided in the supporting document. The variation of the experimental proton shifts with predicted values for a molecule (Compound 1) is shown in the graphs in the supporting documents. The comparison of the calculated and experimental 1 H NMR chemical shift values for all the compounds are provided in the SI.
MAPE was calculated by using the equations in (1) and (2), while RMSD was calculated by using formula (3).
Percentage error (%) (1) where x i = experimental 1 H chemical shift values, MAPE was chosen as a parameter because it normalizes the absolute error between predicted and experimental values over a range of 0% to 100%. This allows comparison between datasets of different scales, as the absolute error of each predicted chemical shift is normalized by the experimental value, and an average is obtained [10]. RMSD, on the other hand, approximates the spread of predicted data from experimental data without restricting its values to any range.

Tests of statistical significance between the software
Three methods were selected to test for the significant difference between collected data, which are one-way ANOVA with a post hoc test, Tukey's HSD, and t -test. The four sets of significance tests are shown in Table 1. Both t -test and ANOVA determine the significant difference between populations by observing the mean and variance. T -test analyses two populations, whereas ANOVA is more suited for analysis of three or more populations. Thus, ANOVA was used for the first two datasets to compare the four NMR predictor programs. t -test was not repeatedly carried out as it can contribute to significant Type I error (false positive) which causes a true null hypothesis to be rejected. As ANOVA simultaneously compares all four NMR predictors, the rate of Type I error is maintained at the significance level of the hypothesis test, α = 0.05. One-way ANOVA was used because the analysis had one independent variable: the type of NMR prediction program. Tukey's HSD (post hoc analysis) was conducted alongside ANOVA to determine which specific NMR predictor was statistically different. Similar to t -test, Tukey's HSD utilizes pairwise comparisons as well, however its data is presented in adjusted p values and simultaneous confidence intervals. These representations maintain the rate of occurrence of Type I errors by limiting the family-wise error rate to the significance level used, α = 0.05 [11].
T -test was used in the third and fourth dataset where two NMR prediction software were compared, namely MestReNova and ACD Workbook Suite. The interpretation of data was based on two-tailed prediction, as the deviation of predicted chemical shift from experimental chemical shift is not unidirectional.

ANOVA and Tukey's HSD
In performing the following ANOVA and Tukey's HSD, the null hypothesis (H 0 ) was that all predictor programs are equal in performance. The alternative hypothesis (H 1 ) was that at least one of the predictor programs is significantly different in performance than the others.
The process of analysis is as follows: ANOVA (omnibus test) was separately performed on the datasets of MAPE and RMSD obtained from observing the deviations between predicted and experimental values. The MAPE and RMSD data of 30 molecules was analysed. ANOVA produces the F value (F ), F -critical value (F crit ), and P value (P ). If both conditions of F > F crit and P < 0.05 are achieved, it indicates that at least one of the programs is significantly different in performance from the others, affirming H 1 and rejecting H 0 .
To verify the conclusion from ANOVA and to identify which specific predictor software is significantly different, Tukey's HSD (post hoc analysis) was performed by calculating and comparing the Q-stat value (Q stat ) and Q-critical value (Q crit ) for each pair combination of predictor programs (pairwise comparisons). For pair combinations where Q stat > Q crit , it suggests that one program is significantly different from the other in the pair. The performance of which predictor is better or worse can be determined based on their average MAPE or RMSD. Additionally, simultaneous confidence interval (CI) of each pair combination of predictor programs was calculated and a graph was plotted. The difference in performance between the programs is significant if CI does not encompass zero in the graph.

RMSD
The chemical structure of the 30 molecules is provided in the supporting information. The comparison of the calculated and experimental 1 H NMR chemical shift values is shown in Appendix B in the supporting information. RMSD was calculated from the difference between the 1 H NMR of the experimental value and predicted value as outlined in (3). The variation of the experimental proton shifts with the corresponding predicted values using the four software was carried out for all the molecules.

ANOVA, Tukey's HSD and CI from RMSD of the 30 organic molecules
To test for the significant difference in RMSD from Table 2, one-way ANOVA with a post hoc test and Tukey's HSD were carried out. The results are shown in Table 3a, 3b and 3c and Figure 2.
ANOVA of the dataset showed that F = 9.38, F crit = 2.68, and P = 0.0000133. As P < 0.05 and F > F crit , it suggests that H 0 is false, whereas H 1 is true: at least one of the predictor programs is significantly different in performance than the others. Tukey's HSD indicates that Q stat > Q crit between NMRShiftDB paired with MestReNova, with ChemDraw, and with ACD Workbook Suite, affirming the conclusion reached by ANOVA. Results from Figure 2 shows the CI analysis where the CI between NMRShiftDB and all other programs does not intersect zero, indicating that there is significant difference between NMRShiftDB and all other programs, verifying the conclusion of Tukey's HSD.
The results obtained from RMSD lead to the conclusion that NMRShiftDB (with the highest average RMSD) is the weakest in 1 H NMR prediction whereas the remaining three predictors are equally good in performance.

MAPE (%)
MAPE (%) was calculated from the total percentage error as outlined in (2). The average MAPE (%) was calculated from the MAPE (%) of the 30 molecules and is shown in Table 4.
The results show the following trend in order of increasing MAPE (%), MestReNova < ACD Workbook NMRShiftDB. Molecules such as compound 4 showed higher MAPE (%). The 1 H NMR of the protons in the chair conformation of its cyclohexane contributed to the high error. For molecule 22, the highest error was shown by the hydrogen in the cyclopentane. The 1 H NMR assumed a flat cyclopentane, however it could be either in

ANOVA, Tukey's HSD and CI from MAPE (%) of the 30 organic molecules
To test for the significant difference in MAPE (%) from Table 4, one-way ANOVA with a post hoc test and Tukey's HSD were carried out. The results are shown in Table 5a, 5b and 5c and Figure 3.
ANOVA of dataset shows that F = 5.49, F crit = 2.68, and P = 0.00145. As P < 0.05 and F > F crit , it indicates that H 0 is false, whereas H 1 is true: at least one of the predictor programs is significantly different in performance from the others. In Tukey's HSD, it is found that Q stat > Q crit between MestReNova and NMRShiftDB, and between NMRShiftDB and ACD Workbook Suite, in line with the conclusion reached by ANOVA.
Results in Figure 3 show the CI analysis where the CI between MestReNova and NMRShiftDB, and between NMRShiftDB and ACD Workbook Suite do not encompass zero, indicating that there exists significant difference between MestReNova and NMR-ShiftDB, and between NMRShiftDB and ACD, affirming the conclusion obtained by Tukey's HSD. The results obtained from MAPE lead to conclude that MestReNova and ACD Workbook Suite are equally good in performance, followed by ChemDraw and NMRShiftDB. This conclusion differs slightly from that obtained using RMSD.

Conclusion from RMSD and MAPE statistical studies
The conclusion based on MAPE is that MestReNova and ACD Workbook Suite are equally good in performance, followed by ChemDraw and NMRShiftDB, because MAPE is more reliable as it normalizes our data over the experimental value to a range of 0% to 100%, as compared to RMSD which is unscaled. Based on Table 4, although MestReNova exhibits the lowest MAPE, it could be due to chance, as ANOVA and Tukey's HSD indicate that the difference in its mean is not significant as compared to ACD Workbook Suite.

Using NMR software to predict chemical shifts
Using the best NMR predictors, the 1 H NMR chemical shifts of three published compounds, with 60 data points namely epitetrathiodiketopiperazine (referred to as Compound 31), bis(p-fluorobenzyl)disulfide (Compound 32), and tosylate (Compound 33) [12], were predicted and compared with literature. The structural formulae of the compounds are shown in Figure 4.
The predicted 1 H NMR chemical shifts and experimental data are shown in Table 6. 3.4.1. RMSD and MAPE (%) of three organic compounds from literature [12] Table 7 shows the average RMSD and MAPE (%) for MestReNova and ACD Workbook Suite calculated from the findings in Table 6. Table 7 shows that MestReNova has a lower average MAPE and the same RMSD as compared to ACD Workbook Suite.

T -test of RMSD and MAPE
T -test was performed to determine the statistical difference between the average values obtained from MestReNova and ACD Workbook Suite. Statistical difference is significant when both following conditions are met: (a) TSTAT value is lesser than TCRIT value, (b) P (T ≤ t ) two-tail value is more than 0.05. When it is the case, the NMR prediction software with either the lower average RMSD or the lower average MAPE (%) has performed significantly better than the other. Based on the values obtained in both Table 8a and 8b, there was no instance in which both conditions proved that statistical difference was simultaneously reached.
The t -test showed that the difference between the average MAPE and RMSD of MestReNova and ACD Workbook Suite is not significant. Based on this result, we concluded that MestReNova and ACD Workbook Suite are equally good in terms of 1 H NMR prediction, even though the average MAPE of MestReNova is lower than that of ACD Workbook Suite, as the statistical tests have shown that there is no significant difference between their average MAPEs.

Conclusion
In contrast to NMR spectrometers, NMR prediction programs are certainly more cost-effective, versa-   The ability to employ high frequencies in NMR prediction allows the chemical shift data to be more pronounced while saving cost on acquiring high-end spectroscopic instruments. NMR prediction programs can rapidly generate chemical shift data with high accuracy using wellstudied and reliable computational methods and include other functions such as peak processor, multiplet analysis, structure elucidation programs, and embedded chemical shift databases. The ability to rapidly generate highly accurate chemical shift data without using any spectroscopic instruments makes NMR prediction programs an invaluable tool for scientists.
The data obtained from this project suggest that among the four 1 H NMR software, MestReNova and   predictions having equally less accuracy than the previous two programs. The equal level of accuracy between MestReNova and ACD Workbook Suite, and between ChemDraw and NMRShiftDB is due to the inability of the data collected to prove statistical significance between the two programs within the respective pairings, and thus we cannot definitively confirm that one program is better or worse than the other within each pairing. The results of this study can prove extremely useful in deciding which software to be prioritized in order to obtain more accurate 1 H NMR predictions, encouraging the use of NMR software that provide accurate predictions, and discouraging the use of programs that do not.

Conflicts of interest
Authors have no conflict of interest to declare.