We train on a single format (format 0 in Table 10), test on all formats, and
break the results by format. We report results with all four operation types. Table
3 shows that the generalization across formats is perfect in most cases. Format 4,
which is the only one not phrased as a question (see Table 10) appears to be the most
challenging to generalize to. We note that there are numerous ways to phrase such
questions, hence the ability to generalize across formats, even for the case where the
model was trained only on one format, is critical for the model’s success.

add
1.0 ± 0
format 0
1.0 ± 0
format 1
1.0 ± 0
format 2
1.0 ± 0
format 3
format 4 0.993 ± 0.012

sub
1.0 ± 0
1.0 ± 0
1.0 ± 0
0.993 ± 0.012
0.863 ± 0.237

mul
1.0 ± 0
1.0 ± 0
1.0 ± 0
0.997 ± 0.006
0.977 ± 0.04

div
0.997 ± 0.006
1.0 ± 0
1.0 ± 0
1.0 ± 0
0.727 ± 0.465

Table 3: Robustness to phrasing. Accuracy when training with one format (format 0)
and evaluating on each of the 5 formats from Table 10.
MRKL Systems