Independent run: our MNLI 0.7417 against your published 0.856 - which is right?
Independent run of decider-2b, and a gap on MNLI we would like to resolve.
Setup. Revision 533964dae8be, one sealed manifest, seed 42, 1,190 cases / 1,240 scored evaluation decisions across 9 typed-decision suites, bf16 on one T4.
What we measured (overall 0.7895):
| suite | ours |
|---|---|
| MNLI | 0.7417 |
| our moderation suite | 0.8819 |
| our guardrails suite | 0.6354 |
| emotion | 0.6719 |
The gap. Your card reports multinli (choice; trained) at 0.856 / 0.853. Ours is 0.7417 on MNLI - about 0.11 lower. Since both are choice questions over MNLI, we could not explain it from task shape alone.
Three candidates on our side, in the order we suspect them:
- Prompt shape. We present the full label set as declared options and score with
per_option_conditional_logprob. A card that reports a trained/choice row may use a different verbalisation. - Label set. We use the 3-way MNLI test split. A different label grouping changes the ceiling.
- Split membership. Our manifest pins case IDs and order indices, so we can diff row-by-row rather than argue in aggregate.
What would help. The exact MNLI prompt and label set behind the 0.856 / 0.853 row. We will re-run and report whichever number is correct, including if yours holds and ours is the bug.
Note our guardrails (0.6354) and moderation (0.8819) spread far more than your aegis2 (noul) 0.728 / 0.716 would suggest, but our two suites are different datasets from aegis2, so we are not treating that as a comparison.
Full data, per-suite matrices, per-row predictions: https://sysone.sdad.pro/
Update: we have corrected the gold labels and re-scored the panel as dataset v2.1.0.
The overall figure quoted above - 0.7895 - was scored against the old gold, which had our reviewer's label drift on about 14% of public-source cases. Under the corrected gold, the same stored answers for decider-2b re-score to 0.7976.
| v2.0.0 gold | v2.1.0 gold | |
|---|---|---|
| decider-2b overall accuracy | 0.7895 | 0.7976 |
No inference was re-run. The correction moves the gold; the answers are unchanged. Full detail: https://sysone.sdad.pro/
Thank you for the careful report, and sorry for the slow reply. I re-ran your MNLI items and the full MNLI validation set. Your harness is correct; the gap comes from which items are scored.
Your harness. runners/vendor/decider.py calls our Decider.system_one with your questions unchanged, so the readout is our lettered-option softmax, not a per-option log-probability. I ran your 120 MNLI evaluation cases (manifest v2.1.0) at your pinned revisions, bf16 on a B300:
| your report | our re-run, your wording | our re-run, our suite's wording | |
|---|---|---|---|
decider-4b (eb5fbdf) |
0.842 | 0.850 | 0.825 |
decider-2b (533964d) |
0.792 | 0.783 | 0.783 |
Our numbers differ from yours by one item, which fits T4 versus B300 bf16 arithmetic. The wording of the question does not explain the gap.
The item sets. All 120 of your cases are from MNLI validation_matched, and your labels equal the MNLI gold labels on all 120. The figure on our card comes from a different sample of the same split: 299 items from the public JevBench suite (multinli). To find the value on the whole split, I scored all 9,815 validation_matched items:
| your 120 items | our 299-item suite | all 9,815 validation_matched | |
|---|---|---|---|
| decider-4b | 0.850 | 0.933 | 0.904 |
| decider-2b | 0.783 | 0.860 | 0.864 |
- Your 120 items are harder than the split as a whole, mainly on neutral. decider-4b gets 0.725 on your 40 neutral items and 0.830 on all neutral items. With 120 items the 95% interval is about ±6 points, so a sample this far below the mean is unusual but possible.
- Our suite is easier than the split for decider-4b: 0.933 against 0.904. For decider-2b the suite matches the split.
- MNLI's training split is in our training mixture, which is why the card marks the row "trained".
What changed on our side. Both cards now give the full validation_matched accuracy next to the suite number: 0.904 for decider-4b and 0.864 for decider-2b. Your figures are correct for your items. Because every model in your panel is scored on the same 120 items, the comparison between models is unaffected.
Thanks also for the corrected-gold update. The scripts and per-item outputs are available if you want to check them.