Independent run: our MNLI 0.7667 against your published 0.926
Following up on our independent run of decider-4b - a large gap on MNLI we cannot yet explain.
Setup. Revision 533964dae8be, one sealed manifest, seed 42, 1,190 cases / 1,240 scored evaluation decisions across 9 typed-decision suites, bf16 on one T4. Same harness and same pinned revision as our decider-2b post.
What we measured (overall 0.8411):
| suite | ours | your card |
|---|---|---|
| MNLI | 0.7667 | 0.926 |
| our moderation suite | 0.8790 | - |
| our guardrails suite | 0.8333 | - |
The gap. Both are choice questions over MNLI. Ours is 0.1593 below yours. That is far outside anything our label grouping or option-count could explain, so we do not think we can attribute it to task shape.
What we suspect on our side, most likely first.
- Prompt and option verbalisation. We declare the label set per case and score with
per_option_conditional_logprob. If your harness instead renders options as an enumerated list and reads a single next-token distribution, the two are not measuring the same function, and MNLI is where that diverges most for us. - Scoring path. Our
choicescoring for decider-4b went through the vendor-default readout with no calibration applied. - Split. Our manifest pins case IDs and order indices, so we can diff row-by-row instead of arguing in aggregate.
What would help. The exact MNLI prompt, label set and answer-extraction rule behind your 0.926 figure. We will re-run and publish whichever number holds, including if yours is right and our harness is the bug. We would rather correct our record than leave a 0.17 gap unexplained.
Full data, per-suite matrices and per-row predictions: https://sysone.sdad.pro/
Correction to the setup line above. Two fields in my post were wrong, both mine, neither affecting the MNLI comparison:
| field | stated above | correct |
|---|---|---|
| revision | 533964dae8be |
eb5fbdfc9448 |
| scoring path | no calibration applied |
vendor-default readout, dtype bf16 |
| our moderation suite | 0.8790 |
0.9444 |
| our guardrails suite | 0.8333 |
0.8333 |
The revision is the one to care about: it was transcribed from the wrong row of our results, so the
checkpoint I named was not the checkpoint I ran. eb5fbdfc9448 is the pinned revision fromdecider-4b-t4-d0. MNLI remains 0.7667 overall 0.8411, and that is unchanged.
Apologies for the noise. Source of truth for every field is the run directory named above.
Update: we have corrected the gold labels and re-scored the panel as dataset v2.1.0.
The overall figure quoted above - 0.8411 - was scored against the old gold, which had our reviewer's label drift on about 14% of public-source cases. Under the corrected gold, the same stored answers for decider-4b re-score to 0.8581.
| v2.0.0 gold | v2.1.0 gold | |
|---|---|---|
| decider-4b overall accuracy | 0.8411 | 0.8581 |
No inference was re-run. The correction moves the gold; the answers are unchanged. Full detail: https://sysone.sdad.pro/
Thank you for the careful report, and sorry for the slow reply. I re-ran your MNLI items and the full MNLI validation set. Your harness is correct; the gap comes from which items are scored.
Your harness. runners/vendor/decider.py calls our Decider.system_one with your questions unchanged, so the readout is our lettered-option softmax, not a per-option log-probability. I ran your 120 MNLI evaluation cases (manifest v2.1.0) at your pinned revisions, bf16 on a B300:
| your report | our re-run, your wording | our re-run, our suite's wording | |
|---|---|---|---|
decider-4b (eb5fbdf) |
0.842 | 0.850 | 0.825 |
decider-2b (533964d) |
0.792 | 0.783 | 0.783 |
Our numbers differ from yours by one item, which fits T4 versus B300 bf16 arithmetic. The wording of the question does not explain the gap.
The item sets. All 120 of your cases are from MNLI validation_matched, and your labels equal the MNLI gold labels on all 120. The figure on our card comes from a different sample of the same split: 299 items from the public JevBench suite (multinli). To find the value on the whole split, I scored all 9,815 validation_matched items:
| your 120 items | our 299-item suite | all 9,815 validation_matched | |
|---|---|---|---|
| decider-4b | 0.850 | 0.933 | 0.904 |
| decider-2b | 0.783 | 0.860 | 0.864 |
- Your 120 items are harder than the split as a whole, mainly on neutral. decider-4b gets 0.725 on your 40 neutral items and 0.830 on all neutral items. With 120 items the 95% interval is about ±6 points, so a sample this far below the mean is unusual but possible.
- Our suite is easier than the split for decider-4b: 0.933 against 0.904. For decider-2b the suite matches the split.
- MNLI's training split is in our training mixture, which is why the card marks the row "trained".
What changed on our side. Both cards now give the full validation_matched accuracy next to the suite number: 0.904 for decider-4b and 0.864 for decider-2b. Your figures are correct for your items. Because every model in your panel is scored on the same 120 items, the comparison between models is unaffected.
Thanks also for the corrected-gold update. The scripts and per-item outputs are available if you want to check them.