loads the tokenizer for each checkpoint, to solve the reproducability…
Hi
I observed that if you run "run_glue" code with the same parameters in the following ways:
1) run with both --do_train and --do_eval
2) run without --do_train but only --do_eval, but set the modelpath to use the trained models from step 1

The obtained evaluation results in these two cases are not the same, and to make the results reproducible it is needed to reload tokenizer from checkpoints.
 
Thanks.
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026?src=pr&el=h1) Report
> Merging [#1026](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/a7b4cfe9194bf93c7044a42c9f1281260ce6279e?src=pr&el=desc) will **decrease** coverage by `0.31%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #1026      +/-   ##
=========================================
- Coverage   79.22%   78.9%   -0.32%     
=========================================
  Files          38      34       -4     
  Lines        6406    6192     -214     
=========================================
- Hits         5075    4886     -189     
+ Misses       1331    1306      -25
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/file\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZmlsZV91dGlscy5weQ==) | `71.01% <0%> (-3.09%)` | :arrow_down: |
| [pytorch\_transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2JlcnQucHk=) | `93.2% <0%> (-1.95%)` | :arrow_down: |
| [...transformers/tests/tokenization\_transfo\_xl\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3RyYW5zZm9feGxfdGVzdC5weQ==) | `96.42% <0%> (-0.55%)` | :arrow_down: |
| [...rch\_transformers/tests/tokenization\_openai\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX29wZW5haV90ZXN0LnB5) | `96.77% <0%> (-0.45%)` | :arrow_down: |
| [...ytorch\_transformers/tests/tokenization\_xlm\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3hsbV90ZXN0LnB5) | `96.77% <0%> (-0.45%)` | :arrow_down: |
| [...orch\_transformers/tests/tokenization\_xlnet\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3hsbmV0X3Rlc3QucHk=) | `97.05% <0%> (-0.45%)` | :arrow_down: |
| [...torch\_transformers/tests/tokenization\_gpt2\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX2dwdDJfdGVzdC5weQ==) | `96.87% <0%> (-0.43%)` | :arrow_down: |
| [pytorch\_transformers/tests/optimization\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvb3B0aW1pemF0aW9uX3Rlc3QucHk=) | `98.57% <0%> (-0.41%)` | :arrow_down: |
| [pytorch\_transformers/optimization.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvb3B0aW1pemF0aW9uLnB5) | `96.29% <0%> (-0.34%)` | :arrow_down: |
| [...torch\_transformers/tests/tokenization\_bert\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX2JlcnRfdGVzdC5weQ==) | `98.38% <0%> (-0.13%)` | :arrow_down: |
| ... and [14 more](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026?src=pr&el=footer). Last update [a7b4cfe...3d47a7f](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1026?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>Ok, looks good to me, thanks @rabeehk <|||||>Hi Thomas
There is really some reproducibility issue in the codes, and this solves it
only
for the case when I think one does not evaluate on all the checkpoints,
please
undo this commit just to be sure not to break the codes, I will send you a
new
pull request soon when it is test for both cases.
thank you.
Best regards,
Rabeeh

On Fri, Aug 30, 2019 at 2:16 PM Thomas Wolf <notifications@github.com>
wrote:

> Merged #1026
> <https://github.com/huggingface/pytorch-transformers/pull/1026> into
> master.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/pull/1026?email_source=notifications&email_token=ABP4ZCFNY7Y4FNDVAV6CWK3QHEFQBA5CNFSM4ILSUUMKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOTLFNQXI#event-2596984925>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCBELPM2QN44SHXIZMTQHEFQBANCNFSM4ILSUUMA>
> .
>
