david-waterworth:

Do you mean with the GitHub - awslabs/mlm-scoring: Python library & examples for Masked Language Model Scoring (ACL 2020) implementation?


yes, i was wondering whether you could adapt their code to match the current transformers API.



 david-waterworth:

Specifically my tokeniser is in the new single json file format and as far as I can see the 3.3 library is trying to load from the legacy format


can you point me to the line of code where this is done? i might be able to suggest a workaround this way