Do you mean with the GitHub - awslabs/mlm-scoring: Python library & examples for Masked Language Model Scoring (ACL 2020) 14 implementation? I’m assuming there’s not much I can do to try and get a 3rd party library which is specifically designed for transformers 3.3 to work with a transformer / tokeniser trained with version 4.5. Specifically my tokeniser is in the new single json file format and as far as I can see the 3.3 library is trying to load from the legacy format. The main issue is the setup.py of the mlm-scoring library requires ==3.3 rather than >=3.3 so installing it downgrades. I suppose I could try removing the version requirement and see what happens.
But ideally the metric would be available via a library which is more up to date. I’ll probably code it up myself altouhg it wont be overly efficient, you need to compute the MLM objective masking each token in order and then sum the log likelyhoods to compute PLL for a single sentance.