Thanks for the info Sylvain!
I agree that the config is a bit weird, so I’ve posted a question on the FastFormers repo (link 16) whose experiments I am trying to reproduce and who somehow found a way to make TinyBERT work with  Transformers.
Failing that, I’ll ask the TinyBERT authors directly and report back here