@lewtun - Regarding TinyBERT, have you checked Albert joint model from GitHub - legacyai/tf-transformers: State of the art faster Natural Language Processing in Tensorflow 2.0 . 12 .
Glue score on Albert base 14M and 6 layer seems to have 81, which is better than Tinybert, Mobilebert, distillbert, which has 60M parameter.