This sounds like a great project! Love it!
And the tokenizer is ByteLevelBPETokenizer and not WordPiece so I have very little to add =D
As I understand byte pair this is clever because we have very common spelling substitutions between languages e.g. s/c, æ/ä, ø/ö, and å/aa.
With regard to model size there’s obviously some benefits from multiples of 8 (or 32) but could you maybe also take graphics cards RAM sizes into account? Something that can sit comfortably on 8 GB and 12 GB would play pretty well with the current (30) and previous generation of Nvidia cards (too bad for my 2080 Ti with 11 GB).
And do you have anything planned to figure out if Icelandic will actually benefit from this? It might be too small a data set and too different a language?