I am also really interested in this one, or maybe even ALBERT. But the available dataset on OSCAR might not be enough. Using mC4 might add almost 100GB of clean-ish text. I have worked on Norwegian BERT-based models before 3, and we (the National Library of Norway) have a big corpus that unfortunately we are not allowed to share. We could maybe share parts of it to complement mC4/OSCAR, but not all it. Not sure if there is a way to train on a private corpus in this event, I guess not, but worth asking.
There are also datasets for extrinsic evaluation on Norwegian for POS, NER, and sentiment (the sentiment one I feel was kinda designed for word2vec-like models) from the UiO LTG group, another on political speech party affiliation we built from the Talks of Norway, and a recent one on hate speech we could use. Not sure if evaluating on WikiANN could be an option.