Will it be possible for you to create colab for this ?
Also, to take advantage of .from_pretrained  and .save_pretrained you can sub-class the BertPretrainedModel  and add the additional layers in it.  See these 203 task specific bert models, they use bert and additional layer on top of it and subclass BertPreTrainedModel.
let me know if this solves your problem.