Hi rgwatwormhill,
thanks for taking the time to answer this. I put in an example in the main question which might help explain what I am trying to do.
Yes, the two completely separate BERT models seems to be the cleanest way to go! I would try to of course avoid that if the hugginface library does provide a way to specify per-head masks, which I may be overlooking.