You might need to modify the encoder (and/or) the decoder in this:
https://huggingface.co/transformers/_modules/transformers/models/bart/modeling_bart.html#BartModel 2
Copy-paste the entire file, save it locally as ‘modifiedbart.py’, and call it instead of the default “BARTModel” in transformers.