Thank you for your help. I was able to get the model coded up with the following specs:
Encoder - google/vit-base-patch16-224-in21k
Decoder - bert-base-uncased.
Any suggestions for looking inside the model from an explainability standpoint? I was thinking about using something like shap - Welcome to the SHAP documentation — SHAP latest documentation 1 but wasn’t sure if there were other options.