You can do what @valhalla said. Perhaps the most clean way is changing the config.
config = AutoConfig.from_pretrained(MODEL_ID, output_attentions=True)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID, config=config)
output = model(**inputs) # inputs from default_data_collator

output will consist of logits, pooled output and attention.