Is attention_mask supposed to be added to attention_weights? Based on functions docstring mask values are either 0 or 1.  
https://github.com/huggingface/transformers/blame/476be08c4aa96f8c1cae4200d2677bbe8f12cf80/src/transformers/models/autoformer/modeling_autoformer.py#L619
cc @kashif <|||||>thanks @sgugger and @rpanackal let me check