Sources of randomness for Longformer
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.1
- Platform: Ubuntu 20.04.2 LTS 
- Python version: 3.7.10
- PyTorch version: 1.7.1
- Using GPU in script?: Yes

### Who can help
@patrickvonplaten
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information
Results when using `Longformer` for sequence classification are not consistent across runs after setting a random seed.

## To reproduce

I include below a simple script derived from [this](https://huggingface.co/transformers/custom_datasets.html#seq-imdb) example to reproduce the behavior.

When using `bert-base-uncased`, the generated results (and loss values) will be *exactly* the same across runs. However, when using `allenai/longformer-base-4096` (just swap the comment for lines starting with `MODEL_NAME`), results (and loss values) will vary across runs. In this example, results happen to be very similar because of the simplicity of the problem, but I have experienced higher variability in cases with longer training schedules and larger and more 'complex' datasets. Still, I think this example suffices to illustrate the issue.

P.S. Using the commented `set_seed` function does not help.

```
import torch
from datasets import load_dataset
from sklearn.model_selection import train_test_split
from transformers import Trainer, TrainingArguments, set_seed
from transformers import LongformerTokenizerFast, LongformerForSequenceClassification
from transformers import BertTokenizerFast, BertForSequenceClassification

import os
import random
import numpy as np
from typing import Optional


MODEL_NAME = 'bert-base-uncased'
# MODEL_NAME = 'allenai/longformer-base-4096'
SEED = 42

class IMDbDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx])
        return item

    def __len__(self):
        return len(self.labels)


# def set_seed(seed: Optional[int]):
#     """ Set all seeds to make results reproducible (deterministic mode).
#          When seed is None, disables deterministic mode. """
#     if seed is not None:
#         torch.manual_seed(seed)
#         torch.cuda.manual_seed_all(seed)
#         torch.backends.cudnn.deterministic = True
#         torch.backends.cudnn.benchmark = False
#         np.random.seed(seed)
#         random.seed(seed)
#         os.environ['PYTHONHASHSEED'] = str(seed)

def main():
    set_seed(SEED)

    # Load IMDb
    train = load_dataset("imdb", split="train")[:50]
    test = load_dataset("imdb", split="test")[:10]

    train_texts = train['text']
    train_labels= train['label']

    test_texts = test['text']
    test_labels= test['label']

    # Split train into train and val
    train_texts, val_texts, train_labels, val_labels = train_test_split(train_texts, train_labels, test_size=.2,
                                                                        random_state=SEED)

    # Load tokenizer
    if MODEL_NAME == 'bert-base-uncased':
        tokenizer = BertTokenizerFast.from_pretrained(MODEL_NAME)
    elif MODEL_NAME == 'allenai/longformer-base-4096':
        tokenizer = LongformerTokenizerFast.from_pretrained(MODEL_NAME)
    else:
        raise ValueError


    # Generate encodings
    train_encodings = tokenizer(train_texts, truncation=True, padding=True)
    val_encodings = tokenizer(val_texts, truncation=True, padding=True)
    test_encodings = tokenizer(test_texts, truncation=True, padding=True)

    # Create datasets
    train_dataset = IMDbDataset(train_encodings, train_labels)
    val_dataset = IMDbDataset(val_encodings, val_labels)
    test_dataset = IMDbDataset(test_encodings, test_labels)

    # Training
    training_args = TrainingArguments(
        output_dir='../tutorial_results',  # output directory
        num_train_epochs=5,  # total number of training epochs
        per_device_train_batch_size=1,  # batch size per device during training
        per_device_eval_batch_size=1,  # batch size for evaluation
        warmup_steps=500,  # number of warmup steps for learning rate scheduler
        weight_decay=0.01,  # strength of weight decay
        logging_dir='../tutorial_logs',  # directory for storing logs
        logging_steps=10,
        seed=SEED
    )

    if MODEL_NAME == 'bert-base-uncased':
        model = BertForSequenceClassification.from_pretrained(MODEL_NAME)
    elif MODEL_NAME == 'allenai/longformer-base-4096':
        model = LongformerForSequenceClassification.from_pretrained(MODEL_NAME)
    else:
        raise ValueError

    trainer = Trainer(
        model=model,  # the instantiated 🤗 Transformers model to be trained
        args=training_args,  # training arguments, defined above
        train_dataset=train_dataset,  # training dataset
        eval_dataset=val_dataset  # evaluation dataset
    )

    trainer.train()

    # Test set
    test_results = trainer.predict(test_dataset)
    print(test_results)


if __name__ == '__main__':
    main()
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
<!-- A clear and concise description of what you would expect to happen. -->
I would expect that, having set a random seed, results would be the same across runs for longformers too. Is there any source of randomness for longformers not 'covered' by the `set_seed` function?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi, I am facing the same issues with reproducibility for the Longformer model for sequence classification.
I posted my example in the [Huggingface discussion forum here](https://discuss.huggingface.co/t/how-can-i-enforce-reproducibility-for-longformer/8862).

Setting the seeds as recommended [here](https://huggingface.co/transformers/testing.html#getting-reproducible-results) produces the exact same training loss in multiple training iterations (each time starting the finetuning from scratch) when using the `roberta-base` model, but not with `allenai/longformer-base-4096`.

<|||||>Hmm, that's interesting! Also pinging the original author @ibeltagy here.
I don't see any weird functionality that's used in Longformer maybe except `torch.Tensor.stride(...)` that is in Longformer but not in other models like RoBERTa. Sadly I won't have the time to dive deep into this reproducible problem, but a good start to find the bug would be to verify that the same random model weights are loaded before training and then working with `print(...)` to see after what layer results start to differ
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@abhishekkrthakur do you know where the randomness comes from?<|||||>I have recently run into this issue. One of the things I've noticed is if you drop the max length down to 512 I start to get reproducible behaviour - not sure if that helps at all<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@abhishekkrthakur @patrickvonplaten
I'm facing same issue. Any update on this behaviour ?

**Environment info**
transformers Version: 4.11.0
Platform: Amazon Linux AMI 2018.03
torch Version: 1.9.0
Number GPU: 1

more information, when activating torch.use_deterministic_algorithms(True) during training. 
roberta-base and bert-base-uncased works fine. However with Longformer I get this error:

Loading features from cached file ./training-data/processed-data/text/toy/cached_train_longformer_512
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Validation sanity check: 0it [00:00, ?it/s]Loading features from cached file ./training-data/processed-data/text/toy/cached_dev_longformer_512
Validation sanity check:   0%|                                                                                                                                                       | 0/2 [00:00<?, ?it/s]inputs={'input_ids': tensor([[    0,  7202,  3063,  ...,     1,     1,     1],
        [    0,  1301,  1723,  ..., 44828, 15555,     2],
        [    0, 27201,  1000,  ...,  3706,     6,     2],
        ...,
        [    0, 41188,  2444,  ...,     1,     1,     1],
        [    0,  3384,   591,  ...,     1,     1,     1],
        [    0,   495,  2492,  ...,   700, 28607,     2]], device='cuda:0'), 'attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 1, 1, 1],
        [1, 1, 1,  ..., 1, 1, 1],
        ...,
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 1, 1, 1]], device='cuda:0'), 'labels': tensor([1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1,
        0, 0, 1, 0, 1, 0, 1, 1], device='cuda:0')}
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/venv-transformer/bin/nlpsaf-transformer-train", line 33, in <module>
    sys.exit(load_entry_point('nlpsaf-transformer', 'console_scripts', 'nlpsaf-transformer-train')())
  File "/home/ec2-user/SageMaker/nlpsaf-transformers/nlpsaf_transformer/run_safety_signal.py", line 31, in main
    _ = train(args)
  File "/home/ec2-user/SageMaker/nlpsaf-transformers/nlpsaf_transformer/run_safety_signal.py", line 17, in train
    generic_train(model, args)
  File "/home/ec2-user/SageMaker/nlpsaf-transformers/nlpsaf_transformer/model.py", line 462, in generic_train
    trainer.fit(model)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 552, in fit
    self._run(model)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 922, in _run
    self._dispatch()
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _dispatch
    self.accelerator.start_training(self)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1000, in run_stage
    return self._run_train()
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run_train
    self._run_sanity_check(self.lightning_module)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1122, in _run_sanity_check
    self._evaluation_loop.run()
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
    dl_outputs = self.epoch_loop.run(
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 111, in advance
    output = self.evaluation_step(batch, batch_idx, dataloader_idx)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 158, in evaluation_step
    output = self.trainer.accelerator.validation_step(step_kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 211, in validation_step
    return self.training_type_plugin.validation_step(*step_kwargs.values())
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 392, in validation_step
    return self.model(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 799, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 93, in forward
    output = self.module.validation_step(*inputs, **kwargs)
  File "/home/ec2-user/SageMaker/nlpsaf-transformers/nlpsaf_transformer/model.py", line 345, in validation_step
    outputs = self(**inputs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/SageMaker/nlpsaf-transformers/nlpsaf_transformer/model.py", line 179, in forward
    return self.model(**inputs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 1858, in forward
    outputs = self.longformer(
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 1677, in forward
    encoder_outputs = self.encoder(
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 1280, in forward
    layer_outputs = layer_module(
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 1205, in forward
    self_attn_outputs = self.attention(
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 1141, in forward
    self_outputs = self.self(
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/venv-transformer/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 708, in forward
    attn_probs[is_index_global_attn_nonzero] = 0
RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/native/cuda/Indexing.cu":253, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor1973761<|||||>Facing the same issue with [allenai/led-large-16384](https://huggingface.co/allenai/led-large-16384) via the [run_summarization.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization.py) script. Both `seed` and `data_seed` are set, and there are no randomly initialized weights:

```
[INFO|modeling_utils.py:3032] 2023-03-30 10:44:26,097 >> All model checkpoint weights were used when initializing LEDForConditionalGeneration.
[INFO|modeling_utils.py:3040] 2023-03-30 10:44:26,098 >> All the weights of LEDForConditionalGeneration were initialized from the model checkpoint at allenai/led-large-16384.
```

The same script with another model (e.g. `flan-t5-large`) is perfectly reproducible across runs with the same seed. I don't have a sense of where the remaining sources of randomness would be for Longformer/LED.