(TF) InternalError: Multiple CPU devices 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.5
- Platform: Linux-5.4.0-90-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): 2.3.4 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed


### Who can help
@patrickvonplaten @Rocketknight1
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Longformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. instantiate `TFAutoModelForSequenceClassification.from_pretrained("allenai/longformer-base-4096")`
2. compile the model under `tf.distribute.experimental.MultiWorkerMirroredStrategy` strategy
3. run `fit()`

```
Traceback (most recent call last):
  File "training.py", line 20, in <module>
    classifier.fit()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/training.py", line 27, in fit
    self._fit()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/_mixins.py", line 50, in _fit
    self._fit_model()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/_mixins.py", line 430, in _fit_model
    class_weight=self.class_weight_
  File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/opt/miniconda/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InternalError: Multiple CPU devices [/job:worker/replica:0/task:0/device:GPU:7,/job:worker/replica:0/task:0/device:CPU:0,/job:localhost/replica:0/task:0/device:CPU:0] [Op:__inference_train_function_800368]
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I would expect the model to begin training however it errors with the traceback above. I have tested this same script with `TFDistilBertForSequenceClassification` and the model successfully trains, so I'm kinda stuck here.

EDIT: This isn't running locally on my machine, but in Azure on the [A100 VM series](https://docs.microsoft.com/en-us/azure/virtual-machines/nda100-v4-series)

<!-- A clear and concise description of what you would expect to happen. -->

This is unusual - is this issue unique to Hugging Face models? Does it still occur with a basic Keras model? <|||||>Hi @Rocketknight1! I haven't tried with basic Keras models. I can try that if you think it would help, however, `TFDistilBertForSequenceClassification` trained successfully, so I suspect it is something specific to `Longformer`<|||||>So I traced it to [here](https://github.com/tensorflow/tensorflow/blob/b1b76e24284d55156ad4e12fb847bfbc637b9a20/tensorflow/compiler/jit/device_util.cc#L173-L176). Must be some issue with the JIT compilation. I'll test some stuff out.<|||||>Still testing stuff, but I didn't pay close attention when I ran the cli tool to get system info. I'm actually using a docker container on the A100 VMs that is based on Debian + NVIDIA (cudnn 8/ cuda 11.2).

The `tensorflow` version is actually 2.7.*
The `torch` version is actually 1.10.0<|||||>[this](https://github.com/tensorflow/tensorflow/issues/53192) seems suspiciously related as they're having JIT issues on the A100. I think in my case though, TF isn't sure which device to use.<|||||>I was able to train the model successfully in the end by disabling XLA JIT compilation with `ENV TF_XLA_FLAGS="--tf_xla_auto_jit=-1"`<|||||>Ok, so it works for a single epoch. When running more than 1 epoch, the first epoch succeeds and upon beginning the second epoch, we get this (first epoch logging included here for completeness).

```
1/232 [..............................] - ETA: 35:39:11 - loss: 0.5014 - Accuracy: 0.8750
  2/232 [..............................] - ETA: 4:16 - loss: 0.4386 - Accuracy: 0.8125    
  3/232 [..............................] - ETA: 4:02 - loss: 0.3492 - Accuracy: 0.6250
  4/232 [..............................] - ETA: 3:51 - loss: 0.3182 - Accuracy: 0.6250
  5/232 [..............................] - ETA: 3:46 - loss: 0.2899 - Accuracy: 0.6750
  6/232 [..............................] - ETA: 3:42 - loss: 0.2548 - Accuracy: 0.7083
  7/232 [..............................] - ETA: 3:38 - loss: 0.2454 - Accuracy: 0.6786
  8/232 [>.............................] - ETA: 3:37 - loss: 0.2363 - Accuracy: 0.6719
  9/232 [>.............................] - ETA: 3:37 - loss: 0.2329 - Accuracy: 0.6389
 10/232 [>.............................] - ETA: 3:36 - loss: 0.2262 - Accuracy: 0.6375
 11/232 [>.............................] - ETA: 3:34 - loss: 0.2139 - Accuracy: 0.6705
 12/232 [>.............................] - ETA: 3:33 - loss: 0.2021 - Accuracy: 0.6875
 13/232 [>.............................] - ETA: 3:32 - loss: 0.1965 - Accuracy: 0.6923
 14/232 [>.............................] - ETA: 3:30 - loss: 0.1912 - Accuracy: 0.7054
 15/232 [>.............................] - ETA: 3:30 - loss: 0.1894 - Accuracy: 0.7000
 16/232 [=>............................] - ETA: 3:29 - loss: 0.1802 - Accuracy: 0.7188
 17/232 [=>............................] - ETA: 3:27 - loss: 0.1793 - Accuracy: 0.7132
 18/232 [=>............................] - ETA: 3:25 - loss: 0.1753 - Accuracy: 0.7153
 19/232 [=>............................] - ETA: 3:24 - loss: 0.1798 - Accuracy: 0.6974
 20/232 [=>............................] - ETA: 3:22 - loss: 0.1771 - Accuracy: 0.7063
 21/232 [=>............................] - ETA: 3:21 - loss: 0.1741 - Accuracy: 0.7083
 22/232 [=>............................] - ETA: 3:20 - loss: 0.1751 - Accuracy: 0.6989
 23/232 [=>............................] - ETA: 3:19 - loss: 0.1753 - Accuracy: 0.6902
 24/232 [==>...........................] - ETA: 3:17 - loss: 0.1702 - Accuracy: 0.7031
 25/232 [==>...........................] - ETA: 3:16 - loss: 0.1677 - Accuracy: 0.7050
 26/232 [==>...........................] - ETA: 3:15 - loss: 0.1656 - Accuracy: 0.7067
 27/232 [==>...........................] - ETA: 3:14 - loss: 0.1619 - Accuracy: 0.7130
 28/232 [==>...........................] - ETA: 3:12 - loss: 0.1616 - Accuracy: 0.7098
 29/232 [==>...........................] - ETA: 3:11 - loss: 0.1619 - Accuracy: 0.7069
 30/232 [==>...........................] - ETA: 3:10 - loss: 0.1644 - Accuracy: 0.6958
 31/232 [===>..........................] - ETA: 3:09 - loss: 0.1616 - Accuracy: 0.7016
 32/232 [===>..........................] - ETA: 3:08 - loss: 0.1605 - Accuracy: 0.7031
 33/232 [===>..........................] - ETA: 3:08 - loss: 0.1612 - Accuracy: 0.7008
 34/232 [===>..........................] - ETA: 3:06 - loss: 0.1588 - Accuracy: 0.7059
 35/232 [===>..........................] - ETA: 3:06 - loss: 0.1563 - Accuracy: 0.7143
 36/232 [===>..........................] - ETA: 3:05 - loss: 0.1550 - Accuracy: 0.7153
 37/232 [===>..........................] - ETA: 3:04 - loss: 0.1549 - Accuracy: 0.7128
 38/232 [===>..........................] - ETA: 3:03 - loss: 0.1535 - Accuracy: 0.7138
 39/232 [====>.........................] - ETA: 3:02 - loss: 0.1527 - Accuracy: 0.7147
 40/232 [====>.........................] - ETA: 3:01 - loss: 0.1533 - Accuracy: 0.7125
 41/232 [====>.........................] - ETA: 2:59 - loss: 0.1500 - Accuracy: 0.7195
 42/232 [====>.........................] - ETA: 2:59 - loss: 0.1524 - Accuracy: 0.7113
 43/232 [====>.........................] - ETA: 2:58 - loss: 0.1523 - Accuracy: 0.7122
 44/232 [====>.........................] - ETA: 2:57 - loss: 0.1526 - Accuracy: 0.7102
 45/232 [====>.........................] - ETA: 2:56 - loss: 0.1528 - Accuracy: 0.7083
 46/232 [====>.........................] - ETA: 2:55 - loss: 0.1509 - Accuracy: 0.7147
 47/232 [=====>........................] - ETA: 2:54 - loss: 0.1496 - Accuracy: 0.7181
 48/232 [=====>........................] - ETA: 2:53 - loss: 0.1507 - Accuracy: 0.7135
 49/232 [=====>........................] - ETA: 2:52 - loss: 0.1507 - Accuracy: 0.7117
 50/232 [=====>........................] - ETA: 2:51 - loss: 0.1508 - Accuracy: 0.7100
 51/232 [=====>........................] - ETA: 2:50 - loss: 0.1508 - Accuracy: 0.7083
 52/232 [=====>........................] - ETA: 2:49 - loss: 0.1522 - Accuracy: 0.7019
 53/232 [=====>........................] - ETA: 2:48 - loss: 0.1504 - Accuracy: 0.7075
 54/232 [=====>........................] - ETA: 2:47 - loss: 0.1500 - Accuracy: 0.7083
 55/232 [======>.......................] - ETA: 2:46 - loss: 0.1494 - Accuracy: 0.7091
 56/232 [======>.......................] - ETA: 2:45 - loss: 0.1487 - Accuracy: 0.7098
 57/232 [======>.......................] - ETA: 2:44 - loss: 0.1491 - Accuracy: 0.7061
 58/232 [======>.......................] - ETA: 2:44 - loss: 0.1485 - Accuracy: 0.7069
 59/232 [======>.......................] - ETA: 2:43 - loss: 0.1473 - Accuracy: 0.7097
 60/232 [======>.......................] - ETA: 2:42 - loss: 0.1467 - Accuracy: 0.7104
 61/232 [======>.......................] - ETA: 2:41 - loss: 0.1458 - Accuracy: 0.7111
 62/232 [=======>......................] - ETA: 2:40 - loss: 0.1447 - Accuracy: 0.7137
 63/232 [=======>......................] - ETA: 2:39 - loss: 0.1449 - Accuracy: 0.7123
 64/232 [=======>......................] - ETA: 2:38 - loss: 0.1446 - Accuracy: 0.7129
 65/232 [=======>......................] - ETA: 2:37 - loss: 0.1470 - Accuracy: 0.7038
 66/232 [=======>......................] - ETA: 2:36 - loss: 0.1468 - Accuracy: 0.7027
 67/232 [=======>......................] - ETA: 2:35 - loss: 0.1472 - Accuracy: 0.6996
 68/232 [=======>......................] - ETA: 2:34 - loss: 0.1472 - Accuracy: 0.6967
 69/232 [=======>......................] - ETA: 2:33 - loss: 0.1481 - Accuracy: 0.6957
 70/232 [========>.....................] - ETA: 2:32 - loss: 0.1482 - Accuracy: 0.7000
 71/232 [========>.....................] - ETA: 2:31 - loss: 0.1488 - Accuracy: 0.6972
 72/232 [========>.....................] - ETA: 2:30 - loss: 0.1481 - Accuracy: 0.6997
 73/232 [========>.....................] - ETA: 2:29 - loss: 0.1485 - Accuracy: 0.6969
 74/232 [========>.....................] - ETA: 2:28 - loss: 0.1496 - Accuracy: 0.6926
 75/232 [========>.....................] - ETA: 2:28 - loss: 0.1515 - Accuracy: 0.6867
 76/232 [========>.....................] - ETA: 2:26 - loss: 0.1505 - Accuracy: 0.6891
 77/232 [========>.....................] - ETA: 2:25 - loss: 0.1511 - Accuracy: 0.6867
 78/232 [=========>....................] - ETA: 2:24 - loss: 0.1506 - Accuracy: 0.6875
 79/232 [=========>....................] - ETA: 2:23 - loss: 0.1513 - Accuracy: 0.6835
 80/232 [=========>....................] - ETA: 2:22 - loss: 0.1516 - Accuracy: 0.6812
 81/232 [=========>....................] - ETA: 2:21 - loss: 0.1514 - Accuracy: 0.6806
 82/232 [=========>....................] - ETA: 2:20 - loss: 0.1513 - Accuracy: 0.6799
 83/232 [=========>....................] - ETA: 2:20 - loss: 0.1507 - Accuracy: 0.6822
 84/232 [=========>....................] - ETA: 2:19 - loss: 0.1512 - Accuracy: 0.6786
 85/232 [=========>....................] - ETA: 2:18 - loss: 0.1515 - Accuracy: 0.6750
 86/232 [==========>...................] - ETA: 2:17 - loss: 0.1512 - Accuracy: 0.6744
 87/232 [==========>...................] - ETA: 2:16 - loss: 0.1509 - Accuracy: 0.6753
 88/232 [==========>...................] - ETA: 2:15 - loss: 0.1506 - Accuracy: 0.6776
 89/232 [==========>...................] - ETA: 2:14 - loss: 0.1505 - Accuracy: 0.6784
 90/232 [==========>...................] - ETA: 2:13 - loss: 0.1501 - Accuracy: 0.6806
 91/232 [==========>...................] - ETA: 2:12 - loss: 0.1496 - Accuracy: 0.6827
 92/232 [==========>...................] - ETA: 2:11 - loss: 0.1501 - Accuracy: 0.6793
 93/232 [===========>..................] - ETA: 2:10 - loss: 0.1495 - Accuracy: 0.6815
 94/232 [===========>..................] - ETA: 2:09 - loss: 0.1497 - Accuracy: 0.6795
 95/232 [===========>..................] - ETA: 2:08 - loss: 0.1493 - Accuracy: 0.6803
 96/232 [===========>..................] - ETA: 2:07 - loss: 0.1489 - Accuracy: 0.6810
 97/232 [===========>..................] - ETA: 2:06 - loss: 0.1494 - Accuracy: 0.6791
 98/232 [===========>..................] - ETA: 2:05 - loss: 0.1486 - Accuracy: 0.6811
 99/232 [===========>..................] - ETA: 2:04 - loss: 0.1478 - Accuracy: 0.6831
100/232 [===========>..................] - ETA: 2:03 - loss: 0.1479 - Accuracy: 0.6825
101/232 [============>.................] - ETA: 2:02 - loss: 0.1483 - Accuracy: 0.6807
102/232 [============>.................] - ETA: 2:01 - loss: 0.1496 - Accuracy: 0.6765
103/232 [============>.................] - ETA: 2:01 - loss: 0.1492 - Accuracy: 0.6772
104/232 [============>.................] - ETA: 2:00 - loss: 0.1493 - Accuracy: 0.6767
105/232 [============>.................] - ETA: 1:59 - loss: 0.1497 - Accuracy: 0.6750
106/232 [============>.................] - ETA: 1:58 - loss: 0.1493 - Accuracy: 0.6757
107/232 [============>.................] - ETA: 1:57 - loss: 0.1503 - Accuracy: 0.6717
108/232 [============>.................] - ETA: 1:56 - loss: 0.1496 - Accuracy: 0.6736
109/232 [=============>................] - ETA: 1:55 - loss: 0.1496 - Accuracy: 0.6732
110/232 [=============>................] - ETA: 1:54 - loss: 0.1500 - Accuracy: 0.6716
111/232 [=============>................] - ETA: 1:53 - loss: 0.1496 - Accuracy: 0.6723
112/232 [=============>................] - ETA: 1:52 - loss: 0.1496 - Accuracy: 0.6719
113/232 [=============>................] - ETA: 1:51 - loss: 0.1496 - Accuracy: 0.6715
114/232 [=============>................] - ETA: 1:50 - loss: 0.1490 - Accuracy: 0.6732
115/232 [=============>................] - ETA: 1:49 - loss: 0.1490 - Accuracy: 0.6728
116/232 [==============>...............] - ETA: 1:48 - loss: 0.1486 - Accuracy: 0.6746
117/232 [==============>...............] - ETA: 1:47 - loss: 0.1484 - Accuracy: 0.6752
118/232 [==============>...............] - ETA: 1:47 - loss: 0.1480 - Accuracy: 0.6769
119/232 [==============>...............] - ETA: 1:46 - loss: 0.1472 - Accuracy: 0.6796
120/232 [==============>...............] - ETA: 1:45 - loss: 0.1469 - Accuracy: 0.6802
121/232 [==============>...............] - ETA: 1:44 - loss: 0.1471 - Accuracy: 0.6787
122/232 [==============>...............] - ETA: 1:43 - loss: 0.1468 - Accuracy: 0.6793
123/232 [==============>...............] - ETA: 1:42 - loss: 0.1462 - Accuracy: 0.6809
124/232 [===============>..............] - ETA: 1:41 - loss: 0.1459 - Accuracy: 0.6815
125/232 [===============>..............] - ETA: 1:40 - loss: 0.1456 - Accuracy: 0.6820
126/232 [===============>..............] - ETA: 1:39 - loss: 0.1450 - Accuracy: 0.6835
127/232 [===============>..............] - ETA: 1:38 - loss: 0.1452 - Accuracy: 0.6831
128/232 [===============>..............] - ETA: 1:37 - loss: 0.1450 - Accuracy: 0.6836
129/232 [===============>..............] - ETA: 1:36 - loss: 0.1454 - Accuracy: 0.6822
130/232 [===============>..............] - ETA: 1:35 - loss: 0.1452 - Accuracy: 0.6827
131/232 [===============>..............] - ETA: 1:34 - loss: 0.1449 - Accuracy: 0.6832
132/232 [================>.............] - ETA: 1:33 - loss: 0.1450 - Accuracy: 0.6828
133/232 [================>.............] - ETA: 1:32 - loss: 0.1448 - Accuracy: 0.6833
134/232 [================>.............] - ETA: 1:32 - loss: 0.1448 - Accuracy: 0.6828
135/232 [================>.............] - ETA: 1:31 - loss: 0.1446 - Accuracy: 0.6833
136/232 [================>.............] - ETA: 1:30 - loss: 0.1440 - Accuracy: 0.6847
137/232 [================>.............] - ETA: 1:29 - loss: 0.1435 - Accuracy: 0.6861
138/232 [================>.............] - ETA: 1:28 - loss: 0.1436 - Accuracy: 0.6857
139/232 [================>.............] - ETA: 1:27 - loss: 0.1433 - Accuracy: 0.6862
140/232 [=================>............] - ETA: 1:26 - loss: 0.1432 - Accuracy: 0.6866
141/232 [=================>............] - ETA: 1:25 - loss: 0.1427 - Accuracy: 0.6879
142/232 [=================>............] - ETA: 1:24 - loss: 0.1425 - Accuracy: 0.6884
143/232 [=================>............] - ETA: 1:23 - loss: 0.1423 - Accuracy: 0.6888
144/232 [=================>............] - ETA: 1:22 - loss: 0.1419 - Accuracy: 0.6901
145/232 [=================>............] - ETA: 1:21 - loss: 0.1419 - Accuracy: 0.6897
146/232 [=================>............] - ETA: 1:20 - loss: 0.1423 - Accuracy: 0.6884
147/232 [==================>...........] - ETA: 1:19 - loss: 0.1418 - Accuracy: 0.6896
148/232 [==================>...........] - ETA: 1:18 - loss: 0.1422 - Accuracy: 0.6883
149/232 [==================>...........] - ETA: 1:17 - loss: 0.1420 - Accuracy: 0.6888
150/232 [==================>...........] - ETA: 1:16 - loss: 0.1423 - Accuracy: 0.6875
151/232 [==================>...........] - ETA: 1:15 - loss: 0.1430 - Accuracy: 0.6846
152/232 [==================>...........] - ETA: 1:15 - loss: 0.1427 - Accuracy: 0.6859
153/232 [==================>...........] - ETA: 1:14 - loss: 0.1431 - Accuracy: 0.6838
154/232 [==================>...........] - ETA: 1:13 - loss: 0.1427 - Accuracy: 0.6851
155/232 [===================>..........] - ETA: 1:12 - loss: 0.1427 - Accuracy: 0.6847
156/232 [===================>..........] - ETA: 1:11 - loss: 0.1429 - Accuracy: 0.6835
157/232 [===================>..........] - ETA: 1:10 - loss: 0.1430 - Accuracy: 0.6823
158/232 [===================>..........] - ETA: 1:09 - loss: 0.1429 - Accuracy: 0.6835
159/232 [===================>..........] - ETA: 1:08 - loss: 0.1428 - Accuracy: 0.6832
160/232 [===================>..........] - ETA: 1:07 - loss: 0.1427 - Accuracy: 0.6836
161/232 [===================>..........] - ETA: 1:06 - loss: 0.1426 - Accuracy: 0.6840
162/232 [===================>..........] - ETA: 1:05 - loss: 0.1425 - Accuracy: 0.6844
163/232 [====================>.........] - ETA: 1:04 - loss: 0.1426 - Accuracy: 0.6840
164/232 [====================>.........] - ETA: 1:03 - loss: 0.1423 - Accuracy: 0.6845
165/232 [====================>.........] - ETA: 1:02 - loss: 0.1427 - Accuracy: 0.6826
166/232 [====================>.........] - ETA: 1:01 - loss: 0.1425 - Accuracy: 0.6830
167/232 [====================>.........] - ETA: 1:00 - loss: 0.1420 - Accuracy: 0.6849
168/232 [====================>.........] - ETA: 59s - loss: 0.1420 - Accuracy: 0.6845 
169/232 [====================>.........] - ETA: 59s - loss: 0.1428 - Accuracy: 0.6812
170/232 [====================>.........] - ETA: 58s - loss: 0.1433 - Accuracy: 0.6794
171/232 [=====================>........] - ETA: 57s - loss: 0.1429 - Accuracy: 0.6806
172/232 [=====================>........] - ETA: 56s - loss: 0.1428 - Accuracy: 0.6802
173/232 [=====================>........] - ETA: 55s - loss: 0.1427 - Accuracy: 0.6806
174/232 [=====================>........] - ETA: 54s - loss: 0.1433 - Accuracy: 0.6782
175/232 [=====================>........] - ETA: 53s - loss: 0.1430 - Accuracy: 0.6793
176/232 [=====================>........] - ETA: 52s - loss: 0.1429 - Accuracy: 0.6790
177/232 [=====================>........] - ETA: 51s - loss: 0.1430 - Accuracy: 0.6787
178/232 [======================>.......] - ETA: 50s - loss: 0.1429 - Accuracy: 0.6791
179/232 [======================>.......] - ETA: 49s - loss: 0.1429 - Accuracy: 0.6788
180/232 [======================>.......] - ETA: 48s - loss: 0.1429 - Accuracy: 0.6785
181/232 [======================>.......] - ETA: 47s - loss: 0.1427 - Accuracy: 0.6789
182/232 [======================>.......] - ETA: 46s - loss: 0.1426 - Accuracy: 0.6793
183/232 [======================>.......] - ETA: 45s - loss: 0.1426 - Accuracy: 0.6790
184/232 [======================>.......] - ETA: 44s - loss: 0.1428 - Accuracy: 0.6780
185/232 [======================>.......] - ETA: 44s - loss: 0.1429 - Accuracy: 0.6777
186/232 [=======================>......] - ETA: 43s - loss: 0.1432 - Accuracy: 0.6761
187/232 [=======================>......] - ETA: 42s - loss: 0.1431 - Accuracy: 0.6765
188/232 [=======================>......] - ETA: 41s - loss: 0.1434 - Accuracy: 0.6749
189/232 [=======================>......] - ETA: 40s - loss: 0.1431 - Accuracy: 0.6759
190/232 [=======================>......] - ETA: 39s - loss: 0.1431 - Accuracy: 0.6757
191/232 [=======================>......] - ETA: 38s - loss: 0.1429 - Accuracy: 0.6767
192/232 [=======================>......] - ETA: 37s - loss: 0.1426 - Accuracy: 0.6777
193/232 [=======================>......] - ETA: 36s - loss: 0.1423 - Accuracy: 0.6794
194/232 [========================>.....] - ETA: 35s - loss: 0.1419 - Accuracy: 0.6811
195/232 [========================>.....] - ETA: 34s - loss: 0.1419 - Accuracy: 0.6808
196/232 [========================>.....] - ETA: 33s - loss: 0.1418 - Accuracy: 0.6811
197/232 [========================>.....] - ETA: 32s - loss: 0.1414 - Accuracy: 0.6821
198/232 [========================>.....] - ETA: 31s - loss: 0.1413 - Accuracy: 0.6824
199/232 [========================>.....] - ETA: 30s - loss: 0.1412 - Accuracy: 0.6828
200/232 [========================>.....] - ETA: 30s - loss: 0.1413 - Accuracy: 0.6825
201/232 [========================>.....] - ETA: 29s - loss: 0.1416 - Accuracy: 0.6816
202/232 [=========================>....] - ETA: 28s - loss: 0.1415 - Accuracy: 0.6819
203/232 [=========================>....] - ETA: 27s - loss: 0.1414 - Accuracy: 0.6823
204/232 [=========================>....] - ETA: 26s - loss: 0.1411 - Accuracy: 0.6832
205/232 [=========================>....] - ETA: 25s - loss: 0.1409 - Accuracy: 0.6835
206/232 [=========================>....] - ETA: 24s - loss: 0.1414 - Accuracy: 0.6820
207/232 [=========================>....] - ETA: 23s - loss: 0.1415 - Accuracy: 0.6818
208/232 [=========================>....] - ETA: 22s - loss: 0.1414 - Accuracy: 0.6821
209/232 [==========================>...] - ETA: 21s - loss: 0.1409 - Accuracy: 0.6836
210/232 [==========================>...] - ETA: 20s - loss: 0.1407 - Accuracy: 0.6839
211/232 [==========================>...] - ETA: 19s - loss: 0.1405 - Accuracy: 0.6848
212/232 [==========================>...] - ETA: 18s - loss: 0.1400 - Accuracy: 0.6863
213/232 [==========================>...] - ETA: 17s - loss: 0.1400 - Accuracy: 0.6860
214/232 [==========================>...] - ETA: 16s - loss: 0.1401 - Accuracy: 0.6857
215/232 [==========================>...] - ETA: 15s - loss: 0.1403 - Accuracy: 0.6849
216/232 [==========================>...] - ETA: 14s - loss: 0.1400 - Accuracy: 0.6858
217/232 [===========================>..] - ETA: 14s - loss: 0.1404 - Accuracy: 0.6843
218/232 [===========================>..] - ETA: 13s - loss: 0.1402 - Accuracy: 0.6846
219/232 [===========================>..] - ETA: 12s - loss: 0.1403 - Accuracy: 0.6844
220/232 [===========================>..] - ETA: 11s - loss: 0.1403 - Accuracy: 0.6841
221/232 [===========================>..] - ETA: 10s - loss: 0.1403 - Accuracy: 0.6844
222/232 [===========================>..] - ETA: 9s - loss: 0.1407 - Accuracy: 0.6824 
223/232 [===========================>..] - ETA: 8s - loss: 0.1406 - Accuracy: 0.6822
224/232 [===========================>..] - ETA: 7s - loss: 0.1404 - Accuracy: 0.6830
225/232 [============================>.] - ETA: 6s - loss: 0.1404 - Accuracy: 0.6828
226/232 [============================>.] - ETA: 5s - loss: 0.1404 - Accuracy: 0.6825
227/232 [============================>.] - ETA: 4s - loss: 0.1404 - Accuracy: 0.6823
228/232 [============================>.] - ETA: 3s - loss: 0.1405 - Accuracy: 0.6815
229/232 [============================>.] - ETA: 2s - loss: 0.1406 - Accuracy: 0.6807
230/232 [============================>.] - ETA: 1s - loss: 0.1408 - Accuracy: 0.6799
231/232 [============================>.] - ETA: 0s - loss: 0.1409 - Accuracy: 0.6802
232/232 [==============================] - ETA: 0s - loss: 0.1408 - Accuracy: 0.68002021-12-08 03:38:06.835804: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:766] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
  key: "Targuments"
  value {
    list {
    }
  }
}
attr {
  key: "_cardinality"
  value {
    i: -2
  }
}
attr {
  key: "f"
  value {
    func {
      name: "__inference_Dataset_flat_map_slice_batch_indices_819036"
    }
  }
}
attr {
  key: "metadata"
  value {
    s: "\n\021FlatMapDataset:53"
  }
}
attr {
  key: "output_shapes"
  value {
    list {
      shape {
        dim {
          size: -1
        }
      }
    }
  }
}
attr {
  key: "output_types"
  value {
    list {
      type: DT_INT64
    }
  }
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2021-12-08 03:38:06.891410: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.893979: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.907358: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.910977: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.924507: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.927559: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.939629: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.941576: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.108656: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.111741: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.125587: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.128333: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.129748: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.143568: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.146326: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.159566: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.162239: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.163579: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.177174: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.179804: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.193266: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.195900: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.197273: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.210907: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.213517: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.226549: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.229243: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.230606: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.244240: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.246853: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.259984: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.262666: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.264019: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.277687: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.280270: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.293432: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.296074: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.297418: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.310926: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.313478: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.326484: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.329043: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.330331: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.344024: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.346536: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.359540: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.362145: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.363468: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:39:50.853125: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 19176/53881 nodes to float16 precision using 992 cast(s) to float16 (excluding Const and Variable casts)
2021-12-08 03:40:12.677238: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 0/50369 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts)

232/232 [==============================] - 924s 2s/step - loss: 0.1408 - Accuracy: 0.6800 - val_loss: 0.9255 - val_Accuracy: 0.6700
Epoch 2/5
9 root error(s) found.
  (0) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_7/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Cast_1/_775]]
  (1) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_5/tf_longformer_for_sequence_classification/longformer/embeddings/Cumsum/_524]]
  (2) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_4/tf_longformer_for_sequence_classification/longformer/Greater/_658]]
  (3) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_3/tf_longformer_for_sequence_classification/longformer/Abs/_797]]
  (4) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_2/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Tile_2/_2065]]
  (5) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_1/tf_longformer_for_sequence_classification/longformer/Any/_741]]
  (6) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[div_no_nan_1/ReadVariableOp_2/_12654]]
  (7) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[cond/output/_50/_393]]
  (8) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_817844]

Errors may have originated from an input operation.
Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Function call stack:
train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function



[2021-12-08T03:40:38.420250] The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 900.0 seconds
2 items cleaning up...
Cleanup took 0.1139984130859375 seconds
Traceback (most recent call last):
  File "training.py", line 20, in <module>
    classifier.fit()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/training.py", line 27, in fit
    self._fit()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/_mixins.py", line 50, in _fit
    self._fit_model()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/_mixins.py", line 418, in _fit_model
    class_weight=self.class_weight_
  File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/opt/miniconda/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 9 root error(s) found.
  (0) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_7/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Cast_1/_775]]
  (1) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_5/tf_longformer_for_sequence_classification/longformer/embeddings/Cumsum/_524]]
  (2) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_4/tf_longformer_for_sequence_classification/longformer/Greater/_658]]
  (3) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_3/tf_longformer_for_sequence_classification/longformer/Abs/_797]]
  (4) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_2/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Tile_2/_2065]]
  (5) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_1/tf_longformer_for_sequence_classification/longformer/Any/_741]]
  (6) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[div_no_nan_1/ReadVariableOp_2/_12654]]
  (7) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[cond/output/_50/_393]]
  (8) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_817844]

Errors may have originated from an input operation.
Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Function call stack:
train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function


[2021-12-08T03:40:38.708621] Finished context manager injector with Exception.
```<|||||>Any thoughts @Rocketknight1? I was able to get it to successfully train for 2 epochs if I limit the visible devices to 2 GPUs relevant logs below

```
[5 rows x 5 columns]
2021-12-08 21:15:29.463814: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-12-08 21:15:30.377640: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-12-08 21:15:30.377729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38414 MB memory:  -> device: 0, name: A100-SXM4-40GB, pci bus id: 0001:00:00.0, compute capability: 8.0
2021-12-08 21:15:30.380280: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-12-08 21:15:30.380309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38414 MB memory:  -> device: 1, name: A100-SXM4-40GB, pci bus id: 0002:00:00.0, compute capability: 8.0

Downloading:   0%|          | 0.00/694 [00:00<?, ?B/s]
Downloading: 100%|██████████| 694/694 [00:00<00:00, 566kB/s]

Downloading:   0%|          | 0.00/729M [00:00<?, ?B/s]
Downloading:   1%|          | 6.10M/729M [00:00<00:11, 63.9MB/s]
Downloading:   2%|▏         | 13.0M/729M [00:00<00:10, 69.1MB/s]
Downloading:   3%|▎         | 19.9M/729M [00:00<00:10, 70.3MB/s]
Downloading:   4%|▎         | 26.7M/729M [00:00<00:10, 71.0MB/s]
Downloading:   5%|▍         | 33.6M/729M [00:00<00:10, 71.4MB/s]
Downloading:   6%|▌         | 40.5M/729M [00:00<00:10, 71.6MB/s]
Downloading:   6%|▋         | 47.3M/729M [00:00<00:10, 71.3MB/s]
Downloading:   7%|▋         | 54.1M/729M [00:00<00:09, 71.4MB/s]
Downloading:   8%|▊         | 61.1M/729M [00:00<00:09, 71.8MB/s]
Downloading:   9%|▉         | 68.0M/729M [00:01<00:09, 72.0MB/s]
Downloading:  10%|█         | 75.0M/729M [00:01<00:09, 72.6MB/s]
Downloading:  11%|█         | 82.0M/729M [00:01<00:09, 72.7MB/s]
Downloading:  12%|█▏        | 88.9M/729M [00:01<00:09, 72.7MB/s]
Downloading:  13%|█▎        | 95.9M/729M [00:01<00:09, 72.7MB/s]
Downloading:  14%|█▍        | 103M/729M [00:01<00:09, 72.7MB/s] 
Downloading:  15%|█▌        | 110M/729M [00:01<00:08, 72.9MB/s]
Downloading:  16%|█▌        | 117M/729M [00:01<00:08, 72.9MB/s]
Downloading:  17%|█▋        | 124M/729M [00:01<00:08, 72.9MB/s]
Downloading:  18%|█▊        | 131M/729M [00:01<00:08, 72.9MB/s]
Downloading:  19%|█▉        | 138M/729M [00:02<00:08, 72.7MB/s]
Downloading:  20%|█▉        | 145M/729M [00:02<00:08, 72.7MB/s]
Downloading:  21%|██        | 151M/729M [00:02<00:08, 72.7MB/s]
Downloading:  22%|██▏       | 158M/729M [00:02<00:08, 72.9MB/s]
Downloading:  23%|██▎       | 165M/729M [00:02<00:08, 72.8MB/s]
Downloading:  24%|██▎       | 172M/729M [00:02<00:07, 73.0MB/s]
Downloading:  25%|██▍       | 179M/729M [00:02<00:07, 72.8MB/s]
Downloading:  26%|██▌       | 186M/729M [00:02<00:07, 72.7MB/s]
Downloading:  27%|██▋       | 193M/729M [00:02<00:07, 72.7MB/s]
Downloading:  27%|██▋       | 200M/729M [00:02<00:07, 72.7MB/s]
Downloading:  28%|██▊       | 207M/729M [00:03<00:07, 72.7MB/s]
Downloading:  29%|██▉       | 214M/729M [00:03<00:07, 72.7MB/s]
Downloading:  30%|███       | 221M/729M [00:03<00:07, 72.7MB/s]
Downloading:  31%|███▏      | 228M/729M [00:03<00:07, 72.7MB/s]
Downloading:  32%|███▏      | 235M/729M [00:03<00:07, 73.0MB/s]
Downloading:  33%|███▎      | 242M/729M [00:03<00:07, 72.8MB/s]
Downloading:  34%|███▍      | 249M/729M [00:03<00:06, 72.9MB/s]
Downloading:  35%|███▌      | 256M/729M [00:03<00:06, 72.9MB/s]
Downloading:  36%|███▌      | 263M/729M [00:03<00:06, 72.7MB/s]
Downloading:  37%|███▋      | 270M/729M [00:03<00:06, 72.6MB/s]
Downloading:  38%|███▊      | 277M/729M [00:04<00:06, 72.4MB/s]
Downloading:  39%|███▉      | 284M/729M [00:04<00:06, 72.5MB/s]
Downloading:  40%|███▉      | 291M/729M [00:04<00:06, 72.7MB/s]
Downloading:  41%|████      | 298M/729M [00:04<00:06, 72.6MB/s]
Downloading:  42%|████▏     | 304M/729M [00:04<00:06, 72.5MB/s]
Downloading:  43%|████▎     | 311M/729M [00:04<00:06, 72.5MB/s]
Downloading:  44%|████▎     | 318M/729M [00:04<00:05, 72.5MB/s]
Downloading:  45%|████▍     | 325M/729M [00:04<00:05, 72.7MB/s]
Downloading:  46%|████▌     | 332M/729M [00:04<00:05, 72.6MB/s]
Downloading:  47%|████▋     | 339M/729M [00:04<00:05, 72.7MB/s]
Downloading:  47%|████▋     | 346M/729M [00:05<00:05, 72.5MB/s]
Downloading:  48%|████▊     | 353M/729M [00:05<00:05, 72.5MB/s]
Downloading:  49%|████▉     | 360M/729M [00:05<00:05, 72.5MB/s]
Downloading:  50%|█████     | 367M/729M [00:05<00:05, 72.6MB/s]
Downloading:  51%|█████▏    | 374M/729M [00:05<00:05, 72.6MB/s]
Downloading:  52%|█████▏    | 381M/729M [00:05<00:05, 72.3MB/s]
Downloading:  53%|█████▎    | 388M/729M [00:05<00:04, 72.0MB/s]
Downloading:  54%|█████▍    | 395M/729M [00:05<00:04, 72.2MB/s]
Downloading:  55%|█████▌    | 402M/729M [00:05<00:04, 72.4MB/s]
Downloading:  56%|█████▌    | 408M/729M [00:05<00:04, 72.5MB/s]
Downloading:  57%|█████▋    | 415M/729M [00:06<00:04, 72.5MB/s]
Downloading:  58%|█████▊    | 422M/729M [00:06<00:04, 72.6MB/s]
Downloading:  59%|█████▉    | 429M/729M [00:06<00:04, 72.8MB/s]
Downloading:  60%|█████▉    | 436M/729M [00:06<00:04, 72.8MB/s]
Downloading:  61%|██████    | 443M/729M [00:06<00:04, 73.0MB/s]
Downloading:  62%|██████▏   | 450M/729M [00:06<00:04, 72.9MB/s]
Downloading:  63%|██████▎   | 457M/729M [00:06<00:03, 72.9MB/s]
Downloading:  64%|██████▎   | 464M/729M [00:06<00:03, 72.7MB/s]
Downloading:  65%|██████▍   | 471M/729M [00:06<00:03, 72.5MB/s]
Downloading:  66%|██████▌   | 478M/729M [00:06<00:03, 72.7MB/s]
Downloading:  67%|██████▋   | 485M/729M [00:07<00:03, 72.7MB/s]
Downloading:  67%|██████▋   | 492M/729M [00:07<00:03, 72.7MB/s]
Downloading:  68%|██████▊   | 499M/729M [00:07<00:03, 72.6MB/s]
Downloading:  69%|██████▉   | 506M/729M [00:07<00:03, 72.6MB/s]
Downloading:  70%|███████   | 513M/729M [00:07<00:03, 72.9MB/s]
Downloading:  71%|███████▏  | 520M/729M [00:07<00:03, 72.8MB/s]
Downloading:  72%|███████▏  | 527M/729M [00:07<00:02, 72.9MB/s]
Downloading:  73%|███████▎  | 534M/729M [00:07<00:02, 72.8MB/s]
Downloading:  74%|███████▍  | 541M/729M [00:07<00:02, 72.5MB/s]
Downloading:  75%|███████▌  | 548M/729M [00:07<00:02, 72.5MB/s]
Downloading:  76%|███████▌  | 554M/729M [00:08<00:02, 72.1MB/s]
Downloading:  77%|███████▋  | 561M/729M [00:08<00:02, 72.4MB/s]
Downloading:  78%|███████▊  | 568M/729M [00:08<00:02, 72.5MB/s]
Downloading:  79%|███████▉  | 575M/729M [00:08<00:02, 72.6MB/s]
Downloading:  80%|███████▉  | 582M/729M [00:08<00:02, 72.7MB/s]
Downloading:  81%|████████  | 589M/729M [00:08<00:02, 72.7MB/s]
Downloading:  82%|████████▏ | 596M/729M [00:08<00:01, 72.6MB/s]
Downloading:  83%|████████▎ | 603M/729M [00:08<00:01, 72.6MB/s]
Downloading:  84%|████████▎ | 610M/729M [00:08<00:01, 72.6MB/s]
Downloading:  85%|████████▍ | 617M/729M [00:08<00:01, 72.9MB/s]
Downloading:  86%|████████▌ | 624M/729M [00:09<00:01, 72.7MB/s]
Downloading:  87%|████████▋ | 631M/729M [00:09<00:01, 72.6MB/s]
Downloading:  87%|████████▋ | 638M/729M [00:09<00:01, 72.7MB/s]
Downloading:  88%|████████▊ | 645M/729M [00:09<00:01, 72.6MB/s]
Downloading:  89%|████████▉ | 652M/729M [00:09<00:01, 72.8MB/s]
Downloading:  90%|█████████ | 659M/729M [00:09<00:01, 72.6MB/s]
Downloading:  91%|█████████▏| 666M/729M [00:09<00:00, 72.8MB/s]
Downloading:  92%|█████████▏| 673M/729M [00:09<00:00, 72.7MB/s]
Downloading:  93%|█████████▎| 680M/729M [00:09<00:00, 72.7MB/s]
Downloading:  94%|█████████▍| 687M/729M [00:09<00:00, 72.7MB/s]
Downloading:  95%|█████████▌| 693M/729M [00:10<00:00, 72.6MB/s]
Downloading:  96%|█████████▌| 700M/729M [00:10<00:00, 72.7MB/s]
Downloading:  97%|█████████▋| 707M/729M [00:10<00:00, 72.6MB/s]
Downloading:  98%|█████████▊| 714M/729M [00:10<00:00, 72.6MB/s]
Downloading:  99%|█████████▉| 721M/729M [00:10<00:00, 72.5MB/s]
Downloading: 100%|█████████▉| 728M/729M [00:10<00:00, 72.6MB/s]
Downloading: 100%|██████████| 729M/729M [00:10<00:00, 72.5MB/s]
2021-12-08 21:15:42.678981: I tensorflow/stream_executor/cuda/cuda_blas.cc:1774] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
Some layers from the model checkpoint at allenai/longformer-base-4096 were not used when initializing TFLongformerForSequenceClassification: ['lm_head']
- This IS expected if you are initializing TFLongformerForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFLongformerForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some layers of TFLongformerForSequenceClassification were not initialized from the model checkpoint at allenai/longformer-base-4096 and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some layers from the model checkpoint at allenai/longformer-base-4096 were not used when initializing TFLongformerForSequenceClassification: ['lm_head']
- This IS expected if you are initializing TFLongformerForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFLongformerForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some layers of TFLongformerForSequenceClassification were not initialized from the model checkpoint at allenai/longformer-base-4096 and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2021-12-08 21:15:47.914719: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:766] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
  key: "Targuments"
  value {
    list {
    }
  }
}
attr {
  key: "_cardinality"
  value {
    i: -2
  }
}
attr {
  key: "f"
  value {
    func {
      name: "__inference_Dataset_flat_map_slice_batch_indices_59118"
    }
  }
}
attr {
  key: "metadata"
  value {
    s: "\n\020FlatMapDataset:4"
  }
}
attr {
  key: "output_shapes"
  value {
    list {
      shape {
        dim {
          size: -1
        }
      }
    }
  }
}
attr {
  key: "output_types"
  value {
    list {
      type: DT_INT64
    }
  }
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2021-12-08 21:15:47.967533: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.969043: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.978416: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.980483: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.985689: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.987135: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.991900: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.993490: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.997905: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.999031: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.038998: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.040759: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.048322: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.049956: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.050807: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.056971: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.058550: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.064085: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.065707: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.066561: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
Epoch 1/2
WARNING:tensorflow:Efficient allreduce is not supported for 3 IndexedSlices
WARNING:tensorflow:Efficient allreduce is not supported for 3 IndexedSlices
2021-12-08 21:17:31.182628: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 10772/50331 nodes to float16 precision using 1486 cast(s) to float16 (excluding Const and Variable casts)
2021-12-08 21:17:49.083818: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 0/42321 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO Bootstrap : Using eth0:10.0.0.5<0>
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NET/IB : Using [0]mlx5_ib4:1/IB [1]mlx5_ib2:1/IB [2]mlx5_ib0:1/IB [3]mlx5_ib7:1/IB [4]mlx5_ib5:1/IB [5]mlx5_ib3:1/IB [6]mlx5_ib1:1/IB [7]mlx5_ib6:1/IB ; OOB eth0:10.0.0.5<0>
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO Using network IB
NCCL version 2.8.3+cudaCUDA_MAJOR.CUDA_MINOR
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 16.
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/microsoft/ndv4-topo.xml
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/microsoft/ndv4-topo.xml
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Loading unnamed topology
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Loading unnamed topology
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO === System : maxWidth 252.0 totalWidth 252.0 ===
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO CPU/0 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + PCI[24.0] - PCI/FFFFFF010
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - GPU/100000 (0)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO                             + NVL[252.0] - NVS/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10100000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - GPU/200000 (1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO                             + NVL[252.0] - NVS/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10200000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO CPU/1 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + PCI[24.0] - PCI/FFFFFF020
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10300000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10400000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO CPU/2 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + PCI[24.0] - PCI/FFFFFF030
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10500000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10600000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO CPU/3 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + PCI[24.0] - PCI/FFFFFF040
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10700000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10800000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO ==========================================
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO GPU/100000 :GPU/100000 (0/5000.000000/LOC) GPU/200000 (2/252.000000/NVL) CPU/0 (2/24.000000/PHB) CPU/1 (3/24.000000/SYS) CPU/2 (3/24.000000/SYS) CPU/3 (3/24.000000/SYS) 
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO GPU/200000 :GPU/100000 (2/252.000000/NVL) GPU/200000 (0/5000.000000/LOC) CPU/0 (2/24.000000/PHB) CPU/1 (3/24.000000/SYS) CPU/2 (3/24.000000/SYS) CPU/3 (3/24.000000/SYS) 
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, speed 21.000000/21.000000, type NVL/PIX, sameChannels 1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  6 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  7 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  8 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  9 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 10 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 11 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Pattern 3, crossNic 0, nChannels 12, speed 21.000000/21.000000, type NVL/PIX, sameChannels 1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  6 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  7 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  8 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  9 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 10 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 11 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Pattern 3, crossNic 0, nChannels 12, speed 42.000000/42.000000, type NVL/PIX, sameChannels 0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  6 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  7 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  8 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  9 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 10 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 11 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO === System : maxWidth 252.0 totalWidth 252.0 ===
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO CPU/0 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + PCI[24.0] - PCI/FFFFFF010
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - GPU/100000 (0)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO                             + NVL[252.0] - NVS/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10100000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - GPU/200000 (1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO                             + NVL[252.0] - NVS/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10200000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO CPU/1 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + PCI[24.0] - PCI/FFFFFF020
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10300000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10400000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO CPU/2 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + PCI[24.0] - PCI/FFFFFF030
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10500000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10600000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO CPU/3 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + PCI[24.0] - PCI/FFFFFF040
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10700000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10800000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO ==========================================
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO GPU/100000 :GPU/100000 (0/5000.000000/LOC) GPU/200000 (2/252.000000/NVL) CPU/0 (2/24.000000/PHB) CPU/1 (3/24.000000/SYS) CPU/2 (3/24.000000/SYS) CPU/3 (3/24.000000/SYS) 
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO GPU/200000 :GPU/100000 (2/252.000000/NVL) GPU/200000 (0/5000.000000/LOC) CPU/0 (2/24.000000/PHB) CPU/1 (3/24.000000/SYS) CPU/2 (3/24.000000/SYS) CPU/3 (3/24.000000/SYS) 
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, speed 21.000000/21.000000, type NVL/PIX, sameChannels 1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  6 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  7 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  8 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  9 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 10 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 11 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Pattern 3, crossNic 0, nChannels 12, speed 21.000000/21.000000, type NVL/PIX, sameChannels 1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  6 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  7 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  8 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  9 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 10 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 11 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Pattern 3, crossNic 0, nChannels 12, speed 42.000000/42.000000, type NVL/PIX, sameChannels 0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  6 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  7 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  8 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  9 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 10 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 11 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 6 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 18 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 7 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 19 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 8 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 20 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 9 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] -1/-1/-1->1->0 [7] -1/-1/-1->1->0 [8] -1/-1/-1->1->0 [9] -1/-1/-1->1->0 [10] -1/-1/-1->1->0 [11] -1/-1/-1->1->0 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] -1/-1/-1->1->0 [19] -1/-1/-1->1->0 [20] -1/-1/-1->1->0 [21] -1/-1/-1->1->0 [22] -1/-1/-1->1->0 [23] -1/-1/-1->1->0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 21 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 10 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 22 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 11 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 23 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 00/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 01/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 02/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 03/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 04/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 05/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 06/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 07/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 08/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 09/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 10/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 11/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 12/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 13/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 14/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 15/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 16/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 17/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 18/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 19/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 20/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 21/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 22/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 23/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 00 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 00 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 01 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 01 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 02 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 02 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 03 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 03 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 04 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 04 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 05 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 05 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 06 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 06 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 07 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 07 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 08 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 08 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 09 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 09 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 10 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 10 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 11 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 11 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 12 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 12 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 13 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 13 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 14 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 14 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 15 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 15 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 16 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 16 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 17 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 17 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 18 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 18 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 19 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 19 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 20 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 20 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 21 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 21 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 22 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 22 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 23 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 23 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Connected all rings
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Connected all trees
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Connected all rings
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Connected all trees
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO comm 0x7fca28002e00 rank 0 nranks 2 cudaDev 0 busId 100000 - Init COMPLETE
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO comm 0x7fc494002dd0 rank 1 nranks 2 cudaDev 1 busId 200000 - Init COMPLETE
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1326 [0] NCCL INFO Launch mode Group/CGMD

  1/232 [..............................] - ETA: 8:43:48 - loss: 0.5153 - Accuracy: 0.8750
  2/232 [..............................] - ETA: 6:34 - loss: 0.4137 - Accuracy: 0.8125   
  3/232 [..............................] - ETA: 6:37 - loss: 0.3161 - Accuracy: 0.6250
  4/232 [..............................] - ETA: 6:35 - loss: 0.2721 - Accuracy: 0.6250
  5/232 [..............................] - ETA: 6:38 - loss: 0.2351 - Accuracy: 0.6750
  6/232 [..............................] - ETA: 6:31 - loss: 0.2066 - Accuracy: 0.7083
  7/232 [..............................] - ETA: 6:28 - loss: 0.1930 - Accuracy: 0.6786
  8/232 [>.............................] - ETA: 6:26 - loss: 0.1812 - Accuracy: 0.6719
  9/232 [>.............................] - ETA: 6:26 - loss: 0.1753 - Accuracy: 0.6389
 10/232 [>.............................] - ETA: 6:23 - loss: 0.1670 - Accuracy: 0.6375
 11/232 [>.............................] - ETA: 6:20 - loss: 0.1559 - Accuracy: 0.6705
 12/232 [>.............................] - ETA: 6:18 - loss: 0.1467 - Accuracy: 0.6875
 13/232 [>.............................] - ETA: 6:16 - loss: 0.1405 - Accuracy: 0.6923
 14/232 [>.............................] - ETA: 6:16 - loss: 0.1339 - Accuracy: 0.7054
 15/232 [>.............................] - ETA: 6:14 - loss: 0.1317 - Accuracy: 0.7000
 16/232 [=>............................] - ETA: 6:11 - loss: 0.1250 - Accuracy: 0.7188
 17/232 [=>............................] - ETA: 6:10 - loss: 0.1233 - Accuracy: 0.7132
 18/232 [=>............................] - ETA: 6:07 - loss: 0.1199 - Accuracy: 0.7153
 19/232 [=>............................] - ETA: 6:05 - loss: 0.1210 - Accuracy: 0.6974
 20/232 [=>............................] - ETA: 6:04 - loss: 0.1190 - Accuracy: 0.7063
 21/232 [=>............................] - ETA: 6:01 - loss: 0.1176 - Accuracy: 0.7083
 22/232 [=>............................] - ETA: 5:59 - loss: 0.1176 - Accuracy: 0.6989
 23/232 [=>............................] - ETA: 5:57 - loss: 0.1179 - Accuracy: 0.6902
 24/232 [==>...........................] - ETA: 5:56 - loss: 0.1141 - Accuracy: 0.7031
 25/232 [==>...........................] - ETA: 5:54 - loss: 0.1122 - Accuracy: 0.7050
 26/232 [==>...........................] - ETA: 5:52 - loss: 0.1104 - Accuracy: 0.7067
 27/232 [==>...........................] - ETA: 5:49 - loss: 0.1092 - Accuracy: 0.7130
 28/232 [==>...........................] - ETA: 5:48 - loss: 0.1089 - Accuracy: 0.7098
 29/232 [==>...........................] - ETA: 5:46 - loss: 0.1085 - Accuracy: 0.7069
 30/232 [==>...........................] - ETA: 5:44 - loss: 0.1098 - Accuracy: 0.6958
 31/232 [===>..........................] - ETA: 5:42 - loss: 0.1080 - Accuracy: 0.7016
 32/232 [===>..........................] - ETA: 5:40 - loss: 0.1069 - Accuracy: 0.7031
 33/232 [===>..........................] - ETA: 5:38 - loss: 0.1065 - Accuracy: 0.7008
 34/232 [===>..........................] - ETA: 5:36 - loss: 0.1048 - Accuracy: 0.7059
 35/232 [===>..........................] - ETA: 5:35 - loss: 0.1026 - Accuracy: 0.7143
 36/232 [===>..........................] - ETA: 5:33 - loss: 0.1017 - Accuracy: 0.7153
 37/232 [===>..........................] - ETA: 5:32 - loss: 0.1016 - Accuracy: 0.7128
 38/232 [===>..........................] - ETA: 5:30 - loss: 0.1007 - Accuracy: 0.7138
 39/232 [====>.........................] - ETA: 5:28 - loss: 0.0998 - Accuracy: 0.7147
 40/232 [====>.........................] - ETA: 5:26 - loss: 0.0997 - Accuracy: 0.7125
 41/232 [====>.........................] - ETA: 5:25 - loss: 0.0981 - Accuracy: 0.7195
 42/232 [====>.........................] - ETA: 5:23 - loss: 0.0988 - Accuracy: 0.7113
 43/232 [====>.........................] - ETA: 5:22 - loss: 0.0982 - Accuracy: 0.7122
 44/232 [====>.........................] - ETA: 5:20 - loss: 0.0980 - Accuracy: 0.7102
 45/232 [====>.........................] - ETA: 5:18 - loss: 0.0979 - Accuracy: 0.7083
 46/232 [====>.........................] - ETA: 5:16 - loss: 0.0966 - Accuracy: 0.7147
 47/232 [=====>........................] - ETA: 5:15 - loss: 0.0955 - Accuracy: 0.7181
 48/232 [=====>........................] - ETA: 5:13 - loss: 0.0958 - Accuracy: 0.7135
 49/232 [=====>........................] - ETA: 5:11 - loss: 0.0959 - Accuracy: 0.7117
 50/232 [=====>........................] - ETA: 5:10 - loss: 0.0959 - Accuracy: 0.7100
 51/232 [=====>........................] - ETA: 5:09 - loss: 0.0959 - Accuracy: 0.7083
 52/232 [=====>........................] - ETA: 5:07 - loss: 0.0967 - Accuracy: 0.7019
 53/232 [=====>........................] - ETA: 5:05 - loss: 0.0956 - Accuracy: 0.7075
 54/232 [=====>........................] - ETA: 5:03 - loss: 0.0954 - Accuracy: 0.7083
 55/232 [======>.......................] - ETA: 5:02 - loss: 0.0949 - Accuracy: 0.7091
 56/232 [======>.......................] - ETA: 5:00 - loss: 0.0945 - Accuracy: 0.7098
 57/232 [======>.......................] - ETA: 4:59 - loss: 0.0948 - Accuracy: 0.7061
 58/232 [======>.......................] - ETA: 4:57 - loss: 0.0944 - Accuracy: 0.7069
 59/232 [======>.......................] - ETA: 4:55 - loss: 0.0936 - Accuracy: 0.7097
 60/232 [======>.......................] - ETA: 4:54 - loss: 0.0932 - Accuracy: 0.7104
 61/232 [======>.......................] - ETA: 4:52 - loss: 0.0929 - Accuracy: 0.7111
 62/232 [=======>......................] - ETA: 4:50 - loss: 0.0921 - Accuracy: 0.7137
 63/232 [=======>......................] - ETA: 4:48 - loss: 0.0922 - Accuracy: 0.7123
 64/232 [=======>......................] - ETA: 4:47 - loss: 0.0918 - Accuracy: 0.7129
 65/232 [=======>......................] - ETA: 4:48 - loss: 0.0932 - Accuracy: 0.7038
 66/232 [=======>......................] - ETA: 4:46 - loss: 0.0931 - Accuracy: 0.7027
 67/232 [=======>......................] - ETA: 4:45 - loss: 0.0934 - Accuracy: 0.6996
 68/232 [=======>......................] - ETA: 4:43 - loss: 0.0935 - Accuracy: 0.6967
 69/232 [=======>......................] - ETA: 4:41 - loss: 0.0937 - Accuracy: 0.6957
 70/232 [========>.....................] - ETA: 4:39 - loss: 0.0928 - Accuracy: 0.7000
 71/232 [========>.....................] - ETA: 4:37 - loss: 0.0932 - Accuracy: 0.6972
 72/232 [========>.....................] - ETA: 4:35 - loss: 0.0925 - Accuracy: 0.6997
 73/232 [========>.....................] - ETA: 4:33 - loss: 0.0929 - Accuracy: 0.6969
 74/232 [========>.....................] - ETA: 4:32 - loss: 0.0936 - Accuracy: 0.6926
 75/232 [========>.....................] - ETA: 4:30 - loss: 0.0945 - Accuracy: 0.6867
 76/232 [========>.....................] - ETA: 4:28 - loss: 0.0939 - Accuracy: 0.6891
 77/232 [========>.....................] - ETA: 4:26 - loss: 0.0942 - Accuracy: 0.6867
 78/232 [=========>....................] - ETA: 4:24 - loss: 0.0940 - Accuracy: 0.6875
 79/232 [=========>....................] - ETA: 4:23 - loss: 0.0944 - Accuracy: 0.6835
 80/232 [=========>....................] - ETA: 4:21 - loss: 0.0945 - Accuracy: 0.6812
 81/232 [=========>....................] - ETA: 4:19 - loss: 0.0945 - Accuracy: 0.6806
 82/232 [=========>....................] - ETA: 4:18 - loss: 0.0944 - Accuracy: 0.6799
 83/232 [=========>....................] - ETA: 4:16 - loss: 0.0940 - Accuracy: 0.6822
 84/232 [=========>....................] - ETA: 4:14 - loss: 0.0944 - Accuracy: 0.6786
 85/232 [=========>....................] - ETA: 4:12 - loss: 0.0948 - Accuracy: 0.6750
 86/232 [==========>...................] - ETA: 4:10 - loss: 0.0948 - Accuracy: 0.6744
 87/232 [==========>...................] - ETA: 4:09 - loss: 0.0946 - Accuracy: 0.6753
 88/232 [==========>...................] - ETA: 4:07 - loss: 0.0942 - Accuracy: 0.6776
 89/232 [==========>...................] - ETA: 4:05 - loss: 0.0939 - Accuracy: 0.6784
 90/232 [==========>...................] - ETA: 4:04 - loss: 0.0935 - Accuracy: 0.6806
 91/232 [==========>...................] - ETA: 4:02 - loss: 0.0930 - Accuracy: 0.6827
 92/232 [==========>...................] - ETA: 4:00 - loss: 0.0935 - Accuracy: 0.6793
 93/232 [===========>..................] - ETA: 3:59 - loss: 0.0930 - Accuracy: 0.6815
 94/232 [===========>..................] - ETA: 3:57 - loss: 0.0932 - Accuracy: 0.6795
 95/232 [===========>..................] - ETA: 3:55 - loss: 0.0929 - Accuracy: 0.6803
 96/232 [===========>..................] - ETA: 3:53 - loss: 0.0927 - Accuracy: 0.6810
 97/232 [===========>..................] - ETA: 3:52 - loss: 0.0930 - Accuracy: 0.6791
 98/232 [===========>..................] - ETA: 3:50 - loss: 0.0925 - Accuracy: 0.6811
 99/232 [===========>..................] - ETA: 3:48 - loss: 0.0920 - Accuracy: 0.6831
100/232 [===========>..................] - ETA: 3:46 - loss: 0.0920 - Accuracy: 0.6825
101/232 [============>.................] - ETA: 3:44 - loss: 0.0922 - Accuracy: 0.6807
102/232 [============>.................] - ETA: 3:43 - loss: 0.0929 - Accuracy: 0.6765
103/232 [============>.................] - ETA: 3:41 - loss: 0.0927 - Accuracy: 0.6772
104/232 [============>.................] - ETA: 3:39 - loss: 0.0927 - Accuracy: 0.6767
105/232 [============>.................] - ETA: 3:37 - loss: 0.0928 - Accuracy: 0.6750
106/232 [============>.................] - ETA: 3:36 - loss: 0.0927 - Accuracy: 0.6757
107/232 [============>.................] - ETA: 3:34 - loss: 0.0931 - Accuracy: 0.6717
108/232 [============>.................] - ETA: 3:32 - loss: 0.0929 - Accuracy: 0.6736
109/232 [=============>................] - ETA: 3:31 - loss: 0.0929 - Accuracy: 0.6732
110/232 [=============>................] - ETA: 3:29 - loss: 0.0930 - Accuracy: 0.6716
111/232 [=============>................] - ETA: 3:27 - loss: 0.0928 - Accuracy: 0.6723
112/232 [=============>................] - ETA: 3:25 - loss: 0.0928 - Accuracy: 0.6719
113/232 [=============>................] - ETA: 3:23 - loss: 0.0927 - Accuracy: 0.6715
114/232 [=============>................] - ETA: 3:22 - loss: 0.0924 - Accuracy: 0.6732
115/232 [=============>................] - ETA: 3:20 - loss: 0.0924 - Accuracy: 0.6728
116/232 [==============>...............] - ETA: 3:18 - loss: 0.0920 - Accuracy: 0.6746
117/232 [==============>...............] - ETA: 3:17 - loss: 0.0918 - Accuracy: 0.6752
118/232 [==============>...............] - ETA: 3:15 - loss: 0.0914 - Accuracy: 0.6769
119/232 [==============>...............] - ETA: 3:13 - loss: 0.0908 - Accuracy: 0.6796
120/232 [==============>...............] - ETA: 3:12 - loss: 0.0906 - Accuracy: 0.6802
121/232 [==============>...............] - ETA: 3:10 - loss: 0.0909 - Accuracy: 0.6787
122/232 [==============>...............] - ETA: 3:08 - loss: 0.0907 - Accuracy: 0.6793
123/232 [==============>...............] - ETA: 3:07 - loss: 0.0903 - Accuracy: 0.6809
124/232 [===============>..............] - ETA: 3:05 - loss: 0.0901 - Accuracy: 0.6815
125/232 [===============>..............] - ETA: 3:03 - loss: 0.0899 - Accuracy: 0.6820
126/232 [===============>..............] - ETA: 3:01 - loss: 0.0896 - Accuracy: 0.6835
127/232 [===============>..............] - ETA: 3:00 - loss: 0.0896 - Accuracy: 0.6831
128/232 [===============>..............] - ETA: 2:58 - loss: 0.0895 - Accuracy: 0.6836
129/232 [===============>..............] - ETA: 2:56 - loss: 0.0897 - Accuracy: 0.6822
130/232 [===============>..............] - ETA: 2:54 - loss: 0.0896 - Accuracy: 0.6827
131/232 [===============>..............] - ETA: 2:53 - loss: 0.0894 - Accuracy: 0.6832
132/232 [================>.............] - ETA: 2:51 - loss: 0.0895 - Accuracy: 0.6828
133/232 [================>.............] - ETA: 2:49 - loss: 0.0893 - Accuracy: 0.6833
134/232 [================>.............] - ETA: 2:47 - loss: 0.0893 - Accuracy: 0.6828
135/232 [================>.............] - ETA: 2:46 - loss: 0.0892 - Accuracy: 0.6833
136/232 [================>.............] - ETA: 2:44 - loss: 0.0889 - Accuracy: 0.6847
137/232 [================>.............] - ETA: 2:42 - loss: 0.0886 - Accuracy: 0.6861
138/232 [================>.............] - ETA: 2:41 - loss: 0.0886 - Accuracy: 0.6857
139/232 [================>.............] - ETA: 2:39 - loss: 0.0885 - Accuracy: 0.6862
140/232 [=================>............] - ETA: 2:37 - loss: 0.0884 - Accuracy: 0.6866
141/232 [=================>............] - ETA: 2:35 - loss: 0.0881 - Accuracy: 0.6879
142/232 [=================>............] - ETA: 2:34 - loss: 0.0879 - Accuracy: 0.6884
143/232 [=================>............] - ETA: 2:32 - loss: 0.0878 - Accuracy: 0.6888
144/232 [=================>............] - ETA: 2:30 - loss: 0.0874 - Accuracy: 0.6901
145/232 [=================>............] - ETA: 2:28 - loss: 0.0875 - Accuracy: 0.6897
146/232 [=================>............] - ETA: 2:27 - loss: 0.0878 - Accuracy: 0.6884
147/232 [==================>...........] - ETA: 2:25 - loss: 0.0875 - Accuracy: 0.6896
148/232 [==================>...........] - ETA: 2:23 - loss: 0.0877 - Accuracy: 0.6883
149/232 [==================>...........] - ETA: 2:22 - loss: 0.0875 - Accuracy: 0.6888
150/232 [==================>...........] - ETA: 2:20 - loss: 0.0877 - Accuracy: 0.6875
151/232 [==================>...........] - ETA: 2:18 - loss: 0.0882 - Accuracy: 0.6846
152/232 [==================>...........] - ETA: 2:16 - loss: 0.0879 - Accuracy: 0.6859
153/232 [==================>...........] - ETA: 2:15 - loss: 0.0882 - Accuracy: 0.6838
154/232 [==================>...........] - ETA: 2:13 - loss: 0.0880 - Accuracy: 0.6851
155/232 [===================>..........] - ETA: 2:11 - loss: 0.0880 - Accuracy: 0.6847
156/232 [===================>..........] - ETA: 2:10 - loss: 0.0881 - Accuracy: 0.6835
157/232 [===================>..........] - ETA: 2:08 - loss: 0.0882 - Accuracy: 0.6823
158/232 [===================>..........] - ETA: 2:06 - loss: 0.0881 - Accuracy: 0.6835
159/232 [===================>..........] - ETA: 2:04 - loss: 0.0881 - Accuracy: 0.6832
160/232 [===================>..........] - ETA: 2:03 - loss: 0.0880 - Accuracy: 0.6836
161/232 [===================>..........] - ETA: 2:01 - loss: 0.0880 - Accuracy: 0.6840
162/232 [===================>..........] - ETA: 1:59 - loss: 0.0878 - Accuracy: 0.6844
163/232 [====================>.........] - ETA: 1:57 - loss: 0.0879 - Accuracy: 0.6840
164/232 [====================>.........] - ETA: 1:56 - loss: 0.0877 - Accuracy: 0.6845
165/232 [====================>.........] - ETA: 1:54 - loss: 0.0881 - Accuracy: 0.6826
166/232 [====================>.........] - ETA: 1:52 - loss: 0.0880 - Accuracy: 0.6830
167/232 [====================>.........] - ETA: 1:51 - loss: 0.0875 - Accuracy: 0.6849
168/232 [====================>.........] - ETA: 1:49 - loss: 0.0876 - Accuracy: 0.6845
169/232 [====================>.........] - ETA: 1:47 - loss: 0.0883 - Accuracy: 0.6812
170/232 [====================>.........] - ETA: 1:46 - loss: 0.0886 - Accuracy: 0.6794
171/232 [=====================>........] - ETA: 1:44 - loss: 0.0883 - Accuracy: 0.6806
172/232 [=====================>........] - ETA: 1:42 - loss: 0.0884 - Accuracy: 0.6802
173/232 [=====================>........] - ETA: 1:40 - loss: 0.0882 - Accuracy: 0.6806
174/232 [=====================>........] - ETA: 1:39 - loss: 0.0886 - Accuracy: 0.6782
175/232 [=====================>........] - ETA: 1:37 - loss: 0.0884 - Accuracy: 0.6793
176/232 [=====================>........] - ETA: 1:35 - loss: 0.0884 - Accuracy: 0.6790
177/232 [=====================>........] - ETA: 1:34 - loss: 0.0884 - Accuracy: 0.6787
178/232 [======================>.......] - ETA: 1:32 - loss: 0.0884 - Accuracy: 0.6791
179/232 [======================>.......] - ETA: 1:30 - loss: 0.0884 - Accuracy: 0.6788
180/232 [======================>.......] - ETA: 1:28 - loss: 0.0884 - Accuracy: 0.6785
181/232 [======================>.......] - ETA: 1:27 - loss: 0.0883 - Accuracy: 0.6789
182/232 [======================>.......] - ETA: 1:25 - loss: 0.0882 - Accuracy: 0.6793
183/232 [======================>.......] - ETA: 1:23 - loss: 0.0882 - Accuracy: 0.6790
184/232 [======================>.......] - ETA: 1:22 - loss: 0.0884 - Accuracy: 0.6780
185/232 [======================>.......] - ETA: 1:20 - loss: 0.0884 - Accuracy: 0.6777
186/232 [=======================>......] - ETA: 1:18 - loss: 0.0886 - Accuracy: 0.6761
187/232 [=======================>......] - ETA: 1:16 - loss: 0.0885 - Accuracy: 0.6765
188/232 [=======================>......] - ETA: 1:15 - loss: 0.0888 - Accuracy: 0.6749
189/232 [=======================>......] - ETA: 1:13 - loss: 0.0886 - Accuracy: 0.6759
190/232 [=======================>......] - ETA: 1:11 - loss: 0.0886 - Accuracy: 0.6757
191/232 [=======================>......] - ETA: 1:10 - loss: 0.0884 - Accuracy: 0.6767
192/232 [=======================>......] - ETA: 1:08 - loss: 0.0882 - Accuracy: 0.6777
193/232 [=======================>......] - ETA: 1:06 - loss: 0.0879 - Accuracy: 0.6794
194/232 [========================>.....] - ETA: 1:04 - loss: 0.0876 - Accuracy: 0.6811
195/232 [========================>.....] - ETA: 1:03 - loss: 0.0876 - Accuracy: 0.6808
196/232 [========================>.....] - ETA: 1:01 - loss: 0.0875 - Accuracy: 0.6811
197/232 [========================>.....] - ETA: 59s - loss: 0.0873 - Accuracy: 0.6821 
198/232 [========================>.....] - ETA: 58s - loss: 0.0872 - Accuracy: 0.6824
199/232 [========================>.....] - ETA: 56s - loss: 0.0871 - Accuracy: 0.6828
200/232 [========================>.....] - ETA: 54s - loss: 0.0872 - Accuracy: 0.6825
201/232 [========================>.....] - ETA: 52s - loss: 0.0874 - Accuracy: 0.6816
202/232 [=========================>....] - ETA: 51s - loss: 0.0873 - Accuracy: 0.6819
203/232 [=========================>....] - ETA: 49s - loss: 0.0872 - Accuracy: 0.6823
204/232 [=========================>....] - ETA: 47s - loss: 0.0870 - Accuracy: 0.6832
205/232 [=========================>....] - ETA: 46s - loss: 0.0869 - Accuracy: 0.6835
206/232 [=========================>....] - ETA: 44s - loss: 0.0872 - Accuracy: 0.6820
207/232 [=========================>....] - ETA: 42s - loss: 0.0873 - Accuracy: 0.6818
208/232 [=========================>....] - ETA: 40s - loss: 0.0872 - Accuracy: 0.6821
209/232 [==========================>...] - ETA: 39s - loss: 0.0869 - Accuracy: 0.6836
210/232 [==========================>...] - ETA: 37s - loss: 0.0868 - Accuracy: 0.6839
211/232 [==========================>...] - ETA: 35s - loss: 0.0866 - Accuracy: 0.6848
212/232 [==========================>...] - ETA: 34s - loss: 0.0863 - Accuracy: 0.6863
213/232 [==========================>...] - ETA: 32s - loss: 0.0864 - Accuracy: 0.6860
214/232 [==========================>...] - ETA: 30s - loss: 0.0864 - Accuracy: 0.6857
215/232 [==========================>...] - ETA: 29s - loss: 0.0865 - Accuracy: 0.6849
216/232 [==========================>...] - ETA: 27s - loss: 0.0863 - Accuracy: 0.6858
217/232 [===========================>..] - ETA: 25s - loss: 0.0866 - Accuracy: 0.6843
218/232 [===========================>..] - ETA: 23s - loss: 0.0865 - Accuracy: 0.6846
219/232 [===========================>..] - ETA: 22s - loss: 0.0865 - Accuracy: 0.6844
220/232 [===========================>..] - ETA: 20s - loss: 0.0865 - Accuracy: 0.6841
221/232 [===========================>..] - ETA: 18s - loss: 0.0865 - Accuracy: 0.6844
222/232 [===========================>..] - ETA: 17s - loss: 0.0867 - Accuracy: 0.6824
223/232 [===========================>..] - ETA: 15s - loss: 0.0868 - Accuracy: 0.6822
224/232 [===========================>..] - ETA: 13s - loss: 0.0867 - Accuracy: 0.6830
225/232 [============================>.] - ETA: 11s - loss: 0.0867 - Accuracy: 0.6828
226/232 [============================>.] - ETA: 10s - loss: 0.0867 - Accuracy: 0.6825
227/232 [============================>.] - ETA: 8s - loss: 0.0867 - Accuracy: 0.6823 
228/232 [============================>.] - ETA: 6s - loss: 0.0869 - Accuracy: 0.6815
229/232 [============================>.] - ETA: 5s - loss: 0.0870 - Accuracy: 0.6807
230/232 [============================>.] - ETA: 3s - loss: 0.0871 - Accuracy: 0.6799
231/232 [============================>.] - ETA: 1s - loss: 0.0870 - Accuracy: 0.6802
232/232 [==============================] - ETA: 0s - loss: 0.0871 - Accuracy: 0.68002021-12-08 21:24:38.031434: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:766] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
  key: "Targuments"
  value {
    list {
    }
  }
}
attr {
  key: "_cardinality"
  value {
    i: -2
  }
}
attr {
  key: "f"
  value {
    func {
      name: "__inference_Dataset_flat_map_slice_batch_indices_244154"
    }
  }
}
attr {
  key: "metadata"
  value {
    s: "\n\021FlatMapDataset:35"
  }
}
attr {
  key: "output_shapes"
  value {
    list {
      shape {
        dim {
          size: -1
        }
      }
    }
  }
}
attr {
  key: "output_types"
  value {
    list {
      type: DT_INT64
    }
  }
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2021-12-08 21:24:38.065957: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.068157: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.074784: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.077624: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.083942: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.085927: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.090917: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.092183: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.135684: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.138288: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.145706: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.148147: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.149358: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.157175: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.159599: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.166636: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.169011: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.170183: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:25:02.591269: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 4794/13482 nodes to float16 precision using 248 cast(s) to float16 (excluding Const and Variable casts)
2021-12-08 21:25:06.903502: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 0/12606 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts)

232/232 [==============================] - 579s 2s/step - loss: 0.0871 - Accuracy: 0.6800 - val_loss: 1.1778 - val_Accuracy: 0.6700
Epoch 2/2

  1/232 [..............................] - ETA: 4:56 - loss: 0.0820 - Accuracy: 0.6667
  2/232 [..............................] - ETA: 6:36 - loss: 0.0660 - Accuracy: 0.7857
  3/232 [..............................] - ETA: 6:24 - loss: 0.0625 - Accuracy: 0.8182
  4/232 [..............................] - ETA: 6:19 - loss: 0.0699 - Accuracy: 0.7667
  5/232 [..............................] - ETA: 6:20 - loss: 0.0697 - Accuracy: 0.7632
  6/232 [..............................] - ETA: 6:20 - loss: 0.0768 - Accuracy: 0.7174
  7/232 [..............................] - ETA: 6:17 - loss: 0.0756 - Accuracy: 0.7222
  8/232 [>.............................] - ETA: 6:15 - loss: 0.0743 - Accuracy: 0.7258
  9/232 [>.............................] - ETA: 6:13 - loss: 0.0759 - Accuracy: 0.7143
 10/232 [>.............................] - ETA: 6:10 - loss: 0.0751 - Accuracy: 0.7179
 11/232 [>.............................] - ETA: 6:09 - loss: 0.0763 - Accuracy: 0.7093
 12/232 [>.............................] - ETA: 6:07 - loss: 0.0757 - Accuracy: 0.7128
 13/232 [>.............................] - ETA: 6:07 - loss: 0.0800 - Accuracy: 0.6863
 14/232 [>.............................] - ETA: 6:05 - loss: 0.0826 - Accuracy: 0.6727
 15/232 [>.............................] - ETA: 6:03 - loss: 0.0820 - Accuracy: 0.6780
 16/232 [=>............................] - ETA: 6:02 - loss: 0.0838 - Accuracy: 0.6667
 17/232 [=>............................] - ETA: 6:04 - loss: 0.0839 - Accuracy: 0.6716
 18/232 [=>............................] - ETA: 6:02 - loss: 0.0833 - Accuracy: 0.6761
 19/232 [=>............................] - ETA: 6:01 - loss: 0.0836 - Accuracy: 0.6733
 20/232 [=>............................] - ETA: 5:59 - loss: 0.0853 - Accuracy: 0.6646
 21/232 [=>............................] - ETA: 5:58 - loss: 0.0866 - Accuracy: 0.6566
 22/232 [=>............................] - ETA: 5:56 - loss: 0.0869 - Accuracy: 0.6552
 23/232 [=>............................] - ETA: 5:55 - loss: 0.0872 - Accuracy: 0.6538
 24/232 [==>...........................] - ETA: 5:53 - loss: 0.0875 - Accuracy: 0.6526
 25/232 [==>...........................] - ETA: 5:51 - loss: 0.0859 - Accuracy: 0.6616
 26/232 [==>...........................] - ETA: 5:51 - loss: 0.0844 - Accuracy: 0.6699
 27/232 [==>...........................] - ETA: 5:48 - loss: 0.0830 - Accuracy: 0.6776
 28/232 [==>...........................] - ETA: 5:47 - loss: 0.0833 - Accuracy: 0.6757
 29/232 [==>...........................] - ETA: 5:45 - loss: 0.0827 - Accuracy: 0.6783
 30/232 [==>...........................] - ETA: 5:43 - loss: 0.0821 - Accuracy: 0.6807
 31/232 [===>..........................] - ETA: 5:41 - loss: 0.0809 - Accuracy: 0.6870
 32/232 [===>..........................] - ETA: 5:39 - loss: 0.0798 - Accuracy: 0.6929
 33/232 [===>..........................] - ETA: 5:38 - loss: 0.0794 - Accuracy: 0.6947
 34/232 [===>..........................] - ETA: 5:36 - loss: 0.0797 - Accuracy: 0.6926
 35/232 [===>..........................] - ETA: 5:34 - loss: 0.0816 - Accuracy: 0.6835
 36/232 [===>..........................] - ETA: 5:32 - loss: 0.0813 - Accuracy: 0.6853
 37/232 [===>..........................] - ETA: 5:30 - loss: 0.0814 - Accuracy: 0.6837
 38/232 [===>..........................] - ETA: 5:29 - loss: 0.0814 - Accuracy: 0.6854
 39/232 [====>.........................] - ETA: 5:27 - loss: 0.0808 - Accuracy: 0.6903
 40/232 [====>.........................] - ETA: 5:26 - loss: 0.0818 - Accuracy: 0.6855
 41/232 [====>.........................] - ETA: 5:24 - loss: 0.0834 - Accuracy: 0.6779
 42/232 [====>.........................] - ETA: 5:23 - loss: 0.0823 - Accuracy: 0.6826
 43/232 [====>.........................] - ETA: 5:21 - loss: 0.0820 - Accuracy: 0.6842
 44/232 [====>.........................] - ETA: 5:19 - loss: 0.0810 - Accuracy: 0.6886
 45/232 [====>.........................] - ETA: 5:18 - loss: 0.0813 - Accuracy: 0.6872
 46/232 [====>.........................] - ETA: 5:16 - loss: 0.0811 - Accuracy: 0.6885
 47/232 [=====>........................] - ETA: 5:14 - loss: 0.0825 - Accuracy: 0.6818
 48/232 [=====>........................] - ETA: 5:13 - loss: 0.0823 - Accuracy: 0.6832
 49/232 [=====>........................] - ETA: 5:12 - loss: 0.0816 - Accuracy: 0.6872
 50/232 [=====>........................] - ETA: 5:10 - loss: 0.0814 - Accuracy: 0.6884
 51/232 [=====>........................] - ETA: 5:09 - loss: 0.0828 - Accuracy: 0.6798
 52/232 [=====>........................] - ETA: 5:08 - loss: 0.0830 - Accuracy: 0.6787
 53/232 [=====>........................] - ETA: 5:06 - loss: 0.0831 - Accuracy: 0.6777
 54/232 [=====>........................] - ETA: 5:04 - loss: 0.0831 - Accuracy: 0.6791
 55/232 [======>.......................] - ETA: 5:03 - loss: 0.0838 - Accuracy: 0.6735
 56/232 [======>.......................] - ETA: 5:01 - loss: 0.0834 - Accuracy: 0.6771
 57/232 [======>.......................] - ETA: 4:59 - loss: 0.0835 - Accuracy: 0.6762
 58/232 [======>.......................] - ETA: 4:57 - loss: 0.0829 - Accuracy: 0.6797
 59/232 [======>.......................] - ETA: 4:55 - loss: 0.0830 - Accuracy: 0.6787
 60/232 [======>.......................] - ETA: 4:53 - loss: 0.0824 - Accuracy: 0.6820
 61/232 [======>.......................] - ETA: 4:52 - loss: 0.0821 - Accuracy: 0.6831
 62/232 [=======>......................] - ETA: 4:50 - loss: 0.0814 - Accuracy: 0.6862
 63/232 [=======>......................] - ETA: 4:48 - loss: 0.0821 - Accuracy: 0.6833
 64/232 [=======>......................] - ETA: 4:46 - loss: 0.0820 - Accuracy: 0.6843
 65/232 [=======>......................] - ETA: 4:45 - loss: 0.0809 - Accuracy: 0.6892
 66/232 [=======>......................] - ETA: 4:43 - loss: 0.0803 - Accuracy: 0.6920
 67/232 [=======>......................] - ETA: 4:41 - loss: 0.0802 - Accuracy: 0.6929
 68/232 [=======>......................] - ETA: 4:39 - loss: 0.0805 - Accuracy: 0.6919
 69/232 [=======>......................] - ETA: 4:38 - loss: 0.0812 - Accuracy: 0.6891
 70/232 [========>.....................] - ETA: 4:36 - loss: 0.0810 - Accuracy: 0.6900
 71/232 [========>.....................] - ETA: 4:34 - loss: 0.0808 - Accuracy: 0.6908
 72/232 [========>.....................] - ETA: 4:32 - loss: 0.0810 - Accuracy: 0.6899
 73/232 [========>.....................] - ETA: 4:30 - loss: 0.0809 - Accuracy: 0.6907
 74/232 [========>.....................] - ETA: 4:28 - loss: 0.0807 - Accuracy: 0.6915
 75/232 [========>.....................] - ETA: 4:27 - loss: 0.0814 - Accuracy: 0.6873
 76/232 [========>.....................] - ETA: 4:25 - loss: 0.0811 - Accuracy: 0.6898
 77/232 [========>.....................] - ETA: 4:23 - loss: 0.0816 - Accuracy: 0.6857
 78/232 [=========>....................] - ETA: 4:21 - loss: 0.0820 - Accuracy: 0.6849
 79/232 [=========>....................] - ETA: 4:20 - loss: 0.0820 - Accuracy: 0.6857
 80/232 [=========>....................] - ETA: 4:18 - loss: 0.0815 - Accuracy: 0.6881
 81/232 [=========>....................] - ETA: 4:16 - loss: 0.0816 - Accuracy: 0.6873
 82/232 [=========>....................] - ETA: 4:14 - loss: 0.0811 - Accuracy: 0.6896
 83/232 [=========>....................] - ETA: 4:12 - loss: 0.0815 - Accuracy: 0.6873
 84/232 [=========>....................] - ETA: 4:11 - loss: 0.0816 - Accuracy: 0.6866
 85/232 [=========>....................] - ETA: 4:09 - loss: 0.0824 - Accuracy: 0.6829
 86/232 [==========>...................] - ETA: 4:07 - loss: 0.0828 - Accuracy: 0.6808
 87/232 [==========>...................] - ETA: 4:06 - loss: 0.0824 - Accuracy: 0.6830
 88/232 [==========>...................] - ETA: 4:04 - loss: 0.0825 - Accuracy: 0.6823
 89/232 [==========>...................] - ETA: 4:02 - loss: 0.0825 - Accuracy: 0.6817
 90/232 [==========>...................] - ETA: 4:00 - loss: 0.0820 - Accuracy: 0.6852
 91/232 [==========>...................] - ETA: 3:59 - loss: 0.0826 - Accuracy: 0.6818
 92/232 [==========>...................] - ETA: 3:57 - loss: 0.0829 - Accuracy: 0.6798
 93/232 [===========>..................] - ETA: 3:55 - loss: 0.0830 - Accuracy: 0.6792
 94/232 [===========>..................] - ETA: 3:54 - loss: 0.0832 - Accuracy: 0.6773
 95/232 [===========>..................] - ETA: 3:52 - loss: 0.0835 - Accuracy: 0.6755
 96/232 [===========>..................] - ETA: 3:50 - loss: 0.0835 - Accuracy: 0.6762
 97/232 [===========>..................] - ETA: 3:49 - loss: 0.0837 - Accuracy: 0.6757
 98/232 [===========>..................] - ETA: 3:47 - loss: 0.0834 - Accuracy: 0.6777
 99/232 [===========>..................] - ETA: 3:45 - loss: 0.0833 - Accuracy: 0.6785
100/232 [===========>..................] - ETA: 3:44 - loss: 0.0835 - Accuracy: 0.6767
101/232 [============>.................] - ETA: 3:42 - loss: 0.0834 - Accuracy: 0.6774
102/232 [============>.................] - ETA: 3:40 - loss: 0.0837 - Accuracy: 0.6757
103/232 [============>.................] - ETA: 3:38 - loss: 0.0838 - Accuracy: 0.6752
104/232 [============>.................] - ETA: 3:37 - loss: 0.0837 - Accuracy: 0.6759
105/232 [============>.................] - ETA: 3:35 - loss: 0.0835 - Accuracy: 0.6766
106/232 [============>.................] - ETA: 3:33 - loss: 0.0834 - Accuracy: 0.6773
107/232 [============>.................] - ETA: 3:32 - loss: 0.0837 - Accuracy: 0.6756
108/232 [============>.................] - ETA: 3:30 - loss: 0.0840 - Accuracy: 0.6740
109/232 [=============>................] - ETA: 3:28 - loss: 0.0838 - Accuracy: 0.6747
110/232 [=============>................] - ETA: 3:27 - loss: 0.0835 - Accuracy: 0.6765
111/232 [=============>................] - ETA: 3:25 - loss: 0.0834 - Accuracy: 0.6772
112/232 [=============>................] - ETA: 3:23 - loss: 0.0833 - Accuracy: 0.6779
113/232 [=============>................] - ETA: 3:22 - loss: 0.0836 - Accuracy: 0.6763
114/232 [=============>................] - ETA: 3:20 - loss: 0.0836 - Accuracy: 0.6758
115/232 [=============>................] - ETA: 3:18 - loss: 0.0833 - Accuracy: 0.6776
116/232 [==============>...............] - ETA: 3:16 - loss: 0.0834 - Accuracy: 0.6771
117/232 [==============>...............] - ETA: 3:15 - loss: 0.0833 - Accuracy: 0.6777
118/232 [==============>...............] - ETA: 3:13 - loss: 0.0837 - Accuracy: 0.6752
119/232 [==============>...............] - ETA: 3:11 - loss: 0.0836 - Accuracy: 0.6758
120/232 [==============>...............] - ETA: 3:10 - loss: 0.0834 - Accuracy: 0.6775
121/232 [==============>...............] - ETA: 3:08 - loss: 0.0836 - Accuracy: 0.6760
122/232 [==============>...............] - ETA: 3:06 - loss: 0.0831 - Accuracy: 0.6786
123/232 [==============>...............] - ETA: 3:05 - loss: 0.0832 - Accuracy: 0.6782
124/232 [===============>..............] - ETA: 3:03 - loss: 0.0829 - Accuracy: 0.6798
125/232 [===============>..............] - ETA: 3:01 - loss: 0.0828 - Accuracy: 0.6804
126/232 [===============>..............] - ETA: 3:00 - loss: 0.0829 - Accuracy: 0.6799
127/232 [===============>..............] - ETA: 2:58 - loss: 0.0824 - Accuracy: 0.6824
128/232 [===============>..............] - ETA: 2:56 - loss: 0.0822 - Accuracy: 0.6830
129/232 [===============>..............] - ETA: 2:54 - loss: 0.0822 - Accuracy: 0.6835
130/232 [===============>..............] - ETA: 2:53 - loss: 0.0818 - Accuracy: 0.6850
131/232 [===============>..............] - ETA: 2:51 - loss: 0.0817 - Accuracy: 0.6855
132/232 [================>.............] - ETA: 2:49 - loss: 0.0816 - Accuracy: 0.6860
133/232 [================>.............] - ETA: 2:48 - loss: 0.0821 - Accuracy: 0.6836
134/232 [================>.............] - ETA: 2:46 - loss: 0.0820 - Accuracy: 0.6841
135/232 [================>.............] - ETA: 2:44 - loss: 0.0823 - Accuracy: 0.6827
136/232 [================>.............] - ETA: 2:43 - loss: 0.0821 - Accuracy: 0.6842
137/232 [================>.............] - ETA: 2:41 - loss: 0.0821 - Accuracy: 0.6837
138/232 [================>.............] - ETA: 2:39 - loss: 0.0825 - Accuracy: 0.6815
139/232 [================>.............] - ETA: 2:38 - loss: 0.0825 - Accuracy: 0.6811
140/232 [=================>............] - ETA: 2:36 - loss: 0.0825 - Accuracy: 0.6816
141/232 [=================>............] - ETA: 2:34 - loss: 0.0823 - Accuracy: 0.6829
142/232 [=================>............] - ETA: 2:33 - loss: 0.0821 - Accuracy: 0.6843
143/232 [=================>............] - ETA: 2:31 - loss: 0.0818 - Accuracy: 0.6856
144/232 [=================>............] - ETA: 2:29 - loss: 0.0817 - Accuracy: 0.6861
145/232 [=================>............] - ETA: 2:27 - loss: 0.0814 - Accuracy: 0.6874
146/232 [=================>............] - ETA: 2:26 - loss: 0.0816 - Accuracy: 0.6870
147/232 [==================>...........] - ETA: 2:24 - loss: 0.0819 - Accuracy: 0.6857
148/232 [==================>...........] - ETA: 2:22 - loss: 0.0818 - Accuracy: 0.6861
149/232 [==================>...........] - ETA: 2:21 - loss: 0.0819 - Accuracy: 0.6857
150/232 [==================>...........] - ETA: 2:19 - loss: 0.0826 - Accuracy: 0.6828
151/232 [==================>...........] - ETA: 2:17 - loss: 0.0829 - Accuracy: 0.6816
152/232 [==================>...........] - ETA: 2:16 - loss: 0.0829 - Accuracy: 0.6812
153/232 [==================>...........] - ETA: 2:14 - loss: 0.0828 - Accuracy: 0.6817
154/232 [==================>...........] - ETA: 2:12 - loss: 0.0826 - Accuracy: 0.6829
155/232 [===================>..........] - ETA: 2:10 - loss: 0.0827 - Accuracy: 0.6826
156/232 [===================>..........] - ETA: 2:09 - loss: 0.0824 - Accuracy: 0.6846
157/232 [===================>..........] - ETA: 2:07 - loss: 0.0825 - Accuracy: 0.6834
158/232 [===================>..........] - ETA: 2:05 - loss: 0.0829 - Accuracy: 0.6807
159/232 [===================>..........] - ETA: 2:04 - loss: 0.0829 - Accuracy: 0.6803
160/232 [===================>..........] - ETA: 2:02 - loss: 0.0832 - Accuracy: 0.6784
161/232 [===================>..........] - ETA: 2:00 - loss: 0.0833 - Accuracy: 0.6788
162/232 [===================>..........] - ETA: 1:59 - loss: 0.0832 - Accuracy: 0.6793
163/232 [====================>.........] - ETA: 1:57 - loss: 0.0832 - Accuracy: 0.6797
164/232 [====================>.........] - ETA: 1:56 - loss: 0.0833 - Accuracy: 0.6794
165/232 [====================>.........] - ETA: 1:54 - loss: 0.0834 - Accuracy: 0.6783
166/232 [====================>.........] - ETA: 1:52 - loss: 0.0834 - Accuracy: 0.6780
167/232 [====================>.........] - ETA: 1:50 - loss: 0.0835 - Accuracy: 0.6777
168/232 [====================>.........] - ETA: 1:49 - loss: 0.0836 - Accuracy: 0.6766
169/232 [====================>.........] - ETA: 1:47 - loss: 0.0835 - Accuracy: 0.6770
170/232 [====================>.........] - ETA: 1:45 - loss: 0.0835 - Accuracy: 0.6775
171/232 [=====================>........] - ETA: 1:44 - loss: 0.0834 - Accuracy: 0.6779
172/232 [=====================>........] - ETA: 1:42 - loss: 0.0834 - Accuracy: 0.6776
173/232 [=====================>........] - ETA: 1:40 - loss: 0.0835 - Accuracy: 0.6773
174/232 [=====================>........] - ETA: 1:39 - loss: 0.0833 - Accuracy: 0.6784
175/232 [=====================>........] - ETA: 1:37 - loss: 0.0835 - Accuracy: 0.6774
176/232 [=====================>........] - ETA: 1:35 - loss: 0.0835 - Accuracy: 0.6771
177/232 [=====================>........] - ETA: 1:33 - loss: 0.0836 - Accuracy: 0.6768
178/232 [======================>.......] - ETA: 1:32 - loss: 0.0835 - Accuracy: 0.6772
179/232 [======================>.......] - ETA: 1:30 - loss: 0.0836 - Accuracy: 0.6762
180/232 [======================>.......] - ETA: 1:28 - loss: 0.0835 - Accuracy: 0.6766
181/232 [======================>.......] - ETA: 1:27 - loss: 0.0834 - Accuracy: 0.6770
182/232 [======================>.......] - ETA: 1:25 - loss: 0.0834 - Accuracy: 0.6774
183/232 [======================>.......] - ETA: 1:23 - loss: 0.0836 - Accuracy: 0.6765
184/232 [======================>.......] - ETA: 1:21 - loss: 0.0837 - Accuracy: 0.6762
185/232 [======================>.......] - ETA: 1:20 - loss: 0.0837 - Accuracy: 0.6759
186/232 [=======================>......] - ETA: 1:18 - loss: 0.0837 - Accuracy: 0.6763
187/232 [=======================>......] - ETA: 1:16 - loss: 0.0836 - Accuracy: 0.6767
188/232 [=======================>......] - ETA: 1:15 - loss: 0.0834 - Accuracy: 0.6778
189/232 [=======================>......] - ETA: 1:13 - loss: 0.0834 - Accuracy: 0.6781
190/232 [=======================>......] - ETA: 1:11 - loss: 0.0836 - Accuracy: 0.6765
191/232 [=======================>......] - ETA: 1:09 - loss: 0.0839 - Accuracy: 0.6750
192/232 [=======================>......] - ETA: 1:08 - loss: 0.0840 - Accuracy: 0.6747
193/232 [=======================>......] - ETA: 1:06 - loss: 0.0839 - Accuracy: 0.6751
194/232 [========================>.....] - ETA: 1:04 - loss: 0.0839 - Accuracy: 0.6748
195/232 [========================>.....] - ETA: 1:03 - loss: 0.0841 - Accuracy: 0.6739
196/232 [========================>.....] - ETA: 1:01 - loss: 0.0841 - Accuracy: 0.6737
197/232 [========================>.....] - ETA: 59s - loss: 0.0839 - Accuracy: 0.6747 
198/232 [========================>.....] - ETA: 57s - loss: 0.0838 - Accuracy: 0.6751
199/232 [========================>.....] - ETA: 56s - loss: 0.0837 - Accuracy: 0.6761
200/232 [========================>.....] - ETA: 54s - loss: 0.0836 - Accuracy: 0.6765
201/232 [========================>.....] - ETA: 52s - loss: 0.0837 - Accuracy: 0.6762
202/232 [=========================>....] - ETA: 51s - loss: 0.0836 - Accuracy: 0.6766
203/232 [=========================>....] - ETA: 49s - loss: 0.0836 - Accuracy: 0.6763
204/232 [=========================>....] - ETA: 47s - loss: 0.0836 - Accuracy: 0.6767
205/232 [=========================>....] - ETA: 46s - loss: 0.0838 - Accuracy: 0.6758
206/232 [=========================>....] - ETA: 44s - loss: 0.0839 - Accuracy: 0.6750
207/232 [=========================>....] - ETA: 42s - loss: 0.0838 - Accuracy: 0.6753
208/232 [=========================>....] - ETA: 40s - loss: 0.0836 - Accuracy: 0.6769
209/232 [==========================>...] - ETA: 39s - loss: 0.0837 - Accuracy: 0.6760
210/232 [==========================>...] - ETA: 37s - loss: 0.0836 - Accuracy: 0.6770
211/232 [==========================>...] - ETA: 35s - loss: 0.0836 - Accuracy: 0.6767
212/232 [==========================>...] - ETA: 34s - loss: 0.0835 - Accuracy: 0.6771
213/232 [==========================>...] - ETA: 32s - loss: 0.0836 - Accuracy: 0.6769
214/232 [==========================>...] - ETA: 30s - loss: 0.0835 - Accuracy: 0.6772
215/232 [==========================>...] - ETA: 28s - loss: 0.0833 - Accuracy: 0.6781
216/232 [==========================>...] - ETA: 27s - loss: 0.0830 - Accuracy: 0.6796
217/232 [===========================>..] - ETA: 25s - loss: 0.0831 - Accuracy: 0.6794
218/232 [===========================>..] - ETA: 23s - loss: 0.0829 - Accuracy: 0.6803
219/232 [===========================>..] - ETA: 22s - loss: 0.0829 - Accuracy: 0.6800
220/232 [===========================>..] - ETA: 20s - loss: 0.0831 - Accuracy: 0.6792
221/232 [===========================>..] - ETA: 18s - loss: 0.0830 - Accuracy: 0.6795
222/232 [===========================>..] - ETA: 17s - loss: 0.0831 - Accuracy: 0.6793
223/232 [===========================>..] - ETA: 15s - loss: 0.0830 - Accuracy: 0.6796
224/232 [===========================>..] - ETA: 13s - loss: 0.0829 - Accuracy: 0.6799
225/232 [============================>.] - ETA: 11s - loss: 0.0833 - Accuracy: 0.6780
226/232 [============================>.] - ETA: 10s - loss: 0.0832 - Accuracy: 0.6783
227/232 [============================>.] - ETA: 8s - loss: 0.0833 - Accuracy: 0.6781 
228/232 [============================>.] - ETA: 6s - loss: 0.0833 - Accuracy: 0.6778
229/232 [============================>.] - ETA: 5s - loss: 0.0833 - Accuracy: 0.6776
230/232 [============================>.] - ETA: 3s - loss: 0.0833 - Accuracy: 0.6774
231/232 [============================>.] - ETA: 1s - loss: 0.0832 - Accuracy: 0.6782
232/232 [==============================] - ETA: 0s - loss: 0.0831 - Accuracy: 0.67912021-12-08 21:32:02.219818: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.222070: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.228852: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.232040: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.238568: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.240707: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.246288: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.247601: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.294783: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.297749: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.305270: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.307566: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.308656: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.316060: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.318295: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.324953: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.327110: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.328203: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do

232/232 [==============================] - 412s 2s/step - loss: 0.0831 - Accuracy: 0.6791 - val_loss: 1.2342 - val_Accuracy: 0.6700
```<|||||>Training also works with 3 GPUs.<|||||>Gently pinging @Rocketknight1 here

<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Been doing library upgrade testing for our stuff lately (we were still using `transformers==2.11.0`) :sweat_smile: and that is almost complete, so hoping to get back to more testing around this in the next few weeks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I still have planned testing coming up, so let's re-open