Hugging Face's logo

ckadirt
/
backup

Model card Files
xet
Community
backup / 541342.err
ckadirt's picture
ckadirt
Add files using upload-large-folder tool
19c938f verified
raw
history blame contribute delete
28.8 kB
 [NbConvertApp] Converting notebook HCP_downstream_finetune.ipynb to python
[NbConvertApp] Writing 31940 bytes to HCP_downstream_finetune.py
/weka/proj-fmri/ckadirt/fMRI-foundation-model/src/HCP_downstream_finetune.py:658: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load(checkpoint_path)
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: ckadirt. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.18.3
wandb: Run data is saved locally in /weka/proj-fmri/ckadirt/fMRI-foundation-model/src/wandb/run-20241127_015634-HCPflat_large_gsrFalse__beta_age_HCPFT_e7c8af61-0ee0-4235-bcdb-bd61bb32c3b5
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run HCPflat_large_gsrFalse__beta_age_HCPFT
wandb: ⭐️ View project at https://stability.wandb.io/ckadirt/fMRI-foundation-model
wandb: 🚀 View run at https://stability.wandb.io/ckadirt/fMRI-foundation-model/runs/HCPflat_large_gsrFalse__beta_age_HCPFT_e7c8af61-0ee0-4235-bcdb-bd61bb32c3b5
Epoch 1/20 - Training: 0%| | 0/6957 [00:00<?, ?it/s]/admin/home-ckadirt/foundation_env/lib/python3.11/site-packages/torch/nn/modules/loss.py:538: UserWarning: Using a target size (torch.Size([16, 1])) that is different to the input size (torch.Size([16])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
return F.mse_loss(input, target, reduction=self.reduction)
Epoch 1/20 - Training: 0%| | 1/6957 [00:09<19:13:47, 9.95s/it] Epoch 1/20 - Training: 0%| | 2/6957 [00:10<8:41:38, 4.50s/it] Epoch 1/20 - Training: 0%| | 3/6957 [00:11<5:18:58, 2.75s/it] Epoch 1/20 - Training: 0%| | 4/6957 [00:11<3:43:42, 1.93s/it] Epoch 1/20 - Training: 0%| | 5/6957 [00:12<2:51:01, 1.48s/it] Epoch 1/20 - Training: 0%| | 6/6957 [00:13<2:19:15, 1.20s/it] Epoch 1/20 - Training: 0%| | 7/6957 [00:13<1:59:04, 1.03s/it] Epoch 1/20 - Training: 0%| | 8/6957 [00:14<1:45:53, 1.09it/s] Epoch 1/20 - Training: 0%| | 9/6957 [00:15<1:37:02, 1.19it/s] Epoch 1/20 - Training: 0%| | 10/6957 [00:16<1:31:03, 1.27it/s] Epoch 1/20 - Training: 0%| | 11/6957 [00:16<1:35:18, 1.21it/s] Epoch 1/20 - Training: 0%| | 12/6957 [00:17<1:29:57, 1.29it/s] Epoch 1/20 - Training: 0%| | 13/6957 [00:18<1:26:09, 1.34it/s] Epoch 1/20 - Training: 0%| | 14/6957 [00:18<1:23:38, 1.38it/s] Epoch 1/20 - Training: 0%| | 15/6957 [00:19<1:21:48, 1.41it/s] Epoch 1/20 - Training: 0%| | 16/6957 [00:20<1:20:27, 1.44it/s] Epoch 1/20 - Training: 0%| | 17/6957 [00:20<1:19:31, 1.45it/s] Epoch 1/20 - Training: 0%| | 18/6957 [00:21<1:18:54, 1.47it/s] Epoch 1/20 - Training: 0%| | 19/6957 [00:22<1:18:23, 1.48it/s] Epoch 1/20 - Training: 0%| | 20/6957 [00:22<1:18:04, 1.48it/s] Epoch 1/20 - Training: 0%| | 21/6957 [00:23<1:18:09, 1.48it/s] Epoch 1/20 - Training: 0%| | 22/6957 [00:24<1:17:52, 1.48it/s] Epoch 1/20 - Training: 0%| | 23/6957 [00:24<1:17:39, 1.49it/s] Epoch 1/20 - Training: 0%| | 24/6957 [00:25<1:17:32, 1.49it/s] Epoch 1/20 - Training: 0%| | 25/6957 [00:26<1:17:25, 1.49it/s] Epoch 1/20 - Training: 0%| | 26/6957 [00:26<1:17:20, 1.49it/s] Epoch 1/20 - Training: 0%| | 27/6957 [00:27<1:17:16, 1.49it/s] Epoch 1/20 - Training: 0%| | 28/6957 [00:28<1:17:12, 1.50it/s] Epoch 1/20 - Training: 0%| | 29/6957 [00:28<1:17:08, 1.50it/s] Epoch 1/20 - Training: 0%| | 30/6957 [00:29<1:17:04, 1.50it/s] Epoch 1/20 - Training: 0%| | 31/6957 [00:30<1:17:05, 1.50it/s] Epoch 1/20 - Training: 0%| | 32/6957 [00:30<1:17:05, 1.50it/s] Epoch 1/20 - Training: 0%| | 33/6957 [00:31<1:17:08, 1.50it/s] Epoch 1/20 - Training: 0%| | 34/6957 [00:32<1:17:05, 1.50it/s] Epoch 1/20 - Training: 1%| | 35/6957 [00:32<1:17:01, 1.50it/s] Epoch 1/20 - Training: 1%| | 36/6957 [00:33<1:17:02, 1.50it/s] Epoch 1/20 - Training: 1%| | 37/6957 [00:34<1:17:04, 1.50it/s] Epoch 1/20 - Training: 1%| | 38/6957 [00:34<1:17:05, 1.50it/s] Epoch 1/20 - Training: 1%| | 39/6957 [00:35<1:17:08, 1.49it/s] Epoch 1/20 - Training: 1%| | 40/6957 [00:36<1:17:09, 1.49it/s] Epoch 1/20 - Training: 1%| | 41/6957 [00:36<1:17:05, 1.50it/s] Epoch 1/20 - Training: 1%| | 42/6957 [00:37<1:17:18, 1.49it/s] Epoch 1/20 - Training: 1%| | 43/6957 [00:38<1:17:16, 1.49it/s] Epoch 1/20 - Training: 1%| | 44/6957 [00:38<1:17:15, 1.49it/s] Epoch 1/20 - Training: 1%| | 45/6957 [00:39<1:17:12, 1.49it/s] Epoch 1/20 - Training: 1%| | 46/6957 [00:40<1:17:12, 1.49it/s] Epoch 1/20 - Training: 1%| | 47/6957 [00:41<1:17:14, 1.49it/s] Epoch 1/20 - Training: 1%| | 48/6957 [00:41<1:17:11, 1.49it/s] Epoch 1/20 - Training: 1%| | 49/6957 [00:42<1:17:06, 1.49it/s] Epoch 1/20 - Training: 1%| | 50/6957 [00:43<1:17:07, 1.49it/s] Epoch 1/20 - Training: 1%| | 51/6957 [00:43<1:17:13, 1.49it/s] Epoch 1/20 - Training: 1%| | 52/6957 [00:44<1:17:12, 1.49it/s] Epoch 1/20 - Training: 1%| | 53/6957 [00:45<1:17:12, 1.49it/s] Epoch 1/20 - Training: 1%| | 54/6957 [00:45<1:17:20, 1.49it/s] Epoch 1/20 - Training: 1%| | 55/6957 [00:46<1:17:17, 1.49it/s] Epoch 1/20 - Training: 1%| | 56/6957 [00:47<1:17:13, 1.49it/s] Epoch 1/20 - Training: 1%| | 57/6957 [00:47<1:17:09, 1.49it/s] Epoch 1/20 - Training: 1%| | 58/6957 [00:48<1:17:07, 1.49it/s] Epoch 1/20 - Training: 1%| | 59/6957 [00:49<1:17:05, 1.49it/s] Epoch 1/20 - Training: 1%| | 60/6957 [00:49<1:17:07, 1.49it/s] Epoch 1/20 - Training: 1%| | 61/6957 [00:50<1:17:08, 1.49it/s] Epoch 1/20 - Training: 1%| | 62/6957 [00:51<1:17:13, 1.49it/s] Epoch 1/20 - Training: 1%| | 63/6957 [00:51<1:17:15, 1.49it/s] Epoch 1/20 - Training: 1%| | 64/6957 [00:52<1:17:10, 1.49it/s] Epoch 1/20 - Training: 1%| | 65/6957 [00:53<1:17:05, 1.49it/s] Epoch 1/20 - Training: 1%| | 66/6957 [00:53<1:17:00, 1.49it/s] Epoch 1/20 - Training: 1%| | 67/6957 [00:54<1:16:57, 1.49it/s] Epoch 1/20 - Training: 1%| | 68/6957 [00:55<1:16:52, 1.49it/s] Epoch 1/20 - Training: 1%| | 69/6957 [00:55<1:16:49, 1.49it/s] Epoch 1/20 - Training: 1%| | 70/6957 [00:56<1:16:57, 1.49it/s] Epoch 1/20 - Training: 1%| | 71/6957 [00:57<1:16:59, 1.49it/s] Epoch 1/20 - Training: 1%| | 72/6957 [00:57<1:17:00, 1.49it/s] Epoch 1/20 - Training: 1%| | 73/6957 [00:58<1:17:01, 1.49it/s] Epoch 1/20 - Training: 1%| | 74/6957 [00:59<1:17:02, 1.49it/s] Epoch 1/20 - Training: 1%| | 75/6957 [00:59<1:17:01, 1.49it/s] Epoch 1/20 - Training: 1%| | 76/6957 [01:00<1:16:57, 1.49it/s] Epoch 1/20 - Training: 1%| | 77/6957 [01:01<1:16:51, 1.49it/s] Epoch 1/20 - Training: 1%| | 78/6957 [01:01<1:16:47, 1.49it/s] Epoch 1/20 - Training: 1%| | 79/6957 [01:02<1:16:45, 1.49it/s] Epoch 1/20 - Training: 1%| | 80/6957 [01:03<1:16:46, 1.49it/s] Epoch 1/20 - Training: 1%| | 81/6957 [01:03<1:16:52, 1.49it/s] Epoch 1/20 - Training: 1%| | 82/6957 [01:04<1:16:57, 1.49it/s] Epoch 1/20 - Training: 1%| | 83/6957 [01:05<1:16:54, 1.49it/s] Epoch 1/20 - Training: 1%| | 84/6957 [01:05<1:16:57, 1.49it/s] Epoch 1/20 - Training: 1%| | 85/6957 [01:06<1:16:53, 1.49it/s] Epoch 1/20 - Training: 1%| | 86/6957 [01:07<1:16:52, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 87/6957 [01:07<1:16:45, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 88/6957 [01:08<1:16:42, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 89/6957 [01:09<1:16:37, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 90/6957 [01:09<1:16:39, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 91/6957 [01:10<1:16:46, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 92/6957 [01:11<1:16:48, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 93/6957 [01:11<1:16:50, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 94/6957 [01:12<1:16:50, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 95/6957 [01:13<1:16:46, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 96/6957 [01:13<1:16:38, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 97/6957 [01:14<1:16:33, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 98/6957 [01:15<1:16:29, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 99/6957 [01:15<1:16:25, 1.50it/s] Epoch 1/20 - Training: 1%|▏ | 100/6957 [01:16<1:16:32, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 101/6957 [01:17<1:16:36, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 102/6957 [01:17<1:16:38, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 103/6957 [01:18<1:16:40, 1.49it/s] Epoch 1/20 - Training: 1%|▏ | 104/6957 [01:19<1:16:36, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 105/6957 [01:19<1:16:32, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 106/6957 [01:20<1:16:27, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 107/6957 [01:21<1:16:24, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 108/6957 [01:21<1:16:21, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 109/6957 [01:22<1:16:18, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 110/6957 [01:23<1:16:18, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 111/6957 [01:23<1:16:18, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 112/6957 [01:24<1:16:19, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 113/6957 [01:25<1:16:20, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 114/6957 [01:25<1:16:20, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 115/6957 [01:26<1:16:17, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 116/6957 [01:27<1:16:14, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 117/6957 [01:27<1:16:12, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 118/6957 [01:28<1:16:10, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 119/6957 [01:29<1:16:08, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 120/6957 [01:29<1:16:08, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 121/6957 [01:30<1:16:11, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 122/6957 [01:31<1:16:14, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 123/6957 [01:31<1:16:10, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 124/6957 [01:32<1:16:06, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 125/6957 [01:33<1:16:12, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 126/6957 [01:33<1:16:08, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 127/6957 [01:34<1:16:05, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 128/6957 [01:35<1:16:01, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 129/6957 [01:35<1:15:58, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 130/6957 [01:36<1:15:57, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 131/6957 [01:37<1:16:02, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 132/6957 [01:37<1:16:07, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 133/6957 [01:38<1:16:12, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 134/6957 [01:39<1:16:13, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 135/6957 [01:39<1:16:15, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 136/6957 [01:40<1:16:09, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 137/6957 [01:41<1:16:04, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 138/6957 [01:41<1:16:01, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 139/6957 [01:42<1:15:59, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 140/6957 [01:43<1:15:57, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 141/6957 [01:43<1:15:58, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 142/6957 [01:44<1:15:59, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 143/6957 [01:45<1:16:00, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 144/6957 [01:45<1:15:59, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 145/6957 [01:46<1:15:57, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 146/6957 [01:47<1:15:54, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 147/6957 [01:47<1:15:52, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 148/6957 [01:48<1:15:52, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 149/6957 [01:49<1:15:50, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 150/6957 [01:50<1:16:01, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 151/6957 [01:50<1:15:58, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 152/6957 [01:51<1:16:00, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 153/6957 [01:52<1:15:57, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 154/6957 [01:52<1:15:57, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 155/6957 [01:53<1:15:53, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 156/6957 [01:54<1:15:57, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 157/6957 [01:54<1:15:53, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 158/6957 [01:55<1:15:49, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 159/6957 [01:56<1:15:46, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 160/6957 [01:56<1:15:43, 1.50it/s] Epoch 1/20 - Training: 2%|▏ | 161/6957 [01:57<1:15:52, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 162/6957 [01:58<1:15:54, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 163/6957 [01:58<1:16:04, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 164/6957 [01:59<1:15:58, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 165/6957 [02:00<1:15:54, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 166/6957 [02:00<1:15:52, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 167/6957 [02:01<1:15:51, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 168/6957 [02:02<1:15:50, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 169/6957 [02:02<1:15:48, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 170/6957 [02:03<1:15:55, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 171/6957 [02:04<1:15:52, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 172/6957 [02:04<1:15:50, 1.49it/s] Epoch 1/20 - Training: 2%|▏ | 173/6957 [02:05<1:15:49, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 174/6957 [02:06<1:15:45, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 175/6957 [02:06<1:15:46, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 176/6957 [02:07<1:15:44, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 177/6957 [02:08<1:15:41, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 178/6957 [02:08<1:15:38, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 179/6957 [02:09<1:15:45, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 180/6957 [02:10<1:15:42, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 181/6957 [02:10<1:15:40, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 182/6957 [02:11<1:15:46, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 183/6957 [02:12<1:15:42, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 184/6957 [02:12<1:15:40, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 185/6957 [02:13<1:15:38, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 186/6957 [02:14<1:15:34, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 187/6957 [02:14<1:15:35, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 188/6957 [02:15<1:15:33, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 189/6957 [02:16<1:15:32, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 190/6957 [02:16<1:15:40, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 191/6957 [02:17<1:15:40, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 192/6957 [02:18<1:15:35, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 193/6957 [02:18<1:15:32, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 194/6957 [02:19<1:15:30, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 195/6957 [02:20<1:15:29, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 196/6957 [02:20<1:15:29, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 197/6957 [02:21<1:15:28, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 198/6957 [02:22<1:15:27, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 199/6957 [02:22<1:15:25, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 200/6957 [02:23<1:15:30, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 201/6957 [02:24<1:15:27, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 202/6957 [02:24<1:15:26, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 203/6957 [02:25<1:15:28, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 204/6957 [02:26<1:15:25, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 205/6957 [02:26<1:15:24, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 206/6957 [02:27<1:15:26, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 207/6957 [02:28<1:15:40, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 208/6957 [02:28<1:15:33, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 209/6957 [02:29<1:15:26, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 210/6957 [02:30<1:15:33, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 211/6957 [02:30<1:15:30, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 212/6957 [02:31<1:15:28, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 213/6957 [02:32<1:15:24, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 214/6957 [02:32<1:15:20, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 215/6957 [02:33<1:15:23, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 216/6957 [02:34<1:15:20, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 217/6957 [02:34<1:15:16, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 218/6957 [02:35<1:15:10, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 219/6957 [02:36<1:15:06, 1.50it/s] Epoch 1/20 - Training: 3%|▎ | 220/6957 [02:36<1:15:06, 1.50it/s] Epoch 1/20 - Training: 3%|▎ | 221/6957 [02:37<1:15:04, 1.50it/s] Epoch 1/20 - Training: 3%|▎ | 222/6957 [02:38<1:15:07, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 223/6957 [02:38<1:15:05, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 224/6957 [02:39<1:15:02, 1.50it/s] Epoch 1/20 - Training: 3%|▎ | 225/6957 [02:40<1:15:10, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 226/6957 [02:40<1:15:08, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 227/6957 [02:41<1:15:05, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 228/6957 [02:42<1:15:02, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 229/6957 [02:42<1:14:58, 1.50it/s] Epoch 1/20 - Training: 3%|▎ | 230/6957 [02:43<1:15:04, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 231/6957 [02:44<1:15:02, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 232/6957 [02:44<1:15:03, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 233/6957 [02:45<1:15:05, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 234/6957 [02:46<1:15:06, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 235/6957 [02:46<1:15:04, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 236/6957 [02:47<1:15:05, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 237/6957 [02:48<1:15:03, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 238/6957 [02:48<1:15:02, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 239/6957 [02:49<1:15:01, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 240/6957 [02:50<1:14:59, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 241/6957 [02:50<1:15:01, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 242/6957 [02:51<1:14:59, 1.49it/s] Epoch 1/20 - Training: 3%|▎ | 243/6957 [02:52<1:14:59, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 244/6957 [02:53<1:14:58, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 245/6957 [02:53<1:14:56, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 246/6957 [02:54<1:14:59, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 247/6957 [02:55<1:14:58, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 248/6957 [02:55<1:14:57, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 249/6957 [02:56<1:14:53, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 250/6957 [02:57<1:15:01, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 251/6957 [02:57<1:15:00, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 252/6957 [02:58<1:14:57, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 253/6957 [02:59<1:14:53, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 254/6957 [02:59<1:14:50, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 255/6957 [03:00<1:14:46, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 256/6957 [03:01<1:14:49, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 257/6957 [03:01<1:14:49, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 258/6957 [03:02<1:14:46, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 259/6957 [03:03<1:14:45, 1.49it/s] Epoch 1/20 - Training: 4%|▎ | 260/6957 [03:03<1:14:52, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 261/6957 [03:04<1:14:48, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 262/6957 [03:05<1:14:49, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 263/6957 [03:05<1:14:48, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 264/6957 [03:06<1:14:45, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 265/6957 [03:07<1:14:42, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 266/6957 [03:07<1:14:40, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 267/6957 [03:08<1:14:40, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 268/6957 [03:09<1:14:39, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 269/6957 [03:09<1:14:37, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 270/6957 [03:10<1:14:44, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 271/6957 [03:11<1:14:43, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 272/6957 [03:11<1:14:40, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 273/6957 [03:12<1:14:45, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 274/6957 [03:13<1:14:41, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 275/6957 [03:13<1:14:41, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 276/6957 [03:14<1:14:40, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 277/6957 [03:15<1:14:44, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 278/6957 [03:15<1:14:42, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 279/6957 [03:16<1:14:40, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 280/6957 [03:17<1:14:41, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 281/6957 [03:17<1:14:38, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 282/6957 [03:18<1:14:37, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 283/6957 [03:19<1:14:34, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 284/6957 [03:19<1:14:32, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 285/6957 [03:20<1:14:31, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 286/6957 [03:21<1:14:28, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 287/6957 [03:21<1:14:27, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 288/6957 [03:22<1:14:27, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 289/6957 [03:23<1:14:28, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 290/6957 [03:23<1:14:29, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 291/6957 [03:24<1:14:29, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 292/6957 [03:25<1:14:27, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 293/6957 [03:25<1:14:24, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 294/6957 [03:26<1:14:23, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 295/6957 [03:27<1:14:22, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 296/6957 [03:27<1:14:24, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 297/6957 [03:28<1:14:26, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 298/6957 [03:29<1:14:24, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 299/6957 [03:29<1:14:23, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 300/6957 [03:30<1:14:27, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 301/6957 [03:31<1:14:24, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 302/6957 [03:31<1:14:22, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 303/6957 [03:32<1:14:23, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 304/6957 [03:33<1:14:23, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 305/6957 [03:33<1:14:22, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 306/6957 [03:34<1:14:20, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 307/6957 [03:35<1:14:16, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 308/6957 [03:35<1:14:13, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 309/6957 [03:36<1:14:11, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 310/6957 [03:37<1:14:17, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 311/6957 [03:37<1:14:23, 1.49it/s] Epoch 1/20 - Training: 4%|▍ | 312/6957 [03:38<1:14:36, 1.48it/s] Epoch 1/20 - Training: 4%|▍ | 313/6957 [03:39<1:14:36, 1.48it/s] Epoch 1/20 - Training: 5%|▍ | 314/6957 [03:39<1:14:30, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 315/6957 [03:40<1:14:26, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 316/6957 [03:41<1:14:20, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 317/6957 [03:41<1:14:20, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 318/6957 [03:42<1:14:16, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 319/6957 [03:43<1:14:13, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 320/6957 [03:43<1:14:14, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 321/6957 [03:44<1:14:22, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 322/6957 [03:45<1:14:25, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 323/6957 [03:45<1:14:22, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 324/6957 [03:46<1:14:18, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 325/6957 [03:47<1:14:14, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 326/6957 [03:48<1:14:11, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 327/6957 [03:48<1:14:09, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 328/6957 [03:49<1:14:16, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 329/6957 [03:50<1:14:11, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 330/6957 [03:50<1:14:26, 1.48it/s] Epoch 1/20 - Training: 5%|▍ | 331/6957 [03:51<1:14:18, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 332/6957 [03:52<1:14:13, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 333/6957 [03:52<1:14:09, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 334/6957 [03:53<1:14:05, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 335/6957 [03:54<1:14:01, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 336/6957 [03:54<1:14:00, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 337/6957 [03:55<1:14:00, 1.49it/s] Epoch 1/20 - Training: 5%|▍ | 338/6957 [03:56<1:13:58, 1.49it/s]slurmstepd: error: *** REASON: burst_buffer/lua: Stage-out in progress ***
slurmstepd: error: *** JOB 541342 ON ip-10-0-136-5 CANCELLED AT 2024-11-27T02:00:31 ***
slurmstepd: error: *** REASON: burst_buffer/lua: Stage-out in progress ***
Epoch 1/20 - Training: 5%|▍ | 339/6957 [03:56<1:14:00, 1.49it/s]