541284.err

[NbConvertApp] Converting notebook HCP_downstream_finetune.ipynb to python
[NbConvertApp] Writing 31825 bytes to HCP_downstream_finetune.py
/weka/proj-fmri/ckadirt/fMRI-foundation-model/src/HCP_downstream_finetune.py:658: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  state = torch.load(checkpoint_path)
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: ckadirt. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.18.3
wandb: Run data is saved locally in /weka/proj-fmri/ckadirt/fMRI-foundation-model/src/wandb/run-20241126_214117-HCPflat_large_gsrFalse__beta_sex_HCPFT_83810
wandb: Run `wandb offline` to turn off syncing.
wandb: Resuming run HCPflat_large_gsrFalse__beta_sex_HCPFT
wandb: ⭐️ View project at https://stability.wandb.io/ckadirt/fMRI-foundation-model
wandb: 🚀 View run at https://stability.wandb.io/ckadirt/fMRI-foundation-model/runs/HCPflat_large_gsrFalse__beta_sex_HCPFT_83810

Epoch 1/20 - Training:   0%|          | 0/6957 [00:00<?, ?it/s]
Epoch 1/20 - Training:   0%|          | 1/6957 [00:04<7:57:58,  4.12s/it]
Epoch 1/20 - Training:   0%|          | 2/6957 [00:04<4:02:42,  2.09s/it]
Epoch 1/20 - Training:   0%|          | 3/6957 [00:05<2:47:24,  1.44s/it]
Epoch 1/20 - Training:   0%|          | 4/6957 [00:06<2:11:57,  1.14s/it]
Epoch 1/20 - Training:   0%|          | 5/6957 [00:06<1:52:26,  1.03it/s]
Epoch 1/20 - Training:   0%|          | 6/6957 [00:07<1:40:33,  1.15it/s]
Epoch 1/20 - Training:   0%|          | 7/6957 [00:08<1:32:58,  1.25it/s]
Epoch 1/20 - Training:   0%|          | 8/6957 [00:08<1:28:14,  1.31it/s]
Epoch 1/20 - Training:   0%|          | 9/6957 [00:09<1:24:52,  1.36it/s]
Epoch 1/20 - Training:   0%|          | 10/6957 [00:10<1:22:35,  1.40it/s]
Epoch 1/20 - Training:   0%|          | 11/6957 [00:10<1:21:03,  1.43it/s]
Epoch 1/20 - Training:   0%|          | 12/6957 [00:11<1:19:59,  1.45it/s]
Epoch 1/20 - Training:   0%|          | 13/6957 [00:12<1:19:14,  1.46it/s]
Epoch 1/20 - Training:   0%|          | 14/6957 [00:12<1:18:40,  1.47it/s]
Epoch 1/20 - Training:   0%|          | 15/6957 [00:13<1:18:19,  1.48it/s]
Epoch 1/20 - Training:   0%|          | 16/6957 [00:14<1:18:18,  1.48it/s]
Epoch 1/20 - Training:   0%|          | 17/6957 [00:14<1:18:04,  1.48it/s]
Epoch 1/20 - Training:   0%|          | 18/6957 [00:15<1:17:55,  1.48it/s]
Epoch 1/20 - Training:   0%|          | 19/6957 [00:16<1:17:57,  1.48it/s]
Epoch 1/20 - Training:   0%|          | 20/6957 [00:16<1:17:58,  1.48it/s]
Epoch 1/20 - Training:   0%|          | 21/6957 [00:17<1:17:49,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 22/6957 [00:18<1:17:39,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 23/6957 [00:18<1:17:32,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 24/6957 [00:19<1:17:28,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 25/6957 [00:20<1:17:23,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 26/6957 [00:20<1:17:21,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 27/6957 [00:21<1:17:17,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 28/6957 [00:22<1:17:13,  1.50it/s]
Epoch 1/20 - Training:   0%|          | 29/6957 [00:22<1:17:15,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 30/6957 [00:23<1:17:15,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 31/6957 [00:24<1:17:15,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 32/6957 [00:24<1:17:14,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 33/6957 [00:25<1:17:13,  1.49it/s]
Epoch 1/20 - Training:   0%|          | 34/6957 [00:26<1:17:18,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 35/6957 [00:26<1:17:20,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 36/6957 [00:27<1:17:21,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 37/6957 [00:28<1:17:19,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 38/6957 [00:28<1:17:15,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 39/6957 [00:29<1:17:12,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 40/6957 [00:30<1:17:11,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 41/6957 [00:30<1:17:16,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 42/6957 [00:31<1:17:19,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 43/6957 [00:32<1:17:21,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 44/6957 [00:32<1:17:18,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 45/6957 [00:33<1:17:15,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 46/6957 [00:34<1:17:09,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 47/6957 [00:34<1:17:06,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 48/6957 [00:35<1:17:04,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 49/6957 [00:36<1:17:05,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 50/6957 [00:36<1:17:04,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 51/6957 [00:37<1:17:03,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 52/6957 [00:38<1:17:02,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 53/6957 [00:38<1:17:34,  1.48it/s]
Epoch 1/20 - Training:   1%|          | 54/6957 [00:39<1:17:24,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 55/6957 [00:40<1:17:15,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 56/6957 [00:41<1:17:10,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 57/6957 [00:41<1:17:23,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 58/6957 [00:42<1:17:35,  1.48it/s]
Epoch 1/20 - Training:   1%|          | 59/6957 [00:43<1:17:24,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 60/6957 [00:43<1:17:15,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 61/6957 [00:44<1:17:08,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 62/6957 [00:45<1:17:06,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 63/6957 [00:45<1:17:05,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 64/6957 [00:46<1:17:04,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 65/6957 [00:47<1:17:05,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 66/6957 [00:47<1:17:19,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 67/6957 [00:48<1:17:11,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 68/6957 [00:49<1:17:31,  1.48it/s]
Epoch 1/20 - Training:   1%|          | 69/6957 [00:49<1:17:35,  1.48it/s]
Epoch 1/20 - Training:   1%|          | 70/6957 [00:50<1:17:25,  1.48it/s]
Epoch 1/20 - Training:   1%|          | 71/6957 [00:51<1:17:19,  1.48it/s]
Epoch 1/20 - Training:   1%|          | 72/6957 [00:51<1:17:13,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 73/6957 [00:52<1:17:09,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 74/6957 [00:53<1:17:05,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 75/6957 [00:53<1:17:02,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 76/6957 [00:54<1:17:00,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 77/6957 [00:55<1:16:57,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 78/6957 [00:55<1:16:55,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 79/6957 [00:56<1:16:55,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 80/6957 [00:57<1:16:55,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 81/6957 [00:57<1:16:53,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 82/6957 [00:58<1:16:52,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 83/6957 [00:59<1:16:51,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 84/6957 [00:59<1:17:02,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 85/6957 [01:00<1:16:56,  1.49it/s]
Epoch 1/20 - Training:   1%|          | 86/6957 [01:01<1:16:53,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 87/6957 [01:01<1:16:51,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 88/6957 [01:02<1:16:51,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 89/6957 [01:03<1:16:51,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 90/6957 [01:03<1:16:47,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 91/6957 [01:04<1:16:44,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 92/6957 [01:05<1:16:42,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 93/6957 [01:05<1:16:41,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 94/6957 [01:06<1:16:42,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 95/6957 [01:07<1:16:48,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 96/6957 [01:07<1:16:45,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 97/6957 [01:08<1:16:42,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 98/6957 [01:09<1:16:42,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 99/6957 [01:09<1:16:53,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 100/6957 [01:10<1:16:50,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 101/6957 [01:11<1:16:45,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 102/6957 [01:11<1:16:43,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 103/6957 [01:12<1:16:42,  1.49it/s]
Epoch 1/20 - Training:   1%|▏         | 104/6957 [01:13<1:16:41,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 105/6957 [01:13<1:16:40,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 106/6957 [01:14<1:16:38,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 107/6957 [01:15<1:16:36,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 108/6957 [01:15<1:16:34,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 109/6957 [01:16<1:16:32,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 110/6957 [01:17<1:16:29,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 111/6957 [01:17<1:16:27,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 112/6957 [01:18<1:16:24,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 113/6957 [01:19<1:16:24,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 114/6957 [01:19<1:16:28,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 115/6957 [01:20<1:16:30,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 116/6957 [01:21<1:16:29,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 117/6957 [01:21<1:16:31,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 118/6957 [01:22<1:16:38,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 119/6957 [01:23<1:16:41,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 120/6957 [01:23<1:16:44,  1.48it/s]
Epoch 1/20 - Training:   2%|▏         | 121/6957 [01:24<1:16:40,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 122/6957 [01:25<1:16:36,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 123/6957 [01:26<1:16:32,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 124/6957 [01:26<1:16:28,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 125/6957 [01:27<1:16:26,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 126/6957 [01:28<1:16:25,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 127/6957 [01:28<1:16:22,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 128/6957 [01:29<1:16:20,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 129/6957 [01:30<1:16:22,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 130/6957 [01:30<1:16:19,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 131/6957 [01:31<1:16:17,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 132/6957 [01:32<1:16:23,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 133/6957 [01:32<1:16:21,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 134/6957 [01:33<1:16:21,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 135/6957 [01:34<1:16:18,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 136/6957 [01:34<1:16:21,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 137/6957 [01:35<1:16:18,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 138/6957 [01:36<1:16:16,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 139/6957 [01:36<1:16:14,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 140/6957 [01:37<1:16:13,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 141/6957 [01:38<1:16:11,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 142/6957 [01:38<1:16:09,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 143/6957 [01:39<1:16:05,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 144/6957 [01:40<1:16:06,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 145/6957 [01:40<1:16:06,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 146/6957 [01:41<1:16:06,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 147/6957 [01:42<1:16:09,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 148/6957 [01:42<1:16:08,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 149/6957 [01:43<1:16:07,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 150/6957 [01:44<1:16:05,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 151/6957 [01:44<1:16:08,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 152/6957 [01:45<1:16:10,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 153/6957 [01:46<1:16:05,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 154/6957 [01:46<1:16:04,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 155/6957 [01:47<1:16:01,  1.49it/s]slurmstepd: error: *** REASON: burst_buffer/lua: Stage-out in progress ***
slurmstepd: error: *** JOB 541284 ON ip-10-0-133-32 CANCELLED AT 2024-11-26T21:43:06 ***
slurmstepd: error: *** REASON: burst_buffer/lua: Stage-out in progress ***

Epoch 1/20 - Training:   2%|▏         | 156/6957 [01:48<1:15:59,  1.49it/s]
Epoch 1/20 - Training:   2%|▏         | 157/6957 [01:48<1:15:57,  1.49it/s]