File size: 48,104 Bytes
19c938f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
[NbConvertApp] Converting notebook HCP_downstream_raw_flatmaps.ipynb to python
[NbConvertApp] Writing 36445 bytes to HCP_downstream_raw_flatmaps.py
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: ckadirt. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.18.3
wandb: Run data is saved locally in /weka/proj-fmri/ckadirt/fMRI-foundation-model/src/wandb/run-20241126_220105-HCPflat_raw_beta_age_7cc4d250-ac94-488c-bec4-39b422ee70de
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run HCPflat_raw_beta_age
wandb: ⭐️ View project at https://stability.wandb.io/ckadirt/fMRI-foundation-model
wandb: 🚀 View run at https://stability.wandb.io/ckadirt/fMRI-foundation-model/runs/HCPflat_raw_beta_age_7cc4d250-ac94-488c-bec4-39b422ee70de
Epoch 1/20 - Training: 0%| | 0/870 [00:00<?, ?it/s]/admin/home-ckadirt/foundation_env/lib/python3.11/site-packages/torch/nn/modules/loss.py:538: UserWarning: Using a target size (torch.Size([128, 1])) that is different to the input size (torch.Size([128])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
return F.mse_loss(input, target, reduction=self.reduction)
Epoch 1/20 - Training: 0%| | 1/870 [00:04<1:07:12, 4.64s/it]
Epoch 1/20 - Training: 0%| | 3/870 [00:04<18:23, 1.27s/it]
Epoch 1/20 - Training: 1%| | 5/870 [00:06<14:58, 1.04s/it]
Epoch 1/20 - Training: 1%| | 7/870 [00:06<09:02, 1.59it/s]
Epoch 1/20 - Training: 1%| | 9/870 [00:06<06:00, 2.39it/s]
Epoch 1/20 - Training: 1%|▏ | 11/870 [00:07<04:43, 3.04it/s]
Epoch 1/20 - Training: 1%|▏ | 13/870 [00:07<03:32, 4.02it/s]
Epoch 1/20 - Training: 2%|▏ | 15/870 [00:09<07:24, 1.92it/s]
Epoch 1/20 - Training: 2%|▏ | 17/870 [00:09<05:22, 2.65it/s]
Epoch 1/20 - Training: 2%|▏ | 19/870 [00:09<03:59, 3.55it/s]
Epoch 1/20 - Training: 2%|▏ | 21/870 [00:09<03:03, 4.63it/s]
Epoch 1/20 - Training: 3%|▎ | 23/870 [00:09<02:24, 5.87it/s]
Epoch 1/20 - Training: 3%|▎ | 25/870 [00:11<05:50, 2.41it/s]
Epoch 1/20 - Training: 3%|▎ | 27/870 [00:12<04:21, 3.23it/s]
Epoch 1/20 - Training: 3%|▎ | 29/870 [00:12<03:19, 4.21it/s]
Epoch 1/20 - Training: 4%|▎ | 31/870 [00:12<02:36, 5.35it/s]
Epoch 1/20 - Training: 4%|▍ | 33/870 [00:12<02:06, 6.62it/s]
Epoch 1/20 - Training: 4%|▍ | 35/870 [00:14<05:30, 2.53it/s]
Epoch 1/20 - Training: 4%|▍ | 37/870 [00:14<04:10, 3.32it/s]
Epoch 1/20 - Training: 4%|▍ | 39/870 [00:14<03:14, 4.27it/s]
Epoch 1/20 - Training: 5%|▍ | 41/870 [00:14<02:35, 5.34it/s]
Epoch 1/20 - Training: 5%|▍ | 43/870 [00:15<02:06, 6.53it/s]
Epoch 1/20 - Training: 5%|▌ | 45/870 [00:17<05:38, 2.44it/s]
Epoch 1/20 - Training: 5%|▌ | 47/870 [00:17<04:31, 3.03it/s]
Epoch 1/20 - Training: 6%|▌ | 49/870 [00:17<03:26, 3.97it/s]
Epoch 1/20 - Training: 6%|▌ | 51/870 [00:17<02:41, 5.07it/s]
Epoch 1/20 - Training: 6%|▌ | 53/870 [00:17<02:09, 6.32it/s]
Epoch 1/20 - Training: 6%|▋ | 55/870 [00:19<05:15, 2.59it/s]
Epoch 1/20 - Training: 7%|▋ | 57/870 [00:19<04:23, 3.09it/s]
Epoch 1/20 - Training: 7%|▋ | 59/870 [00:20<03:21, 4.03it/s]
Epoch 1/20 - Training: 7%|▋ | 61/870 [00:20<02:38, 5.11it/s]
Epoch 1/20 - Training: 7%|▋ | 63/870 [00:20<02:07, 6.33it/s]
Epoch 1/20 - Training: 7%|▋ | 65/870 [00:22<05:07, 2.62it/s]
Epoch 1/20 - Training: 8%|▊ | 67/870 [00:22<04:19, 3.10it/s]
Epoch 1/20 - Training: 8%|▊ | 69/870 [00:22<03:19, 4.01it/s]
Epoch 1/20 - Training: 8%|▊ | 71/870 [00:22<02:38, 5.03it/s]
Epoch 1/20 - Training: 8%|▊ | 73/870 [00:23<02:09, 6.14it/s]
Epoch 1/20 - Training: 9%|▊ | 75/870 [00:24<04:56, 2.68it/s]
Epoch 1/20 - Training: 9%|▉ | 77/870 [00:25<04:07, 3.20it/s]
Epoch 1/20 - Training: 9%|▉ | 79/870 [00:25<03:10, 4.16it/s]
Epoch 1/20 - Training: 9%|▉ | 81/870 [00:25<02:29, 5.26it/s]
Epoch 1/20 - Training: 10%|▉ | 83/870 [00:25<02:01, 6.48it/s]
Epoch 1/20 - Training: 10%|▉ | 85/870 [00:27<04:52, 2.68it/s]
Epoch 1/20 - Training: 10%|█ | 87/870 [00:27<04:01, 3.24it/s]
Epoch 1/20 - Training: 10%|█ | 89/870 [00:27<03:05, 4.21it/s]
Epoch 1/20 - Training: 10%|█ | 91/870 [00:27<02:26, 5.31it/s]
Epoch 1/20 - Training: 11%|█ | 93/870 [00:28<01:59, 6.51it/s]
Epoch 1/20 - Training: 11%|█ | 95/870 [00:29<04:47, 2.70it/s]
Epoch 1/20 - Training: 11%|█ | 97/870 [00:30<04:02, 3.19it/s]
Epoch 1/20 - Training: 11%|█▏ | 99/870 [00:30<03:06, 4.14it/s]
Epoch 1/20 - Training: 12%|█▏ | 101/870 [00:30<02:27, 5.21it/s]
Epoch 1/20 - Training: 12%|█▏ | 103/870 [00:30<01:59, 6.42it/s]
Epoch 1/20 - Training: 12%|█▏ | 105/870 [00:32<04:39, 2.74it/s]
Epoch 1/20 - Training: 12%|█▏ | 107/870 [00:32<03:57, 3.21it/s]
Epoch 1/20 - Training: 13%|█▎ | 109/870 [00:32<03:02, 4.16it/s]
Epoch 1/20 - Training: 13%|█▎ | 111/870 [00:32<02:25, 5.22it/s]
Epoch 1/20 - Training: 13%|█▎ | 113/870 [00:33<01:58, 6.41it/s]
Epoch 1/20 - Training: 13%|█▎ | 115/870 [00:34<04:35, 2.74it/s]
Epoch 1/20 - Training: 13%|█▎ | 117/870 [00:35<03:50, 3.26it/s]
Epoch 1/20 - Training: 14%|█▎ | 119/870 [00:35<02:57, 4.22it/s]
Epoch 1/20 - Training: 14%|█▍ | 121/870 [00:35<02:21, 5.29it/s]
Epoch 1/20 - Training: 14%|█▍ | 123/870 [00:35<01:53, 6.56it/s]
Epoch 1/20 - Training: 14%|█▍ | 125/870 [00:37<04:41, 2.64it/s]
Epoch 1/20 - Training: 15%|█▍ | 127/870 [00:37<03:42, 3.34it/s]
Epoch 1/20 - Training: 15%|█▍ | 129/870 [00:37<02:50, 4.33it/s]
Epoch 1/20 - Training: 15%|█▌ | 131/870 [00:37<02:15, 5.47it/s]
Epoch 1/20 - Training: 15%|█▌ | 133/870 [00:38<01:50, 6.69it/s]
Epoch 1/20 - Training: 16%|█▌ | 135/870 [00:39<04:48, 2.55it/s]
Epoch 1/20 - Training: 16%|█▌ | 137/870 [00:40<03:37, 3.36it/s]
Epoch 1/20 - Training: 16%|█▌ | 139/870 [00:40<02:48, 4.34it/s]
Epoch 1/20 - Training: 16%|█▌ | 141/870 [00:40<02:13, 5.46it/s]
Epoch 1/20 - Training: 16%|█▋ | 143/870 [00:40<01:49, 6.61it/s]
Epoch 1/20 - Training: 17%|█▋ | 145/870 [00:42<04:48, 2.51it/s]
Epoch 1/20 - Training: 17%|█▋ | 147/870 [00:42<03:37, 3.32it/s]
Epoch 1/20 - Training: 17%|█▋ | 149/870 [00:42<02:47, 4.30it/s]
Epoch 1/20 - Training: 17%|█▋ | 151/870 [00:42<02:12, 5.44it/s]
Epoch 1/20 - Training: 18%|█▊ | 153/870 [00:43<01:47, 6.66it/s]
Epoch 1/20 - Training: 18%|█▊ | 155/870 [00:45<05:02, 2.37it/s]
Epoch 1/20 - Training: 18%|█▊ | 157/870 [00:45<03:47, 3.14it/s]
Epoch 1/20 - Training: 18%|█▊ | 159/870 [00:45<02:53, 4.09it/s]
Epoch 1/20 - Training: 19%|█▊ | 161/870 [00:45<02:16, 5.18it/s]
Epoch 1/20 - Training: 19%|█▊ | 163/870 [00:45<01:51, 6.36it/s]
Epoch 1/20 - Training: 19%|█▉ | 165/870 [00:47<04:49, 2.43it/s]
Epoch 1/20 - Training: 19%|█▉ | 167/870 [00:47<03:37, 3.23it/s]
Epoch 1/20 - Training: 19%|█▉ | 169/870 [00:48<02:46, 4.21it/s]
Epoch 1/20 - Training: 20%|█▉ | 171/870 [00:48<02:11, 5.31it/s]
Epoch 1/20 - Training: 20%|█▉ | 173/870 [00:48<01:46, 6.51it/s]
Epoch 1/20 - Training: 20%|██ | 175/870 [00:50<04:52, 2.38it/s]
Epoch 1/20 - Training: 20%|██ | 177/870 [00:50<03:39, 3.16it/s]
Epoch 1/20 - Training: 21%|██ | 179/870 [00:50<02:46, 4.14it/s]
Epoch 1/20 - Training: 21%|██ | 181/870 [00:50<02:12, 5.22it/s]
Epoch 1/20 - Training: 21%|██ | 183/870 [00:51<01:47, 6.40it/s]
Epoch 1/20 - Training: 21%|██▏ | 185/870 [00:53<04:37, 2.47it/s]
Epoch 1/20 - Training: 21%|██▏ | 187/870 [00:53<03:27, 3.28it/s]
Epoch 1/20 - Training: 22%|██▏ | 189/870 [00:53<02:38, 4.28it/s]
Epoch 1/20 - Training: 22%|██▏ | 191/870 [00:53<02:05, 5.39it/s]
Epoch 1/20 - Training: 22%|██▏ | 193/870 [00:53<01:42, 6.59it/s]
Epoch 1/20 - Training: 22%|██▏ | 195/870 [00:55<04:34, 2.46it/s]
Epoch 1/20 - Training: 23%|██▎ | 197/870 [00:55<03:26, 3.27it/s]
Epoch 1/20 - Training: 23%|██▎ | 199/870 [00:55<02:37, 4.25it/s]
Epoch 1/20 - Training: 23%|██▎ | 201/870 [00:56<02:04, 5.36it/s]
Epoch 1/20 - Training: 23%|██▎ | 203/870 [00:56<01:41, 6.55it/s]
Epoch 1/20 - Training: 24%|██▎ | 205/870 [00:58<04:34, 2.42it/s]
Epoch 1/20 - Training: 24%|██▍ | 207/870 [00:58<03:29, 3.17it/s]
Epoch 1/20 - Training: 24%|██▍ | 209/870 [00:58<02:44, 4.01it/s]
Epoch 1/20 - Training: 24%|██▍ | 211/870 [00:58<02:14, 4.90it/s]
Epoch 1/20 - Training: 24%|██▍ | 213/870 [00:58<01:53, 5.78it/s]
Epoch 1/20 - Training: 25%|██▍ | 215/870 [01:00<04:14, 2.58it/s]
Epoch 1/20 - Training: 25%|██▍ | 217/870 [01:00<03:11, 3.41it/s]
Epoch 1/20 - Training: 25%|██▌ | 219/870 [01:01<02:27, 4.42it/s]
Epoch 1/20 - Training: 25%|██▌ | 221/870 [01:01<01:57, 5.54it/s]
Epoch 1/20 - Training: 26%|██▌ | 223/870 [01:01<01:36, 6.71it/s]
Epoch 1/20 - Training: 26%|██▌ | 225/870 [01:03<05:05, 2.11it/s]
Epoch 1/20 - Training: 26%|██▌ | 227/870 [01:03<03:48, 2.82it/s]
Epoch 1/20 - Training: 26%|██▋ | 229/870 [01:04<02:52, 3.72it/s]
Epoch 1/20 - Training: 27%|██▋ | 231/870 [01:04<02:13, 4.78it/s]
Epoch 1/20 - Training: 27%|██▋ | 233/870 [01:04<01:46, 5.99it/s]
Epoch 1/20 - Training: 27%|██▋ | 235/870 [01:07<06:03, 1.75it/s]
Epoch 1/20 - Training: 27%|██▋ | 237/870 [01:07<04:27, 2.37it/s]
Epoch 1/20 - Training: 27%|██▋ | 239/870 [01:07<03:21, 3.14it/s]
Epoch 1/20 - Training: 28%|██▊ | 241/870 [01:07<02:33, 4.09it/s]
Epoch 1/20 - Training: 28%|██▊ | 243/870 [01:08<01:59, 5.23it/s]
Epoch 1/20 - Training: 28%|██▊ | 245/870 [01:10<05:58, 1.75it/s]
Epoch 1/20 - Training: 28%|██▊ | 247/870 [01:11<04:22, 2.37it/s]
Epoch 1/20 - Training: 29%|██▊ | 249/870 [01:11<03:16, 3.16it/s]
Epoch 1/20 - Training: 29%|██▉ | 251/870 [01:11<02:29, 4.13it/s]
Epoch 1/20 - Training: 29%|██▉ | 253/870 [01:11<01:57, 5.27it/s]
Epoch 1/20 - Training: 29%|██▉ | 255/870 [01:14<06:04, 1.69it/s]
Epoch 1/20 - Training: 30%|██▉ | 257/870 [01:14<04:27, 2.29it/s]
Epoch 1/20 - Training: 30%|██▉ | 259/870 [01:14<03:19, 3.06it/s]
Epoch 1/20 - Training: 30%|███ | 261/870 [01:14<02:31, 4.01it/s]
Epoch 1/20 - Training: 30%|███ | 263/870 [01:15<01:59, 5.09it/s]
Epoch 1/20 - Training: 30%|███ | 265/870 [01:18<06:13, 1.62it/s]
Epoch 1/20 - Training: 31%|███ | 267/870 [01:18<04:33, 2.20it/s]
Epoch 1/20 - Training: 31%|███ | 269/870 [01:18<03:23, 2.95it/s]
Epoch 1/20 - Training: 31%|███ | 271/870 [01:18<02:34, 3.87it/s]
Epoch 1/20 - Training: 31%|███▏ | 273/870 [01:18<02:00, 4.96it/s]
Epoch 1/20 - Training: 32%|███▏ | 275/870 [01:22<06:19, 1.57it/s]
Epoch 1/20 - Training: 32%|███▏ | 277/870 [01:22<04:38, 2.13it/s]
Epoch 1/20 - Training: 32%|███▏ | 279/870 [01:22<03:27, 2.85it/s]
Epoch 1/20 - Training: 32%|███▏ | 281/870 [01:22<02:36, 3.76it/s]
Epoch 1/20 - Training: 33%|███▎ | 283/870 [01:22<02:01, 4.84it/s]
Epoch 1/20 - Training: 33%|███▎ | 285/870 [01:26<06:33, 1.48it/s]
Epoch 1/20 - Training: 33%|███▎ | 287/870 [01:26<04:48, 2.02it/s]
Epoch 1/20 - Training: 33%|███▎ | 289/870 [01:26<03:33, 2.72it/s]
Epoch 1/20 - Training: 33%|███▎ | 291/870 [01:26<02:41, 3.59it/s]
Epoch 1/20 - Training: 34%|███▎ | 293/870 [01:26<02:05, 4.58it/s]
Epoch 1/20 - Training: 34%|███▍ | 295/870 [01:30<06:33, 1.46it/s]
Epoch 1/20 - Training: 34%|███▍ | 297/870 [01:30<04:46, 2.00it/s]
Epoch 1/20 - Training: 34%|███▍ | 299/870 [01:30<03:32, 2.69it/s]
Epoch 1/20 - Training: 35%|███▍ | 301/870 [01:30<02:39, 3.56it/s]
Epoch 1/20 - Training: 35%|███▍ | 303/870 [01:31<02:04, 4.55it/s]
Epoch 1/20 - Training: 35%|███▌ | 305/870 [01:35<07:05, 1.33it/s]
Epoch 1/20 - Training: 35%|███▌ | 307/870 [01:35<05:09, 1.82it/s]
Epoch 1/20 - Training: 36%|███▌ | 309/870 [01:35<03:48, 2.45it/s]
Epoch 1/20 - Training: 36%|███▌ | 311/870 [01:35<02:51, 3.26it/s]
Epoch 1/20 - Training: 36%|███▌ | 313/870 [01:35<02:11, 4.24it/s]
Epoch 1/20 - Training: 36%|███▌ | 315/870 [01:39<07:08, 1.30it/s]
Epoch 1/20 - Training: 36%|███▋ | 317/870 [01:39<05:12, 1.77it/s]
Epoch 1/20 - Training: 37%|███▋ | 319/870 [01:39<03:52, 2.37it/s]
Epoch 1/20 - Training: 37%|███▋ | 321/870 [01:40<02:57, 3.09it/s]
Epoch 1/20 - Training: 37%|███▋ | 323/870 [01:40<02:18, 3.96it/s]
Epoch 1/20 - Training: 37%|███▋ | 325/870 [01:44<06:54, 1.32it/s]
Epoch 1/20 - Training: 38%|███▊ | 327/870 [01:44<05:01, 1.80it/s]
Epoch 1/20 - Training: 38%|███▊ | 329/870 [01:44<03:45, 2.40it/s]
Epoch 1/20 - Training: 38%|███▊ | 331/870 [01:44<02:52, 3.13it/s]
Epoch 1/20 - Training: 38%|███▊ | 333/870 [01:44<02:13, 4.01it/s]
Epoch 1/20 - Training: 39%|███▊ | 335/870 [01:48<06:46, 1.32it/s]
Epoch 1/20 - Training: 39%|███▊ | 337/870 [01:48<04:55, 1.80it/s]
Epoch 1/20 - Training: 39%|███▉ | 339/870 [01:49<03:40, 2.41it/s]
Epoch 1/20 - Training: 39%|███▉ | 341/870 [01:49<02:47, 3.15it/s]
Epoch 1/20 - Training: 39%|███▉ | 343/870 [01:49<02:11, 4.01it/s]
Epoch 1/20 - Training: 40%|███▉ | 345/870 [01:53<06:20, 1.38it/s]
Epoch 1/20 - Training: 40%|███▉ | 347/870 [01:53<04:36, 1.89it/s]
Epoch 1/20 - Training: 40%|████ | 349/870 [01:53<03:26, 2.52it/s]
Epoch 1/20 - Training: 40%|████ | 351/870 [01:53<02:37, 3.29it/s]
Epoch 1/20 - Training: 41%|████ | 353/870 [01:53<02:03, 4.19it/s]
Epoch 1/20 - Training: 41%|████ | 355/870 [01:57<06:17, 1.37it/s]
Epoch 1/20 - Training: 41%|████ | 357/870 [01:57<04:35, 1.86it/s]
Epoch 1/20 - Training: 41%|████▏ | 359/870 [01:57<03:25, 2.49it/s]
Epoch 1/20 - Training: 41%|████▏ | 361/870 [01:58<02:36, 3.24it/s]
Epoch 1/20 - Training: 42%|████▏ | 363/870 [01:58<02:03, 4.09it/s]
Epoch 1/20 - Training: 42%|████▏ | 365/870 [02:02<06:25, 1.31it/s]
Epoch 1/20 - Training: 42%|████▏ | 367/870 [02:02<04:40, 1.79it/s]
Epoch 1/20 - Training: 42%|████▏ | 369/870 [02:02<03:29, 2.40it/s]
Epoch 1/20 - Training: 43%|████▎ | 371/870 [02:02<02:38, 3.14it/s]
Epoch 1/20 - Training: 43%|████▎ | 373/870 [02:02<02:05, 3.97it/s]
Epoch 1/20 - Training: 43%|████▎ | 375/870 [02:06<06:19, 1.30it/s]
Epoch 1/20 - Training: 43%|████▎ | 377/870 [02:07<04:35, 1.79it/s]
Epoch 1/20 - Training: 44%|████▎ | 379/870 [02:07<03:25, 2.39it/s]
Epoch 1/20 - Training: 44%|████▍ | 381/870 [02:07<02:35, 3.14it/s]
Epoch 1/20 - Training: 44%|████▍ | 383/870 [02:07<02:05, 3.89it/s]
Epoch 1/20 - Training: 44%|████▍ | 384/870 [02:07<01:52, 4.31it/s]
Epoch 1/20 - Training: 44%|████▍ | 385/870 [02:11<07:45, 1.04it/s]
Epoch 1/20 - Training: 44%|████▍ | 387/870 [02:11<05:08, 1.57it/s]
Epoch 1/20 - Training: 45%|████▍ | 389/870 [02:11<03:36, 2.23it/s]
Epoch 1/20 - Training: 45%|████▍ | 391/870 [02:12<02:39, 3.00it/s]
Epoch 1/20 - Training: 45%|████▌ | 393/870 [02:12<02:00, 3.96it/s]
Epoch 1/20 - Training: 45%|████▌ | 395/870 [02:16<06:14, 1.27it/s]
Epoch 1/20 - Training: 46%|████▌ | 397/870 [02:16<04:28, 1.76it/s]
Epoch 1/20 - Training: 46%|████▌ | 399/870 [02:16<03:18, 2.37it/s]
Epoch 1/20 - Training: 46%|████▌ | 401/870 [02:16<02:30, 3.12it/s]
Epoch 1/20 - Training: 46%|████▋ | 403/870 [02:16<01:55, 4.03it/s]
Epoch 1/20 - Training: 47%|████▋ | 405/870 [02:20<06:03, 1.28it/s]
Epoch 1/20 - Training: 47%|████▋ | 407/870 [02:21<04:23, 1.76it/s]
Epoch 1/20 - Training: 47%|████▋ | 409/870 [02:21<03:16, 2.35it/s]
Epoch 1/20 - Training: 47%|████▋ | 411/870 [02:21<02:29, 3.07it/s]
Epoch 1/20 - Training: 47%|████▋ | 413/870 [02:21<01:55, 3.96it/s]
Epoch 1/20 - Training: 48%|████▊ | 415/870 [02:25<05:49, 1.30it/s]
Epoch 1/20 - Training: 48%|████▊ | 417/870 [02:25<04:13, 1.78it/s]
Epoch 1/20 - Training: 48%|████▊ | 419/870 [02:25<03:08, 2.39it/s]
Epoch 1/20 - Training: 48%|████▊ | 421/870 [02:26<02:23, 3.13it/s]
Epoch 1/20 - Training: 49%|████▊ | 423/870 [02:26<01:51, 4.01it/s]
Epoch 1/20 - Training: 49%|████▉ | 425/870 [02:30<05:45, 1.29it/s]
Epoch 1/20 - Training: 49%|████▉ | 427/870 [02:30<04:11, 1.76it/s]
Epoch 1/20 - Training: 49%|████▉ | 429/870 [02:30<03:07, 2.35it/s]
Epoch 1/20 - Training: 50%|████▉ | 431/870 [02:30<02:22, 3.08it/s]
Epoch 1/20 - Training: 50%|████▉ | 433/870 [02:30<01:49, 3.99it/s]
Epoch 1/20 - Training: 50%|█████ | 435/870 [02:34<05:34, 1.30it/s]
Epoch 1/20 - Training: 50%|█████ | 437/870 [02:35<04:02, 1.78it/s]
Epoch 1/20 - Training: 50%|█████ | 439/870 [02:35<03:00, 2.39it/s]
Epoch 1/20 - Training: 51%|█████ | 441/870 [02:35<02:16, 3.14it/s]
Epoch 1/20 - Training: 51%|█████ | 443/870 [02:35<01:45, 4.05it/s]
Epoch 1/20 - Training: 51%|█████ | 445/870 [02:39<05:29, 1.29it/s]
Epoch 1/20 - Training: 51%|█████▏ | 447/870 [02:39<03:58, 1.77it/s]
Epoch 1/20 - Training: 52%|█████▏ | 449/870 [02:39<02:58, 2.36it/s]
Epoch 1/20 - Training: 52%|█████▏ | 451/870 [02:40<02:16, 3.08it/s]
Epoch 1/20 - Training: 52%|█████▏ | 453/870 [02:40<01:45, 3.96it/s]
Epoch 1/20 - Training: 52%|█████▏ | 455/870 [02:43<05:08, 1.35it/s]
Epoch 1/20 - Training: 53%|█████▎ | 457/870 [02:44<03:44, 1.84it/s]
Epoch 1/20 - Training: 53%|█████▎ | 459/870 [02:44<02:47, 2.46it/s]
Epoch 1/20 - Training: 53%|█████▎ | 461/870 [02:44<02:07, 3.20it/s]
Epoch 1/20 - Training: 53%|█████▎ | 463/870 [02:44<01:39, 4.11it/s]
Epoch 1/20 - Training: 53%|█████▎ | 465/870 [02:48<05:05, 1.33it/s]
Epoch 1/20 - Training: 54%|█████▎ | 467/870 [02:48<03:41, 1.82it/s]
Epoch 1/20 - Training: 54%|█████▍ | 469/870 [02:48<02:44, 2.43it/s]
Epoch 1/20 - Training: 54%|█████▍ | 471/870 [02:49<02:05, 3.18it/s]
Epoch 1/20 - Training: 54%|█████▍ | 473/870 [02:49<01:37, 4.08it/s]
Epoch 1/20 - Training: 55%|█████▍ | 475/870 [02:53<05:07, 1.28it/s]
Epoch 1/20 - Training: 55%|█████▍ | 477/870 [02:53<03:43, 1.76it/s]
Epoch 1/20 - Training: 55%|█████▌ | 479/870 [02:53<02:46, 2.35it/s]
Epoch 1/20 - Training: 55%|█████▌ | 481/870 [02:53<02:06, 3.07it/s]
Epoch 1/20 - Training: 56%|█████▌ | 483/870 [02:53<01:38, 3.94it/s]
Epoch 1/20 - Training: 56%|█████▌ | 485/870 [02:57<04:59, 1.28it/s]
Epoch 1/20 - Training: 56%|█████▌ | 487/870 [02:58<03:37, 1.76it/s]
Epoch 1/20 - Training: 56%|█████▌ | 489/870 [02:58<02:42, 2.35it/s]
Epoch 1/20 - Training: 56%|█████▋ | 491/870 [02:58<02:02, 3.09it/s]
Epoch 1/20 - Training: 57%|█████▋ | 493/870 [02:58<01:35, 3.94it/s]
Epoch 1/20 - Training: 57%|█████▋ | 495/870 [03:02<04:46, 1.31it/s]
Epoch 1/20 - Training: 57%|█████▋ | 497/870 [03:02<03:28, 1.79it/s]
Epoch 1/20 - Training: 57%|█████▋ | 499/870 [03:02<02:34, 2.40it/s]
Epoch 1/20 - Training: 57%|█████▋ | 500/870 [03:03<02:13, 2.77it/s]
Epoch 1/20 - Training: 58%|█████▊ | 502/870 [03:03<01:39, 3.71it/s]
Epoch 1/20 - Training: 58%|█████▊ | 504/870 [03:03<01:17, 4.70it/s]
Epoch 1/20 - Training: 58%|█████▊ | 506/870 [03:07<04:42, 1.29it/s]
Epoch 1/20 - Training: 58%|█████▊ | 508/870 [03:07<03:23, 1.78it/s]
Epoch 1/20 - Training: 59%|█████▊ | 510/870 [03:07<02:30, 2.40it/s]
Epoch 1/20 - Training: 59%|█████▉ | 512/870 [03:07<01:53, 3.15it/s]
Epoch 1/20 - Training: 59%|█████▉ | 514/870 [03:08<01:28, 4.02it/s]
Epoch 1/20 - Training: 59%|█████▉ | 516/870 [03:12<04:32, 1.30it/s]
Epoch 1/20 - Training: 60%|█████▉ | 518/870 [03:12<03:17, 1.78it/s]
Epoch 1/20 - Training: 60%|█████▉ | 519/870 [03:12<02:48, 2.08it/s]
Epoch 1/20 - Training: 60%|█████▉ | 521/870 [03:12<02:02, 2.84it/s]
Epoch 1/20 - Training: 60%|██████ | 523/870 [03:12<01:32, 3.74it/s]
Epoch 1/20 - Training: 60%|██████ | 525/870 [03:16<04:42, 1.22it/s]
Epoch 1/20 - Training: 61%|██████ | 527/870 [03:16<03:21, 1.70it/s]
Epoch 1/20 - Training: 61%|██████ | 529/870 [03:17<02:28, 2.30it/s]
Epoch 1/20 - Training: 61%|██████ | 531/870 [03:17<01:52, 3.02it/s]
Epoch 1/20 - Training: 61%|██████▏ | 533/870 [03:17<01:26, 3.91it/s]
Epoch 1/20 - Training: 61%|██████▏ | 535/870 [03:21<04:23, 1.27it/s]
Epoch 1/20 - Training: 62%|██████▏ | 537/870 [03:21<03:10, 1.75it/s]
Epoch 1/20 - Training: 62%|██████▏ | 539/870 [03:21<02:21, 2.34it/s]
Epoch 1/20 - Training: 62%|██████▏ | 541/870 [03:21<01:47, 3.05it/s]
Epoch 1/20 - Training: 62%|██████▏ | 543/870 [03:22<01:23, 3.90it/s]
Epoch 1/20 - Training: 63%|██████▎ | 545/870 [03:26<04:10, 1.30it/s]
Epoch 1/20 - Training: 63%|██████▎ | 547/870 [03:26<03:01, 1.78it/s]
Epoch 1/20 - Training: 63%|██████▎ | 549/870 [03:26<02:15, 2.36it/s]
Epoch 1/20 - Training: 63%|██████▎ | 551/870 [03:26<01:43, 3.08it/s]
Epoch 1/20 - Training: 63%|██████▎ | 552/870 [03:26<01:30, 3.51it/s]
Epoch 1/20 - Training: 64%|██████▎ | 554/870 [03:26<01:09, 4.58it/s]
Epoch 1/20 - Training: 64%|██████▍ | 556/870 [03:30<04:06, 1.28it/s]
Epoch 1/20 - Training: 64%|██████▍ | 558/870 [03:30<02:55, 1.78it/s]
Epoch 1/20 - Training: 64%|██████▍ | 560/870 [03:31<02:07, 2.42it/s]
Epoch 1/20 - Training: 65%|██████▍ | 562/870 [03:31<01:35, 3.21it/s]
Epoch 1/20 - Training: 65%|██████▍ | 564/870 [03:31<01:15, 4.05it/s]
Epoch 1/20 - Training: 65%|██████▌ | 566/870 [03:35<03:58, 1.27it/s]
Epoch 1/20 - Training: 65%|██████▌ | 568/870 [03:35<02:52, 1.75it/s]
Epoch 1/20 - Training: 65%|██████▌ | 569/870 [03:35<02:27, 2.04it/s]
Epoch 1/20 - Training: 66%|██████▌ | 571/870 [03:35<01:46, 2.82it/s]
Epoch 1/20 - Training: 66%|██████▌ | 573/870 [03:36<01:20, 3.70it/s]
Epoch 1/20 - Training: 66%|██████▌ | 575/870 [03:40<03:54, 1.26it/s]
Epoch 1/20 - Training: 66%|██████▋ | 577/870 [03:40<02:47, 1.75it/s]
Epoch 1/20 - Training: 67%|██████▋ | 579/870 [03:40<02:04, 2.35it/s]
Epoch 1/20 - Training: 67%|██████▋ | 581/870 [03:40<01:34, 3.07it/s]
Epoch 1/20 - Training: 67%|██████▋ | 582/870 [03:40<01:22, 3.50it/s]
Epoch 1/20 - Training: 67%|██████▋ | 584/870 [03:40<01:02, 4.55it/s]
Epoch 1/20 - Training: 67%|██████▋ | 585/870 [03:44<04:10, 1.14it/s]
Epoch 1/20 - Training: 67%|██████▋ | 587/870 [03:44<02:47, 1.69it/s]
Epoch 1/20 - Training: 68%|██████▊ | 589/870 [03:44<02:00, 2.34it/s]
Epoch 1/20 - Training: 68%|██████▊ | 591/870 [03:45<01:28, 3.14it/s]
Epoch 1/20 - Training: 68%|██████▊ | 593/870 [03:45<01:08, 4.03it/s]
Epoch 1/20 - Training: 68%|██████▊ | 595/870 [03:48<03:21, 1.36it/s]
Epoch 1/20 - Training: 69%|██████▊ | 597/870 [03:48<02:25, 1.87it/s]
Epoch 1/20 - Training: 69%|██████▊ | 598/870 [03:49<02:04, 2.19it/s]
Epoch 1/20 - Training: 69%|██████▉ | 600/870 [03:49<01:31, 2.96it/s]
Epoch 1/20 - Training: 69%|██████▉ | 601/870 [03:49<01:18, 3.43it/s]
Epoch 1/20 - Training: 69%|██████▉ | 602/870 [03:49<01:07, 3.98it/s]
Epoch 1/20 - Training: 69%|██████▉ | 603/870 [03:49<00:58, 4.60it/s]
Epoch 1/20 - Training: 70%|██████▉ | 605/870 [03:53<03:55, 1.13it/s]
Epoch 1/20 - Training: 70%|██████▉ | 607/870 [03:53<02:35, 1.69it/s]
Epoch 1/20 - Training: 70%|███████ | 609/870 [03:53<01:50, 2.36it/s]
Epoch 1/20 - Training: 70%|███████ | 610/870 [03:53<01:33, 2.77it/s]
Epoch 1/20 - Training: 70%|███████ | 612/870 [03:54<01:08, 3.78it/s]
Epoch 1/20 - Training: 70%|███████ | 613/870 [03:54<00:59, 4.31it/s]
Epoch 1/20 - Training: 71%|███████ | 615/870 [03:57<03:30, 1.21it/s]
Epoch 1/20 - Training: 71%|███████ | 617/870 [03:57<02:23, 1.76it/s]
Epoch 1/20 - Training: 71%|███████ | 619/870 [03:58<01:43, 2.43it/s]
Epoch 1/20 - Training: 71%|███████▏ | 620/870 [03:58<01:28, 2.83it/s]
Epoch 1/20 - Training: 71%|███████▏ | 621/870 [03:58<01:14, 3.33it/s]
Epoch 1/20 - Training: 71%|███████▏ | 622/870 [03:58<01:03, 3.91it/s]
Epoch 1/20 - Training: 72%|███████▏ | 624/870 [03:58<00:46, 5.27it/s]
Epoch 1/20 - Training: 72%|███████▏ | 625/870 [04:02<03:47, 1.08it/s]
Epoch 1/20 - Training: 72%|███████▏ | 627/870 [04:02<02:26, 1.66it/s]
Epoch 1/20 - Training: 72%|███████▏ | 629/870 [04:02<01:41, 2.38it/s]
Epoch 1/20 - Training: 72%|███████▏ | 630/870 [04:02<01:26, 2.78it/s]
Epoch 1/20 - Training: 73%|███████▎ | 631/870 [04:02<01:12, 3.31it/s]
Epoch 1/20 - Training: 73%|███████▎ | 633/870 [04:02<00:52, 4.54it/s]
Epoch 1/20 - Training: 73%|███████▎ | 634/870 [04:03<00:46, 5.10it/s]
Epoch 1/20 - Training: 73%|███████▎ | 635/870 [04:06<03:48, 1.03it/s]
Epoch 1/20 - Training: 73%|███████▎ | 637/870 [04:06<02:22, 1.64it/s]
Epoch 1/20 - Training: 73%|███████▎ | 639/870 [04:06<01:37, 2.36it/s]
Epoch 1/20 - Training: 74%|███████▎ | 641/870 [04:07<01:11, 3.19it/s]
Epoch 1/20 - Training: 74%|███████▍ | 642/870 [04:07<01:02, 3.68it/s]
Epoch 1/20 - Training: 74%|███████▍ | 643/870 [04:07<00:53, 4.22it/s]
Epoch 1/20 - Training: 74%|███████▍ | 645/870 [04:10<02:59, 1.25it/s]
Epoch 1/20 - Training: 74%|███████▍ | 647/870 [04:10<02:01, 1.84it/s]
Epoch 1/20 - Training: 75%|███████▍ | 649/870 [04:11<01:26, 2.54it/s]
Epoch 1/20 - Training: 75%|███████▍ | 651/870 [04:11<01:05, 3.37it/s]
Epoch 1/20 - Training: 75%|███████▍ | 652/870 [04:11<00:56, 3.83it/s]
Epoch 1/20 - Training: 75%|███████▌ | 653/870 [04:11<00:49, 4.37it/s]
Epoch 1/20 - Training: 75%|███████▌ | 654/870 [04:11<00:43, 5.01it/s]
Epoch 1/20 - Training: 75%|███████▌ | 655/870 [04:15<03:38, 1.02s/it]
Epoch 1/20 - Training: 76%|███████▌ | 657/870 [04:15<02:13, 1.60it/s]
Epoch 1/20 - Training: 76%|███████▌ | 659/870 [04:15<01:30, 2.34it/s]
Epoch 1/20 - Training: 76%|███████▌ | 661/870 [04:15<01:05, 3.19it/s]
Epoch 1/20 - Training: 76%|███████▌ | 662/870 [04:15<00:56, 3.68it/s]
Epoch 1/20 - Training: 76%|███████▌ | 663/870 [04:15<00:52, 3.92it/s]
Epoch 1/20 - Training: 76%|███████▋ | 665/870 [04:19<02:49, 1.21it/s]
Epoch 1/20 - Training: 77%|███████▋ | 667/870 [04:19<01:53, 1.78it/s]
Epoch 1/20 - Training: 77%|███████▋ | 669/870 [04:19<01:21, 2.48it/s]
Epoch 1/20 - Training: 77%|███████▋ | 671/870 [04:19<01:00, 3.29it/s]
Epoch 1/20 - Training: 77%|███████▋ | 672/870 [04:20<00:52, 3.75it/s]
Epoch 1/20 - Training: 77%|███████▋ | 673/870 [04:20<00:53, 3.67it/s]
Epoch 1/20 - Training: 78%|███████▊ | 675/870 [04:23<02:38, 1.23it/s]
Epoch 1/20 - Training: 78%|███████▊ | 677/870 [04:23<01:47, 1.79it/s]
Epoch 1/20 - Training: 78%|███████▊ | 679/870 [04:24<01:17, 2.46it/s]
Epoch 1/20 - Training: 78%|███████▊ | 681/870 [04:24<00:57, 3.26it/s]
Epoch 1/20 - Training: 79%|███████▊ | 683/870 [04:24<00:52, 3.55it/s]
Epoch 1/20 - Training: 79%|███████▊ | 685/870 [04:27<02:09, 1.43it/s]
Epoch 1/20 - Training: 79%|███████▉ | 687/870 [04:28<01:33, 1.97it/s]
Epoch 1/20 - Training: 79%|███████▉ | 689/870 [04:28<01:09, 2.60it/s]
Epoch 1/20 - Training: 79%|███████▉ | 691/870 [04:28<00:53, 3.37it/s]
Epoch 1/20 - Training: 80%|███████▉ | 692/870 [04:28<00:46, 3.82it/s]
Epoch 1/20 - Training: 80%|███████▉ | 693/870 [04:29<00:58, 3.00it/s]
Epoch 1/20 - Training: 80%|███████▉ | 695/870 [04:32<02:15, 1.29it/s]
Epoch 1/20 - Training: 80%|████████ | 697/870 [04:32<01:32, 1.87it/s]
Epoch 1/20 - Training: 80%|████████ | 699/870 [04:32<01:06, 2.56it/s]
Epoch 1/20 - Training: 80%|████████ | 700/870 [04:32<00:57, 2.97it/s]
Epoch 1/20 - Training: 81%|████████ | 701/870 [04:32<00:48, 3.49it/s]
Epoch 1/20 - Training: 81%|████████ | 703/870 [04:33<00:57, 2.90it/s]
Epoch 1/20 - Training: 81%|████████ | 705/870 [04:36<02:01, 1.36it/s]
Epoch 1/20 - Training: 81%|████████▏ | 707/870 [04:36<01:24, 1.93it/s]
Epoch 1/20 - Training: 81%|████████▏ | 709/870 [04:37<01:01, 2.62it/s]
Epoch 1/20 - Training: 82%|████████▏ | 711/870 [04:37<00:46, 3.42it/s]
Epoch 1/20 - Training: 82%|████████▏ | 712/870 [04:37<00:40, 3.88it/s]
Epoch 1/20 - Training: 82%|████████▏ | 713/870 [04:38<01:00, 2.60it/s]
Epoch 1/20 - Training: 82%|████████▏ | 715/870 [04:41<01:59, 1.29it/s]
Epoch 1/20 - Training: 82%|████████▏ | 717/870 [04:41<01:21, 1.88it/s]
Epoch 1/20 - Training: 83%|████████▎ | 719/870 [04:41<00:58, 2.57it/s]
Epoch 1/20 - Training: 83%|████████▎ | 721/870 [04:41<00:43, 3.40it/s]
Epoch 1/20 - Training: 83%|████████▎ | 722/870 [04:41<00:38, 3.85it/s]
Epoch 1/20 - Training: 83%|████████▎ | 723/870 [04:42<00:57, 2.55it/s]
Epoch 1/20 - Training: 83%|████████▎ | 724/870 [04:42<00:47, 3.08it/s]
Epoch 1/20 - Training: 83%|████████▎ | 725/870 [04:45<02:12, 1.09it/s]
Epoch 1/20 - Training: 84%|████████▎ | 727/870 [04:45<01:22, 1.74it/s]
Epoch 1/20 - Training: 84%|████████▍ | 729/870 [04:45<00:56, 2.50it/s]
Epoch 1/20 - Training: 84%|████████▍ | 730/870 [04:45<00:47, 2.95it/s]
Epoch 1/20 - Training: 84%|████████▍ | 731/870 [04:46<00:39, 3.51it/s]
Epoch 1/20 - Training: 84%|████████▍ | 732/870 [04:46<00:33, 4.18it/s]
Epoch 1/20 - Training: 84%|████████▍ | 733/870 [04:46<00:54, 2.53it/s]
Epoch 1/20 - Training: 84%|████████▍ | 734/870 [04:47<00:42, 3.17it/s]
Epoch 1/20 - Training: 84%|████████▍ | 735/870 [04:49<02:14, 1.00it/s]
Epoch 1/20 - Training: 85%|████████▍ | 736/870 [04:49<01:39, 1.35it/s]
Epoch 1/20 - Training: 85%|████████▍ | 738/870 [04:50<00:59, 2.21it/s]
Epoch 1/20 - Training: 85%|████████▍ | 739/870 [04:50<00:48, 2.70it/s]
Epoch 1/20 - Training: 85%|████████▌ | 740/870 [04:50<00:39, 3.30it/s]
Epoch 1/20 - Training: 85%|████████▌ | 741/870 [04:50<00:32, 3.98it/s]
Epoch 1/20 - Training: 85%|████████▌ | 742/870 [04:50<00:26, 4.77it/s]
Epoch 1/20 - Training: 85%|████████▌ | 743/870 [04:51<00:46, 2.73it/s]
Epoch 1/20 - Training: 86%|████████▌ | 744/870 [04:51<00:36, 3.44it/s]
Epoch 1/20 - Training: 86%|████████▌ | 745/870 [04:54<02:13, 1.07s/it]
Epoch 1/20 - Training: 86%|████████▌ | 746/870 [04:54<01:37, 1.28it/s]
Epoch 1/20 - Training: 86%|████████▌ | 747/870 [04:54<01:11, 1.72it/s]
Epoch 1/20 - Training: 86%|████████▌ | 748/870 [04:54<00:53, 2.28it/s]
Epoch 1/20 - Training: 86%|████████▌ | 749/870 [04:54<00:41, 2.94it/s]
Epoch 1/20 - Training: 86%|████████▌ | 750/870 [04:54<00:32, 3.72it/s]
Epoch 1/20 - Training: 86%|████████▋ | 751/870 [04:54<00:26, 4.55it/s]
Epoch 1/20 - Training: 86%|████████▋ | 752/870 [04:55<00:22, 5.32it/s]
Epoch 1/20 - Training: 87%|████████▋ | 753/870 [04:56<00:47, 2.46it/s]
Epoch 1/20 - Training: 87%|████████▋ | 755/870 [04:59<01:59, 1.04s/it]
Epoch 1/20 - Training: 87%|████████▋ | 757/870 [04:59<01:15, 1.50it/s]
Epoch 1/20 - Training: 87%|████████▋ | 759/870 [04:59<00:51, 2.17it/s]
Epoch 1/20 - Training: 87%|████████▋ | 760/870 [05:00<00:42, 2.58it/s]
Epoch 1/20 - Training: 87%|████████▋ | 761/870 [05:00<00:35, 3.08it/s]
Epoch 1/20 - Training: 88%|████████▊ | 763/870 [05:01<00:44, 2.39it/s]
Epoch 1/20 - Training: 88%|████████▊ | 765/870 [05:04<01:35, 1.10it/s]
Epoch 1/20 - Training: 88%|████████▊ | 767/870 [05:05<01:04, 1.59it/s]
Epoch 1/20 - Training: 88%|████████▊ | 769/870 [05:05<00:46, 2.19it/s]
Epoch 1/20 - Training: 89%|████████▊ | 770/870 [05:05<00:38, 2.57it/s]
Epoch 1/20 - Training: 89%|████████▊ | 771/870 [05:05<00:32, 3.03it/s]
Epoch 1/20 - Training: 89%|████████▊ | 772/870 [05:05<00:27, 3.61it/s]
Epoch 1/20 - Training: 89%|████████▉ | 773/870 [05:06<00:47, 2.05it/s]
Epoch 1/20 - Training: 89%|████████▉ | 775/870 [05:10<01:35, 1.00s/it]
Epoch 1/20 - Training: 89%|████████▉ | 777/870 [05:10<01:01, 1.50it/s]
Epoch 1/20 - Training: 90%|████████▉ | 779/870 [05:10<00:42, 2.13it/s]
Epoch 1/20 - Training: 90%|████████▉ | 780/870 [05:10<00:35, 2.52it/s]
Epoch 1/20 - Training: 90%|████████▉ | 781/870 [05:10<00:29, 3.01it/s]
Epoch 1/20 - Training: 90%|█████████ | 783/870 [05:12<00:39, 2.22it/s]
Epoch 1/20 - Training: 90%|█████████ | 785/870 [05:15<01:16, 1.11it/s]
Epoch 1/20 - Training: 90%|█████████ | 787/870 [05:15<00:52, 1.59it/s]
Epoch 1/20 - Training: 91%|█████████ | 789/870 [05:15<00:36, 2.19it/s]
Epoch 1/20 - Training: 91%|█████████ | 790/870 [05:15<00:31, 2.57it/s]
Epoch 1/20 - Training: 91%|█████████ | 791/870 [05:16<00:25, 3.04it/s]
Epoch 1/20 - Training: 91%|█████████ | 793/870 [05:17<00:35, 2.18it/s]
Epoch 1/20 - Training: 91%|█████████▏| 795/870 [05:20<01:06, 1.12it/s]
Epoch 1/20 - Training: 92%|█████████▏| 797/870 [05:20<00:45, 1.60it/s]
Epoch 1/20 - Training: 92%|█████████▏| 799/870 [05:21<00:32, 2.19it/s]
Epoch 1/20 - Training: 92%|█████████▏| 800/870 [05:21<00:27, 2.56it/s]
Epoch 1/20 - Training: 92%|█████████▏| 801/870 [05:21<00:22, 3.03it/s]
Epoch 1/20 - Training: 92%|█████████▏| 803/870 [05:22<00:31, 2.15it/s]
Epoch 1/20 - Training: 93%|█████████▎| 805/870 [05:26<00:57, 1.12it/s]
Epoch 1/20 - Training: 93%|█████████▎| 807/870 [05:26<00:39, 1.60it/s]
Epoch 1/20 - Training: 93%|█████████▎| 809/870 [05:26<00:27, 2.19it/s]
Epoch 1/20 - Training: 93%|█████████▎| 810/870 [05:26<00:23, 2.56it/s]
Epoch 1/20 - Training: 93%|█████████▎| 811/870 [05:26<00:19, 3.03it/s]
Epoch 1/20 - Training: 93%|█████████▎| 813/870 [05:28<00:26, 2.12it/s]
Epoch 1/20 - Training: 94%|█████████▎| 815/870 [05:31<00:48, 1.14it/s]
Epoch 1/20 - Training: 94%|█████████▍| 817/870 [05:31<00:32, 1.63it/s]
Epoch 1/20 - Training: 94%|█████████▍| 819/870 [05:31<00:22, 2.24it/s]
Epoch 1/20 - Training: 94%|█████████▍| 820/870 [05:31<00:19, 2.62it/s]
Epoch 1/20 - Training: 94%|█████████▍| 821/870 [05:31<00:15, 3.09it/s]
Epoch 1/20 - Training: 95%|█████████▍| 823/870 [05:33<00:22, 2.06it/s]
Epoch 1/20 - Training: 95%|█████████▍| 825/870 [05:36<00:39, 1.13it/s]
Epoch 1/20 - Training: 95%|█████████▌| 827/870 [05:36<00:26, 1.61it/s]
Epoch 1/20 - Training: 95%|█████████▌| 829/870 [05:37<00:18, 2.20it/s]
Epoch 1/20 - Training: 95%|█████████▌| 830/870 [05:37<00:15, 2.58it/s]
Epoch 1/20 - Training: 96%|█████████▌| 831/870 [05:37<00:12, 3.05it/s]
Epoch 1/20 - Training: 96%|█████████▌| 833/870 [05:38<00:17, 2.06it/s]
Epoch 1/20 - Training: 96%|█████████▌| 835/870 [05:42<00:30, 1.14it/s]
Epoch 1/20 - Training: 96%|█████████▌| 837/870 [05:42<00:20, 1.63it/s]
Epoch 1/20 - Training: 96%|█████████▋| 839/870 [05:42<00:13, 2.23it/s]
Epoch 1/20 - Training: 97%|█████████▋| 840/870 [05:42<00:11, 2.61it/s]
Epoch 1/20 - Training: 97%|█████████▋| 841/870 [05:42<00:09, 3.09it/s]
Epoch 1/20 - Training: 97%|█████████▋| 843/870 [05:44<00:13, 2.03it/s]
Epoch 1/20 - Training: 97%|█████████▋| 845/870 [05:47<00:21, 1.15it/s]
Epoch 1/20 - Training: 97%|█████████▋| 847/870 [05:47<00:14, 1.63it/s]
Epoch 1/20 - Training: 98%|█████████▊| 849/870 [05:47<00:09, 2.24it/s]
Epoch 1/20 - Training: 98%|█████████▊| 850/870 [05:47<00:07, 2.61it/s]
Epoch 1/20 - Training: 98%|█████████▊| 851/870 [05:47<00:06, 3.08it/s]
Epoch 1/20 - Training: 98%|█████████▊| 853/870 [05:49<00:07, 2.20it/s]
Epoch 1/20 - Training: 98%|█████████▊| 855/870 [05:51<00:10, 1.50it/s]
Epoch 1/20 - Training: 99%|█████████▊| 857/870 [05:51<00:06, 2.09it/s]
Epoch 1/20 - Training: 99%|█████████▊| 858/870 [05:51<00:04, 2.47it/s]
Epoch 1/20 - Training: 99%|█████████▊| 859/870 [05:51<00:03, 2.92it/s]
Epoch 1/20 - Training: 99%|█████████▉| 860/870 [05:51<00:02, 3.49it/s]
Epoch 1/20 - Training: 99%|█████████▉| 861/870 [05:51<00:02, 4.14it/s]
Epoch 1/20 - Training: 99%|█████████▉| 863/870 [05:52<00:01, 3.97it/s]
Epoch 1/20 - Training: 99%|█████████▉| 865/870 [05:53<00:01, 2.66it/s]
Epoch 1/20 - Training: 100%|█████████▉| 867/870 [05:53<00:00, 3.64it/s]
Epoch 1/20 - Training: 100%|█████████▉| 869/870 [05:54<00:00, 4.68it/s]/admin/home-ckadirt/foundation_env/lib/python3.11/site-packages/torch/nn/modules/loss.py:538: UserWarning: Using a target size (torch.Size([70, 1])) that is different to the input size (torch.Size([70])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
return F.mse_loss(input, target, reduction=self.reduction)
Epoch 1/20 - Training: 100%|██████████| 870/870 [05:54<00:00, 2.46it/s]
Epoch 1/20 - Validation: 0%| | 0/95 [00:00<?, ?it/s]
Epoch 1/20 - Validation: 1%| | 1/95 [00:02<03:15, 2.08s/it]
Epoch 1/20 - Validation: 2%|▏ | 2/95 [00:04<03:18, 2.13s/it]
Epoch 1/20 - Validation: 3%|▎ | 3/95 [00:06<03:16, 2.13s/it]
Epoch 1/20 - Validation: 4%|▍ | 4/95 [00:08<03:25, 2.26s/it]
Epoch 1/20 - Validation: 5%|▌ | 5/95 [00:11<03:26, 2.29s/it]
Epoch 1/20 - Validation: 6%|▋ | 6/95 [00:13<03:18, 2.23s/it]
Epoch 1/20 - Validation: 7%|▋ | 7/95 [00:15<03:15, 2.22s/it]
Epoch 1/20 - Validation: 8%|▊ | 8/95 [00:17<03:10, 2.19s/it]
Epoch 1/20 - Validation: 9%|▉ | 9/95 [00:20<03:16, 2.28s/it]
Epoch 1/20 - Validation: 11%|█ | 10/95 [00:22<03:11, 2.25s/it]
Epoch 1/20 - Validation: 12%|█▏ | 11/95 [00:24<03:01, 2.16s/it]
Epoch 1/20 - Validation: 13%|█▎ | 12/95 [00:26<02:54, 2.10s/it]
Epoch 1/20 - Validation: 14%|█▎ | 13/95 [00:28<02:50, 2.07s/it]
Epoch 1/20 - Validation: 15%|█▍ | 14/95 [00:30<02:45, 2.04s/it]
Epoch 1/20 - Validation: 16%|█▌ | 15/95 [00:32<02:41, 2.02s/it]
Epoch 1/20 - Validation: 17%|█▋ | 16/95 [00:34<02:38, 2.01s/it]
Epoch 1/20 - Validation: 18%|█▊ | 17/95 [00:35<02:30, 1.93s/it]
Epoch 1/20 - Validation: 19%|█▉ | 18/95 [00:37<02:20, 1.82s/it]
Epoch 1/20 - Validation: 20%|██ | 19/95 [00:39<02:12, 1.75s/it]
Epoch 1/20 - Validation: 21%|██ | 20/95 [00:40<02:07, 1.69s/it]
Epoch 1/20 - Validation: 22%|██▏ | 21/95 [00:42<02:00, 1.63s/it]
Epoch 1/20 - Validation: 23%|██▎ | 22/95 [00:43<02:00, 1.64s/it]
Epoch 1/20 - Validation: 24%|██▍ | 23/95 [00:45<01:53, 1.57s/it]
Epoch 1/20 - Validation: 25%|██▌ | 24/95 [00:46<01:47, 1.52s/it]
Epoch 1/20 - Validation: 26%|██▋ | 25/95 [00:47<01:43, 1.48s/it]
Epoch 1/20 - Validation: 27%|██▋ | 26/95 [00:49<01:46, 1.54s/it]
Epoch 1/20 - Validation: 28%|██▊ | 27/95 [00:51<01:42, 1.51s/it]
Epoch 1/20 - Validation: 29%|██▉ | 28/95 [00:52<01:39, 1.48s/it]
Epoch 1/20 - Validation: 31%|███ | 29/95 [00:53<01:37, 1.47s/it]
Epoch 1/20 - Validation: 32%|███▏ | 30/95 [00:55<01:35, 1.46s/it]
Epoch 1/20 - Validation: 33%|███▎ | 31/95 [00:56<01:32, 1.45s/it]
Epoch 1/20 - Validation: 34%|███▎ | 32/95 [00:58<01:30, 1.44s/it]
Epoch 1/20 - Validation: 35%|███▍ | 33/95 [00:59<01:28, 1.42s/it]
Epoch 1/20 - Validation: 36%|███▌ | 34/95 [01:00<01:26, 1.42s/it]
Epoch 1/20 - Validation: 37%|███▋ | 35/95 [01:02<01:23, 1.40s/it]
Epoch 1/20 - Validation: 38%|███▊ | 36/95 [01:03<01:21, 1.39s/it]
Epoch 1/20 - Validation: 39%|███▉ | 37/95 [01:05<01:20, 1.38s/it]
Epoch 1/20 - Validation: 40%|████ | 38/95 [01:06<01:18, 1.38s/it]
Epoch 1/20 - Validation: 41%|████ | 39/95 [01:07<01:17, 1.38s/it]
Epoch 1/20 - Validation: 42%|████▏ | 40/95 [01:09<01:16, 1.39s/it]
Epoch 1/20 - Validation: 43%|████▎ | 41/95 [01:10<01:14, 1.39s/it]
Epoch 1/20 - Validation: 44%|████▍ | 42/95 [01:11<01:13, 1.38s/it]
Epoch 1/20 - Validation: 45%|████▌ | 43/95 [01:13<01:11, 1.37s/it]
Epoch 1/20 - Validation: 46%|████▋ | 44/95 [01:14<01:10, 1.38s/it]
Epoch 1/20 - Validation: 47%|████▋ | 45/95 [01:16<01:09, 1.39s/it]
Epoch 1/20 - Validation: 48%|████▊ | 46/95 [01:17<01:07, 1.39s/it]
Epoch 1/20 - Validation: 49%|████▉ | 47/95 [01:19<01:09, 1.45s/it]
Epoch 1/20 - Validation: 51%|█████ | 48/95 [01:20<01:07, 1.43s/it]
Epoch 1/20 - Validation: 52%|█████▏ | 49/95 [01:21<01:06, 1.44s/it]
Epoch 1/20 - Validation: 53%|█████▎ | 50/95 [01:23<01:04, 1.43s/it]
Epoch 1/20 - Validation: 54%|█████▎ | 51/95 [01:24<01:02, 1.42s/it]
Epoch 1/20 - Validation: 55%|█████▍ | 52/95 [01:26<00:59, 1.40s/it]
Epoch 1/20 - Validation: 56%|█████▌ | 53/95 [01:27<01:00, 1.43s/it]
Epoch 1/20 - Validation: 57%|█████▋ | 54/95 [01:29<00:58, 1.44s/it]
Epoch 1/20 - Validation: 58%|█████▊ | 55/95 [01:30<00:57, 1.43s/it]
Epoch 1/20 - Validation: 59%|█████▉ | 56/95 [01:31<00:55, 1.43s/it]
Epoch 1/20 - Validation: 60%|██████ | 57/95 [01:33<00:54, 1.43s/it]
Epoch 1/20 - Validation: 61%|██████ | 58/95 [01:35<00:57, 1.54s/it]
Epoch 1/20 - Validation: 62%|██████▏ | 59/95 [01:36<00:58, 1.61s/it]
Epoch 1/20 - Validation: 63%|██████▎ | 60/95 [01:38<00:55, 1.59s/it]
Epoch 1/20 - Validation: 64%|██████▍ | 61/95 [01:39<00:51, 1.51s/it]
Epoch 1/20 - Validation: 65%|██████▌ | 62/95 [01:41<00:50, 1.53s/it]
Epoch 1/20 - Validation: 66%|██████▋ | 63/95 [01:42<00:48, 1.52s/it]
Epoch 1/20 - Validation: 67%|██████▋ | 64/95 [01:44<00:47, 1.54s/it]
Epoch 1/20 - Validation: 68%|██████▊ | 65/95 [01:45<00:45, 1.50s/it]
Epoch 1/20 - Validation: 69%|██████▉ | 66/95 [01:47<00:44, 1.53s/it]
Epoch 1/20 - Validation: 71%|███████ | 67/95 [01:49<00:44, 1.57s/it]
Epoch 1/20 - Validation: 72%|███████▏ | 68/95 [01:50<00:43, 1.61s/it]
Epoch 1/20 - Validation: 73%|███████▎ | 69/95 [01:52<00:44, 1.70s/it]slurmstepd: error: *** REASON: burst_buffer/lua: Stage-out in progress ***
slurmstepd: error: *** JOB 541290 ON ip-10-0-136-5 CANCELLED AT 2024-11-26T22:08:55 ***
slurmstepd: error: *** REASON: burst_buffer/lua: Stage-out in progress ***
|