training works fine for a full epoch. The loss spike issue i mentioned earlier was unrelated and I resolved it. Validation after epoch 1 is where I am seeing the error which is what is so perplexing.
Here is some of my training config:
  # loaders
  train_dataloader:
    batch_size: 40
    shuffle: true
    drop_last: true
    pin_memory: true
    num_workers: 8

  valid_dataloader:
    batch_size: 1
    shuffle: false
    drop_last: false
    pin_memory: true
    num_workers: 16


here’s around where it fails during validation:
valid:  17%|█▋        | 2987/17897 [02:42<13:04, 18.99it/s, val_loss_mask: 0.1158, val_loss: 0.1158, val_mask_micro_iou: 0.7913]
valid:  17%|█▋        | 2988/17897 [02:42<13:00, 19.10it/s, val_loss_mask: 0.1158, val_loss: 0.1158, val_mask_micro_iou: 0.7913]
valid:  17%|█▋        | 2988/17897 [02:42<13:31, 18.38it/s, val_loss_mask: 0.1158, val_loss: 0.1158, val_mask_micro_iou: 0.7913]