During training, my model output occasionally flips and loss skyrockets.
From the fact that r^2 is maintained and AUC becomes 1-AUC, I can the output have flipped.
Can you guess what is the problem?
Here is my model
 Custom_Model(
  (submodule_0): Sequential(
    (0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=34.066230764894875)
    (1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ELU(alpha=1.0)
    (3): Dropout(p=0.4, inplace=False)
  )
  (submodule_1): Sequential(
    (0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=21.358266163124966)
    (1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ELU(alpha=1.0)
    (3): Dropout(p=0.6, inplace=False)
  )
  (submodule_2): Sequential(
    (0): Masked_Linear(input_features=17741, output_features=17741, bias=True, avg_mask_on=7.067527196888563)
    (1): BatchNorm1d(17741, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ELU(alpha=1.0)
    (3): Dropout(p=0.6, inplace=False)
  )
  (submodule_3): Sequential(
    (0): Linear(in_features=17741, out_features=1024, bias=True)
    (1): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ELU(alpha=1.0)
    (3): Dropout(p=0.5, inplace=False)
    (4): Linear(in_features=1024, out_features=256, bias=True)
    (5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (6): ELU(alpha=1.0)
    (7): Dropout(p=0.2, inplace=False)
    (8): Linear(in_features=256, out_features=1, bias=True)
  )
)

Optimizer
Adagrad (
Parameter Group 0
    eps: 1e-10
    initial_accumulator_value: 0
    lr: 0.01
    lr_decay: 0
    weight_decay: 3e-07
)

Log
VALID: epoch: 14-852(99.88%), loss: 0.16498, cached: 63.937
VALID: output           r2: 0.000241, auc: 0.56510
TRAIN: epoch: 15-2556(99.96%), loss: 0.00577, cached: 63.986
TRAIN: output           r2: 0.097297, auc: 0.99790
VALID: epoch: 15-852(99.88%), loss: 0.16588, cached: 63.937
VALID: output           r2: 0.000255, auc: 0.56665
TRAIN: epoch: 16-2556(99.96%), loss: 0.00535, cached: 63.986
TRAIN: output           r2: 0.092888, auc: 0.99835
VALID: epoch: 16-852(99.88%), loss: 5.11513, cached: 63.937
VALID: output           r2: 0.000259, auc: 0.56450
TRAIN: epoch: 17-2556(99.96%), loss: 0.00452, cached: 63.986
TRAIN: output           r2: 0.100936, auc: 0.99893
VALID: epoch: 17-852(99.88%), loss: 14.84378, cached: 63.937
VALID: output           r2: 0.000285, auc: 0.43133
TRAIN: epoch: 18-2556(99.96%), loss: 0.00371, cached: 63.986
TRAIN: output           r2: 0.115433, auc: 0.99923
VALID: epoch: 18-852(99.88%), loss: 0.07149, cached: 63.937
VALID: output           r2: 0.000277, auc: 0.56678