Here I am showing both training batch statistics as well as statistics of feature maps activations right before the classification layer. Again I use epochs 300, 700 and final (1015) for before, during and after the breakdown. For each epoch, I draw 5 batches and show the aforementioned statistics for training, validation (network.train = False) and validation with network.train = True (I am including the latter to demonstrate that the problem must be the running stats as feature map activations for train=False are off while they are OK for train=Tru). As you can see, input data statistics are consistent for all batches and epochs, so no surprise here. For epoch 300 and final, feature maps statistics are consistent-ish for train, validation (train=False) and validation (train=True). For epoch 700 however (which is where the network breaks down), the feature map activations match between train and val (train=True) but don’t match for validation (train=False; this is where the running stats are applied).
I am also printing the loss for all epochs and phases (train. val, …). Don’t worry about the loss being negative. That is because the F1 loss ranges from 0 to -1 while the BCE is [0, inf] (total loss is the sum and thus ranges from [-1, inf]). In epochs 300 and final the loss between train and validations is basically identical (with some overfitting) while in epoch 700 the loss for validation (train=False) is way too high.

epoch:  300
training: (network is train=True and samples are drawn from training DataLoader)
Input data: max: 8.606085 min: -0.7775649 mean: 0.11011692 var: 1.1633884
Features before classification: max: 17.142632 min: 0.0 mean: 0.24466729 var: 0.29341334
Input data: max: 8.5557785 min: -0.7629295 mean: 0.04259184 var: 1.0552053
Features before classification: max: 18.69158 min: 0.0 mean: 0.24318185 var: 0.29009783
Input data: max: 8.638606 min: -0.7287697 mean: 0.10248642 var: 1.160278
Features before classification: max: 22.366526 min: 0.0 mean: 0.24295413 var: 0.29546708
Input data: max: 8.53746 min: -0.76369715 mean: 0.14483288 var: 1.2324796
Features before classification: max: 15.086981 min: 0.0 mean: 0.24739826 var: 0.28974423
Input data: max: 8.563448 min: -0.7089811 mean: 0.08948848 var: 1.133431
Features before classification: max: 32.820637 min: 0.0 mean: 0.24512042 var: 0.28797802
train loss : -0.5370
validation: (network is train=False and samples are drawn from validation DataLoader)
Input data: max: 8.53746 min: -0.6521306 mean: 0.07451663 var: 1.0982525
Features before classification: max: 19.99752 min: 0.0 mean: 0.24418607 var: 0.2950879
Input data: max: 8.53746 min: -0.6521306 mean: 0.06573128 var: 1.0647457
Features before classification: max: 21.51592 min: 0.0 mean: 0.23669797 var: 0.2616221
Input data: max: 8.53746 min: -0.6521306 mean: 0.07230965 var: 1.1099536
Features before classification: max: 34.280563 min: 0.0 mean: 0.26418424 var: 0.3719333
Input data: max: 8.53746 min: -0.6521306 mean: 0.16783251 var: 1.3045272
Features before classification: max: 26.123634 min: 0.0 mean: 0.25989175 var: 0.37081867
Input data: max: 8.53746 min: -0.6521306 mean: 0.110146575 var: 1.2274413
Features before classification: max: 26.462603 min: 0.0 mean: 0.22915748 var: 0.25221625
val loss (train=False): -0.4176
validation (train=True): (network is train=True and samples are drawn from validation DataLoader)
Input data: max: 8.53746 min: -0.6521306 mean: 0.041268155 var: 1.1999401
Features before classification: max: 21.246727 min: 0.0 mean: 0.24235177 var: 0.2917685
Input data: max: 8.53746 min: -0.6521306 mean: 0.1290773 var: 1.2299877
Features before classification: max: 23.499233 min: 0.0 mean: 0.2495341 var: 0.28842986
Input data: max: 8.53746 min: -0.6521306 mean: 0.077322766 var: 1.0789642
Features before classification: max: 20.976784 min: 0.0 mean: 0.24999543 var: 0.28809312
Input data: max: 8.53746 min: -0.6521306 mean: 0.12412989 var: 1.2118222
Features before classification: max: 20.250025 min: 0.0 mean: 0.25311542 var: 0.28560916
Input data: max: 8.53746 min: -0.6521306 mean: 0.058781143 var: 1.1097133
Features before classification: max: 16.501434 min: 0.0 mean: 0.2545616 var: 0.28281626
val loss (train=True): -0.5057
epoch:  700
training: (network is train=True and samples are drawn from training DataLoader)
Input data: max: 8.55883 min: -0.77725536 mean: 0.11748674 var: 1.1631266
Features before classification: max: 15.908981 min: 0.0 mean: 0.2101385 var: 0.21815181
Input data: max: 8.53746 min: -0.7591846 mean: 0.16579734 var: 1.1916564
Features before classification: max: 16.09063 min: 0.0 mean: 0.21404302 var: 0.21558203
Input data: max: 8.555547 min: -0.6906217 mean: 0.101644285 var: 1.1464278
Features before classification: max: 17.098934 min: 0.0 mean: 0.2126852 var: 0.21343498
Input data: max: 8.594714 min: -0.74816304 mean: 0.10817674 var: 1.1627903
Features before classification: max: 17.051395 min: 0.0 mean: 0.20846856 var: 0.21932985
Input data: max: 8.563169 min: -0.78391445 mean: 0.051834863 var: 1.0531709
Features before classification: max: 18.298471 min: 0.0 mean: 0.20677677 var: 0.21607172
train loss : -0.6972
validation: (network is train=False and samples are drawn from validation DataLoader)
Input data: max: 8.53746 min: -0.6521306 mean: 0.13776894 var: 1.1895192
Features before classification: max: 9.838846 min: 0.0 mean: 0.14135005 var: 0.07449843
Input data: max: 8.53746 min: -0.6521306 mean: 0.16435458 var: 1.2554524
Features before classification: max: 10.988285 min: 0.0 mean: 0.1407173 var: 0.0791312
Input data: max: 8.53746 min: -0.6521306 mean: -0.014078559 var: 0.9748203
Features before classification: max: 11.078625 min: 0.0 mean: 0.13703124 var: 0.0729197
Input data: max: 8.53746 min: -0.6521306 mean: 0.07416497 var: 1.1208515
Features before classification: max: 26.292133 min: 0.0 mean: 0.14437278 var: 0.093446925
Input data: max: 8.53746 min: -0.6521306 mean: 0.06462406 var: 1.1092615
Features before classification: max: 10.157297 min: 0.0 mean: 0.14090487 var: 0.08402736
val loss (train=False): 0.1353
validation (train=True): (network is train=True and samples are drawn from validation DataLoader)
Input data: max: 8.53746 min: -0.6521306 mean: 0.10422288 var: 1.2318301
Features before classification: max: 16.510902 min: 0.0 mean: 0.20996732 var: 0.21453643
Input data: max: 8.53746 min: -0.6521306 mean: 0.058634315 var: 1.1580375
Features before classification: max: 20.746746 min: 0.0 mean: 0.20406458 var: 0.21710423
Input data: max: 8.53746 min: -0.6521306 mean: 0.096261665 var: 1.1046003
Features before classification: max: 20.254076 min: 0.0 mean: 0.20849131 var: 0.21499783
Input data: max: 8.53746 min: -0.6521306 mean: 0.124039836 var: 1.2028973
Features before classification: max: 19.05787 min: 0.0 mean: 0.2078148 var: 0.2171791
Input data: max: 8.53746 min: -0.6521306 mean: 0.13756868 var: 1.1901097
Features before classification: max: 17.98934 min: 0.0 mean: 0.21233286 var: 0.21371831
val loss (train=True): -0.4974
epoch:  1015
training: (network is train=True and samples are drawn from training DataLoader)
Input data: max: 8.591203 min: -0.744614 mean: 0.12869895 var: 1.2715969
Features before classification: max: 15.38456 min: 0.0 mean: 0.22904472 var: 0.23167339
Input data: max: 8.543597 min: -0.75908107 mean: 0.04424698 var: 1.0194602
Features before classification: max: 20.93504 min: 0.0 mean: 0.22539279 var: 0.22878553
Input data: max: 8.600206 min: -0.7582485 mean: 0.08429426 var: 1.2217238
Features before classification: max: 16.953554 min: 0.0 mean: 0.22585449 var: 0.23169553
Input data: max: 8.589985 min: -0.74655396 mean: 0.009551339 var: 0.94260657
Features before classification: max: 15.600396 min: 0.0 mean: 0.22332579 var: 0.22921476
Input data: max: 8.6038065 min: -0.75667745 mean: 0.11769733 var: 1.1169093
Features before classification: max: 21.059433 min: 0.0 mean: 0.22816601 var: 0.23191863
train loss : -0.7191
validation: (network is train=False and samples are drawn from validation DataLoader)
Input data: max: 8.53746 min: -0.6521306 mean: 0.057770584 var: 1.0422962
Features before classification: max: 18.795538 min: 0.0 mean: 0.21927223 var: 0.20996067
Input data: max: 8.53746 min: -0.6521306 mean: 0.03980498 var: 1.0491408
Features before classification: max: 19.233257 min: 0.0 mean: 0.2139626 var: 0.1971896
Input data: max: 8.53746 min: -0.6521306 mean: 0.05019422 var: 1.0788214
Features before classification: max: 22.418144 min: 0.0 mean: 0.23339844 var: 0.25270167
Input data: max: 8.53746 min: -0.6521306 mean: 0.12727928 var: 1.2005315
Features before classification: max: 18.214798 min: 0.0 mean: 0.23262215 var: 0.25595525
Input data: max: 8.53746 min: -0.6521306 mean: 0.09293718 var: 1.1846416
Features before classification: max: 18.957647 min: 0.0 mean: 0.22007352 var: 0.21793382
val loss (train=False): -0.5545
validation (train=True): (network is train=True and samples are drawn from validation DataLoader)
Input data: max: 8.53746 min: -0.6521306 mean: 0.121076666 var: 1.3058403
Features before classification: max: 19.245342 min: 0.0 mean: 0.23451522 var: 0.22419384
Input data: max: 8.53746 min: -0.6521306 mean: 0.15040742 var: 1.227032
Features before classification: max: 19.016422 min: 0.0 mean: 0.23178016 var: 0.2272836
Input data: max: 8.53746 min: -0.6521306 mean: 0.18541424 var: 1.3306359
Features before classification: max: 18.303911 min: 0.0 mean: 0.2370149 var: 0.22376394
Input data: max: 8.53746 min: -0.6521306 mean: 0.036099356 var: 0.971277
Features before classification: max: 21.868843 min: 0.0 mean: 0.23125803 var: 0.22653475
Input data: max: 8.53746 min: -0.6521306 mean: 0.07943766 var: 1.2029198
Features before classification: max: 23.589802 min: 0.0 mean: 0.23089185 var: 0.22577839
val loss (train=True): -0.5044

If you need anything else please let me know!
Best,
Fabian