Hi there,
I could finally look at the network. I used epoch 300, 700 and final for before, during and after the breakdown.
Interestingly I don’t see any weirdness in the batch norm moving stats at the first bn layer:

epoch 300:
In [9]: self.network.bn1.running_mean
Out[9]:
tensor([ 0.0265, -0.1110, -0.1160,  0.0190,  0.1058, -0.0179, -0.0286,  0.0524,
-0.0180, -0.1844,  0.0287,  0.2987,  0.0370, -0.6230,  0.0031,  0.0256,
-0.0679,  0.1194, -0.2744,  0.0251, -0.0909,  0.0986, -0.0009, -0.2132,
0.0537,  0.1174, -0.0971, -0.0125, -0.0445, -0.0024, -0.3101, -0.0073],
device=‘cuda:0’)
In [10]: self.network.bn1.running_var
Out[10]:
tensor([ 1.2123,  1.7482,  2.6450,  1.9762,  1.9177,  4.5315,  0.4663,  1.5161,
1.1491,  6.9815,  1.1021,  9.3386,  1.2492, 29.5786,  1.1680,  0.4657,
4.0417,  2.0542, 16.2966,  2.7019,  2.1935,  1.9766,  0.9401,  5.6444,
1.5645,  1.7632,  6.9528,  1.8997,  1.1977,  1.5128, 20.3052,  0.8678],
device=‘cuda:0’)
In [11]:
epoch 700:
In [12]: self.network.bn1.running_mean
Out[12]:
tensor([ 0.0273, -0.1171, -0.0941,  0.0393,  0.1386,  0.0285, -0.0054,  0.0831,
-0.0080, -0.1630,  0.0332,  0.3122,  0.0628, -0.4963, -0.0061,  0.0065,
-0.1043,  0.1000, -0.2297,  0.0593, -0.0857,  0.0967,  0.0244, -0.1920,
0.0275,  0.0837, -0.2192,  0.0217, -0.0661, -0.0337, -0.4571, -0.0109],
device=‘cuda:0’)
In [13]: self.network.bn1.running_var
Out[13]:
tensor([ 1.1938,  2.7972,  4.3059,  3.4096,  2.8702,  5.8498,  0.5527,  2.3203,
1.7257, 10.2496,  1.3604, 15.1904,  2.0068, 42.5539,  1.6873,  0.5314,
6.4726,  3.1628, 22.7749,  4.3518,  2.8136,  2.8452,  1.9811,  7.4680,
2.3911,  2.3746, 18.5319,  3.0089,  1.7593,  2.4780, 40.4268,  1.4430],
device=‘cuda:0’)
In [14]:
final:
In [15]: self.network.bn1.running_mean
Out[15]:
tensor([ 0.0027, -0.1487, -0.1678,  0.0462,  0.1889,  0.0641,  0.0002,  0.0834,
-0.0166, -0.1836,  0.0333,  0.4439,  0.0763, -0.7683,  0.0203,  0.0281,
-0.1092,  0.1382, -0.2974,  0.0031, -0.0821,  0.1381,  0.0201, -0.2344,
0.0412,  0.1326, -0.2208,  0.0218, -0.0966, -0.0137, -0.5406, -0.0126],
device=‘cuda:0’)
In [16]: self.network.bn1.running_var
Out[16]:
tensor([ 1.1598,  3.0503,  4.5197,  3.4007,  3.1393,  5.8271,  0.5361,  2.2878,
1.7715, 10.5495,  1.2760, 15.2090,  1.9926, 43.3876,  1.8111,  0.5640,
5.4958,  3.2575, 22.9858,  4.0293,  2.7336,  3.1246,  2.0181,  7.6566,
2.3407,  2.4714, 17.4709,  2.9588,  1.6889,  2.3654, 37.0451,  1.4181],
device=‘cuda:0’)

Here are some descriptive statistics of the running mean and var of the last bn layer (the one that comes after the last convolution in the very last residual block. It has 1024 entries which is why I am providing stats only here)

epoch 300:
In [33]: self.network.layers[-1][-1].bn2.running_mean.max()
Out[33]: tensor(0.2174, device=‘cuda:0’)
In [34]: self.network.layers[-1][-1].bn2.running_mean.min()
Out[34]: tensor(-0.6384, device=‘cuda:0’)
In [35]: self.network.layers[-1][-1].bn2.running_mean.mean()
Out[35]: tensor(-0.1016, device=‘cuda:0’)
In [36]: self.network.layers[-1][-1].bn2.running_mean.var()
Out[36]: tensor(0.0184, device=‘cuda:0’)
In [39]: self.network.layers[-1][-1].bn2.running_var.max()
Out[39]: tensor(0.8339, device=‘cuda:0’)
In [40]: self.network.layers[-1][-1].bn2.running_var.min()
Out[40]: tensor(4.8037e-34, device=‘cuda:0’)
In [41]: self.network.layers[-1][-1].bn2.running_var.mean()
Out[41]: tensor(0.0686, device=‘cuda:0’)
In [42]: self.network.layers[-1][-1].bn2.running_var.var()
Out[42]: tensor(0.0122, device=‘cuda:0’)
epoch 700:
In [48]: self.network.layers[-1][-1].bn2.running_mean.max()
Out[48]: tensor(0.1805, device=‘cuda:0’)
In [49]: self.network.layers[-1][-1].bn2.running_mean.min()
Out[49]: tensor(-0.4298, device=‘cuda:0’)
In [50]: self.network.layers[-1][-1].bn2.running_mean.mean()
Out[50]: tensor(-0.0727, device=‘cuda:0’)
In [51]: self.network.layers[-1][-1].bn2.running_mean.var()
Out[51]: tensor(0.0118, device=‘cuda:0’)
In [44]: self.network.layers[-1][-1].bn2.running_var.max()
Out[44]: tensor(0.2591, device=‘cuda:0’)
In [45]: self.network.layers[-1][-1].bn2.running_var.min()
Out[45]: tensor(5.6052e-45, device=‘cuda:0’)
In [46]: self.network.layers[-1][-1].bn2.running_var.mean()
Out[46]: tensor(0.0242, device=‘cuda:0’)
In [47]: self.network.layers[-1][-1].bn2.running_var.var()
Out[47]: tensor(0.0016, device=‘cuda:0’)
final:
In [53]: self.network.layers[-1][-1].bn2.running_mean.max()
Out[53]: tensor(0.0899, device=‘cuda:0’)
In [54]: self.network.layers[-1][-1].bn2.running_mean.min()
Out[54]: tensor(-0.4327, device=‘cuda:0’)
In [55]: self.network.layers[-1][-1].bn2.running_mean.mean()
Out[55]: tensor(-0.0624, device=‘cuda:0’)
In [56]: self.network.layers[-1][-1].bn2.running_mean.var()
Out[56]: tensor(0.0093, device=‘cuda:0’)
In [57]: self.network.layers[-1][-1].bn2.running_var.max()
Out[57]: tensor(0.1923, device=‘cuda:0’)
In [58]: self.network.layers[-1][-1].bn2.running_var.min()
Out[58]: tensor(5.6052e-45, device=‘cuda:0’)
In [59]: self.network.layers[-1][-1].bn2.running_var.mean()
Out[59]: tensor(0.0170, device=‘cuda:0’)
In [60]: self.network.layers[-1][-1].bn2.running_var.var()
Out[60]: tensor(0.0009, device=‘cuda:0’)

Batch norm statistics look normal-ish to me although I think running variances of 1e-45 are dangerous (almost division by zero). How can that happen?
I will plot batch statistics in my next post