Hmm I think my architecture is ok. Here is the first few lines but the rest follows the same pattern
           Conv2d-1         [-1, 64, 112, 112]           4,864
       BatchNorm2d-2         [-1, 64, 112, 112]             128
              ReLU-3         [-1, 64, 112, 112]               0
 AdaptiveMaxPool2d-4           [-1, 64, 56, 56]               0
  InitialConvblock-5           [-1, 64, 56, 56]               0
            Conv2d-6          [-1, 256, 56, 56]          16,640
       BatchNorm2d-7          [-1, 256, 56, 56]             512
            Conv2d-8           [-1, 64, 56, 56]           4,160
       BatchNorm2d-9           [-1, 64, 56, 56]             128
             ReLU-10           [-1, 64, 56, 56]               0
           Conv2d-11           [-1, 64, 54, 54]          36,928
      BatchNorm2d-12           [-1, 64, 54, 54]             128
             ReLU-13           [-1, 64, 54, 54]               0
           Conv2d-14          [-1, 256, 56, 56]          16,640
      BatchNorm2d-15          [-1, 256, 56, 56]             512
             ReLU-16          [-1, 256, 56, 56]               0
         ResBlock-17          [-1, 256, 56, 56]               0
           Conv2d-18          [-1, 256, 56, 56]          65,792
      BatchNorm2d-19          [-1, 256, 56, 56]             512

The strange thing is I am able to train MNIST on this model to a good degree of accuracy. It only seems to be with imagenet that I run into difficulty. Can you think of any other reason I might not be seeing my loss decrease/what I can do to fix my training?