Hello, I am trying to train this network on pytorch ignite but the loss doesn’t increase a single point after more than 20 epochs…
I am working with Deep Neural Networks and I’ve been trying to emulate ELU network
class CIFARClassifierELUPaper(nn.Module):
    def __init__(self, num_classes=100):
        super(CIFARClassifierELUPaper, self).__init__()
        self.main = nn.Sequential(
            # Block 1
            nn.Conv2d(3, 384, 3, padding=1), # 32
            nn.ELU(),
            nn.MaxPool2d(2, 2), # 16
            # Block 2
            nn.Conv2d(384, 384, 1, padding=0), # 16
            nn.ELU(),
            nn.Conv2d(384, 384, 2, padding=1), # 17
            nn.ELU(),
            nn.Conv2d(384, 640, 2, padding=1), # 18
            nn.ELU(),
            nn.MaxPool2d(2, 2), # 9
            nn.Dropout2d(0.1),
            # Block 3
            nn.Conv2d(640, 640, 1, padding=0), # 9
            nn.ELU(),
            nn.Conv2d(640, 768, 2, padding=1), # 10
            nn.ELU(),
            nn.Conv2d(768, 768, 2, padding=1), # 11
            nn.ELU(),
            nn.Conv2d(768, 768, 2, padding=1), # 12
            nn.ELU(),
            nn.MaxPool2d(2, 2), # 6
            nn.Dropout2d(0.2),
            # Block 4
            nn.Conv2d(768, 768, 1, padding=0), # 6
            nn.ELU(),
            nn.Conv2d(768, 896, 2, padding=1), # 7
            nn.ELU(),
            nn.Conv2d(896, 896, 2, padding=1), # 8
            nn.ELU(),
            nn.MaxPool2d(2, 2), # 4
            nn.Dropout2d(0.3),
            # Block 5num_classes
            nn.Conv2d(896, 896, 1, padding=0), # 4
            nn.ELU(),
            nn.Conv2d(896, 1024, 2, padding=1), # 5
            nn.ELU(),
            nn.Conv2d(1024, 1024, 2, padding=1), # 6
            nn.ELU(),
            nn.MaxPool2d(2, 2), # 3
            nn.Dropout2d(0.4),
            # Block 6
            nn.Conv2d(1024, 1024, 1, padding=0), # 3
            nn.ELU(),
            nn.Conv2d(1024, 1152, 2, padding=0), # 2
            nn.ELU(),
            nn.MaxPool2d(2, 2), # 1
            nn.Dropout2d(0.5),
            # Block 7
            nn.Conv2d(1152, 1152, 1, padding=0), #  1
            nn.ELU(),
            nn.Dropout2d(0.0),
            # Block 8
            nn.Conv2d(1152, num_classes, 1, padding=0), # 1
        )
 
    def forward(self, x):
        return self.main(x).view(x.size(0),-1)

model=CIFARClassifierELUPaper()

I am trying to train CIFAR 100 and I am training with:
SGD
learning rate 0.1
ZCA whitening and all the others preprocessing (Random horizontal flip and randomcrops) steps
but both eval and train losses doesn’t increase or decrease ever (I tested with 20 epochs of 80)…
This are the metrics
] The metrics on training are {‘Loss’: 4.6055209875106815, ‘Accuracy’: 0.0082, ‘Precision’: 8.2e-05, ‘Recall’: 0.01, ‘Top-5 Accuracy’: 0.0474}
The metrics on testing are {‘Loss’: 4.6053833532333375, ‘Accuracy’: 0.01, ‘Precision’: 0.0001, ‘Recall’: 0.01, ‘Top-5 Accuracy’: 0.05}
Should I wait longer or there is something wrong with my network? I trained this before getting 64% but on the other one the only difference is that I added batch normalization…but I want to train from the begging and try to get the 73% …Also the  the publication suggest that BN is not necessary, so it shouldn’t be a problem right?