@ptrblck, used the following architecture For my case, I needed to train the network with no bias applied on the layers. Hence, I could not use the pretrained weights I just need to train the weights.

Conv2d(output dim=128,input dim=3,kernel_size=3,padding=1,bias=False)

2.batchnorm(128)
3 relu
4.Conv2d(output dim=128,input dim=128,kernel_size=3,padding=1,bias=False)


Batchnorm(128)


Relu


7.Maxpool(kernel_size=2,stride=2)

Conv2d(output dim=256,input dim=128,kernel_size=3,padding=1,bias=False)

9.batchnorm(256)
10 relu
11 conv2d(output dim=256,input dim=256,kernel_size=3,padding=1,bias=False)

batchnorm(256)

13 relu
14 Maxpool(kernel_size=2,stride=2)
15 conv2d(output dim=512,input dim=256,kernel_size=3,padding=1,bias=False)
16 batch norm(512)
17 relu
18 conv2d(output dim=512,input dim=512,kernel_size=3,padding=1,bias=False)
19 batchnorm(512)
20 relu
21 maxpool
22 dropout(0.5)


fully connected layer1(ouput dim=1024,input dim=8192,bias=False)


relu


dropout(0.5)


26 softmax(output dim=10,input dim=1024,bias=False)
This setup gave me a 92.2 percent  on the validation set. I ran it for 300 epochs using SGD with learning rate 0.05 which reduces by half the previous value every 30 epochs, momentum=0.9,weight_decay=5e-4, and batch size=50