I get another error running your model with CIFAR10:

RuntimeError: Calculated padded input size per channel: (7 x 7). Kernel size: (11 x 11). Kernel size can’t be greater than actual input size at /opt/conda/conda-bld/pytorch_1532581333611/work/aten/src/THNN/generic/SpatialConvolutionMM.c:48

If you print the shape of conv1 in the forward pass, you will see, that it’s just [batch_size, 96, 3, 3], which is too small for conv2 using kernel_size=11.