Please verify that the weights of the layers you edited (conv1 and fc) are indeed initialized to non-zero values.
2,400 examples is too little for training a large model from scratch. Can you obtain more data?