You could reduce the spatial size of the activation with pooling layers or a generally deeper architecture.
However, I would focus on the other two points, i.e. the softmax layer and dropout at the end of the model.
Did you remove them and did anything change?