You seem to have only convolutions only in your network without any maxpooling . I would suggest applying strided-convolution or maxpooling . In case of Dropout apply it after the convolution and just before a dense layer and in-between some dense layers