Check the gradients and weights, are they updating? This is fairly common in deep learning. Your error won’t decrease at all. Try a different architecture, kernel size. For image classification you might need more layers, Try batch normalization to see if that helps.