at some step in your forward pass your input gets downsampled too much, likely because of one extra pooling layer. during your forward pass report shape of tensor, and find where the problem is that way