Is this a sequential model? Please show your forward definition.
When building a CNN, you have to make sure you know what size you’re getting out of each part. You can calculate the sizes by looking at the formula on the bottom of the documentation page for each type of module(i.e. Conv2d, MaxPool2d, etc.).
Or you can just add in a bunch of print statements at each step in your forward pass in order to understand the size of the tensor moving thru.
x=self.relu(self.conv1(x))
print(x.size())
x=pool(x)
print(x.size())
...

Once you know what the size is, you might decide to remove some layers or add some padding to accommodate. If the height and width are (1, 1) any further reduction layers will give errors.