@RayJ,
My issue was exactly as @stes suggested; a kernel size in the latter layers of my network was too big for the intermediate feature map size.
I found where that was by printing the size of the tensor after each layer in the forward() function. As a quick sanity check, you might try setting all of your kernel sizes to 1. If the error goes away, then you could dig further into which kernel is too large for your feature representation.