Solved by ptrblck in post #2 
              
              
                Your second conv layer is returning an activation with a single output channel, while the next conv layer expects 64 input channels (same issue in all following layers). 
You should set the number of output channels to 64 and your model should work.