Solved by moskomule in post #2 
              
              
                We found ReLu solves this issue. ReLu is all we need. 
conv = nn.Sequential(nn.Conv2d(3, 64, 3),
                     nn.ReLU())