Solved by ptrblck in post #4 
              
              
                I think the slower runtime is due to the higher number of modules (71 vs. 240, if I’m not mistaken). 
Even if your first model uses much more parameters, the number of layers (and thus kernel launches) is much lower, which could explain the difference in the runtime.