Nothing in the code you pasted looks particularly slow. Perhaps the larger model you’re using contains many small layers/kernels? The nn.DataParallel wrapper replicates a module to N devices and runs forward on each of them. This overhead can dominate the runtime if your model is very small or has many very small kernels.