Sure, my training loop code is now:
	torch.cuda.synchronize()
	t = default_timer()
	for data, target in data_loader:
		torch.cuda.synchronize(); T = default_timer(); print(f"Get data {1000*(T - t):.3f}ms"); t = default_timer()
		data = data.to(device)
		target = target.to(device)
		torch.cuda.synchronize(); T = default_timer(); print(f"Data to device {1000*(T - t):.3f}ms"); t = default_timer()
		optimizer.zero_grad()
		torch.cuda.synchronize(); T = default_timer(); print(f"Zero grad {1000*(T - t):.3f}ms"); t = default_timer()
		output = model(data)
		torch.cuda.synchronize(); T = default_timer(); print(f"Model eval {1000*(T - t):.3f}ms"); t = default_timer()
		loss = criterion(output, target)
		torch.cuda.synchronize(); T = default_timer(); print(f"Criterion {1000*(T - t):.3f}ms"); t = default_timer()
		loss.backward()
		torch.cuda.synchronize(); T = default_timer(); print(f"Backward {1000*(T - t):.3f}ms"); t = default_timer()
		optimizer.step()
		torch.cuda.synchronize(); T = default_timer(); print(f"Optim step {1000*(T - t):.3f}ms"); t = default_timer()

The timings for the 1900K model becomes:

Model 1
Get data 0.012ms
Data to device 0.542ms
Zero grad 0.313ms
Model eval 10.577ms
Criterion 0.136ms
Backward 15.327ms
Optim step 4.073ms

And for the 20K model:

Model 2
Get data 0.012ms
Data to device 0.576ms
Zero grad 1.259ms
Model eval 26.028ms
Criterion 0.167ms
Backward 42.614ms
Optim step 15.681ms

These timings are now pretty nice and consistent, and show how zero grad, model eval, the backwards step and the optimisation step consistently take much longer for the much smaller model. Any ideas?