The thing is that internally. We try and never call .to under the hook to avoid performance pitfalls. DataParallel is an exception has it needs to move stuff between gpus.
So one forward in your model still calls 4500 times .to() ?
Note as well that if you do many .float() or .long(), etc, they get converted to .to operators I think.