Maybe you are running out of memory on the default device which will gather and scatter some parameters, thus usually using a bit more memory than the other devices.
This behavior is described in @Thomas_Wolf’s blog post 30.