Usually model training on two GPUs is there to help you get a bigger batch size: what the Trainer and the example scripts do automatically is that each GPU will treat batch of the given --pre_device_train_batch_size which will result on a training with 2 * per_device_train_batch_size. This still requires the model to fit on each GPU.
What you want to use is model parallelism but this is still very experimental. You can check the deepspeed or fairscale integrations 23 to use ZeRO-DP3 and split your model accross your two GPUs.