Would you like to apply some data parallelism (cloning the model on each GPU and splitting the data) or would you like to shard the model somehow (different GPUs compute different parts of your model)?