According to the PyTorch tutorials at https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html 10
which says data parallelism can be done if there are multiple-GPUs, by creating replicas of the model on different GPUs.
Is it possible to use data parallelism on a single GPU device by using more memory on the same device to create replicas of the model and parallelizing the training of different batches on these replicas of the model?
My model is three convolutional layers deep and but I have larger batch size. So, it can parallelize by processing smaller batches in different replicas of the model.
Thanks in advance.