Currently trying to do it as a part of collate_fn. Not working quite well:
I use a lock to manage the access of gpu. However, after the first process releases the lock, the mem on gpu is NOT released, and therefore it causes OOM when the second process acquires the lock and start to work on gpu.