Nope, not yet. I have singled out the problem to dataloading.
I created a custom dataset and just wrap it with the pytorch DataLoader. So I dont know if the problem is with the pytorch dataloader in general, an issue with pytorch dataloader with AMD or something specific to my hardware configuration.
The data is basically a collection of videos. I randomly choose a video and from that choose a random frame. To do that I create a dataset for each video and combine all of them using ConcatDataset.