The issue was actually a really silly one. When i loaded the ground truths in the dataset, it was loaded into a tensor. For example,
self.all_joints = read_joints()
other reading stuff

When i loaded the data in the __getitem__ function, i would perform random translation/scaling/rotating and normalization on the joints.
joint = self.all_joints[index]
joint = translate(joint) and so on

As i did not know about tensors not being copied as a new tensor but as a reference, the numbers in self.all_joints[index] kept getting smaller and smaller with every loading of that index. When debugging, i only loaded each index once and hence the issue slipped past me.
As a quick fix, i simply changed saved self.all_joints as a numpy array instead