Thanks for the code.
Yes, I think you are correct and it seems your custom dataset stores the already transformed samples in these lines of code:
    def __getitem__(self, index):
        if self.transform:
            self.x[index] = self.transform(self.x[index])
        
        return self.x[index], self.y[index]

Avoid the assignment to self.x and just return a new tensor via:
    def __getitem__(self, index):
        if self.transform:
            x = self.transform(self.x[index])
        
        return x, self.y[index]