I will continue to debug this, any suggestions?

Then the bug is unlikely to be relevant to the pipelining logic. One thing I would check is all the RRef objects. Have you used any of the RRef objects or its field temporarily? If any, you can try to delete it after the usage. For example, you probably can try del pixel_values after the line x = pixel_values.to_here().to(self.device).