What kind of error do you get? Are you really running out of memory?
A single image in torch.uint8 would take approx 5100*3500*3/1024**2 = 51MB.
So even with a batch size of 4 and even if the next batch is already loaded using multiple workers, you’ll use less than 500MB.
Could you post the error message and the stack trace if possible?