Thanks for your input.

I tried sending tensors to GPU in collate_fn or in my custom Dataset, but this causes problems (described in the first post at the bottom) when I use num_workers > 0. It seem to run out of shared memory too easily. Did you have problems like that? How did you get around this?
I haven’t seen that part of documentation, although it also mentions the data should be a float16 which is not my case (seems a bit too imprecise to me).