That is OOM. To be honest I think every error is OOM.
The problem is kinda solved (I hope) by reducing the number of workers to 3 (12 h of training right now).
With 3 workers I have 5 GB of free memory, not really sure why it would need so much free memory, but with 4 workers it crashes after hours.
It’s not a memory leak either, since the memory usage doesn’t change overtime.
I don’t think It’s worth investigating any further, since it works fine with 3 workers and I wouldn’t even know where to start.
Thanks for the help.