Thanks for helping!
Sorry I was unclear.
If I completely shut off and start I likely get a new gpu assigned, no error. Like colab’s factory reset.
Could it be the issue that I’m storing the whole array in memory?
That’s the only major thing that is different this time arround, I repurposed old code I made which worked fine but loaded images from disk.
I’ll try that next with this dataset too.
Edit Update:
Get the same with reading from disk, weird.
I think I have issue in my code.
Above it points at:
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
something similar in tensorflow woudl mean the graph is disconnected.
What I can’t make out is how this relates to batch_size
Edit update 2:
Actually can do up to batch_size 6