Okay found the error. It was happening in the lines
targets.extend(list(gt))
outputs.extend(list(out_mask))

As all predictions are stored in the ram, the space gets exhausted. But still, this doesn’t explain why the 1st epoch runs without any problems.