Rui_Liu:

Even though the error message reports a format issue, I don’t think that’s the file format issue because in the previous epochs I have already successfully loaded the same file with the same code.
I guess this issue just pops out suddenly.
Does anyone have some thoughts on that?


One thing you could check if it is a memory issue. My thinking around this is that “unknown format” might really be something like “something went wrong while decoding” and if your training leaks memory, it might end up not being able to load whatever it needs for decoding.
It’s just a guess, though.
Best regards
Thomas