I faced a similar issue. My issue turned out to be I was collecting validation stats across batches, but this was all happening on the GPU device side. I solved the problem by detaching from the GPU and sending my collected validation data to the CPU.