If your GPU memory is nearly filled you might run OOM in the last compare operation.
You could push the tensors first to the CPU and calculate the accuracy later, as you are apparently storing them on the CPU anyway.