This was a subtle issue.
Change this line:
correct_count += (pred == y).sum().data[0]

to
correct_count += (pred == y).double().sum().data[0]

The problem is that pred == y returns a ByteTensor, which has only an 8-bit range. Hence, after a particular batch-size, the sum was overflowing, and hence the wrong results.