The keras dataset is unnormalized, i.e. pixel values are in [0, 255].
ToTensor() normalizes the data to be in the range [0, 1]. This often helps the model to learn.
As you can see, normalizing indeed helps a lot! 
Just divide the input by 255. in load_mnist and your model also learns with the keras data.
def load_mnist():
    (x, y), (x_test, y_test) = mnist.load_data()
    x = x.reshape((-1, 1, 28, 28)).astype(np.float32)
    x_test = x_test.reshape((-1, 1, 28, 28)).astype(np.float32)
    y = y.astype(np.int64)
    y_test = y_test.astype(np.int64)
    x = x / 255.
    x_test = x_test / 255.
    
    print("x.shape", x.shape, "y.shape", y.shape,
          "\nx_test.shape", x_test.shape, "y_test.shape", y_test.shape,
          )

    return x, y, x_test, y_test