The model itself is faster on my machine using the GPU:
x = torch.randn(1000, 64)
model = CNN()

cpu_times = []

for epoch in range(100):
    t0 = time.perf_counter()
    output = model(x)
    t1 = time.perf_counter()
    cpu_times.append(t1 - t0)

device = 'cuda'
model = model.to(device)
x = x.to(device)
torch.cuda.synchronize()

gpu_times = []
for epoch in range(100):
    torch.cuda.synchronize()
    t0 = time.perf_counter()
    output = model(x)
    torch.cuda.synchronize()
    t1 = time.perf_counter()
    gpu_times.append(t1 - t0)

print('CPU {}, GPU {}'.format(
    torch.tensor(cpu_times).mean(),
    torch.tensor(gpu_times).mean()))
> CPU 0.0018446099711582065, GPU 0.0003588759864214808

Try to use a DataLoader with pin_memory=True if you are using the GPU, as this will pre-load the next batches to the page-locked memory while the GPU is busy.
What numbers do you get running my script?
PS: I’ve formatted your code. You can add code blocks using three backticks `.