If you increase the workload (e.g. more filters in the conv layers or more conv layers in general), you should see the benefits of using the GPU. As explained above, tiny workloads might suffer from the overheads of pushing the data to the device as well as the kernel launches.