Tiny workloads can be faster on the CPU than on a GPU.
I’m not sure how all submodules are defined or how they are used in your model, so I can’t really guess what might be the reason besides the workload.