I’m not sure what exactly you’ve profiled and how the model is created, but based on the currently posted model architecture it seems that the actual workload is small.
In that case, you might see the overhead of the kernel launches etc. and the GPU might not be able to provide a speedup compared to the CPU run.
This would be especially visible if you have a lot of small kernels (e.g. a lot of linear layers each with a tiny workload).
CUDA graphs 5 which is an upcoming feature, could reduce this overhead as a single launch would be triggered.