You should make sure to call .cuda() or .to(device='cuda') on your data and your model.
Depending on the model architecture, pushing it to the GPU won’t give you any speedups.
This is usually the case for small models.
Could you post your model architecture?