You could verify that the model parameters are indeed being updated and e.g. also check the gradients etc. If you cannot spot any issues, you could try to start with a working example (i.e. change the dataset to images and targets, which the model can learn) and then try to check the changes between the working example and your custom workflow.