Make sure that the target vocab is created correctly. The target labels for a batch are as expected. Also, if you’ve created your custom loss function that might have an issue.
Is possible share your code.