Yep, that worked for me as well.
If your Custom Data set working fine? I can share mine, if you used Labelme to create annotation.
Did you test the data loader?
If both above are ok, so you just need to run the training