You can manually check if your dataset class (let it be denoted as TheeDatasetClass) is working correctly before using Mask RCNN, this is what I usually do. Just create an instance of the dataset, and verify if you are getting the correct image, boxes, etc. Something like:
TheeDatasetClass x
zz = x[1]

So, if you are Spyder or PyCharm, you can check if zz has what you wanted.
Tip: Always use debugging to make sure your code is working as you want it to, and this is why PyTorch is very handy, among other things. For example, if your dataset is causing a problem due to some out of your will reason, just hoover a for-loop over the entire dataset and check which indices are causing the problem.