It depends on your current workflow.
Basically I would try to filter out all files without any annotations.
E.g. if your annotations are stored in a .csv file, you could load it and just pass all files with a valid annotation to the Dataset.
If you could explain your current approach a bit (e.g. what kind of Dataset are you using currently), there might be better suited approaches.