Do you see this OOM error only using your custom dataset or also the original one?
If this error is raised only for your custom dataset, are you using the same image resolution or are you working with bigger images? In the latter case, could you resize your custom images to the same size and try to run the code again?
Since the batch size is already set to 1, you cannot lower is further and would need to save some memory in another part of your training.