It worked, i trained my model for 10 epoch by adding all the ignore labels to extra class(class id:22). I have attached the results i got after 10 epochs.
Thanks @ptrblck for your support, i have closed the issue with your last answer.

Screenshot from 2020-01-13 18-32-21951×330 112 KB