Hi, were you able to resolve your problem?
I am also facing a similar issue, I have removed an aditional 3000 images from training set, as they were overlapping with my validation and test set, total training images after filtering are 92,034. Best Mean IOU accuray of 53.5 occured at 8th epoch, and it has been fluctuating between 50-53 since 2nd epoch.
I am using the reference scripts for segmentation and following command line for training
OMP_NUM_THREADS=4 CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 --use_env train.py --lr 0.02 --dataset coco -b 4 -j 4 --model deeplabv3_resnet101 --aux-loss