at the site of error i have put two statements
print('mask fwd ','xleng',len(x),'roi',
                  len(x[:mask_roi_extractor.num_inputs]),mask_roi_extractor.num_inputs,
                   ,'x[0] shape',x[0].shape)
print('mask fwd not train',x[:mask_roi_extractor.num_inputs])

This is error i start getting

0/122, elapsed: 0s, ETA:torch.Size([100, 5]) mask_rois simple test (tensor([[[[ 1.2903e-01,  8.9258e-01,  7.3486e-01,  ...,  2.6465e+00,...  <this is printing some tensor some big value>
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [0,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

Below is the further error after this. During Train i dont get the error , only during the validation part at the same point I did turn on cuda launch
return self.roi_head.simple_test(
--> 184             x, proposal_list, img_metas, rescale=rescale)
    185 
    186     def aug_test(self, imgs, img_metas, rescale=False):

/kaggle/working/mmdetection/mmdet/models/roi_heads/dsc_roi_head.py in simple_test(self, x, proposal_list, img_metas, rescale)
    570                 print(mask_rois.shape,'mask_rois simple test',  x)
    571                 mask_results = self._mask_forward(self.num_stages, x, mask_rois, 
--> 572                                                   mpn_results['enlarged_rois'][det_rois_inds]
    573                                                   ,semantic_feat,
    574                                                   mpn_results['res_feat'][det_rois_inds],img_shape,

RuntimeError: CUDA error: device-side assert triggered

@ptrblck   any possible issue with tensor type or so, During Training ,print prints all things but during validation only this issue come up. i.e print dsnt gives all values only ffirst few values