hi, thank you for your reply!
i have tried what you said, yes, you are right ! That helps me a lot, and some images is too large, which makes training crash, and the reason why training can complete on the small dataset is there is no large image!
But now i wonder why it always crash on Normalization layer, like what you can see above, the end of ERROR shows it crash in group_norm.
And i have try batchnorm, it also crash on this place.
Traceback (most recent call last):
  File "/home/xsl/code/ResNet50/ModifyBatchLayer/ResnetWithGNTrainVal.py", line 255, in <module>
    model_ft = train_model(model_ft, dataloaders, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=num_epochs)
  File "/home/xsl/code/ResNet50/ModifyBatchLayer/ResnetWithGNTrainVal.py", line 165, in train_model
    outputs = model(inputs)
  File "/home/xsl/.conda/envs/xsltf/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xsl/code/ResNet50/ModifyBatchLayer/ResnetWithGN.py", line 182, in forward
    x = self.layer3(x)
  File "/home/xsl/.conda/envs/xsltf/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xsl/.conda/envs/xsltf/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/xsl/.conda/envs/xsltf/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xsl/code/ResNet50/ModifyBatchLayer/ResnetWithGN.py", line 96, in forward
    out = self.bn3(out)
  File "/home/xsl/.conda/envs/xsltf/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xsl/.conda/envs/xsltf/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 76, in forward
    exponential_average_factor, self.eps)
  File "/home/xsl/.conda/envs/xsltf/lib/python3.6/site-packages/torch/nn/functional.py", line 1623, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 126.62 MiB (GPU 0; 11.91 GiB total capacity; 11.07 GiB already allocated; 107.06 MiB free; 93.97 MiB cached)