Just checked on my system (don’t forget to set CUDA_VISIBLE_DEVICES=0, use examples provided in the container)
root@7fbaac51be2c:/opt/pytorch/examples/imagenet# python main.py -j 10 -a resnet50 --batch-size 128 --fp16 ./
=> creating model 'resnet50'
Epoch: [0][10/10010]    Time 0.184 (0.479)      Data 0.000 (0.162)      Loss 11.3594 (8.9347)   Prec@1 0.781 (0.284)    Prec@5 1.562 (0.568)
Epoch: [0][20/10010]    Time 0.183 (0.338)      Data 0.000 (0.085)      Loss 7.6914 (8.7134)    Prec@1 0.000 (0.186)    Prec@5 0.000 (0.335)
Epoch: [0][30/10010]    Time 0.184 (0.289)      Data 0.000 (0.058)      Loss 7.0508 (8.2504)    Prec@1 0.000 (0.126)    Prec@5 1.562 (0.378)
Epoch: [0][40/10010]    Time 0.183 (0.263)      Data 0.000 (0.044)      Loss 7.7812 (8.0275)    Prec@1 0.000 (0.114)    Prec@5 0.000 (0.362)
Epoch: [0][50/10010]    Time 0.183 (0.247)      Data 0.000 (0.035)      Loss 6.9414 (7.8232)    Prec@1 0.000 (0.107)    Prec@5 0.000 (0.383)
Epoch: [0][60/10010]    Time 0.183 (0.237)      Data 0.000 (0.029)      Loss 6.8828 (7.6770)    Prec@1 0.000 (0.090)    Prec@5 0.000 (0.346)
Epoch: [0][70/10010]    Time 0.184 (0.230)      Data 0.000 (0.025)      Loss 6.9219 (7.5694)    Prec@1 0.000 (0.099)    Prec@5 0.000 (0.385)
Epoch: [0][80/10010]    Time 0.184 (0.224)      Data 0.000 (0.022)      Loss 6.9336 (7.4884)    Prec@1 0.000 (0.096)    Prec@5 1.562 (0.415)
Epoch: [0][90/10010]    Time 0.183 (0.220)      Data 0.000 (0.020)      Loss 6.9375 (7.4253)    Prec@1 0.000 (0.094)    Prec@5 0.000 (0.412)
Epoch: [0][100/10010]   Time 0.183 (0.216)      Data 0.000 (0.018)      Loss 6.9141 (7.3742)    Prec@1 0.000 (0.108)    Prec@

so batch time is ~183ms, average will gradually come down. For batch size of 256 I also don’t go OOM:
 root@7fbaac51be2c:/opt/pytorch/examples/imagenet# python main.py -j 10 -a resnet50 --batch-size 256 --fp16 ./
=> creating model 'resnet50'
Epoch: [0][10/5005]     Time 0.344 (0.916)      Data 0.000 (0.255)      Loss 9.5859 (8.3917)    Prec@1 0.000 (0.036)    Prec@5 0.000 (0.320)
Epoch: [0][20/5005]     Time 0.342 (0.644)      Data 0.000 (0.134)      Loss 7.2852 (8.1789)    Prec@1 0.000 (0.037)    Prec@5 0.391 (0.372)
Epoch: [0][30/5005]     Time 0.347 (0.548)      Data 0.000 (0.091)      Loss 7.3242 (7.9689)    Prec@1 0.000 (0.088)    Prec@5 0.781 (0.454)
Epoch: [0][40/5005]     Time 0.344 (0.498)      Data 0.000 (0.069)      Loss 7.2109 (7.7749)    Prec@1 0.000 (0.086)    Prec@5 0.000 (0.467)
Epoch: [0][50/5005]     Time 0.344 (0.468)      Data 0.000 (0.055)      Loss 7.0781 (7.6285)    Prec@1 0.000 (0.123)    Prec@5 0.781 (0.536)
Epoch: [0][60/5005]     Time 0.344 (0.448)      Data 0.000 (0.046)      Loss 7.4844 (7.5353)    Prec@1 0.000 (0.109)    Prec@5 0.781 (0.570)
Epoch: [0][70/5005]     Time 0.344 (0.433)      Data 0.000 (0.040)      Loss 6.9570 (7.4699)    Prec@1 0.000 (0.105)    Prec@5 0.391 (0.567)

Could be that you are bottlenecked by data loader? Usually it is not a problem for a single GPU, though. Also, if spectre/meltdown patches are applied on your system, that could decrease performance a bit, but also, for a single GPU difference should not be big.