Hi @tom, I made test and this is result.
just for info-
When I am loading photos every loop and pushing them to Model training the training lasts 5 hours(test on 700 images).
Test 1: - Still slow
I deleted from training loop model, backward, loss, and left there only images loader, (no GPU work) - “training” lasts 4 hours.
Test 2: – Extremely fast
I deleted from training loop images loader and I loaded batch only once before training loop. So GPU was working with only one batch whole training. And training lasts only 1 hour and GPU Utilization was 100% all time.
From this we know, loading of images is slowing down whole training. When CPU is loading images for so long GPU does nothing.
Can you now confirm or refute if this is weak CPU or bad images loader optimization  ?
If you want to see this is images loader -PyTorch-AI-road-detection-classification/Model_loader.py at main · Samuel-Bachorik/PyTorch-AI-road-detection-classification · GitHub 2