Yes, I have this problem. If I use 200 images for training, it will take about 10 seconds for the GPU to use from 1800MB to 4848MB, but if I choose 10,000 images for training, it will take about 1 hour for the GPU to go from 1800MB to out of memory.