If it really is out of memory, then it should be normal for you to see a OOM. Is there a reason you expect otherwise?
Running large CNNs in CPU is especially memory demanding. If you have a GPU, use that with cuDNN instead.