Solved by ptrblck in post #2 
              
              
                The forward pass would calculate the forward activations, which would also need to be stored temporarily, so the OOM issue could be expected. What was the peak memory usage using the 512x512 images?