Solved by tom in post #2 
              
              
                The latter. 
In particular, the intermediate results needed for backward are typically on the GPU where you did the forward. 
Best regards 
Thomas