Ah, actually I see it now.
By the way does defining a loss on this net and doing loss.backward() create a copy of the parameters? I run out of memory on doing loss.backward. My gpu has 12 Gigs of memory and before the call to loss.backward I have exhausted 10 gigs.