In the SimpleNet forward() function, the zeros tensor z is being created on CPU, which is why the out tensor returned by the forward function of that block returns out.is_cuda = False. You must either place the z tensor explicitly on the correct rank or make it an nn.Parameter or similar type that will be moved to GPU when the entire module is placed on GPU.