Try to use nn.Parameter for your return values in whole_filter and one_filter, as this will properly register these filters as internal parameters, and will thus push them also to the GPU in the model.to(device) call.