The following cuda call works perfectly in creating each layer on the device:
for layer, p in zip(model.children(), model.named_children()):
    if type(layer) == nn.Linear:
        setattr(model, p[0], type(layer)(2, 2).cuda(device))