Hi All,
Got the answer myself. Because we are using TORCH_MODULE() wrapper.  Which creates a shared pointer for us. Hence instead of m.to(device), we should use m->to(device). Notice the -> insetd of .(dot). That builds as expected.