Like others said, your approach/idea proposal sounds fine, but I think it would be more suited for external PyTorch wrapper libraries as coupling these would make research much harder if you lose control over these things. But also in many practical scenarios it would be a bit limiting if you have multiple networks and want to optimize different parts of a network (or module) at separate times.