I wonder how we are supposed to use registered variables in optimizers (e.g. SGD). .parameters() call does not return None parameters, so if you want to register parameter it in advance, you can create optimizer only after first forward pass?