This is a niche question given my esoteric training strategy - I’m wondering if there is a straightforward way to initialize a parameter before every SGD update if you already have a really good guess what the parameter should be.
To briefly provide some context, I have an alternating minimization scheme, where for a given batch I apply multiple SGD updates between 2 optimizers as follows
    # see pytorch-lightning for module organization
    def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i,
                       second_order_closure, on_tpu=False,
                       using_native_amp=False, using_lbfgs=False):
        # perform multiple steps with SGD to optimize eta
        if optimizer_i == 0:
            for _ in range(self.hparams.steps_per_batch):
                loss = second_order_closure()
                optimizer.step(second_order_closure)
                optimizer.zero_grad()

        # update all of the other parameters once
        # eta is optimized
        if optimizer_i == 1:
            for _ in range(self.hparams.steps_per_batch):
                loss = second_order_closure()
                optimizer.step(second_order_closure)
                optimizer.zero_grad()

    def training_step(self, batch, batch_idx, optimizer_idx):
        self.model.train()
        counts = batch
        self.model.reset(batch)
        loss = self.model(counts)
        assert torch.isnan(loss).item() is False
        if len(self.trainer.lr_schedulers) >= 1:
            lr = self.trainer.lr_schedulers[0]['scheduler'].get_last_lr()[0]
            current_lr = lr
        else:
            current_lr = self.hparams.learning_rate
        tensorboard_logs = {
            'train_loss': loss, 'elbo': -loss, 'lr': current_lr
        }
        # log the learning rate
        return {'loss': loss, 'log': tensorboard_logs}

Here’s the actual question - I have a function that can provide a good guess on what the parameter should me, all I want to do is initialize this parameter so that I can refine the estimate of this parameter with following SGD updates - I’m trying to do this as follows.
    def reset(self, x): 
        hx = my_awesome_guess(x)
        self.eta.data.copy_(hx)

However, I’m noticing that this sort of operation essentially freezes the weights of eta, and I cannot apply SGD updates to update eta.
I should think this initialization strategy should be possible, right?