I don’t understand your use case clearly, especially the dual variables of the size of the dataset.
If the dual variable you mentioned is learnable, you could set requires_grad=True and add them to the optimizer.
Does this method fit your use case?