Ok, I have done this like this for now, unless anyone has a better idea?
p = (
    set(encoder.parameters()) |
    set(decoder.parameters()) |
    set(embedding.parameters()))
opt = optimizer_fn(p, lr=0.001)