I guess you want to check what param[0] is and why they use that.
Then make sure that this requires gradients.
Note that if gradients are needed to compute your gradient penalty loss but you don’t want to update your embedding, you can keep the embedding as requiring gradient but don’t give it to the optimizer so that it will never be updated.