Hello all, I am trying to train an LSTM in the half-precision setting. The LSTM takes an encoded input from a pre-trained autoencoder(Not trained in fp16). I am using  torch.amp instead of apex and scaling the losses as suggested in the documentation.
Here is my training loop -
    def train_model(self, model, dataloader, num_epochs):
        model.cuda()
        least_loss = 5
        model.train()
        optimizer = torch.optim.Adam(model.parameters(), lr =1e-5)
        scaler = amp.GradScaler()
        training_loss = []
        for i in range(0, num_epochs + 1):
            st = time.time()
            training_acc = 0
            epoch_loss = 0
            for _, (x, y) in enumerate(dataloader):
                optimizer.zero_grad()
                sst = time.time()
                x = x.float().half().cuda()
                x, out = self.autoencoder(x)
                x = x.permute(0,2, 1)
                model.init_Hidden()
                y = y.cuda()
                output = model(x)
                loss = self.criterion(output, y)
                scaler.scale(loss).backward()
                scaler.step(optimizer)
                scaler.update()

I call my model as -
lstm =lstm(features=1024, hidden_size=512, sequence_length=313, autoencoder=model).half().cuda()

I am getting the followiing error -
ValueError: Attempting to unscale FP16 gradients.

Could someone please tell why would this be happening
TIA