Thanks, @mcarilli. A few iterations in this context is about 1000 iterations.
And you’re right scaler.scale(loss).backward() should be outside the autocast context. The actual fix was due to how PyTorch did dynamic scaling. For my specific use-case, I had to set the growth_interval parameter to something smaller like 10 iterations. The issue was the scale value was not high enough causing a flat loss. Once the model started learning I’ve reset the growth_interval to the default value 2000 iterations.
my updated code is as follows.

scaler = GradScaler(growth_interval=10)
for i, _data in enumerate(train_loader):
    spectrograms, labels, input_lengths, label_lengths = _data 
    spectrograms, labels = spectrograms.cuda(), labels.cuda()
    with autocast():
        output = model(spectrograms)  # (batch, time, n_class)
        output = F.log_softmax(output, dim=2)

        loss = ctc_loss(output, labels, input_lengths, label_lengths)
        loss = loss / args.grad_acc_steps
    scaler.scale(loss).backward()

    if i % args.grad_acc_steps == 0:
        scaler.unscale_(optimizer)  # unscale to clip gradient
        nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        scaler.step(optimizer)
        scaler.update()
        scheduler.step()
   if scaler.get_growth_interval() != 2000 and total_iter > 1000:
        scaler.set_growth_interval(2000)

This solution works for me and hopefully it can help someone who runs into the same issue.