Try these following steps to figure out where the problem is:

Use the model with a simple most basic training loop. If the problem is solved check your training loop for any accumulation or (maybe make sure you’re not calling model.to(device) for each epoch)
If the error persists try using a different model. To find out if the error is in your model. (Has happened to me before and was  due to some linear layers)

Simple training loop: (no autograd or scalers)

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        pred = model(X)
        loss = loss_fn(pred, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
        if batch%100 ==0:
            loss, current = loss.item(), batch*len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

epochs = 10
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train_loop(train_dataloader, model, loss_fn, optimizer)
print("Done!")