Hi @ptrblck
I confirm that using scheduler will reproduce the issue (after about 8 hours of training), and removing scheduler will avoid the issue. Currently I’m running the model without the scheduler for 48 hours and I don’t observe such problem.
Here it is all scheduler related lines of my code:
# optimizer
optimizer = torch.optim.Adam(
    [
        {'params': model.parameters(), 'lr': args.lr},
        {'params': criterion.parameters(), 'lr': args.criterion_lr}
    ]
)

# lr schedule
scheduler = torch.optim.lr_scheduler.StepLR(
    optimizer,
    step_size=args.step_size,
    gamma=args.gamma
)

for epoch in range(scheduler.last_epoch, args.num_epochs):
    for x, target in tqdm(dataloader):
        x = x.to(device)
        target = target.to(device)
        y = model(x)
        scores, loss = criterion(y, target)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    scheduler.step()
    lr = optimizer.state_dict()['param_groups'][0]['lr']
    log.add_scalar('train-lr', lr, epoch + 1)

Could you please inform me if I used scheduler in a wrong way?!
regards