try to use sam
  
      
      github.com
  
  
    

davda54/sam


  SAM: Sharpness-Aware Minimization (PyTorch). Contribute to davda54/sam development by creating an account on GitHub.

  
  
    
    
  
  


torchversion: 1.7.0+cu110
ubuntu : 18.04
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1280, 5]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
following issue with code
def train_model_sam(model, epoch,train_loader,val_loader,optimizer,criterion,scheduler):
    model.train() 
    
    losses = AverageMeter()
    accs = AverageMeter()
    
    tk = tqdm(train_loader, total=len(train_loader), position=0, leave=True)
    for idx, (imgs, labels) in enumerate(tk):
        imgs_train, labels_train = imgs.cuda(), labels.cuda().long()
        output_train = model(imgs_train)

        #loss = criterion(output_train, labels_train)

        # first forward-backward pass
        optimizer.zero_grad() 
        loss = criterion(output_train, labels_train)  # use this loss for any training statistics
        loss.backward(retain_graph = True)
        optimizer.first_step(zero_grad=True)
        
        # second forward-backward pass
        optimizer.zero_grad() 
        criterion(output_train, labels_train).backward()
        optimizer.second_step(zero_grad=True)

        
        #optimizer.zero_grad() 
        #loss.backward()
        #optimizer.step() 
        
        accs.update((output_train.argmax(1)==labels_train).sum().item()/imgs_train.size(0),imgs_train.size(0))
        losses.update(loss.item(), imgs_train.size(0))

        tk.set_postfix(loss=losses.avg,acc=accs.avg)
        
    return losses.avg