Indeed that was the syntax.
Though, when I try to store the weights of say my first fully connected layer as in
all_weights = []

for epoch in range(start_epoch, start_epoch+epochs):
    train_loss = 0
    correct = 0
    total = 0
    ip_loss = []; 
    grad_loss = []
    for batch_idx, (inputs, targets) in enumerate(trainloader):
        inputs, targets = inputs.to(device), targets.to(device)
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        _, diag_args = optimizer.step()
        ip_loss.append(diag_args['ip_loss'])
        grad_loss.append(diag_args['grad_loss'])
        train_loss += loss.item()
        _, predicted = outputs.max(1)
        total += targets.size(0)
        correct += predicted.eq(targets).sum().item()
        logger_diag.append([lr, momentum,newstat, train_loss, 100.*correct/total, np.sum(ip_loss)])

        #Append weights of fc1 through the iterations
        all_weights.append(net.fc1.weight.data.detach().numpy())

Those weights appear to be constant through the iterations/epochs.
The weights should be updated at each optimizer.step()
Any idea why?