Hey, I wanted to ask if there is a difference of feeding a batch to the model and then update the weights or feeding data solely to the model and sum up the loss and update after the batch size. I am aware that this depends also on the used criterion, but can I do the following example in praxis? (My GPU-mem is too small to train with big batches and I would like to know if this is a workaround)
batch = load_batch()
out = model(batch)
loss = criterion(out, label)
loss.backward()
optimizer.step()

loss = 0
for data in batch:
    out = model(data.unsqueeze(0))
    loss += criterion(out, label)
loss.backward()
optimizer.step()