I have implemented a custom DataLoader batch_sampler to have dynamic batch sizes during training. The first batch has a fixed size but the rest do not. e.g:
original_batch_size = 5

iteration 1: original_batch_size samples
iteration 2: 8 samples
iteration 3: 13 samples

Does that mean that I should scale the learning rate every iteration before calling optimizer.step()?
# pseudocode:
prev_lr = optimizer["lr"]
# scale the learning rate depending on the current batch size
# for example, one batch may have 2 samples but another may have 100
# and we dont want to use the same learning rate in both cases
optimizer["lr"] = prev_lr / original_batch_size * current_batch_size
# update parameters
optimizer.step()
# keep original value
optimizer["lr"] = prev_lr

This would also apply when using a DataLoader with drop_last=False, where the last batch might have less samples than the rest of the batches, correct?