Yes, your explanation is correct.
The first approach of multiplying the averaged batch loss by the batch size and dividing by the number of samples gives you the correct average sample loss for this particular epoch.
The second approach of dividing the averaged batch loss by the number of batches would yield the same result, if each batch in the epoch contains batch_size samples. This might not always be the case, if the length of the dataset is not divisible by the batch_size without a remainder. The last batch would thus contain less samples and the loss calculation would introduce a small bias.