I have a similar question here 13. I simultaneously opened a query in pytorch/fairseq#779 10 to which the response was that there is built in averaging.
How about trying some black box experiments to figure out?