I guess you can compute these averages on the fly.
Accumulate the values for each policy and the number of samples per policy. At the end just divide the accumulated value by the number of samples for each policy.