The problem of EpochMetric within distributed data parallelism is that each process collects its own list of _predictions and _targets. There is no syncronization of the whole seen histories across participating processes. We can also imagine that this can potentially raise OOM when we do all_gather operation (which is missing currently, but we still plan to add it).
In your Mape implementation, _num_examples and _sum_percentages looks like to be scalars, so sync_all_reduce will collect the data across the processes and in compute method all processes will manipulate “total” self._sum_percentages, self._num_examples. So, it is OK.

Does device here mean machine or gpu?

Device means single gpu here