Therefore, can I modify your sentence to say that currently, there is no way of getting Metric to work across multiple gpus in all case scenarios easily?

Well, majority of ignite’s metrics like Accuracy, Precision, Recall, ConfusionMatrix etc are working in DDP for their use-cases 

And is it safe to say that to fix this, every time the user needs to make a custom metric that does not use @sync_all_reduce and instead uses dist.all_gather?

It depends on the metric and how it should be computed: how to compute accumulators for an iteration. If in update function you do += on tensors, scalars etc. @sync_all_reduce  will work without problems and the result will be correct. If you do something else like concat of lists etc, @sync_all_reduce  wont work and at compute time we had to collect all the data.

Sure I’ll look into it, it’ll depend on time and if someone can help me because I’m still just learning

We can help with that. Just if anyone could initialize the work and put some code etc…