A few other thoughts:
Do you have a chance to set async_op=True in dist.all_reduce, so that you can overlap some computation with allreduce?
Additionally, probably you can try to tune some Gloo environment variables. I am not familiar with Gloo, but I see tuning some NCCL parameters can help communication performance a lot.

I’m integrating with another library that doesn’t play very well with DDP

Also interested in knowing the reasons that cause DDP not feasible for this case.

If I used the NCCL backend, would the communication speed decrease at the same rate as Gloo (so that e.g. doubling the number of processes / process groups causes all_reduce to run at half speed)?

NCCL will have both a better scaling efficiency and higher raw speed than Gloo.