@DanielWang Thanks for posting the question, in the second case, did you found there’re any errors when using it? My guess is that since map() is written in C and did some optimizations, its implied loop can be more efficient than a regular Python for loop, so the order might be different, and since all_reduce is happening in SPMD fashion, a mis-order might incur some issues.