Hi Yingding,
Tensorflow Metal plugin utilizes all the core of M1 Max GPU. It appears as a single Device in TF which gets utilized fully to accelerate the training. Distributed training is used for the multi-host scenario. Where different Hosts (with single or multi-gpu) are connected through different network topologies.