@lesscomfortable can you share the code and the command you execute to reproduce the issue.
Here is how it would run CIFAR10 script on CPU multi-core (single node) in distributed way:
CUDA_VISIBLE_DEVICES="" python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py run --backend=gloo

To ensure that it is not a visual effect that program gets stuck as a single epoch on cifar10 on CPU can several minutes to execute. You can add the progress bar:
    trainer = Engine(train_step)

    if idist.get_rank() == 0 and (not config["with_clearml"]):
        common.ProgressBar(persist=False).attach(trainer)

In my case it looks like:
Epoch [1/24]: [4/97]   4%|██████▏                              [00:07<03:45]

HTH