Thank you for your kind advice. I tried your advice, but problems still happened.
I created two files in my github(GitHub - cs123951/temp_public_files 1)  to explain my problem.
I constructed a bidirectional RNN model using model parallelism like this.

image1034×442 28.7 KB

The difference is that the RNN is bidirectional so I made some modifications.
I distributed the GPU-0 to the first several layers, and the GPU-1 to the last layer.
The forward process is normal. But when it comes to the loss.backward(), error will happen:
RuntimeError: grad.device() == bucket_view.device() INTERNAL ASSERT FAILED at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:206, please report a bug to PyTorch. 

You can reproduce my results using the simple_bug_file2.py in my github by the following command:
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=1 --node_rank=0 --master_addr=xxx.xxx.xx.xx --master_port=31234 simple_bug_file2.py
Don’t forget to change the master address in the command and the visible GPUs in the file.
My environment is
PyTorch version: 1.4.0
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A
GPU: GeForce GTX 1080 Ti

OS: Ubuntu 18.04.3 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.3.2

Python version: 3.7 (64-bit runtime)
Nvidia driver version: 455.28
HIP runtime version: N/A
MIOpen runtime version: N/A

pip 21.0.1

By the way, when I changed down_device = self.device_ids[1] into down_device = self.device_ids[0]. It performs well.
I guess maybe bucket_view is related to the unroll operation of the RNN. But I am not familiar with it. Could you give me some more advice?
Thank you very much.