fedml@ip-172-31-46-221:/home/ec2-user/FedML/fedml_core/distributed/test/test_rpc$ sh run_rpc.sh TRPC 0
rank - 0 - 2021-10-13,03:21:54.277 main.py[line:86] INFO Namespace(backend='TRPC', enable_cuda_rpc=False, gpu_mapping_file='gpu_mapping.yaml', gpu_mapping_key='mapping_default', grpc_ipconfig_path='grpc_ipconfig.csv', rank=0, trpc_master_config_path='trp
c_master_config.csv')
rank - 0 - 2021-10-13,03:21:54.277 trpc_comm_manager.py[line:38] INFO using TRPC backend
Worker rank 0 initializing RPC
Creating the object
rank - 0 - 2021-10-13,03:21:54.277 trpc_comm_manager.py[line:58] INFO /home/ec2-user/FedML/fedml_core/distributed/test/test_rpc
rank - 0 - 2021-10-13,03:21:54.277 trpc_comm_manager.py[line:76] INFO str_init_method = tcp://172.31.46.221:9999
terminate called after throwing an instance of 'c10::Error'
  what():  device_index >= 0 && device_index < num_gpusINTERNAL ASSERT FAILED at "../c10/cuda/CUDAStream.cpp":254, please report a bug to PyTorch. 
Exception raised from check_gpu at ../c10/cuda/CUDAStream.cpp:254 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fdbf4447a22 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x5f (0x7fdbf44444af in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::getStreamFromPool(bool, signed char) + 0x177 (0x7fdbf469e187 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x8d05 (0x7fdbf46a0d05 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xe40cc4 (0x7fdc4b3b4cc4 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xe40e48 (0x7fdc4b3b4e48 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0xe93170 (0x7fdc4b407170 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0xe9334f (0x7fdc4b40734f in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #8: tensorpipe::PipeImpl::callReadDescriptorCallback(tensorpipe::OpsStateMachine<tensorpipe::PipeImpl, tensorpipe::ReadOperation>::Iter) + 0x209 (0x7fdc4b409ac9 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0xea2608 (0x7fdc4b416608 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #10: tensorpipe::PipeImpl::advanceReadOperation(tensorpipe::OpsStateMachine<tensorpipe::PipeImpl, tensorpipe::ReadOperation>::Iter, tensorpipe::ReadOperation::State) + 0xf3 (0x7fdc4b400d33 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorc
h_python.so)
frame #11: <unknown function> + 0xea69c2 (0x7fdc4b41a9c2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #12: <unknown function> + 0xe9857d (0x7fdc4b40c57d in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #13: tensorpipe::ContextImpl::deferToLoop(std::function<void ()>) + 0x154 (0x7fdc4b3e43a4 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0xe8ef31 (0x7fdc4b402f31 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #15: <unknown function> + 0xf10c95 (0x7fdc4b484c95 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #16: <unknown function> + 0xf11dc3 (0x7fdc4b485dc3 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #17: tensorpipe::transport::uv::ConnectionImpl::readCallbackFromLoop(long, uv_buf_t const*) + 0x420 (0x7fdc4b502680 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #18: <unknown function> + 0xf929cf (0x7fdc4b5069cf in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #19: <unknown function> + 0x108482f (0x7fdc4b5f882f in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0x1084e6c (0x7fdc4b5f8e6c in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #21: uv__io_poll + 0x356 (0x7fdc4b5fd646 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #22: uv_run + 0x107 (0x7fdc4b5f2f27 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #23: tensorpipe::transport::uv::Loop::eventLoop() + 0x1d (0x7fdc4b50bf5d in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #24: <unknown function> + 0xf8074c (0x7fdc4b4f474c in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #25: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tensorpipe::EventLoopDeferredExecutor::*)(std::string), tensorpipe::EventLoopDeferredExecutor*, std::string> > >::_M_run() + 0x41 (0x7fdc4b4f3ff1 in /usr/local/lib/python3.6/dist-
packages/torch/lib/libtorch_python.so)
frame #26: <unknown function> + 0xbd6df (0x7fdc4e7b96df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #27: <unknown function> + 0x76db (0x7fdc630026db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #28: clone + 0x3f (0x7fdc6333ba3f in /lib/x86_64-linux-gnu/libc.so.6)


[ip-172-31-46-221:01840] *** Process received signal ***
[ip-172-31-46-221:01840] Signal: Aborted (6)
[ip-172-31-46-221:01840] Signal code:  (-6)
[ip-172-31-46-221:01840] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x3efd0)[0x7fdc63258fd0]
[ip-172-31-46-221:01840] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fdc63258f47]
[ip-172-31-46-221:01840] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fdc6325a8b1]
[ip-172-31-46-221:01840] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c957)[0x7fdc4e788957]
[ip-172-31-46-221:01840] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae6)[0x7fdc4e78eae6]
[ip-172-31-46-221:01840] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b21)[0x7fdc4e78eb21]
[ip-172-31-46-221:01840] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92d54)[0x7fdc4e78ed54]
[ip-172-31-46-221:01840] [ 7] /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so(_ZN3c106detail14torchCheckFailEPKcS2_jS2_+0x8a)[0x7fdbf44444da]
[ip-172-31-46-221:01840] [ 8] /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so(_ZN3c104cuda17getStreamFromPoolEba+0x177)[0x7fdbf469e187]
[ip-172-31-46-221:01840] [ 9] /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so(+0x8d05)[0x7fdbf46a0d05]
[ip-172-31-46-221:01840] [10] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xe40cc4)[0x7fdc4b3b4cc4]
[ip-172-31-46-221:01840] [11] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xe40e48)[0x7fdc4b3b4e48]
[ip-172-31-46-221:01840] [12] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xe93170)[0x7fdc4b407170]
[ip-172-31-46-221:01840] [13] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xe9334f)[0x7fdc4b40734f]
[ip-172-31-46-221:01840] [14] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(_ZN10tensorpipe8PipeImpl26callReadDescriptorCallbackENS_15OpsStateMachineIS0_NS_13ReadOperationEE4IterE+0x209)[0x7fdc4b409ac9]
[ip-172-31-46-221:01840] [15] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xea2608)[0x7fdc4b416608]
[ip-172-31-46-221:01840] [16] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(_ZN10tensorpipe8PipeImpl20advanceReadOperationENS_15OpsStateMachineIS0_NS_13ReadOperationEE4IterENS2_5StateE+0xf3)[0x7fdc4b400d33]
[ip-172-31-46-221:01840] [17] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xea69c2)[0x7fdc4b41a9c2]
[ip-172-31-46-221:01840] [18] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xe9857d)[0x7fdc4b40c57d]
[ip-172-31-46-221:01840] [19] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(_ZN10tensorpipe11ContextImpl11deferToLoopESt8functionIFvvEE+0x154)[0x7fdc4b3e43a4]
[ip-172-31-46-221:01840] [20] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xe8ef31)[0x7fdc4b402f31]
[ip-172-31-46-221:01840] [21] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xf10c95)[0x7fdc4b484c95]
[ip-172-31-46-221:01840] [22] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xf11dc3)[0x7fdc4b485dc3]
[ip-172-31-46-221:01840] [23] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(_ZN10tensorpipe9transport2uv14ConnectionImpl20readCallbackFromLoopElPK8uv_buf_t+0x420)[0x7fdc4b502680]
[ip-172-31-46-221:01840] [24] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0xf929cf)[0x7fdc4b5069cf]
[ip-172-31-46-221:01840] [25] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0x108482f)[0x7fdc4b5f882f]
[ip-172-31-46-221:01840] [26] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(+0x1084e6c)[0x7fdc4b5f8e6c]
[ip-172-31-46-221:01840] [27] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(uv__io_poll+0x356)[0x7fdc4b5fd646]
[ip-172-31-46-221:01840] [28] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(uv_run+0x107)[0x7fdc4b5f2f27]
[ip-172-31-46-221:01840] [29] /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so(_ZN10tensorpipe9transport2uv4Loop9eventLoopEv+0x1d)[0x7fdc4b50bf5d]
[ip-172-31-46-221:01840] *** End of error message ***
Aborted (core dumped)