Thanks for the quick response @colesbury.
Just to clarify, the other processes in the pipeline are not python processes (they are C/C++/CUDA).  So it’s important that I can access/create IPC handles with device pointers to the raw underlying tensor data.  My confusion is how to work with these handles within the python/pytorch process.  Correct me if I’m wrong, but it seems that multiprocessing.SimpleQueue will only work the way you describe if both processes are using Pytorch.
So, just to be absolutely clear, the full plan is to use shared memory to pass IPC handles between processes.  For example, the shared memory file will include a 64byte cudaIpcMemHandle_t (containing a pointer to the raw data in GPU memory), plus additional bytes to specify the number of rows and columns in the tensor.