Hello Sangyeon_Kim,
This might help you with your implementation.



 Sangyeon_Kim:

I’d like to make a GPU dedicated to data processing


You can use multiple processes and assign a process to be the data processer. The data processer would use a GPU device.



 Sangyeon_Kim:

pass them to other GPUs


You can communicate the processed data over RPC.



 Sangyeon_Kim:

while other GPUs learn using the processed data in parallel.


Each trainer will be a process. GPU devices will be assigned to the process. Each trainer will create a DDP instance.
ddp doc Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.8.1+cu102 documentation
distributed rpc doc Distributed RPC Framework — PyTorch 1.8.1 documentation 1