This happens because when using RPC with CUDA tensors we need to inspect the values held by the Future objects in order to extract the tensors they contain (because we need to “track” those tensors across CUDA streams). Since we inspect those values from C++, we need to first convert Python objects to JIT objects in order to be able to handle them. This apparently is more expensive than we thought. Note that this only happens for CUDA tensors, and it only happens for callbacks (i.e., when you use the then method) because in those cases the value is a user-provided one and thus we need to re-inspect it every time as we cannot assume anything about it.
We were discussing possible changes to how CUDA Futures work, and this consideration will factor in, but I don’t yet know what the final outcome will be.