The way I would do is it to push the output batches into a multiprocessing.queue. Allow the network forward pass to be in a separate process.
Then, have a few threads of post-processor workers listen to this queue.