I set rpc_timeout = 3600 (1 hr) and it ran for around 2 mins 11 seconds (after rpc_init) then timed out.
I also got following (After rpc_init) on the workers that I forgot to mention in my question,
Failed to respond to 'Shutdown Proceed' in time, got error RPCErr:1:RPC ran for more than set timeout (5000 ms) and will now be marked with an error.

This gets printed before the response (for workers) I have shown in my question.