Solved by iffiX in post #2 
              
              
                Actually in real applications, whether to update your actor&critic per step or update them after collecting several episodes does not quite affect the performance of your algorithm. 
However, it could be very inefficient to compute backward pass per step, as you cannot fill up your cpu ALU units or …