Hi,
This line looks suspicious:         target_a = Variable(target_a, requires_grad = True)
Note that Variable are not a thing anymore so you can remove it.
Also it seems like both inputs to criterion(target_a,zeros_like) are actually leaf Tensors with no history? I think you break the graph earlier. In particular, reward_a is computed with estimation_control_numpy which contains a .data breaking the graph (note that you should not use .data anymore. You can replace it with .detach() here).