It sounds very similar to Q-learning - just use .detach() to prevent the gradients from getting propagated:
action_distribution0 = model(state).detach()
# detach() blocks the gradient
# also action_distribution0 doesn't require grad now, so it can be a loss target

# =========================================
# change the parameters of the network here
# =========================================

KL(Network(state), action_distribution0).backward()