It looks like you are storing output Variables, and that is the source of the problem because when you store a Variable, you force python to keep in memory the entire computation graph for that Variable.
You should probably save the underlying tensors instead.
self.saved_actions.append(..., state_value.data, ...)