Hello,
I am currently trying to learn some reinforcement learning and I have some issues with the management of gradient.
I am in a situation where I have several units! on a board, and I create my input this way:
main_input = torch.stack([self.translated(agent_state.main_input, unit.x, unit.y) for unit in agent_state.player.units])

    @staticmethod
    @torch.no_grad()
    def translated(matrix, x, y):
        return torch.roll(torch.roll(matrix, x, dims=-2), y, dims=-1)

I create my input this way in my forward pass because I intend to use memory replay, and saving the whole state would be too big (dim = (n_unit, 5, 20, 20)). Therefore I create a (5,20,20) tensor that I translate.
I wanted to know at which point I should do a “no grad” or else to avoid issue with the gradient step.(while creating the “main_input” in the “AgentState” class ? or maybe it is okay to just put it as a decorator inside the translate function ?)
Moreover I noticed that when I was debugging that my Tensor seems to be repeating in himself in his “data” attribute, which I guess is not normal, and I don’t seem to find how this behavior can happen.
Capture760×437 11.8 KB
Lastly, after a long time of investigation, I still get an error, which is certainly in relation with my first two interrogations.
"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [50, 6]], which is output 0 of TBackward, is at version 3; expected version 1 instead.
If someone have some advices related to these issues, I would be very interested.
Thank you!
Thomas