When I try to use only
preValue = self.value_net(state) # here without .data
preValue = Variable(preValue)  # get rid of this line.
it works. Maybe it is the mechanism of the Variable and tensor is not matched for .backward()