What is action.reinforce(r) doing actually?
Hi, Thanks.
yunjey
apaszke
yunjey
zuoxingdong
apaszke
wjaskowski
jekbradbury
wjaskowski
I think I see what you’re asking now. The gradient estimates from all the examples in the batch are added together in the stochastic node (there’s no implicit division by the batch size) so, depending on your use case, you may want to manually divide your rewards by the batch size.
jekbradbury
abhigenie92
<SCODE>x = Variable(torch.Tensor([0.1, 0.9]), requires_grad=True)
y = torch.multinomial(x, 1)  # here, make sure y.data[0] = 1
r = torch.Tensor([2]).float()
y.inforce(r)
y.backward()
print x.grad.data
<ECODE>
Crazyai
I implemented Vanilla Policy Gradient in this way and it works. However using the .reinforce method seems cleaner
volpato30
To make it simple, let’s say 1D, given a mean and std, we have a sample = mean + std*eps, where eps ~ N(0, 1). It is not very clear to me why it is like that. Since if we have a sample, according to the formula,  d_sample/d_mean = 1. So, grad_mean = upward_gradient * d_sample/d_mean
zuoxingdong
lucasb-eyer
norm
I do not notice the paper you mentioned, can you give a PDF file or the link. Thanks.
zeng
lucasb-eyer
<SCODE>"""
Try some reinforce by hand and similar
"""

import torch
from torch import autograd, nn, optim
import torch.nn.functional as F
from torch.autograd import Variable
import numpy as np


def run_model(x, h1):
    torch.manual_seed(123)
    params = list(h1.parameters())

    x1 = h1(Variable(x))
    x2 = F.softmax(x1)
    a = torch.multinomial(x2)
    print('a', a)
    return x1, x2, a


def run_by_hand(params, x, h1):
    print('')
    print('=======')
    print('by hand')
    h1.zero_grad()
    x1, x2, a = run_model(x, h1)
    g = torch.gather(x2, 1, Variable(a.data))
    log_g = g.log()
    log_g.backward(- torch.ones(4, 1))
    print('params.grad', params.grad)


def run_pytorch_reinforce(params, x, h1):
    print('')
    print('=======')
    print('pytorch reinforce')
    h1.zero_grad()
    x1, x2, a = run_model(x, h1)
    a.reinforce(torch.ones(4, 1))
    autograd.backward([a], [None])
    print('params.grad', params.grad)


def run():
    N = 4
    K = 1
    C = 3

    torch.manual_seed(123)
    x = torch.ones(N, K)
    h1 = nn.Linear(K, C, bias=False)
    params = list(h1.parameters())[0]
    run_by_hand(params, x, h1)
    run_pytorch_reinforce(params, x, h1)


if __name__ == '__main__':
    run()
<ECODE> Result: <SCODE>=======
by hand
a Variable containing:
 1
 1
 0
 1
[torch.LongTensor of size 4x1]

params.grad Variable containing:
 0.6170
-1.3288
 0.7117
[torch.FloatTensor of size 3x1]


=======
pytorch reinforce
a Variable containing:
 1
 1
 0
 1
[torch.LongTensor of size 4x1]

params.grad Variable containing:
 0.6170
-1.3288
 0.7117
[torch.FloatTensor of size 3x1]
<ECODE> (Its actually almost the same amount of code in fact?)
hughperkins