You only calculated loss once… So essentially you only did one training step. PyTorch uses dynamic graph, unlike the static graphs in tf or theano. As you wrote:



 vishalbhalla:

R = torch.nn.functional.softmax(pre_R)
F = torch.stack(Variable(torch.from_numpy(X).type(dtype), requires_grad=True))
U = Variable(torch.from_numpy(gmm.means_).type(dtype), requires_grad=False)
z_pred = torch.max(R, 1)[1]
distances = torch.sum(((F.unsqueeze(1) - U) ** 2), dim=2)
custom_loss = torch.sum(R * distances) / num_samples
learning_rate = 1e-3
opt_train= torch.optim.Adam([pre_R], lr = learning_rate)
U = torch.div(torch.mm(torch.t®, F), torch.sum(R, dim=0).unsqueeze(1)) #In place assignment with a formula over variables and hence no gradient update is needed.


PyTorch is doing computation on actual values, rather than symbols. So you will want to compute loss in each iteration. In each iteration, you should create a graph, and optimize it. So retain_graph shouldn’t be there in your case. In fact, you usually don’t need to set that in general. Hope this helps.