Oh thank you so much, I didn’t realize this mechanic about optimization before your comment, it made me understand many things about Pytorch
I tried to change my code, according to what you said in order to make the two code as similar as I possibly can, but I don’t know why it is still not working
import numpy as np

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Linear(10, 4)

    def forward(self, x):
        x = self.fc(x)
        return x

    
def custom_loss(delta):
    def loss(y_pred, y_true):
        y_pred_softmax = nn.Softmax(dim=1)(y_pred)
        y_pred_softmax_clipped = torch.clamp(y_pred, 1e-8, 1 - 1e-8)
        log_likelihood = y_true * torch.log(y_pred_softmax_clipped)
        return torch.sum(-log_likelihood * delta)
    return loss


delta = 1
batch_size = 32
n_sample = 1000
network = Net()
loss_function = custom_loss(delta)
optimizer = optim.Adam(network.parameters(), lr=0.01)

x = torch.ones((1,10))
print(network(x))
print(nn.Softmax(dim=1)(network(x)), '\n')

target = [0.7, 0.3, 0.0, 0.0]

for i in range(int(n_sample/batch_size)):
    optimizer.zero_grad()
    
    inputs = torch.ones((batch_size,10))
    targets = torch.FloatTensor(batch_size*[target])
    outputs = network(inputs)
    
    loss = loss_function(outputs, targets)
    loss.backward()
    optimizer.step()

x = torch.ones((1,10), dtype=torch.float)
print(network(x))
print(nn.Softmax(dim=1)(network(x)))

I changed from SGD to Adam, just in case that would affect something (even if I doubt it would) as well as removing the momentum. I used the example of the pytorch website to be sure that I was not missing a step in the training procedure.
However it still doesn’t work as intended 
Two examples of outputs:
tensor([[-0.5445,  1.1960, -0.1856,  0.3652]], grad_fn=<AddmmBackward>)
tensor([[0.0942, 0.5370, 0.1349, 0.2340]], grad_fn=<SoftmaxBackward>) 

tensor([[-0.5445,  1.1960, -0.1856,  0.3652]], grad_fn=<AddmmBackward>)
tensor([[0.0942, 0.5370, 0.1349, 0.2340]], grad_fn=<SoftmaxBackward>)

tensor([[-0.1185,  0.5827,  0.7683,  0.4973]], grad_fn=<AddmmBackward>)
tensor([[0.1371, 0.2764, 0.3327, 0.2538]], grad_fn=<SoftmaxBackward>) 

tensor([[-0.1185,  1.8670,  0.7683,  0.4973]], grad_fn=<AddmmBackward>)
tensor([[0.0796, 0.5798, 0.1932, 0.1474]], grad_fn=<SoftmaxBackward>)

output with the TF code:
[[ 0.0284583   0.35027373  0.4858752  -0.61461353]]
[0.2229509  0.30758977 0.35225958 0.11719974]
Train on 1000 samples
1000/1000 [==============================] - 0s 302us/sample - loss: 25.1664
[[ 2.3004684  1.4706173 -1.8008366 -2.9013252]]
[0.68579024 0.29908288 0.01135056 0.00377643]

I also tried to increase the number of sample and modify the learning rate, but the results were still the same
Did you see other mistake in my code that could cause that ?
(again, thank you  )
edit: I checked the forward and backward graph and all the function of the loss function are present except softmax, is it normal ?
from torchviz import make_dot, make_dot_from_trace
make_dot(loss, params=dict(network.named_parameters()))