i am using pytorch 1.0(cpu no cuda), python 3.7
i random three 2048-D Tensors ( i , j , k ) ,
first a Linear layer transfer these Tensor to 128-D Tensor (seem others are also exist this problem)
second, calculate loss=-logsigmoid(i*j-i*k)
but, when  i*j - i*k < 0  , the loss can increase rapidly,  i*j - i*k>0 , the loss can decrease slowly
Theoretically,the loss will be lower in any condition , isn’t it?
here is a simple demo:
import torch as T
from torch.nn import Module, functional as F, Linear
from torch.optim import SGD, Adam
vis = 2048
x = T.randn([vis],requires_grad=True)
W = Linear(vis,128)
z = T.randn([vis])
bb = T.randn([vis])
for i in range(30):
    
    y = -F.logsigmoid((W(x)).unsqueeze(0).mm((W(z)).unsqueeze(0).t())- (W(x)).unsqueeze(0).mm((W(bb)).unsqueeze(0).t()))
    opt = SGD(W.parameters(),lr=0.07,weight_decay=0.01)

    opt.zero_grad()
    print((W(x)).unsqueeze(0).mm((W(z)).unsqueeze(0).t()) , (W(x)).unsqueeze(0).mm((W(bb)).unsqueeze(0).t()))
    print(y)
    y.backward()
    opt.step()

thanks a lot~!