Thanks a lot for your reply. Please, see below a toy code, with a toy layer. This toy layer treats the input in the same manner as my custom layer: creating a numpy array using copies of values from the input. This toy layer has an explicit but simple backward function, which is the case for my custom layer, also. Thus, this code behaves in the same manner as my original code.
So far, I have been testing optimization problems only. Hence, this toy example is also an optimization problem. I can define optimization problems without that custom layer, also and they behave as expected. So, there is a problem with that custom layer.
Yes, I had come across the link that you have shared earlier.

import torch
import numpy as np
class elemProdLayer(torch.nn.Module):
def __init__(self):
    super(elemProdLayer, self).__init__()

def forward(self, scaleVec, oneVec):

    self.oneVec = oneVec

    vecSiz = len(scaleVec)

    scaleVecLocal = torch.zeros(vecSiz)

    for iVec in range(vecSiz):
        scaleVecLocal[iVec] = scaleVec[iVec].item() 

    return torch.mul(scaleVecLocal, ipVec)

def backward(self, opVec):
    return torch.mul(self.oneVec, opVec)

device = torch.device(‘cpu’)
N = 5
oneVec = torch.ones(N, requires_grad = False)
ipVec = torch.randn(N, device=device, requires_grad = True)
tgtVec = torch.randn(N, device=device, requires_grad = False)
stepSiz = 1e-2
myElemProdLayer = elemProdLayer()
for t in range(500):
scaleVec = myElemProdLayer(ipVec, oneVec)
scaleVec = torch.tensor(scaleVec, requires_grad = True)
loss = torch.nn.MSELoss()(tgtVec, scaleVec)
loss.backward()
with torch.no_grad():
  ipVec -= stepSiz*ipVec.grad

  ipVec.grad.zero_()