Just as a reference, this shows that the parameters are still 0 after some epochs during the second iteration of the cross-validation.

I should mention that I am using two nn.modules. The first is a CNN that is always updated and the other is the CRF layer I showed before. I am using an optimizer that looks like this:
params = list(model.parameters())  # CNN parameters
params.extend(list(modelcrf.parameters()))  # Pairwise parameters
optimizer2 = optim.Adadelta(params, lr=1.0)