julianolm:

Such model uses fixed convolutional binary filters, they are “randomly” set during the instantiation of the net and keep the same ever after. The only parameter updates are over linear weights that combine the feature maps generated by the filters (this operation is implemented with 1x1 convolutions).


Assuming the 1x1 convolutions are implemented via the self.fc1 layer in NetLBC, I understand that all preceding layers should not be updated.
If that’s the case, you could wrap the first part of the model into a with torch.no_grad() block to avoid storing intermediate activations, which would be needed to compute the gradients.
Also, lowering the batch size would decrease the memory usage and should help.