Solved by Ranahanocka in post #4 
              
              
                The optimizer is optimizing the model weights, so there needs to be some parameters to optimize. Indeed, conv, or linear layer would work fine. Did that solve your problem?