May I ask another question if you dont mind?
I noticed we have different ways for imposing/enforcing the sparsity constraint. one other way we can achieve sparsity is likely to get the average of the layers output and treat that as the loss for sparsity and add it e.g. to the reconstruction loss and do the backward pass.
That is simply do in the forward pass :
    def forward(self, input):
        input = input.view(input.size(0), -1)
        sparsity_loss = 0.0
        output = self.encoder(input)
        sparsity_loss += torch.mean(abs(output))
        if self.use_l1_penalty:
            output = L1Penalty.apply(output, self.l1_weight)
        output = self.decoder(output)
        sparsity_loss += torch.mean(abs(output))
        output = output.view(input.size(0), 1, 28, 28)
        return output, sparsity_loss

and in the trianing loop :
for e in range(epochs):
    for imgs,_ in dataloader_train:
        imgs = imgs.to(device)
        output, sparsity_loss = sae_model(imgs)
        loss = criterion(output, imgs)
        
        loss_f = loss + (sparsity_ratio * sparsity_loss)
        optimizer.zero_grad()
        loss_f.backward()
        optimizer.step()
    print(f'epoch: {e}/{epochs} loss_f: {loss_f.item():.6f} loss: { loss.item():.6f}'\
          f' sparsity loss: {sparsity_loss.item():.6f} lr = {scheduler.get_lr()}')
    scheduler.step()

However I noticed the outcome of these two methods are not the same exactly . why is that?
This is the weights outputs using the first method(L1Penalty) :

sample reconstructions at each epoch
encoder_weights_trained:
weight_diff (initial weight - trained_weights)
decoder_weights_trained:


1.png772×288 39.9 KB
 
1_1.jpg1513×1477 754 KB
 
1_2.jpg1513×1477 692 KB
 
1_3.jpg1513×1477 1.17 MB

And using the sparsity loss from activations:

encoder_weights_trained:
weight_diff (initial weight - trained_weights)
decoder_weights_trained:

2.jpg1513×1477 1.07 MB
 
2_2.jpg1513×1477 984 KB
 
2_3.jpg1513×1477 1.09 MB
 
2_4.png772×288 42.1 KB



They use the same exact hyperparameters for optimization which is SGD with lr=0.98 for 20 epochs.
shouldn’t they have the same effect on weights? or is it because of the mean() of the activations, that the gradients will be spread more homogeneously and thus their effect is much milder than when we directly add a term to the gradients?
Thank you very much in advance