I use padding=1 normally, I just forgot to add it there 
Now I am testing the weight initialized version. Also, is there a way to disable the weight decay for batch_normalization layer’s learnable parameters?
I implemented it as below based on https://stackoverflow.com/questions/44641976/in-pytorch-how-to-add-l1-regularizer-to-activations 3. But I am not sure if it is correct (I did’t use out1, out5, and out5?) and if there is a cleaner way to do it.
for i, (images, labels) in enumerate(trainloader):
			images = images.cuda()
			labels = labels.cuda()
			lambda2 = 0.0005
			# Forward pass
			out, out1, out5, out9, out13  = model(images)
			loss = criterion(out, labels)

			all_1_params = torch.cat([x.view(-1) for x in model.layer1.parameters()])
			all_5_params = torch.cat([x.view(-1) for x in model.layer5.parameters()])
			all_9_params = torch.cat([x.view(-1) for x in model.layer9.parameters()])
			all_13_params = torch.cat([x.view(-1) for x in model.layer13.parameters()])

			l2_regularization_1 = lambda2 * torch.norm(all_1_params, 2)
			l2_regularization_5 = lambda2 * torch.norm(all_5_params, 2)
			l2_regularization_9 = lambda2 * torch.norm(all_9_params, 2)
			l2_regularization_13 = lambda2 * torch.norm(all_13_params, 2)

			
			loss_all = loss+l2_regularization_1+l2_regularization_5+l2_regularization_9+l2_regularization_13
			# Backward and optimize
			optimizer.zero_grad()
			loss_all.backward()
			optimizer.step()