I’m trying to fine-tune a pretrained resnet model via a triplet loss. Due to the nature of the loss function, sometimes I make multiple forward passes through the model without ever making a backwards pass or stepping the optimizer. It seems that the norm of the final layer decreases even when I do not explicitly optimize.
I have created a small toy example below to illustrate what I’m talking about. I run an image forward through the model multiple times and then pass the same image as a “validation” image and calculate the final layer norm. Maybe this behavior is a result of the BatchNorm layers? If so, I would love to how those layers create this behavior.
import io

import numpy as np
from PIL import Image
import requests

import torch
from torch.autograd import Variable
from torchvision.models import resnet50
import torchvision.transforms as transforms

# Create finetuning model.
model = resnet50(pretrained=True)
for name, child in model.named_children():
    if name != 'fc':
        for p in child.parameters():
            p.requires_grad = False

            
optimizer = torch.optim.SGD(filter(lambda x: x.requires_grad, 
                                   model.parameters()),
                            lr=1e-3)

# Download a random image and transform for input
# to resnet.
url = ('https://raw.githubusercontent.com/pytorch/'
       'pytorch/master/docs/source/_static/img/'
       'pytorch-logo-dark.png')
response = requests.get(url)
img = Image.open(io.BytesIO(response.content))
transform = transforms.Compose([
    transforms.Scale(224),
    transforms.CenterCrop(224),
    transforms.ToTensor()])
img = transform(img)[:3, :, :]

# "Training" and "Validation" data are the exact
# same image.
train = img.unsqueeze(0)
val = Variable(img.unsqueeze(0), volatile=True)

# Run a single forward pass and measure
# norm of validation embedding. No backwards
# passes, no optimizer steps.
for epoch in range(20):
    
    model.train()
    optimizer.zero_grad()
    train_emb = model(Variable(train))
    
    model.eval()
    val_emb = model(val)
    print(val_emb.norm(2).data[0])