Hi all,
I’m interested in a very simple idea, which I’ll try to explain with a toy example. So, I want to use the weights of a 2D convolution (along with its bias terms) in order to initialize the weights (and biases) of another 2D convolution. More specifically, I want to initialize the weights of the second convolution with the element-wise squares of the weights of the first convolution.
However, I want this to mean that the shared between those two convolutions weights refer to the same variables; that is, they are essentially the same weights (they don’t just have the same values at the initialization step).
Please take a look at the following example. I would like to tell PyTorch that conv_2 should be initialized with the squares of conv_1’s weights (and with the same biases). As a result, the number of parameters of this network will be equal to the number of parameters due to conv_1.
Is this possible?
Thank you for your time.
import torch
import torch.nn as nn
import torch.nn.init as init


def weights_init(m):
    if isinstance(m, nn.Conv2d):
        init.kaiming_uniform_(m.weight.data)
        m.bias.data.zero_()


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv_1 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, padding=1)
        self.conv_2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)

    def forward(self, x):
        return self.conv_2(self.conv_1(x))



net = Net()

# Initialization of conv_1
net.conv_1.apply(weights_init)

x = torch.randn(1, 16, 300, 300).requires_grad_(True)
y = net(x)