I’m not sure.
What I’m trying to do is copy the parameters of a language model and use it to pretrain a text classifier. I’m trying to copy the word embedding and a LSTM unit. WHat I’m doing is the following:

I load both modules with pytorch:

checkpoint = torch.load(load_model)
embed = checkpoint[‘embed’]
rnn = checkpoint[‘rnn’]

From the rnn I try to get only the rnn cell:

rnn.layers[0]
Out[13]: LSTMCell(256, 1024)

I try to pass these modules to a new RNN:

self.embedding = copy.deepcopy(pretrained_embed)
self.rnn_cell = copy.deepcopy(pretrained_lstm_cell)
self.rnn_size = pretrained_lstm_cell.hidden_size

Then I try to connect the copied component to a new Linear layer from which I expect to get the logits of my text classifier:

self.fully_connected = nn.Linear(self.rnn_size, num_classes)
Maybe I’m copying the LSTMCell in the wrong way and it is still attached to the linear output of my language model? If that is the case, what would be the correct way to reuse my pretrained parameters in this new graph?