Doubts on modeling_gpt2.py
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I've been going through the gpt2 source code and i was tracing out how the self attention and feed forward work basically we have `Block` which is a decoder consisting of other 2 segments `Attention` and `MLP`. I was also reading a blog where it mentions the `queries` has to be learned i finally saw the class responsible for that is `Conv1D`
```
class Conv1D(nn.Module):
    def __init__(self, nf, nx):
        """ Conv1D layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2)
            Basically works like a Linear layer but the weights are transposed
        """
        super(Conv1D, self).__init__()
        self.nf = nf
        w = torch.empty(nx, nf)
        nn.init.normal_(w, std=0.02)
        self.weight = nn.Parameter(w)
        self.bias = nn.Parameter(torch.zeros(nf))

    def forward(self, x):
        size_out = x.size()[:-1] + (self.nf,)
        x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
        x = x.view(*size_out)
        return x

```

Here we have a affine function `addmm` is how we do... but i was expecting `train optimer.step, loss` basically all the thing that goes into training in pytorch. Can anyone elaborate on it?

Then in `GPT2LMHeadModel` we use a linear layer to represent `vocab_embed` and we produce `logits` by multiplying it with the transformer output so if the linear layer is trained what is the use of `from_pretrained` anyway?

I am sure this may perhaps be silly questions but i'd like to get some help here.

Thanks a lot.
Can anyone answer it please...<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
