Thanks @BramVanroy for clarifying your point. You are right I mentioned the vocabulary size early on which is incorrect. What I mean is the following (and again I think we agree).
The output of the language model is a vector representation of the [CLS] token for each sentence. This representation is a row vector with 768 dimensions. So if you have N documents in your data you can stack all your row vectors into a big matrix A with dimensions N * 768
Then what I think the linear layer is doing is to learn essentially another matrix B, whose dimensions are 768 x k number of classes because when you do the matrix multiplication of A times B you get as output a matrix with N rows (your N input sentences) and k columns (the score for each possible class). I believe this is also what you are saying (except that your matrices are transposed). Does that make sense?
Thanks! It is always interesting to discuss the underlying mechanisms. Helps understand the models better.