No, that is not right. As I said before, its input dimensions are usually the output dimensions of the base model (e.g. 768) and the output dimensions are the number of classes, e.g. 3 for “neutral, positive, negative”. The classification weights are, relatively speaking, quite small in many downstream tasks.
During language modeling, the LM head has the same input dimensions, but the output dimensions are the same size as the vocabulary: it provides you with a probability for each token how well it fits in a given position. This does lead to a large classifier.