Should a language model like GPT-3 be directly used to perform classification? The OpenAI API enables classification by sampling from GPT-3 given a prompt. Is estimating posterior probabilities a more statistically sound approach? Below is a specification of what "estimating posterior probabilities" could mean here. I'd like feedback on how sensible it is (see the question at the bottom).
Definitions

*

*$\mathcal{X}$ is the set of all possible sequences of tokens.


*$X$ is a random variable for a sequence of tokens. For example, a movie review could be $x = (\text{A}, \text{ thrill}, \text{ ride}, \text{ for}, \text{ the}, \text{ ages}, \text{!}, \text{ --}, \text{Peter}, \text{ K}, \text{ Rosenthal})$.


*$\Pr_{\theta}$ is a language model, i.e., a probability distribution over $\mathcal{X}$ parametrized by $\theta$.


*$Y$ is a random variable for the class an observation belongs to, where the set of classes is $\mathcal{Y}$. Let's allow $\mathcal{Y} \subset \mathcal{X}$. So if I'm classifying movie reviews, $\mathcal{Y} = \{ \text{Just another superhero movie}, \text{Generic hype} \}$ represents a classification task.


*The function $\text{prompt}: \mathcal{X} \rightarrow \mathcal{X}$ takes a sequence of tokens (which must include $x$, and optionally $\mathcal{Y}$) and returns a new sequence of tokens. A prompt asks a language model to perform the task.

*

*See this set of documentation for more examples and detail. (You need to pay to hit OpenAI API endpoints.)



Proposed method
GPT-3's estimates, $\Pr_\theta(x_i \: | \: x_{1:i-1})$, are great. So simply selecting the completion from $\mathcal{Y}$ with the highest posterior probability should be more direct than sampling. For example, we'd query the language model for its probability of this prompt and completion:
This movie review
"""
A thrill ride for the ages! --Peter K Rosenthal
"""
is best categorized as {completion}.

where {completion} is replaced w/ "Just another superhero movie", and then replaced w/ "Generic hype".
I'll start w/ a false start to formalizing the meaning of posterior probability here, and then adjust it. I'm dubious about the adjustment though, so I'd like feedback.
First some new definitions: let $c \in \mathcal{Y}$ be a completion containing $n$ tokens, and $p = \text{prompt}(x)$ be the prompt. In the example above, the prompt is the full sequence of tokens up until {completion}.
False start: it's tempting to simply do $\arg\max_{c \in \mathcal{Y}} \Pr_\theta(c \: | \: p)$. This is wrong b/c longer completions trivially result in lower probabilities. It also asks the language model to do too much work, as it doesn't incorporate the prior $\Pr(y)$ from your specific distribution. It implicitly comes from GPT-3's training distribution as $\Pr_\theta(c)$. (But this prior can be manually set for single-token completions.)
Now for the adjustments: start w/ framing the classifier as
$$
\begin{equation}
\arg\max_{y \in \mathcal{Y}} \Pr(y \: | \: p, c) = \arg\max_{y \in \mathcal{Y}} \text{Pr}_\theta(p, c \: | \: y) \Pr(y).
\end{equation}
$$
This reframing fixes the issue with the prior. To fix the completion length issue, let's replace $\Pr_\theta(p, c \: | \: y)$ with an average per completion token. To construct that, we'll first express $\Pr_\theta(p, c \: | \: y)$ as
$$
\begin{align}
\text{Pr}_\theta(p, c \: | \: y) &= \text{Pr}_\theta(p \: | \: y) \text{Pr}_\theta(c \: | \: p, y) && \text{probability chain rule} \\
&= \text{Pr}_\theta(p) \text{Pr}_\theta(c \: | \: p, y) && P=p \perp Y=y \\
&= \text{Pr}_\theta(p) \prod_{i=1}^{n} \text{Pr}_\theta(c_i \: | \: p, y, c_{1:i-1}) && \text{probability chain rule} \\
&= \text{Pr}_\theta(p) \exp \Bigg\{\sum_{i=1}^{n} \log\text{Pr}_\theta(c_i \: | \: p, y, c_{1:i-1}) \Bigg\}.
\end{align}
$$
Note that we can ignore $\Pr_\theta(p)$ as it doesn't depend on $y$. The completion does depend on the class b/c it's set to the class.
Define the average likelihood (i.e., the average inverse perplexity1,2) as
$$
\begin{equation*}
\bar{\text{Pr}}_\theta(c \: | \: p, y) =  \exp \Bigg\{\frac{1}{n} \sum_{i=1}^{n} \log\text{Pr}_\theta(c_i \: | \: p, y, c_{1:i-1}) \Bigg\}.
\end{equation*}
$$
Finally, estimate the posterior probability of a class as
$$
\begin{equation*}
\Pr(y \: | \: p, c) \approx \bar{\text{Pr}}_\theta(c \: | \: p, y) \Pr(y) / Z
\end{equation*}
$$
where $Z$ is the normalizer.
Example
Let's perform the estimation for the $x$ and $\mathcal{Y}$ movie review example. $p = \text{prompt}(x)$ is this sequence of tokens:
This movie review
"""
A thrill ride for the ages! --Peter K Rosenthal
"""
is best categorized as 

Example computation for the second class:
$$
\begin{equation*}
\bar{\text{Pr}}_\theta(c \: | \: p, \text{Generic hype}) = \exp \Bigg\{ \frac{1}{2} \Big[ \log \text{Pr}_\theta(\text{Generic} \: | \: p) + \log \text{Pr}_\theta(\text{hype} \: | \: p, \text{Generic}) \Big] \Bigg\}.
\end{equation*}
$$
Say I magically know the class distribution for my data:
$$
\begin{align*}
\Pr(\text{Just another superhero movie}) = 1/3 \\
\Pr(\text{Generic hype}) = 2/3.
\end{align*}
$$
Then the normalizer is
$$
\begin{equation*}
Z = \frac{1}{3} \bar{\text{Pr}}_\theta(c \: | \: p, \text{Just another superhero movie}) + \frac{2}{3} \bar{\text{Pr}}_\theta(c \: | \: p, \text{Generic hype}).
\end{equation*}
$$
Finally, the posterior probability estimates are
$$
\begin{align*}
\Pr(\text{Just another superhero movie} \: | \: p, c) &\approx \frac{1}{3} \bar{\text{Pr}}_\theta(c \: | \: p, \text{Just another superhero movie}) / Z \\
\Pr(\text{Generic hype} \: | \: p, c) &\approx \frac{2}{3} \bar{\text{Pr}}_\theta(c \: | \: p, \text{Generic hype}) / Z.
\end{align*}
$$
Question
Did it make sense to reframe the classifier from
$$
\arg\max_{c \in \mathcal{Y}} \text{Pr}_\theta(c \: | \: p) \\
\text{to} \\
\arg\max_{y \in \mathcal{Y}} \bar{\text{Pr}}_\theta(c \: | \: p, y) \Pr(y) ?
$$
References

*

*Wikipedia page on perplexity per word

*huggingface's perplexity documentation

I evaluated the proposed method on the Choice of Plausible Alternatives (COPA) task. I picked this task from the SuperGLUE NLP benchmark b/c each $c \in \mathcal{Y}$ is always a few tokens long.
See the evaluation notebook here.
I still need to empirically evaluate calibration, and evaluate on more classification tasks. But a partial, empirical answer to the question in the title and the "Did it make sense..." question is: yes. Zero-shot accuracy on COPA train + validation is 0.916, which is relatively high.
