GPT-3, Esq? Evaluating AI Legal Summaries [pdf]
&gt; Second, text ought to be tokenized (a term used in natural language processing wherein
text is assigned a numerical value) at many different levels (character, word, sentence,
paragraph, section, etc.) in order to make predictions that are relevant to the excerpt of a text,
while also remaining consistent to the broader document. This is challenging because doing so
involves a tremendous amount of computational resources. However, it may be necessary to
accurately capture meaning at different levels of abstraction.<p>I don&#x27;t think this would be an improvement per se. The way Byte Pair Encodings are constructed that GPT-3 (and GPT-2) use is that they already take higher-level text representations and compress it down (into tokens), which are then reflected in the training of the model.