  Prompt engineering is crucial for deploying LLMs but is poorly understood
mathematically. We formalize LLM systems as a class of discrete stochastic
dynamical systems to explore prompt engineering through the lens of control
theory. We investigate the reachable set of output token sequences $R_y(\mathbf
x_0)$ for which there exists a control input sequence $\mathbf u$ for each
$\mathbf y \in R_y(\mathbf x_0)$ that steers the LLM to output $\mathbf y$ from
initial state sequence $\mathbf x_0$. We offer analytic analysis on the
limitations on the controllability of self-attention in terms of reachable set,
where we prove an upper bound on the reachable set of outputs $R_y(\mathbf
x_0)$ as a function of the singular values of the parameter matrices. We
present complementary empirical analysis on the controllability of a panel of
LLMs, including Falcon-7b, Llama-7b, and Falcon-40b. Our results demonstrate a
lower bound on the reachable set of outputs $R_y(\mathbf x_0)$ w.r.t. initial
state sequences $\mathbf x_0$ sampled from the Wikitext dataset. We find that
the correct next Wikitext token following sequence $\mathbf x_0$ is reachable
over 97% of the time with prompts of $k\leq 10$ tokens. We also establish that
the top 75 most likely next tokens, as estimated by the LLM itself, are
reachable at least 85% of the time with prompts of $k\leq 10$ tokens.
Intriguingly, short prompt sequences can dramatically alter the likelihood of
specific outputs, even making the least likely tokens become the most likely
ones. This control-centric analysis of LLMs demonstrates the significant and
poorly understood role of input sequences in steering output probabilities,
offering a foundational perspective for enhancing language model system
capabilities.

Computation and Language;Artificial Intelligence;Machine Learning;Neural and Evolutionary Computing