C.3 CartPole: Additional Details

We use a simplified version of the CartPole enviornment in OpenAI Gym. Observations are two-dimensional
(corresponding to pole angle and velocity, normalized to 0-100) and the maximum time horizon is 200.
There are two possible actions (1, 2) corresponding to (left, right), and the agent gets +1 reward for every
time step that the CartPole is kept upright. In Listing 4, we show an example context excerpt for CartPole,
where a trajectory history is appended with an encoding of the current trajectory.
52: 40 50 , 1, 40 54 , 2, 41 49 , 1, 41 54 , 1, ...
60: 45 50 , 2, 45 45 , 1, 44 50 , 2, 44 45 , 1, ...
75: 52 50 , 1, 52 55 , 2, 53 50 , 2, 53 46 , 2, ...
98: 44 50 , 1, 44 55 , 2, 45 50 ,
Listing 4: Example context format for a CartPole run. A trajectory history (with each trajectory in the format reward:
observation, action, observation, action ...) is followed by an encoding of the current trajectory, up to the current
observation.
LargeLanguageModelsasGeneralPatternMachines