  Position encoding is the primary mechanism which induces notion of sequential
order for input tokens in transformer architectures. Even though this
formulation in the original transformer paper has yielded plausible performance
for general purpose language understanding and generation, several new
frameworks such as Rotary Position Embedding (RoPE) are proposed for further
enhancement. In this paper, we introduce the notion of "geotokens" which are
input elements for transformer architectures, each representing an information
related to a geological location. Unlike the natural language the sequential
position is not important for the model but the geographical coordinates are.
In order to induce the concept of relative position for such a setting and
maintain the proportion between the physical distance and distance on embedding
space, we formulate a position encoding mechanism based on RoPE architecture
which is adjusted for spherical coordinates.

Computation and Language;Artificial Intelligence