title: Geospatial Mechanistic Interpretability of Large Language Models
summary: Large Language Models (LLMs) have demonstrated unprecedented capabilities
  across various natural language processing tasks. Their ability to process and
  generate viable text and code has made them ubiquitous in many fields, while
  their deployment as knowledge bases and "reasoning" tools remains an area of
  ongoing research. In geography, a growing body of literature has been focusing
  on evaluating LLMs' geographical knowledge and their ability to perform spatial
  reasoning. However, very little is still known about the internal functioning
  of these models, especially about how they process geographical information.
    In this chapter, we establish a novel framework for the study of geospatial
  mechanistic interpretability - using spatial analysis to reverse engineer how
  LLMs handle geographical information. Our aim is to advance our understanding
  of the internal representations that these complex models generate while
  processing geographical information - what one might call "how LLMs think about
  geographic information" if such phrasing was not an undue anthropomorphism.
    We first outline the use of probing in revealing internal structures within
  LLMs. We then introduce the field of mechanistic interpretability, discussing
  the superposition hypothesis and the role of sparse autoencoders in
  disentangling polysemantic internal representations of LLMs into more
  interpretable, monosemantic features. In our experiments, we use spatial
  autocorrelation to show how features obtained for placenames display spatial
  patterns related to their geographic location and can thus be interpreted
  geospatially, providing insights into how these models process geographical
  information. We conclude by discussing how our framework can help shape the
  study and use of foundation models in geography.
paper_authors._id: 681b05ab8e325b4097318969
paper_authors.name: Stef De Sabbata
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-07T08:08:24.829Z
paper_authors._id: 661e841972c7030f3fd57eb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/661e841972c7030f3fd57eb9/PysszxrOl__sZ-IToZfJU.jpeg
paper_authors.fullname: Stef De Sabbata
paper_authors.type: user
paper_authors.user: sdesabbata
paper_authors._id: 681b05ab8e325b409731896a
paper_authors.name: Stefano Mizzaro
paper_authors._id: 681b05ab8e325b409731896b
paper_authors.name: Kevin Roitero
paper_title: Geospatial Mechanistic Interpretability of Large Language Models
paper_summary: Large Language Models (LLMs) have demonstrated unprecedented capabilities
  across various natural language processing tasks. Their ability to process and
  generate viable text and code has made them ubiquitous in many fields, while
  their deployment as knowledge bases and "reasoning" tools remains an area of
  ongoing research. In geography, a growing body of literature has been focusing
  on evaluating LLMs' geographical knowledge and their ability to perform spatial
  reasoning. However, very little is still known about the internal functioning
  of these models, especially about how they process geographical information.
    In this chapter, we establish a novel framework for the study of geospatial
  mechanistic interpretability - using spatial analysis to reverse engineer how
  LLMs handle geographical information. Our aim is to advance our understanding
  of the internal representations that these complex models generate while
  processing geographical information - what one might call "how LLMs think about
  geographic information" if such phrasing was not an undue anthropomorphism.
    We first outline the use of probing in revealing internal structures within
  LLMs. We then introduce the field of mechanistic interpretability, discussing
  the superposition hypothesis and the role of sparse autoencoders in
  disentangling polysemantic internal representations of LLMs into more
  interpretable, monosemantic features. In our experiments, we use spatial
  autocorrelation to show how features obtained for placenames display spatial
  patterns related to their geographic location and can thus be interpreted
  geospatially, providing insights into how these models process geographical
  information. We conclude by discussing how our framework can help shape the
  study and use of foundation models in geography.
paper_ai_summary: A framework for understanding how Large Language Models process geographical information using spatial analysis and mechanistic interpretability techniques.
paper_ai_keywords: Large Language Models geospatial mechanistic interpretability spatial analysis probing mechanistic interpretability superposition hypothesis sparse autoencoders polysemantic internal representations monosemantic features spatial autocorrelation placenames geographic location