Platonic Representations for Poverty Mapping: Unified Vision-Language
  Codes or Agent-Induced Novelty?
We investigate whether socio-economic indicators like household wealth leave
recoverable imprints in satellite imagery (capturing physical features) and
Internet-sourced text (reflecting historical/economic narratives). Using
Demographic and Health Survey (DHS) data from African neighborhoods, we pair
Landsat images with LLM-generated textual descriptions conditioned on
location/year and text retrieved by an AI search agent from web sources. We
develop a multimodal framework predicting household wealth (International
Wealth Index) through five pipelines: (i) vision model on satellite images,
(ii) LLM using only location/year, (iii) AI agent searching/synthesizing web
text, (iv) joint image-text encoder, (v) ensemble of all signals. Our framework
yields three contributions. First, fusing vision and agent/LLM text outperforms
vision-only baselines in wealth prediction (e.g., R-squared of 0.77 vs. 0.63 on
out-of-sample splits), with LLM-internal knowledge proving more effective than
agent-retrieved text, improving robustness to out-of-country and out-of-time
generalization. Second, we find partial representational convergence: fused
embeddings from vision/language modalities correlate moderately (median cosine
similarity of 0.60 after alignment), suggesting a shared latent code of
material well-being while retaining complementary details, consistent with the
Platonic Representation Hypothesis. Although LLM-only text outperforms
agent-retrieved data, challenging our Agent-Induced Novelty Hypothesis, modest
gains from combining agent data in some splits weakly support the notion that
agent-gathered information introduces unique representational structures not
fully captured by static LLM knowledge. Third, we release a large-scale
multimodal dataset comprising more than 60,000 DHS clusters linked to satellite
images, LLM-generated descriptions, and agent-retrieved texts.
68921fffc07ad472081610bb
Satiyabooshan Murugaboopathy
68921fffc07ad472081610bc
Connor T. Jerzak
68921fffc07ad472081610bd
Adel Daoud
Platonic Representations for Poverty Mapping: Unified Vision-Language
  Codes or Agent-Induced Novelty?
We investigate whether socio-economic indicators like household wealth leave
recoverable imprints in satellite imagery (capturing physical features) and
Internet-sourced text (reflecting historical/economic narratives). Using
Demographic and Health Survey (DHS) data from African neighborhoods, we pair
Landsat images with LLM-generated textual descriptions conditioned on
location/year and text retrieved by an AI search agent from web sources. We
develop a multimodal framework predicting household wealth (International
Wealth Index) through five pipelines: (i) vision model on satellite images,
(ii) LLM using only location/year, (iii) AI agent searching/synthesizing web
text, (iv) joint image-text encoder, (v) ensemble of all signals. Our framework
yields three contributions. First, fusing vision and agent/LLM text outperforms
vision-only baselines in wealth prediction (e.g., R-squared of 0.77 vs. 0.63 on
out-of-sample splits), with LLM-internal knowledge proving more effective than
agent-retrieved text, improving robustness to out-of-country and out-of-time
generalization. Second, we find partial representational convergence: fused
embeddings from vision/language modalities correlate moderately (median cosine
similarity of 0.60 after alignment), suggesting a shared latent code of
material well-being while retaining complementary details, consistent with the
Platonic Representation Hypothesis. Although LLM-only text outperforms
agent-retrieved data, challenging our Agent-Induced Novelty Hypothesis, modest
gains from combining agent data in some splits weakly support the notion that
agent-gathered information introduces unique representational structures not
fully captured by static LLM knowledge. Third, we release a large-scale
multimodal dataset comprising more than 60,000 DHS clusters linked to satellite
images, LLM-generated descriptions, and agent-retrieved texts.
A multimodal framework using satellite imagery and text data outperforms vision-only models in predicting household wealth, with LLM-generated text proving more effective than agent-retrieved text.
vision model LLM AI agent joint image-text encoder ensemble International Wealth Index R-squared cosine similarity Platonic Representation Hypothesis Agent-Induced Novelty Hypothesis