Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences
Shuchen Wu Mirko Thalmann Peter Dayan Zeynep Akata Eric Schulz
Humans excel at learning abstract patterns across different sequences, filtering out
irrelevant details, and transferring these generalized concepts to new sequences.
In contrast, many sequence learning models lack the ability to abstract, which
leads to memory inefficiency and poor transfer. We introduce a non-parametric
hierarchical variable learning model (HVM) that learns chunks from sequences
and abstracts contextually similar chunks as variables. HVM efficiently organizes
memory while uncovering abstractions, leading to compact sequence representations.
When learning on language datasets such as babyLM, HVM learns a more efficient
dictionary than standard compression algorithms such as Lempel-Ziv. In a sequence
recall task requiring the acquisition and transfer of variables embedded in sequences,
we demonstrate HVM’s sequence likelihood correlates with human recall times. In
contrast, large language models (LLMs) struggle to transfer abstract variables as
effectively as humans. From HVM’s adjustable layer of abstraction, we demonstrate
that the model realizes a precise trade-off between compression and generalization.
Our work offers a cognitive model that captures the learning and transfer of abstract
representations in human cognition and differentiates itself from LLMs.
Abstraction, Chunking, Cognitive Science, LLMs
The paper presents a hierarchical variable learning model (HVM) that efficiently abstracts patterns in sequences, outperforming standard compression methods and large language models in mimicking human memory and generalization.