Learning Universal Predictors
Meta-learning has emerged as a powerful approach to train neural networks to
learn new tasks quickly from limited data. Broad exposure to different tasks
leads to versatile representations enabling general problem solving. But, what
are the limits of meta-learning? In this work, we explore the potential of
amortizing the most powerful universal predictor, namely Solomonoff Induction
(SI), into neural networks via leveraging meta-learning to its limits. We use
Universal Turing Machines (UTMs) to generate training data used to expose
networks to a broad range of patterns. We provide theoretical analysis of the
UTM data generation processes and meta-training protocols. We conduct
comprehensive experiments with neural architectures (e.g. LSTMs, Transformers)
and algorithmic data generators of varying complexity and universality. Our
results suggest that UTM data is a valuable resource for meta-learning, and
that it can be used to train neural networks capable of learning universal
prediction strategies.
65b71334d0ba0a801639494e
Jordi Grau-Moya
65b71334d0ba0a801639494f
Tim Genewein
65b71334d0ba0a8016394950
Marcus Hutter
65b71334d0ba0a8016394951
Laurent Orseau
admin_assigned
2024-01-29T11:15:49.309Z
64ae74ad71b7e849b0aa3404
/avatars/50585fdd3fc2a916814c500ca6bb3c00.svg
Laurent Orseau
user
lorseau
65b71334d0ba0a8016394952
Grégoire Delétang
65b71334d0ba0a8016394953
Elliot Catt
65b71334d0ba0a8016394954
Anian Ruoss
claimed_verified
2024-02-09T08:05:57.105Z
65c51758522ee3a4edd917d0
/avatars/2ea3b64b5eeb34e2bfe821e4210f1b3e.svg
Anian Ruoss
user
anianruoss
65b71334d0ba0a8016394955
Li Kevin Wenliang
65b71334d0ba0a8016394956
Christopher Mattern
65b71334d0ba0a8016394957
Matthew Aitchison
65b71334d0ba0a8016394958
Joel Veness
Learning Universal Predictors
Meta-learning has emerged as a powerful approach to train neural networks to
learn new tasks quickly from limited data. Broad exposure to different tasks
leads to versatile representations enabling general problem solving. But, what
are the limits of meta-learning? In this work, we explore the potential of
amortizing the most powerful universal predictor, namely Solomonoff Induction
(SI), into neural networks via leveraging meta-learning to its limits. We use
Universal Turing Machines (UTMs) to generate training data used to expose
networks to a broad range of patterns. We provide theoretical analysis of the
UTM data generation processes and meta-training protocols. We conduct
comprehensive experiments with neural architectures (e.g. LSTMs, Transformers)
and algorithmic data generators of varying complexity and universality. Our
results suggest that UTM data is a valuable resource for meta-learning, and
that it can be used to train neural networks capable of learning universal
prediction strategies.
Meta-learning can leverage Universal Turing Machine-generated data to train neural networks for universal prediction strategies.
Meta-learning Solomonoff Induction Universal Turing Machines LSTMs Transformers Universal prediction strategies