title: Learning Universal Predictors
summary: Meta-learning has emerged as a powerful approach to train neural networks to
  learn new tasks quickly from limited data. Broad exposure to different tasks
  leads to versatile representations enabling general problem solving. But, what
  are the limits of meta-learning? In this work, we explore the potential of
  amortizing the most powerful universal predictor, namely Solomonoff Induction
  (SI), into neural networks via leveraging meta-learning to its limits. We use
  Universal Turing Machines (UTMs) to generate training data used to expose
  networks to a broad range of patterns. We provide theoretical analysis of the
  UTM data generation processes and meta-training protocols. We conduct
  comprehensive experiments with neural architectures (e.g. LSTMs, Transformers)
  and algorithmic data generators of varying complexity and universality. Our
  results suggest that UTM data is a valuable resource for meta-learning, and
  that it can be used to train neural networks capable of learning universal
  prediction strategies.
paper_authors._id: 65b71334d0ba0a801639494e
paper_authors.name: Jordi Grau-Moya
paper_authors._id: 65b71334d0ba0a801639494f
paper_authors.name: Tim Genewein
paper_authors._id: 65b71334d0ba0a8016394950
paper_authors.name: Marcus Hutter
paper_authors._id: 65b71334d0ba0a8016394951
paper_authors.name: Laurent Orseau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T11:15:49.309Z
paper_authors._id: 64ae74ad71b7e849b0aa3404
paper_authors.avatarUrl: /avatars/50585fdd3fc2a916814c500ca6bb3c00.svg
paper_authors.fullname: Laurent Orseau
paper_authors.type: user
paper_authors.user: lorseau
paper_authors._id: 65b71334d0ba0a8016394952
paper_authors.name: Grégoire Delétang
paper_authors._id: 65b71334d0ba0a8016394953
paper_authors.name: Elliot Catt
paper_authors._id: 65b71334d0ba0a8016394954
paper_authors.name: Anian Ruoss
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-09T08:05:57.105Z
paper_authors._id: 65c51758522ee3a4edd917d0
paper_authors.avatarUrl: /avatars/2ea3b64b5eeb34e2bfe821e4210f1b3e.svg
paper_authors.fullname: Anian Ruoss
paper_authors.type: user
paper_authors.user: anianruoss
paper_authors._id: 65b71334d0ba0a8016394955
paper_authors.name: Li Kevin Wenliang
paper_authors._id: 65b71334d0ba0a8016394956
paper_authors.name: Christopher Mattern
paper_authors._id: 65b71334d0ba0a8016394957
paper_authors.name: Matthew Aitchison
paper_authors._id: 65b71334d0ba0a8016394958
paper_authors.name: Joel Veness
paper_title: Learning Universal Predictors
paper_summary: Meta-learning has emerged as a powerful approach to train neural networks to
  learn new tasks quickly from limited data. Broad exposure to different tasks
  leads to versatile representations enabling general problem solving. But, what
  are the limits of meta-learning? In this work, we explore the potential of
  amortizing the most powerful universal predictor, namely Solomonoff Induction
  (SI), into neural networks via leveraging meta-learning to its limits. We use
  Universal Turing Machines (UTMs) to generate training data used to expose
  networks to a broad range of patterns. We provide theoretical analysis of the
  UTM data generation processes and meta-training protocols. We conduct
  comprehensive experiments with neural architectures (e.g. LSTMs, Transformers)
  and algorithmic data generators of varying complexity and universality. Our
  results suggest that UTM data is a valuable resource for meta-learning, and
  that it can be used to train neural networks capable of learning universal
  prediction strategies.
paper_ai_summary: Meta-learning can leverage Universal Turing Machine-generated data to train neural networks for universal prediction strategies.
paper_ai_keywords: Meta-learning Solomonoff Induction Universal Turing Machines LSTMs Transformers Universal prediction strategies