A neural network system is proposed. The network can be trained by model-based reinforcement learning to select actions to be performed by an agent interacting with an environment, to perform a task in an attempt to achieve a specified result. The system may comprise at least one imagination core which has an input to receive a current observation characterizing a current state of the environment, and optionally historical observations, and which includes a model of the environment. The imagination core may be configured to output trajectory data in response to the current observation, and/or historical observations. The trajectory data comprising a sequence of future features of the environment imagined by the imagination core. The system may also include at least one rollout encoder to encode the sequence of features, and a reinforcement learning output stage to receive data derived from the rollout embedding and to output action policy data for defining an action policy identifying an action based on the current observation.
IMAGINATION-BASED AGENT NEURAL NETWORKS