In order to test whether the agents are being driven towards a critical point, we analyse signatures of critical behaviour in the neural controller of the agent. As we mentioned before, a sufficient indicator for criticality is the presence of a divergence of the heat capacity of the system (as in Fig. 2A). A divergence in the heat capacity indicates the presence of a continuous phase transition presenting a critical point in which the system is maximally sensitive to parametrical changes. Unfortunately, when the neural controller is embodied in the Mountain Car and Acrobot environments, we can no longer access a formal description of the probability distribution of the agent-environment system, thus we cannot directly compute from the energy of the system values as the entropy or heat capacity of the system. Nevertheless, we can still directly compute the entropy \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H(x)=-\,{\sum }_{x}P(x)\,\mathrm{log}\,P(x)$$\end{document}H(x)=−∑xP(x)logP(x) of any variable of the system by estimating its probability distribution P(x) through simulations. In order to compute the entropy and the heat capacity of different variables, we simulate the agent’s behaviour for 101 values of β, logarithmically distributed in the interval [10−1, 101]. We run the 10 agents for each embodiment during 106 simulation steps, reseting the agent’s position and state every 5 · 104 simulation steps.