Aperiodic (Chaotic) Behavior in RNN with Homeostasis as a Source of Behavior Novelty: Theory and Applications

One way to understand cognitive system is to think in terms of dependency relationships between the neural controller or micro level, and the agent’s body configuration or macro level. Neural dynamics, as modeled in Recurrent Neural Networks (RNN), is determined by units and connections self-organization rules. This micro dynamics guides body configuration as it commands muscular action. On the other hand, an agent’s self-perception causes the body configuration state to influence neural dynamics. Cognitive agents thus work in a multi-level causality loop. An apparent limitation of RNN to model neural controllers for cognitive agents is that the dynamics may converge to a small region of neural state space. In the extreme case, this includes convergence to a fixed point or to limit cycles where only a few neural states are visited. Since agent’s body configuration is mostly determined by neural activity, limited neural dynamics also implies a limited dynamic in an agent’s body — as being completely “frozen” or keep doing the the same thing over and over again. Because natural cognitive agents, understood as animals and humans, maintain an almost continuous thread of behavior while they are awake, one can suggest that neural controllers for cognitive modeling and engineering should also allow for this kind of behavior. RNN with adaptive thresholds, modeling neural homeostasis, provide one possible answer. When units in a RNN are endowed with a rule for dynamically changing units thresholds the neural network as a whole behaves in a complex manner, ranging from a close to periodic behavior to aperiodic (or chaotic) behavior. When coupled to an agent’s body the neural dynamics can be used to produce variability in body configuration dynamics — this is the cognitive agent’s behavior at the macro level. This variability is a key requisite to allow agents the unaided discovery of possibilities of action (affordances) of their body in the context of their environment. Behavior habituation to instantaneous body-environment configurations resulting from neural homeostasis, keeps agent continuously exploring the configuration space, thus producing novel body postures and/or move the agent to new locations in the environment. This mechanism while essential for the production of creative or novel behavior, may not be enough. Without sensors to perceive their body and environment neural activity can not be O pe n A cc es s D at ab as e w w w .ite ch on lin e. co m

influenced by the body-configuration. Thus, neural chaos by itself can not guarantee that the agent performs adequately. For example, it may lunch the agent as whole to enter in a selfdestructive non-viable biological, psychological, or social region. Introducing sensors that perturb individual neurons and collective dynamics, offers an additional mechanism to develop structured behavior. At the formal level, one can infer that by having sensors for self and environment perception an agent can change the probability distribution of the neural state space. This change, in turn, changes the dynamics and probability distribution of the agent body configuration, possibly steering the agent into more interesting regions. To illustrate the application of this principles, we show with concrete examples of simple articulated agents how chaos in neural controllers can be used to generate novel behavior and how self-perception can be used to change neural dynamics. Target applications, included muscular control and visual attention. To make the principles general, we also present a conceptual framework for embodied neural agents as models for cognitive systems. We divide this article into four main parts: in section 2, we make an abstract theoretical characterization of cognitive systems that is useful for the remaining parts of the discussion. In section 3, we describe and discuss the proposed Recurrent Neural Network model that uses units with adaptive thresholds to model homeostasis. In section 4, we use this neural model to build a particular model of a minimalist cognitive agent, endowed with a single link and a joint with only one-degree-of-freedom. The experimental results obtained with this cognitive agent are used study muscular control and to illustrate the application of RNN with homeostasis. In particular, we compare the behavior of agent at the micro and macro level when neural units have or do not have adaptive thresholds and self-perception. In section 5, we describe another model that uses RNN with homeostasis, this time modeling visual attention. In section 6, we present our conclusions and relate our results with others.

A meta-model for cognitive systems
For improved understanding of cognitive system, one needs to have a meta-model or metatheory that allows one to think in abstract terms and helps to identify the relevant entities and concepts specific to the problem domain of cognition. In particular, the relation between agents and the environments in which they live, and the relation between its neural controller dynamics and body configuration dynamics needs to be put in appropriate perspective. If this is achieved, we are better equipped to see what are the relevant elements that models of neural networks need to take into consideration to work effectively as models of cognition. In this section, we present such a meta-model organizing the components sub-sections as follows: section 2.1 presents the concepts needed to begin understanding cognition; section 2.2 further develops the relation between neural controllers and behavior of agents; section 2.3 formally characterizes cognitive systems as complex dynamical systems.

Situated cognitive agents and environments
We characterize cognitive agents as complex systems that can be studied at two different complexity levels: the macro-level and the micro-level. The macro-level is defined by the configuration state -a formal description of the agents body posture in space and time, as seen by an external observer or as made apparent to the agent itself through self-perception. A small number of degrees-of-freedom is often required to describe an agent at this level (e.g., the variables of the joint angles plus the parameters of link geometry, as is often used in robotics). The micro-level is a characterization of the state of agent neural controller. In simple neural models, this may include the activation level of neural units, units' thresholds, and neural connections' weights. Usually, the micro-level requires a much higher number of degrees-offreedom to be fully described than the macro-level, since an agent with few links and joints may have a controller with many neural units. Interfacing the micro and macro-levels, agent descriptions include the way the neural controller is connected to the agents' body -both in muscular connections (efferent) and in the way sensation-perception cells/inputs impinge on the neural controller. In complex articulated agents, the number of macro-level variables and parameters needed may be in high number (e.g., on the order of dozens), but we always assume that the micro-level requires a much high number of variables and parameters to be described. A physical (non-cognitive) systems analogy of this, would be a rigid body (object) described at the macro-level by a few variable and parameters (e.g., for geometry, location and orientation in space, and material properties), and that at the microlevel requires much more variables if one wanted to describe in detail where all its constituent particles/elements are located in space at a given time, assuming, for illustration purposes, that this could be done in practice. Figure 1. Conceptual diagram of a cognitive agent, its environment and the external observer Real and virtual agents are often situated in some environment, in such a way that its behavior and interaction with the environment can be observed by an external observer. As pointed out by many classical thinkers and researcher in the AI community, the agent's own perception of the environment may be quite different from an external observer's perspective [10]. Namely, external observers can not make easily educate guesses about the subjective perspective of the observed agent own perception (e.g., the perspective a human and another animal, such as a dog or frog, might have from the same environment, say a tree, might be quite different -assuming for illustrations purposes, that the two of them could somehow be compared). In Figure 1, we make a sketch representation of the relationship between the agent, its environment, the external observer, and the two levels of description. Below, we postulate that an agent can be sensitive to its own actions by means of self-perception, and we use this to provide a causal account of how such self-generated information can be used to guide autonomously the behavior development of the agent through learning at the micro-level. The activity of neural units often dictates the generation of body movements, by commanding internal force to be made by muscular-like structures. Given this, the dynamics of body movements as captured by the formal configuration definitions, is a reflection of the dynamics of the neural units (plus whatever mechanical external perturbations the environment might impose in the agent at a given time). This is micromacro causality mechanism. Moreover, selforganizing mechanisms at the micro-level (e.g., learning and homeostasis) may change the probability distribution of states of the microlevel, and these can also leave a trace at the observed behavior. This macro behavior is emergent from the micro-level activity. An additional consequence of the above characterization, is that the mapping from the (micro) neural level to the (macro) configuration level is not one-toone, since the number of degrees-of-freedom are different. Many different neural states may mandate the same body configuration. Moreover, since body limbs are pulled by several muscular structures each with possibly many different force generating components (e.g., muscular micro-fibers), the coordinated action of a large number of neurons and muscular cells is usually required to generate strong and high-amplitude body movements. We also explore these aspects below. Since agents have component units sensitive to the environment (the sensationperception inputs), agents can receive feedback of the "world-state" (as inferred by their sensorial apparatus). This is interpreted as a macro-micro (downward) causality mechanism. Moreover, because agent's sensitivity of the macro world also applies to its own body state, agents can sense the effects of their own actions (e.g., using input from proprio-perceptive cells in muscles and tendons, by visually looking at body limbs -such as hands, or by listing the sounds produced by itself). Below, we call this type of macro-micro causality mechanism as self-perturbation or self-perception.

Multi-level causality
Once we make a micro-macro characterization of cognitive systems, we need to focus on the causal relations between the two levels. Figure 2, represents these causal relation in agent behavior according to the presented meta-model. X represents the (activation) state of the neural controller of the agent (part of micro-level or internal state), and C ag represents the body configuration of the agent (the macro-level or external state). The self-loop in X represents the internal dynamics of the controller, such as modeled in recurrent neural network models. The arrow from X to the motor units Mc represent the commanding of muscular contraction/distention causing the generation or cessation of internal force. The connection from Mc to C ag , represents the actual changes made in body configuration caused by changes in internal state (if any). Due to self-perturbation (in any sensorial modality) the agent configuration C ag generates input or perturbation to the neural dynamics -represented as Π. This represents part of the macro-micro causality. As a "side-effect" of the controller internal dynamics, changes in body configuration may change the state or configuration of the environment, in the www.intechopen.com diagram represented as C env (e.g., as in a manipulation task). Changes to environment state trigger additional perturbation to the neural controller. For individual neural units, the two types of perturbation (self-generated and other) should be considered (mechanistically) indistinguishable. Additionally, in a complex task-environment, environments may also have complicated dynamics of their own (e.g., gravity, dumping, reaction force, etc.)represented as a self-loop in the box labeled C env in Figure 2. The environment may also impose macro-perturbation in the configuration of the agent, abstracted as mechanical external forces F ext . We represent this as an additional arrow in Figure 2 from C env to C ag . We aggregate the agent configuration C ag and environment configuration C ag , and call it the global configuration or just configuration for short. We represent this as: C = C ag · C env . Changes in neural activity often (but not necessarily always), create body limb movements because they command muscular-like structures that create internal mechanical forces on the body. In most natural situations, body limb movements are also dependent on other mechanical external perturbations on the agent that combine to self-generate internal forces. This may include forces such as gravity, object contact reaction-force, and physical m a n i p u l a t i o n b y s o c i a l o t h e r . T h i s a s p e c t s of agent-environment interaction are not developed in the chapter. Figure 3. Diagrammatic representation of multi-level causality with two types of perturbation: X is the micro state and C is the macro state as seen by an external observer or by the agent itself. Π μ represents the micro-perturbations, mostly due to input to sensorialperception units/cells, and F ext represents macroscopic/mechanical perturbations, also represented as Π M .
Since agents have component units sensitive to the environment (the sensationperception inputs), agents can receive feedback of the "world-state". This input may change internal neural dynamics, and in turn change the internal forces that cause body limb movements. Agents may also have perception of their own body state (e.g. thought proprio-perception of limb displacement, visual perception of own body, or self-produced sounds). In the model presented below, we focus our attention on studies of a simple form of proprioceptive muscular input. Given this characterization, we see that micro and macro level are connected in a two-way causality loop. The state of the micro-level determines/influences body configuration, and the body configuration perturbs the internal dynamics of the neural controller due to selfperturbation. Figure 3, further illustrates the notion multi-level causality in cognitive systems. An upward arrow is used to represent emergence or upward causality, and an downward arrow represents downward causality due to self-perception and perception of the environment. This characterization of embodied neural agents relates to Ashby classical characterization of adaptive agents and agent-environment couplings as dynamical systems [2], further explored in mainstream situated AI literature [3]. The above presentation, although similar in general form, makes additional distinctions. In particular, it makes explicit and gives theoretical significance to the difference between the typical number of degrees-of-freedom at the micro or neural level, and the macro or configuration-level. Namely, |X| >> |C ag |.

Characterization as complex dynamical system
From a formal point of view, the agent body, neural controller, and environment represent a (complex) dynamical system that can be summarized with two (vectorial) coupled differential equations: where f a is the neural units activation function, f Π some (possibly complicated) function that maps agent and world configuration to a particular value of individual units perturbation, and f C and Mc are functions that relate changes in internal state with changes in agent body and world configuration. We are ignoring here and until the next section, second-order dynamics in the neural controller, such as learning and/or homeostasis. When we make the assumption that the neural state fully determines body posture (e.g. due to lack of body inertia), than the differential equation above for the configuration can be simplified to a functional equation: That is the neural state fully determines the instantaneous body configuration. When no confusion in caused, we abbreviate the above equation to: C = f C (X). For simplicity sake, we leave ambiguous whether the information about configuration state the agent uses is the same or comparable with the information a particular external observer might use to characterize the agent and its environment state. For purposes of neural control, the relevant information is the information the agent uses.

A model of RNN with homeostasis
In previous section, we made an abstract characterization of embodied cognitive agents that is independent of the controller and neural model used to generate its behavior. In this section, we propose a model of Recurrent Neural Networks with adaptive threshold capturing homoeostasis behavior in natural neural cell [16]. In section 3.1, we present the equation for neural dynamics. This is a variation of the continuous Hopfield RNN model [7], where units threshold changes to push activation back to a resting value. In section 3.2, we discuss how the non-embodied RNN model can be extended to control an embodied system as is the case of cognitive agents with a body living in some environment.

Equations for neural dynamics
The neural model consist of a set of units whose activation levels is described b a vector X = [x 1 , . . . , x i , . . . , x N ], with |X|as the number of units. Neural units activation x i is constrained to lie in the interval range [x min , x max ], where x max is the saturation value and x min is the lowest/depression value. Units are also assumed to have a rest or natural activation value x 0 . In computer simulation we make neural units start in this rest/natural activation state. Neural units are assumed to be connected in a network/graph as a fully recurrent neural network (all units connect to all) [7]. Connection strengths are represented by a connectivity matrix M, where element c ij represents the connection strength or weight between unit i and www.intechopen.com j. In the simulation results presented below we experiment mostly with fixed connection weights. Neural units are assumed to be initially connected with random weights, using a normal distribution with mean value 0 and variance  2 (M). Neural units have an adaptive threshold that is used to maintain units in a sensitive state. This is equivalent to cellular homeostasis mechanisms in biological neural networks [16]. For unit i we represent its threshold as θ i . When a unit's activation is very high, a slow adaptation process takes place that gradually moves the activation value to a rest or natural activation value x 0 . Likewise, when a unit's activation value is low the same adaptation process takes place to raise the activation level to x 0 . The operation of units is formally defined using two ordinary first-order differential equations [approximated by the Euler method in the simulations below]. The first equation below describes the (fast) dynamics of individual unit's activation. The second equation describes the (slower) dynamics of homeostasis.
above τ 1 , τ 2 , with τ 1 << τ 2 , are constants for the characteristic times of the neural processes modelled. x 0 is the resting or natural activation of units. f is an activation gain function. The simplest case is to have f a linear function with a constant gain G = 1. ξ is some (optional) random noise value.
Solving for equilibrium in the first equation, τ 1 x i = 0, shows that at rest which is a fast quiescent/rest state. Solving for equilibrium for the second equation, τ 2 Simulation results presented below show that full equilibrium (that is, x i = x 0 for all units) is often not reached due to units' interconnections. Neural connections can be made to have weights changed similarly to Hopfield networks by using a Hebb-like learning rule. This level of plasticity allow neural agents to have more adaptation possibilities since it introduces a second-order dynamics in the system. In this chapter, we will focus in networks without learning.

Embodiment neural agents
A straightforward way to give an embodiment to a RNN (with or without homestasic units), is to postulate that each agent actuator is controlled by a sub-set of units. Formally, if agent configuration state and state space is defined by vector C = [ ψ 1 , . . . , ψ i , . . . , ψ N ' ], with |C| the number-of-degree of freedom of the configuration, then we make each degree-offreedom ψ i to be a function of a sub-set of neural units X| i . In mathematical notation: ψ i = f(X| i ). Due to this functional relations, movement in neural state space may produce some kind of movement at the configuration level. On the other hand, since |X|>> |C|, neural dynamics may be sufficiently confined to make changes in agent configuration minimal. To make the neural units to receive feedback about behavioral consequence of neural dynamics sensorial mechanisms need to be used. One way to model this is to think that each agent sensor has the ability to produce a perturbation π i that adds to a units input, with a gain c i . This slightly changes the equation for neural dynamics, as follows:

A minimalist embodied neural agent
In this section, we present a model of an embodied neural agent that relies on a RNN with homeostasis to control its behavior. In particular, we show that the homeostasis introduces aperiodic (chaotic) behavior in the system preventing the agent to ever reach a stationary regime. This is argued to be useful for characterizing cognitive systems, since behavioral exploration and continuous novelty is a distinguishing feature of this type of systems.

Model
We consider an embodied articulated agent with a single link and a single joint. The joint angle fully defines the body configuration of the agent. The joint angle is determined by the contraction of a simplified muscle that works like a mechanical lever. The muscle has a large number of muscular units m i . The contraction/ extension of a muscular unit m i produces a spatial displacement Δs i , and the summation of all displacements determines the joint angle. Formally, ψ = f(Σ i Δsi), where f is a function of the detailed geometry of the agent. We assume that the contraction of a single neural units produces a relatively small link displacement. Therefore, the simultaneous contraction of a large proportion of muscular units is required to generate maximum displacement of the link. Additionally, the joint angle ψ is always constrained to lie within a maximum amplitude interval [− , ]. The agent also contains proprioceptive mechanisms for muscle contraction/distention or link angular position (discussed below). In Figure 4, we show the abstract design of the agent. We also show the graphical design of the agent as visualized in a developed simulator. Muscle contraction (and thus body configuration) is controlled by a neural population with N units. We make a simple attachment between this neural population and the muscle units, by making the number of muscular units equal to the number of neural units, and connecting them one-to-one (unidirectional). Muscular contraction is thus proportional to the total activation of the network. When all units are in a rest/natural activation value, φ takes value 0 (the link is horizontal).

www.intechopen.com
A fixed proportion f π of neural units is sensitive to the angular position of the link. Namely, we define a preferred angular position φ * and cause that sub-set of units to receive additional excitation the closer the link is to that preferred position/angle. Thus, this units work as proprio-perceptive (proprioceptive) neural cells. The concrete equation we use to model this is: where π i is the external perturbation to the cell due to proprioceptive input, K 1 is a parameter for maximum perturbation value, (φ − φ*) 2 is the squared difference of the link's current angular position and the preferred angle φ*, and K 2 is a parameter for how slow perturbation decreases as the current angle moves away from the preferred angle φ*. (K 2 is also represents the variance of a Gaussian curve.) In the simulation results presented below, f π is always set to .3, φ* always set to 80°.

Experimental results
We have performed several experiments to study the behavior of the previous presented model. In these experiments, we generate neural controllers with random connections according to the weight matrix M, using mean 0 and variance  2 (M) = 1. For the presented results, we made connection weights fixed (no learning), and removed internal noise. Parameters for unit's activation were set as follows: x 0 = 1, x max = 3, x min = .1. Perturbation parameters where set as K 1 = 5x max , and K 2 = 2. Neural activation levels x i at time t = 0 are always set to x 0 . In studying model behavior, we look both at the neural (micro) and link configuration (macro) levels. We also look both at the dynamical and the stochastic aspects of model behavior.

Neural Dynamics without Homeostasis
When units homeostasis is not put in the model's operation (τ 2 = +∞), the neural activation state and the configuration angle converges in most simulation runs to a fixed point. In fixed points, a large proportion of units are either fully saturated (x i = x max ) or fully depressed (x i = x min ). In some simulation runs, some units converge to intermediary values (closer to x 0 ). Simulation runs with different random connection matrices produce different fixed points. Figure 5 shows the evolution in neural state space and link-configuration state spaces, along side with corresponding probability distributions, for a particular simulation run during 200 time steps. (High activation of units is coded as red in color plates, low activation as blue, and values near x 0 as green.) This is a similar behavior to that observed in recurrent neural networks with symmetric connections, as in Hopfield RNN [7]. In a small proportion of simulation runs with different connection weight matrices, neural dynamics converges to a small region of state space usually in the form of a periodic cycle. In these scenarios, most neural units are either in fully depressed or saturated regime, as in fixed points solutions, but a proportion of cells oscillates due to non -symmetric and opposite sign connections. In Figure 6 we show the evolution of the configuration state for one of such simulation run, showing a small periodic cycle that corresponds to an oscillation of low amplitude in configuration space. In a set of 10 consecutive runs with the same settings (but different weight matrices), the results obtained were qualitatively similar -following one of these two cases. The results (either fixed points or periodic cycle) also appeared in controllers and networks with different number of neural units, from 5 to 50. Previous work also showed that the introduction of considerable noise is (most of the times) not enough to take the system away from fixed-points or small-regions of state space [14,13]. Formally, this means that fixed points are either attracting or Lyapunov stable (neural states tend to stay within a small distance of a fixed point when perturbed [15]).

Neural Dynamics with Homeostasis
When units have homeostasis, the behavior of the system changes considerably. The proportion of time that units are not saturated or depressed increases, as inspection of the differential equation for the threshold above suggests. However, most units do not remain with an activation value near x 0 all the time since they are taken away from homeostasis due to interconnection with other units. Figure 7 and figure 8 shows two qualitatively typical simulation runs. [Noise is absent,  2 (ξ) = 0.] The system state does not converge to a fixed point or some simple attractor, but exhibits behavior that qualitatively can be categorized between non-periodic behavior and nearly periodic, due to threshold adjustments [15]. When the number of units is small, the behavior of systems tends to be closer to periodic behavior (nearly periodic), and when the number of units increases the behavior tends to be more aperiodic. Following Langton [8], such class of qualitative behaviors may be designated as complex behavior. This is explained considering that although first-order neural dynamics cause the system to move to a small region of state space, individual units' homeostasis (modeled as threshold adjustments) take the system away from this regions (fixed-points or periodic cycles). This creates the conditions for a wider exploration of state space, when compared with setting where the neural controller as only a first-order dynamics.   . The effects of proprioceptive input in probability distribution of neural and configuration state spaces: top) with proprioceptive input the probability distribution become bi-modal, with one high activation region (link up), and one low activation or depressed region (link down); bottom)) without proprioceptive input probability distribution become uni-modal, with the region near ψ = 0 of highest probability due to homeostasis.

Neural Dynamics with Proprioceptive Input
To investigate the behavior of the system when proprioceptive input is used, we compared the behavior of the systems with and without proprioception perturbation for neural controllers with the same connection matrices. In particular, we want to see if increasing neural activity when body configuration angle is near a preferred position would increase the probability of the agent to staying near that region. Figure 9 shows diagrams for the probability distribution of the configuration (left) and neural state spaces (right), for one particular neural network with and without (top and bottom) proprioceptive perturbation. The results show that proprioception cause the link angle distribution to become bimodal; with the region slightly above the preferred angle, at 1.4 rad. (marked with a red vertical bar in the probability distribution graph on the left), to have highest probability, corresponding to a saturated region, and another high probability region corresponding to a region of neural depression (with higher entropy than the saturated region). In contrast to this, for the same connection weights matrix, the neural dynamics without proprioceptive perturbation causes the link distribution to be uni-modal. In this case, the region near ψ = 0 (link at horizontal position) is the highest probability region. (Previous work, suggests that this distribution can be characterized by a (symmetric) power-law distribution [14]). Testing for Robustness Because, the system's behavior changes considerable with different connection weight matrices, we wanted to test the robustness of these findings across multiple runs with different controllers. For this purpose, we divided the total link-configuration state space, ψ∈ [− , ], in three regions: a depressed region, corresponding to a interval range of ψ∈ [− ,− ], a resting region with ψ∈ [− , ], and a saturated and high amplitude region, ψ∈ [ , ]. Figure 10 shows the probability distribution of link angle for each of these three intervals (mean and variance) for 10 different simulation runs. The left-hand graph correspond to the settings with proprioceptive input, and in right-hand the graph represents the setting without proprioceptive input. The results confirm the initial observation that introducing proprioceptive input with higher intensity near a preferred region tends to make that region of higher probability. The bi-modality, induced by proprioception, arrives because positive perturbations tends to increase neural excitation. This can be understood by looking at the equations governing system behavior (here in vector form): X = F(X, Π; M), which can be linearized to X ≈ F(X;M)+G· Π, if most units are in linear (non-saturated, non-depressed regime), a www.intechopen.com condition ensured by homeostasis. Thus, higher input |Π| increases the value of firstderivative X , and higher values of X correspond to higher configuration angles ψ.
Formally, we can say that in the linear regime > 0. For link states ψ = ψ*, the first derivative X is still positive, so the highest probability angle tends to be higher than ψ*.

Other applications: visual attention
To further illustrate the use of RNN endowed with homeostasis, we describe in this section an additional model this time targeting visual attention. It is a variant of the previous model for muscular control, but now the agent-link as a sensor apparatus for visual perturbation.

Model
As previously, our agent model description consist of two parts: the description of the agent body and the description of the neural controller. The body of the agent consists of a single link with a tip with visual-input sensitive cells ( figure 11). The link position or body configuration is controlled by an antagonistic muscle pair (the left and right muscle), and their contraction-extension depends on the activation of the motor units m l and m r , directly connected to them. Activity of motor units fully defines the body configuration of the agent, and consequently the angular position of his visual axes. Formally, we specify the link angle to be: where K m is a proportionality constant. Therefore, the link will turn to the left when m l > m r and to the right when m r > m l . The link is always constrained to lie within a maximum amplitude interval [− , ]. The visual tip of the agent detects external visual stimuli, modeled here as punctual particles fixed or moving in a direction parallel to the horizontal baseline of the link. The particle position is defined by the angle Φ*, which is constrained to a maximum amplitude interval [ − , ]. The presence of the particle imposes a visual input to the neural controller, that includes N v visual units. A visual unit has a maximum activation value when the stimulus is located at a particular angular position in relation to a preferred position. Formally, visual input is defined as: where k 1 and k 2 are two visual input constants. Δ Φ = Φ − Φ* represents the difference between link and particle orientations. ' i Φ corresponds the preferential angle for each visual unit. With the settings above, Δ Φ is constrained to be in the interval [− , ], with limits representing the two extreme situations where the particle and the link are as far away as possible.
We made a simple attachment between the visual input units and a neural control population, by making the number of units of each population equal and connecting them one-to-one (unidirectionally). Control units are connected as a fully recurrent neural network -all units connect to all. The control population is connected to the neural motor population such that half of the N c units connect to the motor unit m l , and the other half connects to the motor unit m r . That is:

Experimental results
In this section, we present simulation results for basic experimental settings. We focus on experiments with a single particle with a fixed position or simple movements. For more elaborated experimental settings see [1]. Figure 11. Diagram of a minimalist embodied neural agent with one degree-offreedom for visual attention tasks, subject to a variety of simple visual stimuli (point particles). The agent consists of a single link and joint, representing the orientation of a visual axes, a tip sensitive to visual stimuli, and an antagonistic muscle pair. left): Graphical visualization in the developed neural agent simulator. right): Abstract design representing the agent's body and the set of units controlling muscular contraction-distension, and receiving visual input. The body configuration is defined by the link angle _ as commanded by left and right muscles, whose contraction/extension is set by two motor units m l and m r . The motor units are connected to a set of control neural units. Control units are connected in a fully recurrent way (complete connection graph), and each control unit receives input from a corresponding visual input unit.

Neural Dynamics without Homeostasis
The experiments described in this section were used to analyze the effect of homeostasic mechanisms in the visual system, with and without visual perturbation. This study was performed with 8 control units (N c = 8), and then with different sizes of neural populations, N c ∈ {4, 8, 16, 32, 64}.
In the first trials, the neural control population was setup with eight units. Units in the motor population were always set with two units (controlling the left and right muscles).
We set as visual stimulus a point particle situated at a fixed distance from the horizontal basis of the agent. Different simulation runs select a different (randomly selected) angular position for the particle. The weight matrices also take different random values for each simulation run with V c = 1. Figure 12 shows the system's behavior during 500 time-steps of a particular run, when there is no visual stimuli present in the agent's environment. Here, neural homeostasis is turn off (τ 2 = +∞). The left-handed side of figure 12 depicts the time-series for configuration angle Φ, with the vertical axes representing time and the horizontal axes representing angular displacement of the link. In the right-handed side is represented the history of neural activation using codes (blue for depression, red for saturation and green for intermediate values

Neural Dynamics with Perception but without Homeostasis
In figure 13 we show the behavior of the visual system for two simulation run when a point particle is present in the visual field. The figure includes the time-series for the configuration state and the time-series for the neural state, and also (in the middle) the time-series for the angular difference between link and particle orientations (Δ Φ). The position of the particle is highlighted as a vertical red line in the plot for Φ. The results show that for the first presented simulation run the angular position of the link is close to the position of the particle. This can also be seen by looking at the data plot for Δ Φ which shows that the angular distance to the particle quickly converges to a value close to zero. The second row in the figure 13, shows that this is not always the case. In this second run the link converges to a position far away from the particle position. The left-handed side of figure 14 shows the time-series of Δ Φ for 10 consecutive simulation runs when the neural controller is configured without homeostasis. The plots confirm the previous observations. Although for an important fraction of simulation runs the angular differences are reduced (6), for several of the simulation runs the link converges to positions far away from the particle. This occurs because the relaxation of the neural state takes the link to certain positions before the visual input is able to significantly influence the neural dynamics.

Neural Dynamics with Homeostasis
The introduction of an adaptive mechanism in the form of homeostasis completely changes the agent's internal and external dynamics. Figure 15 shows that when homeostasic mechanisms are used ( = 0.5), the body configuration of the agent exhibits a non-periodic or chaotic behavior [15]. This means that while the equations for neural dynamics are completely deterministic both the link angle and the neural state seems to move erratically as if a stochastic process is involved. Note that in this trial the point particle is not present yet. Additionally, it can be seen that individual neural units hardly stabilize in particular activation values. This occurs because homeostasis slowly pushes unit's activation to resting value x 0 . However, due to unit's interconnections a global equilibrium is never reached [13]. Therefore, the proportion of cells not saturated or depressed at a given time is much less than when homeostasis is not used. Consequently, the activation state of the neural population does not converge to a fixed point. Instead, we can observe complex oscillating patterns of neural activity.

Neural Dynamics with Perception and Homeostasis
In figure 16, we present data plots for two simulation runs with neural controllers working with homeostasis and a point particle is present in two slightly different positions. The results show that in both simulation runs the link orientation converges to a region close to the particle position. This is confirmed in the middle plot of figure 16 where is shown that Δ Φ converges to values close to zero. Most importantly, the link orientation does not converge to a fixed-point. Instead, it performs small oscillatory movements. Experimentation with model variations, showed that the use of an antagonistic muscle pair, as apposed to a single muscle, is very useful to give robustness to the model's behavior.
Model variations with a single muscle requires parameters to be carefully selected to achieve effective visual fixation behavior. It is interesting to note that while homeostasis tends to move the neural state away from particular regions (e.g. a fixed-point) [14,13], this does not cause the system to loose track of the particle and increase error. This happens because there is redundancy in the neural state-configuration state mapping, with the number of degrees-of-freedom in the former being much higher than the number of degreeof-freedom in the later (|X| >> 1). This explains why in the righthand plots of figure 16 the neural state moves between several states and yet the orientation of the visual axes changes little. Figure 15. Time-series for the link configuration angle Φ, and control units activation state X c over 500 time-steps. Neural controller has 8 homeostasic units.
The right-handed side of figure 14 shows time-series of Δ Φ for 10 consecutive simulation runs when the neural controller is configured with homeostasis. The plot shows that the behavior of the system is qualitatively different from the behavior when no homeostasis is used (left-handed side of figure 14 ). In a significant proportion of runs the link orientation approximately matches the particle orientation (Δ Φ ≈ 0), although we can identify continuous aperiodic oscillations of the link around the particle position. This happens because homeostasis prevents neural state to reach a long-term equilibrium and makes unit's activation to oscillate. On the other hand, the presence of the particle promotes the selection of neural states that corresponds to configuration states of high visual stimulation. Thus, changes at the macro-level are limited while changes in the micro-level can occur. For another important proportion of runs the link orientation moves away from the particle position some proportion of the total number of simulation time-steps (in the case, 200).
Again this happens due to homeostasis, but at particular occasions the neural state moves to regions of the neural states space that do not correspond to a configuration state where the link is aligned with the visual particle. Only in one simulation run (plotted in a blue line) the link is unable to fixate the particle. Figure 16 Time-series for the link configuration angle Φ, the difference between link and particle positions Δ Φ and control units activation state X c over 500 time-steps. Neural controller with 8 homeostasic units. The particle is represented as a vertical red line and is located in random positions.

Summary discussion, related work, and conclusions
Research in Recurrent Neural Networks since Hopfield initial contribution [7] as received very much attention specially in exploring its properties as a model for associative memory. Additionally, theoretical and experimental advances in artificial intelligence and robotics research have identified complexity theory as a promising tool to understand how neural agents can self-organize to produce adaptive behavior [11]. Combining RNN models and behavioral research is thus a promising approach to understand cognitive systems and the role played by recurrent connection in the nervous system. In this article, we make a characterization of cognitive agents that is suggestive of how RNN can control embodied agents, and extend the basic formulation of RNN to include adaptive thresholds to model neural homeostasis. In the proposed approach, adaptive thresholds make neural units to move to a resting activation value although at a slower pace than main activation dynamics. Experimental results show that homeostasis make neural dynamics to produce aperiodic (chaotic) behavior and, for small networks, nearly periodic behavior. We showed that this can be used as a source of behavioral exploration and novelty in embodied neural agents. The concept of neural network originated from neuroscience, and one of its primitive aims is to help us understand the principle of the central nerve system and related behaviors through mathematical modeling. The first part of the book is a collection of three contributions dedicated to this aim. The second part of the book consists of seven chapters, all of which are about system identification and control. The third part of the book is composed of Chapter 11 and Chapter 12, where two interesting RNNs are discussed, respectively.The fourth part of the book comprises four chapters focusing on optimization problems. Doing optimization in a way like the central nerve systems of advanced animals including humans is promising from some viewpoints.

How to reference
In order to correctly reference this scholarly work, feel free to copy and paste the following: Jorge Simão (2008