Optimal Neural Codes for Control and Estimation
Agents acting in the natural world aim at selecting appropriate actions based on noisy and partial sensory observations. Many behaviors leading to decision making and action selection in a closed loop setting are naturally phrased within a control theoretic framework. Within the framework of optimal Control Theory, one is usually given a cost function which is minimized by selecting a control law based on the observations. While in standard control settings the sensors are assumed fixed, biological systems often gain from the extra flexibility of optimizing the sensors themselves. However, this sensory adaptation is geared towards control rather than perception, as is often assumed. In this work we show that sensory adaptation for control differs from sensory adaptation for perception, even for simple control setups. This implies, consistently with recent experimental results, that when studying sensory adaptation, it is essential to account for the task being performed.
to the control task), followed by constructing a controller based on the estimated state. This idea, referred to as the separation principle in Control Theory, while optimal in certain restricted settings (e.g., Linear Quadratic Gaussian (LQG) control) is, in general, sub-optimal [6]. Unfortunately, it is in general very difficult to provide optimal solutions in cases where separation fails. A special case of the separation principle, referred to as Certainty Equivalence (CE), occurs when the controller treats the estimated state as the true state, and forms a controller assuming full state information. It is generally overlooked, however, that although the optimal control policy does not depend directly on the observation model at hand, the expected future costs do depend on the specifics of that model [7]. In this sense, even when CE holds, costs still arise from uncertain estimates of the state and one can optimise the sensory observation model to minimise these costs, leading to sensory adaptation. At first glance, it might seem that the observation model that will minimise the expected future cost will be the observation model that minimises the estimation error. We will show, however, that this is not generally the case.
A great deal of the work in computational neuroscience has dealt independently with the problem of sensory adaptation and control, while, as stated above, these two issues are part and parcel of the same problem. In fact, it is becoming increasingly clear that biological sensory adaptation is task-dependent [8,9]. For example, [9] demonstrates that task-dependent sensory adaptation takes place in purely motor tasks, explaining after-effect phenomena seen in experiments. In [10], the authors show that specific changes occur in sensory regions, implying sensory plasticity in motor learning. In this work we consider a simple setting for control based on spike time sensory coding, and study the optimal coding of sensory information required in order to perform a well-defined motor task. We show that even if CE holds, the optimal encoder strategy, minimising the control cost, differs from the optimal encoder required for state estimation. This result demonstrates, consistently with experiments, that neural encoding must be tailored to the task at hand. In other words, when analyzing sensory neural data, one must pay careful care to the task being performed. Interestingly, work within the distributed control community dealing with optimal assignment and selection of sensors, leads to similar conclusions and to specific schemes for sensory adaptation.
The interplay between information theory and optimal control is a central pillar of modern control theory, and we believe it must be accounted for in the computational neuroscience community. Though statistical estimation theory has become central in neural coding issues, often through the Cramér-Rao bound, there have been few studies bridging the gap between partially observed control and neural coding. We hope to narrow this gap by presenting a simple example where control and estimation yield different conclusions. The remainder of the paper is organised as follows: In section 1.1 we introduce the notation and concepts; In section 2 we derive expressions for the cost-to-go of a linear-quadratic control system observed through spikes from a dense populations of neurons; in section 3 we present the results and compare optimal codes for control and estimation with point-process filtering, Kalman filtering and LQG control; in section 4 we discuss the results and their implications.
Summary-------The authors study stochastic optimal control problems with incomplete state information. In particular, they consider problems where the sensors are adaptive. They compare sensor configurations which are optimal under a standard signal detection paradigm with sensor configurations that are optimal for a given control problem.Comments--------Optimal sensor adaptation has a long history in computational neuroscience. Usually, optimality is defined wrt estimation (infomax etc). The main point of the paper, that optimal sensor adaptation for control problems will yield different results in general, is very important and could potentially have a big impact on the field. Standard optimal coding approaches were often criticized but concrete alternatives were lacking.Here the authors propose optimal control as a guiding principle and illustrate this with a few well-chosen examples. Not being an expert in control theory, the paper seems technically sound (I checked most equations, not all).It is well written, if a little too dense at times. Some results / examples require further elaboration (details below), it might be worth dropping one (eg the harmonic oscillator in 3.1) in favor of more details for the other examples.Specific comments:- l22: ``However, this sensory adaptation is geared towards control rather than perception, as is often assumed.'' In principle I agree with this statement. However, there's an interesting discussion to be had based on the following argument: Assume that the sensors are used in many different control problems, and therefore adapting them for a single one might result in poor performance on the others. In this situation it might be beneficial to adapt the sensory based on optimal estimation after all. I'd be curious to hear the opinion of the authors on this point.- l103: ``MMSE'' is probably the minimum MSE, but it's never defined. Furthermore, the authors seem to use MSE / MMSE inconsistently, eg in the legend of figs 3&4, shouldn't it be just MSE instead of MMSE?- l110: Shouldn't the minimization be over $\mathbf U_0$ instead of $\mathbf U_t$? And similarly the expectation over $\mathbf X_0$, etc...- l117: The notation here is a bit sloppy I think. On the RHS of the eqn, it prob should be $\gamma$ applied to the time-series of filtered estimates of $X_t$ given $Z_{[0,t]}$. By definition in l108 $\mathbf Z_t$ is the future of $Z$, and not the past...- l146: typo: $E[X_t\vert Y_t]$ should prob be $E[X_t\vert Y_{[0,t]}$- l158, eqn 3: the authors state that only the third term depends on the parameters $F,G$ of the observation process $Y$. For $t > 0$, why don't the first two terms depend on $F,G$ via $\nu_t$ and $K_t$? For $t=0$, I agree that only the 3rd term depends on $F,G$.- l182, eqn 5: Concerning the dense coding hypothesis: I was wondering weather it can be achieved with a finite number of neurons each having an RBF / squared-exponential receptive field (my intuition tells might not, but I might be wrong). If one indeed needs infinitely many neurons, than one could possibly replace the spike process $dN_t$ with a diffusion / Wiener process and hence simplify eqns 6a and 6b. Is this possible? The authors might be interested in the use of the dense packing hypothesis in [Implicit embedding of prior probabilities in optimally efficient neural populations. D Ganguli and E P Simoncelli. ArXiv]- l258: typo, this should probably be the mutual information between $X_t$ and $\{N^m_{[0,t]}\}_{m=1,\ldots,M}$- l270: ``This trade-off can be tilted to either side in the case of control depending on the information available at the start of the problem.'' I don't fully understand this point. Furthermore, I cant find the promised detailed discussion of this point in section 3.1.- fig 2: The left and right panels use different labels in the legends for the same quantities. Left panel: the x-axis label should probably be also $\zeta$- l413: ``Note the extreme example of the stochastic oscillator, where the optimal encoder is concentrating all the resources in one dimension, essentially ignoring the second dimension.'' It's important to spell out in detail what this means for the tuning curve width in direction and perpendicular to the ``important control dimension''