Bistability, non-ergodicity, and inhibition in pairwise maximum-entropy models
Correlated activity between pairs of cells was observed early on in the history of neuroscience 1 , 2 ., Immediately the question arose whether there is a functional interpretation of this observation 3 , and this question is still with us ., Hypotheses range from synchronous activation of neurons to bind representations of features into more complex percepts 4–7 , to the involvement of correlations in efficiently gating information 8 ., Direct experimental evidence for a functional role of correlated activity is the observation that the synchronous pairwise activation of neurons significantly deviates from the uncorrelated case in tight correspondence with behaviour ., Such synchronous events have been observed in motor cortex 9 , 10 at time points of expected , task-relevant information ., In primary visual cortex they appear in relation to saccades ( eye movements ) 11 , 12 ., Another argument for the functional relevance of correlations is the robustness of signals represented by synchronous activity against noise 13 ., Non-Gaussian distributions of membrane potentials of neurons indeed point towards the synchronized arrival of synaptic events 14 , 15 ., An opposite view regards correlated activity merely as an unavoidable epiphenomenon of neurons being connected and influencing one another 16 ., In the worst case , both these views are partly true , prompting us to find ways to distinguish functionally relevant correlated events from the uninformative background ., In the context of experimental paradigms that perform repeated trials , the co-variability of neurons across trials has been termed “noise correlation” ., Recurrent network models are able to reproduce and explain the weak magnitude and wide spread across pairs of second-order 17–23 and higher-order correlations 24 , 25 ., These simple dynamical models effectively map the statistics of the connectivity to the statistics of the activity ., Even though they explain the uninformative part of correlated activity , it is unclear how to use them to distinguish this background from departures thereof ., The separation of the noise- or background correlation from functionally meaningful correlation is in addition hampered by the diverse dimensions of information processing’s not being completely orthogonal ., Indeed , correlation transmission may be modulated by changes of firing rate 9 ., Theory 26 , 27 confirmed this entanglement in the regime of Gaussian fluctuating membrane potentials ., The dynamical-model approaches just outlined pivot on a more or less realistic physical description of the network , with some stochastic features ., A complementary approach is also possible , fully pivoting on statistical models ., The latter try to predict and characterize neuronal activity without relying on a definite physical network model ., Statistical models have two convenient features ., First , intuitive statistical working hypotheses usually translate into a unique statistical model 28 , 29; this fact streamlines the construction and selection of a such a model ., For example , the assumption that first- and second-order correlations recorded in an experiments are sufficient to predict the activity recorded in a new experiment , uniquely selects a truncated Gaussian model 29 , 30 ., Second , a successful statistical model implicitly restricts the set of possible dynamical physical models of the network: only those reflecting the well-modelled statistical properties are acceptable ., Statistical models thus help in modelling the actual physical network structure ., A limit case of this kind of statistical models is obtained by choosing probability distributions having maximum entropy under the constraints of experimentally observed quantities 31 , 32; in neuroscience see e . g . 33 ., The suitability of such maximum-entropy distributions for neuronal activities has been tested in various experimental and simulated set-ups ., For example , to explore the sufficiency of pairwise correlations or higher-order moments , or their predictive power for distribution tails e . g . 34–48 , and to characterize dynamical regimes 36 , 49–51 ., The probability distribution thus obtained , which includes the single-unit and pairwise statistics of the observation by construction , could help us to solve the background-correlation problem described above ., In assigning to every observed activity pattern a probability , we obtain a measure of “surprise” for each such pattern; this surprise measure e . g . 52 , 53 is related to the logarithm of the probability and thus to Shannon’s entropy ., Periods of activity with low probability correspond to large surprise: these patterns cannot be explained by the statistical properties that entered the construction of the probability distribution ., In this way , we are able to effectively differentiate expected , less surprising events from those that are unexpected , surprising , and functionally meaningful ., Computing the maximum-entropy distribution from moment constraints—usually called the inverse problem–is simple in principle: it amounts to finding the maximum of a convex function ., Hence optimization is straightforward 54 , 55 ., The maximum can be searched for with a variety of methods ( downhill simplex , direction set , conjugate gradient , etc . 56 , ch . 10 ) ., The convex function , however , involves a sum over exp ( N ) terms , where N is the number of neurons ., For 60 neurons , that is roughly twice the universe’s age in seconds , and modern technologies enable us to record hundreds of neurons simultaneously 57–60 ., Owing to the combinatorial explosion for such large numbers of neurons , the convex function cannot be calculated , not even numerically ., It is therefore “sampled” , usually via Markov-chain Monte Carlo techniques 61 , 62 ., In neuroscience the Glauber dynamics , also known as Gibbs sampling 61 , 63 , chap . 29 , is usually chosen as the Markov chain whose stationary probability distribution is the maximum-entropy one ., Boltzmann learning 64 is the iterative combination of sampling and search for the maximum , and is still considered the most precise method of computing a maximum-entropy distribution ., Alternatively one may try to approximate the convex function by an analytic expression , as done with the mean-field 65 , 66 , Thouless-Anderson-Palmer 66 , 67 , and Sessak-Monasson 68 , 69 approximations ., The goodness of these approximations is usually checked against a Boltzmann-learning calculation cf . 45 ., Moment-constrained maximum-entropy models have also been used 70 , 71 as generators of surrogate data , again via a Glauber dynamics ., Such surrogates are used to implement a null hypothesis to estimate the statistical significance level of correlations between spike trains 70 , 72–77 ., The pairwise maximum-entropy model is applicable to experimentally recorded activities of populations of a couple hundreds neurons at most , so far; but its success , or lack thereof , cannot be automatically extrapolated to larger population sizes ., Roudi et al . 78 gave evidence that the maximized Shannon entropy and other comparative entropies of such a model may present qualitatively different features above a particular population size ., In the present paper we discuss a feature of the pairwise maximum-entropy model that may be problematic or undesirable: the marginal distribution for the population-averaged activity becomes bimodal , and one of the modes may peak at high activities ., In other words , maximum-entropy claims that the population should fluctuate between a regime with a small fraction of simultaneously active neurons , and another regime with a higher fraction of simultaneously active neurons; the fraction of the second regime can be as high as 90% ., This feature of the maximum-entropy model has been observed before in several theoretical studies that assumed a homogeneous neuronal population see e . g . 34 , 41 , 79 , 80 ., Our analysis has several points in common with Bohte & al ., ’s 34 ., Bohte et al . wanted to see whether a maximum-entropy distribution can correctly predict the distribution of total activity , given only firing rates and pairwise correlations from a simulated network model as constraints ., They found that both the simulation and the maximum-entropy model yield a bimodal distribution of total activity within particular ranges of firing rates and correlations ., The fundamental difference from our work is that our experimental data do not show a bimodal distribution , but the maximum entropy model wrongly predicts such bimodality from the measured rates and correlations ., More quantitatively , the pairwise correlation found in our data is much lower than that reported in Bohte et al . ; in particular , it seems to belong to the range in which their simulation yielded a unimodal distribution 34 , p . 169 ., Their simulations therefore seems to corroborate that a second mode is biologically implausible in our correlation regime ., Amari & al ., 79 notice the appearance of bimodal distributions for the averaged activity and analyse some of their features in the N → ∞ limit ., Their focus is on the correlations needed to obtain a “widespread” distribution in that limit ., Our focus is on the bimodality appearing for large but finite N , and we find some mathematical results that might be at variance with Amari & al ., ’s ., They seem to find 79 , p . 135 that the Dirac-delta modes are at values 0 and 1; we find that they can appear also strictly within this range ., They say 79 , p . 138 that the “bigger peak” dominates as N → ∞; we find that the height ratio between the peaks is finite and depends on the single and pairwise average activity , and for our data is about 2000 as N → ∞—an observable value for recording lengths achievable in present-day experiments ., We provide evidence that the bimodality of the pairwise model is bound to appear in applications to populations of more than a hundred neurons ., It renders the pairwise maximum-entropy model problematic for several reasons ., First , in neurobiological data the coexistence of two regimes appears unrealistic—especially if the second regime corresponds to 90% of all units being simultaneously active within few milliseconds ., Second , two complementary problems appear with the Glauber dynamics and the Boltzmann-learning used to find the model’s parameters ., In the Glauber dynamics the activity alternately hovers about either regime for sustained periods , which is again unrealistic and rules out this method to generate meaningful surrogate data ., In addition , the Glauber dynamics becomes practically non-ergodic , and the pairwise model cannot be calculated at all via Boltzmann learning or via the approximations previously mentioned cf . 62 , S 2 . 1 . 3; 61 , chap . 29 ., This case is particularly subtle because it can go undetected: the non-ergodic Boltzmann learning yields a distribution that is not the maximum-entropy distribution one was looking for ., Bohte & al ., 34 remark that their neuronal-network simulation had to incorporate one inhibitory neuron , with the effect of “curtailing population bursts” 34 , p . 175 , because “the absence of inhibitory neurons makes a network very quickly prone to saturation” 34 , p . 162 ., This is something that a standard maximum-entropy distribution cannot do , hence a limitation in its predictive power ., It is intuitively clear that lack of inhibition and bimodality are related problems: we show this in section “Intuitive understanding of the bimodality: Mean-field picture” using a simple mean-field analysis ., In the present work we propose a modified maximum-entropy model; more precisely , we propose a reference probability measure to be used with the method of maximum relative entropy e . g . 31 , 81 ( also called minimum discrimination information 82; see 83 for a comparison of the two entropies ) ., The principle and reference measure can be used with pairwise or higher-order constraints; standard maximum-entropy corresponds to a uniform measure ., The proposed reference measure , presented in section “Inhibited maximum-entropy model” , solves three problems at once: ( 1 ) it leads to distributions without unrealistic modes and eliminates the bistability in the Glauber dynamics; ( 2 ) it leads to a maximum-entropy model that can be calculated via Boltzmann learning; ( 3 ) it can also “rescue” interesting distributions that otherwise would have to be discarded because incorrect ., The reference measure we propose is neurobiologically motivated ., It is a minimal representation of the statistical effects of inhibition naturally appearing in brain activity , and directly translates Bohte & al’s device of including one inhibitory neuron in the simulated network ., Moreover , the reference measure has a simple analytic expression and the resulting maximum-entropy model is still the stationary distribution of a particular Glauber dynamics , so that it can also be used to generate surrogate data ., In the final “Discussion” we argue that the use of such a measure is not just an ad hoc solution , but a choice required by the underlying biology of neuronal networks: the necessity of non-uniform reference measures is similarly well-known in other statistical scientific fields , like radioastronomy and quantum mechanics ., The plan of this paper is the following: after some mathematical and methodological preliminaries , we show the appearance of the bimodality problem in the maximum-entropy model applied to an experimental dataset of the activity of 159 neurons recorded from macaque motor cortex ., Then we use an analytically tractable homogeneous pairwise maximum-entropy model to give evidence that the bimodality problem will affect larger and larger ranges of datasets as the population size increases ., We show that typical experimental datasets of neural activity are prone to this problem ., We then investigate the underlying biological causes of the bimodality problem and propose a way to eliminate it: using a minimal amount of inhibition in the network , represented in a modified Glauber dynamics that includes a minimal asymmetric inhibition ., We show that this correction corresponds to using the method of maximum entropy with a different reference measure , as discussed above , and that the resulting maximum entropy distribution is the stationary distribution of a modified Glauber dynamics ., We finally bring to a close with a summary , a justification and discussion of the maximum-entropy model with the modified reference measure , and a comparison with other statistical models used in the literature ., Our study uses three main mathematical objects: the pairwise maximum-entropy distribution , a “reduced” pairwise maximum-entropy distribution , and the Glauber dynamics associated with them ., We review them here; some remarks about their range of applicability are given in ., Towards the end of the paper we will introduce an additional maximum-entropy distribution ., We first show how the bimodality problem subtly appears with a set of experimental data , then explore its significance for larger population sizes and other samples of experimental data of brain activity ., Let us briefly summarize our results so far and the reason why a maximum-entropy model yielding a bimodal distribution in the population-averaged activity is problematic: We will propose a solution that addresses all three issues at once ., This solution pivots on the idea of inhibition and can be grasped with an intuitive explanation of how the bimodality arises ., In this work we have shown that pairwise maximum-entropy models , widely used as references distributions in the statistical description of the joint activity of hundreds of neurons , are poised to suffer from three interrelated problems when constrained with mean activities and pairwise correlations typically found in cortex: We have given an intuitive explanation of the common cause of these issues: positive pairwise correlations imply positive Lagrange multipliers between pairs of neuron , corresponding to a symmetric network that is excitatory on average ., For typical values of correlations observed in neuroscientific experiments , this network can therefore possess two metastable dynamic regimes , given sufficiently many units ., The mechanism is identical to the ferromagnetic transition in the Ising model , as explained in “Bimodality of the inhomogeneous model for large N” ., An analogous bimodality appears in the statistical mechanics of finite-size systems e . g . 108 , 115 , and refs therein—but it is experimentally expected and verified there , unlike our neurobiological case ., Although we did not study maximum-entropy models typically used in other fields , like structural biology and genetic networks 116–118 , social behavior in mammals 119 , 120 , natural image statistics 121 , 122 , and economics 123 , the problems we have addressed are generic and emerge as soon as we study a large network with positive pairwise correlations on average; hence they might be of relevance to these fields ., In this work we have also suggested a remedy , based on the explanation above: the intuitive idea is to add a minimal asymmetric inhibition to the network , in the guise of an additional , asymmetrically coupled inhibitory neuron ( Fig 8A ) cf . 34 , p . 175 ., This leads to an “inhibited” Glauber dynamics that is free from bistable regimes and has a unimodal stationary distribution Pi ( s ) , Eq ( 22 ) ., This dynamics depends on an inhibition-coupling parameter JI and a threshold parameter θ ., Most important , we have shown that this new stationary distribution Pi ( s ) belongs to the maximum-entropy family: it can be obtained with the maximum-relative-entropy method with respect to a reference measure , Eq ( 25 ) ( Fig 9 ) , that represents the neurobiologically natural presence of inhibition in the network ., We call this model an “inhibited” pairwise maximum-entropy model ., The inhibited pairwise model solves all three problems above: We wish to stress that the presence of bimodality and non-ergodicity can easily go unnoticed ., Sampling from a bimodal distribution , the probability to switch to the second mode may be so small that it occurs over more sampling steps larger than those typically used in the literature , and the high mode is not visited during Boltzmann learning or surrogate generation ., We then face a subtle situation: The obtained distribution is not a pairwise maximum-entropy distribution Eq ( 3 ) —the Lagrange multipliers are incorrect—yet a consistency check ( also affected by undersampling ) may wrongly seem to validate it , and also analytic approximations ( outside of their convergence domain ) may wrongly validate it ., The distribution found in this circumstance is not a standard pairwise distribution , but our inhibited maximum entropy distribution Eq ( 22 ) , for appropriately chosen JI and θ ., In this regard we urge researchers who have calculated pairwise ( and even higher-order ) maximum-entropy distributions for more than 50 neurons using short Boltzmann-learning procedures , to check for the possible presence of higher metastable regimes ., The presence of bimodality and non-ergodicity can be checked , for example , by starting the sampling from different initial conditions , at low and high activities , looking out for bistable regimes cf . 62 , S 2 . 1 . 3 ., Another way out of this problem is to use other sampling techniques or Markov chains different from the Glauber one 61 , 62 , 97 , 98 ., Alternatively , one may use the inhibited model Eq ( 22 ) with the standard approaches ., In the presence of inhomogeneous and randomly chosen parameters and large network sizes , the standard pairwise maximum-entropy distribution is mathematically identical with the Boltzmann distribution of the Sherrington & Kirkpatrick infinite-range spin glass 124 , 125 ., A more systematic analysis of the effect of inhomogeneity on the appearance of the second mode could therefore employ methods developed for spin glasses 126 , which could produce approximate expressions for the inverse problem: the determination of Lagrange multipliers from the data ., One may think of modifying the Thouless-Anderson-Palmer ( TAP ) mean-field approach 67 , 127 , generalizations of which exist for the asymmetric non-equilibrium case 93 appearing here due to the inhibitory unit ., An appropriate modification of the ideas of Sessak and Monasson 68 , 69 could also be an alternative ., Another possibility is the use of cumulant expansions 17 , 128 , which unlike TAP-based approaches have the advantage of being valid also in regimes of strong coupling; recent extensions allow us to obtain the statistics at the level of individual units 129 ., In this work we have not investigated other models , like general linear models or kinetic Ising models for example ., Considering the fundamental mechanism by which the bimodality arises , we expect similar problems in other models ., The reasoning backing this hypothesis is this: Pairwise correlations in cortical areas are on average positive but very weak ., In this limit we expect that these correlations require slightly positive “excitatory” couplings between units in most other models; an independent-pair approximation also suggests this 127 ., As a result of this rough approximation determined at the level of individual pairs , we expect the couplings to be independent of the number of units of a dynamic or statistical model ., With increasing number of units in the model the overall “excitatory feedback” ∑ j N J i j will increase , and a simple mean-field analysis makes us expect the appearance of a second mode at a certain critical number , what in statistical mechanics is called a ferromagnetic transition; cf ., Fig 7B ., We expect similar ferromagnetic transitions to happen in a wide class of statistical models that only represent the observed , on average positively correlated units ., Similar transitions are also reported in Bohte et al . 34 for a biological—as opposed to statistical—neuron model composed of excitatory neurons only ., In fact , they had to introduce one inhibitory neuron in their model to avoid such transitions , which is also the idea behind our inhibitory term ., The bimodality problem could be cured by allowing for asymmetric connections , enabling the implementation of possibly hidden inhibitory units that stabilize the activity ., For example , kinetic Ising models 130–132 , which are maximum-entropy models over the possible histories of network activity 133–135 , can have positive correlations among excitatory units in the asynchronous irregular regime , while their dynamics is stabilized by inhibitory feedback see e . g . 136 , Fig 3A ., Scaling of network properties with the number of units N is often studied in this context ., In the asynchronous regime , mean pairwise correlations decrease as N−1 18 , 22 , 110 , 136 ., This scaling is the result of a fictive experiment , typically used to derive a theoretical results in the N → ∞ limit—any biological neuronal network has of course a certain fixed size N . The mean correlation measured in a sample of size M , with 1 ≪ M ≤ N , is by sampling theory expected to be roughly equal to the mean correlation of the full network , and does not vary much with M; only the variance around this expectation declines to 0 as M approaches N . The inhibited maximum-entropy model Pi , Eq ( 22 ) , solves the problems discussed above; but we may ask if this is enough to motivate its use ., We consider it an interesting model for at least two reasons ., First , it actually is a class of models rather than a single specific model ., In the present work we have focused on its use with pairwise constraints because these are still widely discussed in the literature ., But the inhibition reference measure Eq ( 25 ) can be used with higher-order constraints or other kinds of constraints as well ., We leave to future works the analysis of this possibility ., Second , there are neurobiological reasons why the reference measure Eq ( 25 ) can be methodologically more appropriate than the uniform measure of the standard maximum-entropy method ., Let us argue this point in more depth ., Standard ( i . e . uniform reference measure ) maximum-entropy distributions are often recommended as “maximally noncommittal” 137 ., But this adjective needs qualification ., Jaynes precised: ‘“maximally noncommittal” by a certain criterion’—that the possible events or states be deemed to have a priori equal probabilities before any constraints are enforced 31 ., When the initial probabilities are not deemed equal , for physical or biological reasons for example , reference measures appear ., An important example of reference measure is the “density of states” that multiplies the Boltzmann factor e − E/ ( kT ) in statistical mechanics e . g . 138 , ch . 16: we cannot judge energy levels to be a priori equally probable because each one comprises a different amount of degrees of freedom ., The proper choice of this reference measure is so essential as to be the first manifest difference between classical and quantum statistical mechanics , from “classical counting” to “quantum counting” of phase-space cells 138 , ch . 16 ., Owing to quantized energy exchanges , a quantum density of states is necessary in statistical mechanics; likewise we could say that owing to inhibitory feedback an inhibitory reference measure is necessary in the statistical mechanics of neuronal networks ., The uniform reference measure of standard maximum-entropy expresses that network units have a priori equally probable {0 , 1} states ., But these units are neurons , whose states are not a priori equally likely ., The measure of the inhibited model Pi reflects this a priori asymmetry in a simplified way ., There are surely other reference measures that reflect this asymmetry in a more elaborated way , but the one we have found is likely one of the simplest; cf ., Bohte et al . ’s 34 inhibitory solution ., The choice of an appropriate reference measure is critically important in neuroscientific inferences also for another reason ., When maximum-entropy is used to generate an initial distribution to be updated by Bayes’s theorem , the choice of reference measure is not critical , because a poor choice gets anyway updated and corrected as new data accumulate ., Not so when maximum-entropy is used to generate a sort of reference distribution that will not be updated , as is often done in neuroscience: an unnaturally chosen reference measure will then bias and taint all conclusions derived from comparisons with the maximum-entropy distribution ., The inhibited pairwise model can therefore be quite useful in all applications of the maximum-entropy model mentioned in “Introduction” ., For example , it can serve as a realistic hypothesis against which to check or measure the prominence of correlations in simulated or recorded neural activities , to separate the low baseline level of correlation from the potentially behaviourally relevant departures thereof ., The surprise measure to effect such separation would , according to the inhibited model , take into account the presence of inhibition and the overall low level of activity that are natural in the cortex ., The inhibited model can also be used for the generation of surrogate data which include the natural effect of inhibition besides the observed level of pairwise activity ., It can also be useful in the study of the predictive sufficiency of pairwise correlations as opposed to higher-order moments , for example for distribution tails e . g . 34–36 , 38–44; and in the characterization of dynamical regimes of neuronal activity 36 , 49–51 ., The inhibition reference measure Eq ( 25 ) contains the threshold θ and the inhibitory coupling JI as parameters ., The choice of their values depends on the point of view adopted about the measure ., Three venues seem possible: ( 1 ) One might think of choosing ( θ , JI ) to better fit the specific dataset under study , but this would counter the maximum-entropy spirit: the threshold cannot be a constraint , and the inhibitory coupling would acquire infinite values , as explained in section “Inhibited maximum-entropy model” ., Moreover for our dataset this strategy would only give a worse fit ( cf . Fig 2B ) because the inhibition term flattens the distribution tails ., ( 2 ) One might only want to get rid of the bistability of the Glauber dynamics and the bimodality of the distribution ., In this case the precise choice of ( θ , JI ) is not critical within certain bounds ., The inhibition coupling JI < 0 must be negative and sufficiently large to suppress activity once the population-averaged activity reaches θ ., The self-consistency condition Eq ( 21 ) then gives  1 + exp ( ∑ j j ≠ i J i j m j + h i + J I )  - 1 ⪡ θ for all i ., The threshold θ can be safely set to any value between the highest observed population activity s ¯ and the second fixed point of the self-consistency equation Eq ( 21 ) , which is indicative of the second mode and is beyond s ¯ > 1 / 2 ( see Fig 7B ) for the typically low mean activities observed in the cortex ., ( 3 ) A methodologically sounder possibility , in view of the remarks about maximum-entropy measures given above , is to choose ( θ , JI ) from general neurobiological arguments and observations ., This was implicitly done in Bohte & al ., ’s neuron model 34 for example , but unfortunately they did not publish the values they chose ., We leave the discussion of the neurobiological choice of these parameters to future investigations ., Our inibition term J I N G ( s ¯ - θ ) , Eq ( 22 ) , formally includes Shimazaki et al . ’s “simultaneous silence” constraint 44 as the limit JI → −∞ , θ = 1/N ., Because of this limit their model has a sharp jump in probability at s ¯ = 1 / N: their constraint uniformly removes probability for s ¯ > 1 / N and assigns it to the single point s ¯ = 0 ., In contrast , our inhibited model Pi presents a kink but no jump for s ¯ = θ , with a discontinuity in the derivative proportional to JI ., But besides this mathematical relationship , our inhibition term and the “simultaneous silence” constraint have different motivations and uses ., As discussed at length above and in section “Inhibited maximum-entropy model” , our term is best interpreted as a reference measure expressing the effects of inhibition , providing a biologically more suitable starting point cf . 34 for maximum-entropy , rather than a constraint ., Its goal is not to improve the goodness-of-fit for activities well below threshold , in contrast to earlier works e . g . 35 , 40 , 50 , 78 , 80 and to the “simultaneous silence” constraint 44 ., The goodness-of-fit is determined by the constraints alone ., In this regard we do not present any improvement of the fit compared to a pure pairwise model ., Future work could explore combinations of the here proposed reference measure and additional constraints that improve the fitness of the model ., Maximum-entropy models are an approximate limit case of probability models by exchangeability 139–141 , or sufficiency 141 , 142 , §§4 . 2–5 ., This approximation holds if the constraints are empirical averages ( e . g . time averages in our case ) over enough many data compared with the number of points in the sample space ., How much is “enough” depends on where the empirical averages lie within their physically allowed ranges: If they are well within their ranges , then a number of data values large but still smaller than the number of sample-space points may be enough ., If the empirical averages are close or equal to their physically allowed extreme values , then the number of data values should be much larger than the number of sample-space points ., If these conditions are not met the maximum-entropy method gives unreasonable or plainly wrong results , as can be ascertained by comparison with the non-approximat
Pairwise maximum-entropy models have been used in neuroscience to predict the activity of neuronal populations , given only the time-averaged correlations of the neuron activities ., This paper provides evidence that the pairwise model , applied to experimental recordings , would produce a bimodal distribution for the population-averaged activity , and for some population sizes the second mode would peak at high activities , that experimentally would be equivalent to 90% of the neuron population active within time-windows of few milliseconds ., Several problems are connected with this bimodality:, 1 . The presence of the high-activity mode is unrealistic in view of observed neuronal activity and on neurobiological grounds ., 2 . Boltzmann learning becomes non-ergodic , hence the pairwise maximum-entropy distribution cannot be found: in fact , Boltzmann learning would produce an incorrect distribution; similarly , common variants of mean-field approximations also produce an incorrect distribution ., 3 . The Glauber dynamics associated with the model is unrealistically bistable and cannot be used to generate realistic surrogate data ., This bimodality problem is first demonstrated for an experimental dataset from 159 neurons in the motor cortex of macaque monkey ., Evidence is then provided that this problem affects typical neural recordings of population sizes of a couple of hundreds or more neurons ., The cause of the bimodality problem is identified as the inability of standard maximum-entropy distributions with a uniform reference measure to model neuronal inhibition ., To eliminate this problem a modified maximum-entropy model is presented , which reflects a basic effect of inhibition in the form of a simple but non-uniform reference measure ., This model does not lead to unrealistic bimodalities , can be found with Boltzmann learning , and has an associated Glauber dynamics which incorporates a minimal asymmetric inhibition .
Networks of interacting units are ubiquitous in various fields of biology; e . g . gene regulatory networks , neuronal networks , social structures ., If a limited set of observables is accessible , maximum-entropy models provide a way to construct a statistical model for such networks , under particular assumptions ., The pairwise maximum-entropy model only uses the first two moments among those observables , and can be interpreted as a network with only pairwise interactions ., If correlations are on average positive , we here show that the maximum entropy distribution tends to become bimodal ., In the application to neuronal activity this is a problem , because the bimodality is an artefact of the statistical model and not observed in real data ., This problem could also affect other fields in biology ., We here explain under which conditions bimodality arises and present a solution to the problem by introducing a collective negative feedback , corresponding to a modified maximum-entropy model ., This result may point to the existence of a homeostatic mechanism active in the system that is not part of our set of observable units .
statistical mechanics, neural networks, vertebrates, neuroscience, animals, mammals, primates, probability distribution, mathematics, statistics (mathematics), thermodynamics, old world monkeys, computer and information sciences, entropy, monkeys, animal cells, probability theory, macaque, approximation methods, physics, statistical models, cellular neuroscience, eukaryota, cell biology, neurons, biology and life sciences, cellular types, physical sciences, amniotes, organisms