Input Alignment along Chaotic directions increases Stability in Recurrent Neural Networks
Anatomical studies demonstrate that brain reformats input information to generate reliable responses for performing computations. However, it remains unclear how neural circuits encode complex spatio-temporal patterns. We show that neural dynamics are strongly influenced by the phase alignment between the input and the spontaneous chaotic activity. Input alignment along the dominant chaotic projections causes the chaotic trajectories to become stable channels (or attractors), hence, improving the computational capability of a recurrent network. Using mean field analysis, we derive the impact of input alignment on the overall stability of attractors formed. Our results indicate that input alignment determines the extent of intrinsic noise suppression and hence, alters the attractor state stability, thereby controlling the network*s inference ability.
Overview: Using a reservoir of recurrent, randomly-connected spiking neurons, this paper studies how input stimuli may be modified to produce more robust responses. Specifically, this entails adjusting the phase of a sinusoidal input such that the leading principal components of the variation in the reservoir’s response match the leading principal components of its spontaneous and unstimulated chaotic dynamics. The reservoir’s ability to discriminate different stimuli, even in the presence of noise, is shown to be improved by aligning them to different principal components of its chaotic dynamics, both between stimuli and within a single stimulus in the presence of noise. The authors provide a statistical physics-driven outline of the theoretical analysis underpinning this result, and qualitatively demonstrate the effectiveness of their approach to a simple handwriting generation task. This paper addresses an important question and provides a result that is of interest, however, the overlap between this and previous related work is quite significant. In particular, this paper is heavily indebted to Rajan et al’s work in “Inferring Stimulus Selectivity from the Spatial Structure of Neural Networks” and “Stimulus-Dependent Suppression of Chaos in Recurrent Neural Networks”, two papers from nearly a decade ago from which it borrows most of its approach and theoretical analysis. The work presented here is a logical extension of these papers, and in some respects lacks originality. That being said, the demonstration that merely aligning the phase of a sinusoidal input (as opposed to identifying a resonant frequency, as in Rajan et al) is adequate to improve stability is an interesting result. The theoretical analysis is not at all new, but does provide a nice way to understand the empirical results, and in its presentation here is more understandable to non-statistical physicists than in the original formulation. Specific comments: - “Anatomical studies demonstrate that brain”, “Brain actively…” -> most instances of “brain” should be changed to “the brain” - Given that this is a standard liquid state machine, it should be explicitly called that and some citations to the relevant literature (e.g. Maass, , *Real-time computing without stable states: a new framework for neural computation based on perturbations*) should also be provided. - It’s difficult to understand why the effect of changing input phase changes (granted, somewhat unpredictably) the orientation of the resulting circular oscillation. Providing some intuition would be helpful. In particular, why is the relationship between phase and subspace orientation in the left plot of Fig/ 1 (c) symmetric about 180°? - Reservoir model schematic from fig. 2 should be moved to fig. 1 - Fig. 1 (a), thetas appear to represent the normal vectors of the first two PCs of activity, which should be made explicit. - Equation (2): Notation is unclear. Basis vectors should either be arranged into matrices, or “SingularValueOf” should be changed to an inner product - Visualization of the chaotic activity in 3D PC space would be helpful, consider moving it out of the appendix. - Fig. 2 (d), dashed lines should be noted as direction vectors in caption
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Input Alignment along Chaotic directions increases Stability in Recurrent Neural Networks
Anatomical studies demonstrate that brain reformats input information to generate reliable responses for performing computations. However, it remains unclear how neural circuits encode complex spatio-temporal patterns. We show that neural dynamics are strongly influenced by the phase alignment between the input and the spontaneous chaotic activity. Input alignment along the dominant chaotic projections causes the chaotic trajectories to become stable channels (or attractors), hence, improving the computational capability of a recurrent network. Using mean field analysis, we derive the impact of input alignment on the overall stability of attractors formed. Our results indicate that input alignment determines the extent of intrinsic noise suppression and hence, alters the attractor state stability, thereby controlling the network*s inference ability.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Overview: Using a reservoir of recurrent, randomly-connected spiking neurons, this paper studies how input stimuli may be modified to produce more robust responses. Specifically, this entails adjusting the phase of a sinusoidal input such that the leading principal components of the variation in the reservoir’s response match the leading principal components of its spontaneous and unstimulated chaotic dynamics. The reservoir’s ability to discriminate different stimuli, even in the presence of noise, is shown to be improved by aligning them to different principal components of its chaotic dynamics, both between stimuli and within a single stimulus in the presence of noise. The authors provide a statistical physics-driven outline of the theoretical analysis underpinning this result, and qualitatively demonstrate the effectiveness of their approach to a simple handwriting generation task. This paper addresses an important question and provides a result that is of interest, however, the overlap between this and previous related work is quite significant. In particular, this paper is heavily indebted to Rajan et al’s work in “Inferring Stimulus Selectivity from the Spatial Structure of Neural Networks” and “Stimulus-Dependent Suppression of Chaos in Recurrent Neural Networks”, two papers from nearly a decade ago from which it borrows most of its approach and theoretical analysis. The work presented here is a logical extension of these papers, and in some respects lacks originality. That being said, the demonstration that merely aligning the phase of a sinusoidal input (as opposed to identifying a resonant frequency, as in Rajan et al) is adequate to improve stability is an interesting result. The theoretical analysis is not at all new, but does provide a nice way to understand the empirical results, and in its presentation here is more understandable to non-statistical physicists than in the original formulation. Specific comments: - “Anatomical studies demonstrate that brain”, “Brain actively…” -> most instances of “brain” should be changed to “the brain” - Given that this is a standard liquid state machine, it should be explicitly called that and some citations to the relevant literature (e.g. Maass, , *Real-time computing without stable states: a new framework for neural computation based on perturbations*) should also be provided. - It’s difficult to understand why the effect of changing input phase changes (granted, somewhat unpredictably) the orientation of the resulting circular oscillation. Providing some intuition would be helpful. In particular, why is the relationship between phase and subspace orientation in the left plot of Fig/ 1 (c) symmetric about 180°? - Reservoir model schematic from fig. 2 should be moved to fig. 1 - Fig. 1 (a), thetas appear to represent the normal vectors of the first two PCs of activity, which should be made explicit. - Equation (2): Notation is unclear. Basis vectors should either be arranged into matrices, or “SingularValueOf” should be changed to an inner product - Visualization of the chaotic activity in 3D PC space would be helpful, consider moving it out of the appendix. - Fig. 2 (d), dashed lines should be noted as direction vectors in caption
assistant
Overview: Using a reservoir of recurrent, randomly-connected spiking neurons, this paper studies how input stimuli may be modified to produce more robust responses. Specifically, this entails adjusting the phase of a sinusoidal input such that the leading principal components of the variation in the reservoir’s response match the leading principal components of its spontaneous and unstimulated chaotic dynamics. The reservoir’s ability to discriminate different stimuli, even in the presence of noise, is shown to be improved by aligning them to different principal components of its chaotic dynamics, both between stimuli and within a single stimulus in the presence of noise. The authors provide a statistical physics-driven outline of the theoretical analysis underpinning this result, and qualitatively demonstrate the effectiveness of their approach to a simple handwriting generation task. This paper addresses an important question and provides a result that is of interest, however, the overlap between this and previous related work is quite significant. In particular, this paper is heavily indebted to Rajan et al’s work in “Inferring Stimulus Selectivity from the Spatial Structure of Neural Networks” and “Stimulus-Dependent Suppression of Chaos in Recurrent Neural Networks”, two papers from nearly a decade ago from which it borrows most of its approach and theoretical analysis. The work presented here is a logical extension of these papers, and in some respects lacks originality. That being said, the demonstration that merely aligning the phase of a sinusoidal input (as opposed to identifying a resonant frequency, as in Rajan et al) is adequate to improve stability is an interesting result. The theoretical analysis is not at all new, but does provide a nice way to understand the empirical results, and in its presentation here is more understandable to non-statistical physicists than in the original formulation. Specific comments: - “Anatomical studies demonstrate that brain”, “Brain actively…” -> most instances of “brain” should be changed to “the brain” - Given that this is a standard liquid state machine, it should be explicitly called that and some citations to the relevant literature (e.g. Maass, , *Real-time computing without stable states: a new framework for neural computation based on perturbations*) should also be provided. - It’s difficult to understand why the effect of changing input phase changes (granted, somewhat unpredictably) the orientation of the resulting circular oscillation. Providing some intuition would be helpful. In particular, why is the relationship between phase and subspace orientation in the left plot of Fig/ 1 (c) symmetric about 180°? - Reservoir model schematic from fig. 2 should be moved to fig. 1 - Fig. 1 (a), thetas appear to represent the normal vectors of the first two PCs of activity, which should be made explicit. - Equation (2): Notation is unclear. Basis vectors should either be arranged into matrices, or “SingularValueOf” should be changed to an inner product - Visualization of the chaotic activity in 3D PC space would be helpful, consider moving it out of the appendix. - Fig. 2 (d), dashed lines should be noted as direction vectors in caption