Neural representation of action sequences: how far can a simple snippet-matching model take us?
The macaque Superior Temporal Sulcus (STS) is a brain area that receives and integrates inputs from both the ventral and dorsal visual processing streams (thought to specialize in form and motion processing respectively). For the processing of articulated actions, prior work has shown that even a small population of STS neurons contains sufficient information for the decoding of actor invariant to action, action invariant to actor, as well as the specific conjunction of actor and action. This paper addresses two questions. First, what are the invariance properties of individual neural representations (rather than the population representation) in STS? Second, what are the neural encoding mechanisms that can produce such individual neural representations from streams of pixel images? We find that a simple model, one that simply computes a linear weighted sum of ventral and dorsal responses to short action "snippets", produces surprisingly good fits to the neural data. Interestingly, even using inputs from a single stream, both actor-invariance and action-invariance can be accounted for, by having different linear weights.
SUMMARY This paper makes two points. First 
      individual neurons in superior temporal sulcus have a continuous and 
      distributed encoding of both the actor and action (and their conjunction) 
      during action observation. It then makes the second point that the evoked 
      responses (mean firing rate) can be predicted by a linear mixture of 
      transformed pixel input – where the transform is motivated by existing 
      models of dorsal and ventral stream processing. The report is written 
      extremely nicely; however, it is rather descriptive and could be improved 
      with some clarification of its conceptual direction and analysis details. 
      COMMENTS TO AUTHORS Your report was written very 
      nicely and enjoyable to read. On the other hand, there was a slight 
      ambiguity about the underlying message and the connection between the 
      snippet-matching model and your analyses. Perhaps you could consider the 
      following; 1) Your analyses make two separate points. First, there 
      are action and actor invariant responses in STS neurons. Second, you can 
      predict the responses of STS neurons using a transformation of pixel 
      inputs which – at the last stage – involves a linear mixture. My problem 
      here is that these two points are unconnected. You could have presented 
      the invariance analyses separately from the linear mixture analyses and 
      vice versa. This is not necessarily bad but I think you need to clarify 
      what the contribution of this report is and try to link the two points 
      through some contribution to our understanding of perceptual synthesis? 
      2) It is not clear how your linear regression analysis relates to 
      the snippet-matching model. Your discussion of the snippet-matching model 
      could be interpreted as either a response to a short sequence of inputs 
      (with explicit temporal support) or simply a response to a time average 
      (integrating the input over a small time window). Given that you seem to 
      interpret your regression analysis in reference to the snippet-matching 
      model, I suspect the latter is the case. If this is so, could you make it 
      clear that each time bin or time step in the regression analysis 
      corresponds to a snippet of input and that this is summarized by the 
      average over 120 milliseconds. Furthermore, say how many time steps there 
      were – in other words quantify T. It is confusing because you introduce 
      snippet-matching after the regression analysis and then say: “Again, we 
      try the simplest possibly function:” When you say “Again”, does this mean 
      you have done two analyses and have only reported one or are you simply 
      interpreting regression analysis in terms of the snippet-matching model. 
      If the latter, it might be useful to introduce the snippet-matching model 
      first and then describe how it motivated your linear regression. 
      MINOR POINTS 1) In the abstract, you need to make it clear 
      that this paper is about characterising neuronal responses to action 
      observation. The title and first sentences made me think I was going to 
      read about motor control. 2) I am not sure about your argument in 
      the second paragraph of the introduction. Simply showing that there is 
      information about actions, actors (and their conjunction) in any part of 
      the brain does not mean they have tackled a “(three-fold) challenge”. One 
      would find (presumably) very similar response properties in any part of 
      the mirror neuron system. Furthermore, if one analysed retinal cells, one 
      would also find this information. Perhaps you could highlight the fact 
      that you have found information or invariance properties at the level of 
      the single neuron - that could not be found at low levels in the visual 
      hierarchy - to make your point more clearly? 3) Your use of cross 
      validation to establish the utility of the linear regression model of 
      neuronal responses is fine for the machine learning community. However, it 
      is statistically sub-optimal in relation to inference based directly on 
      your general linear model: by central limit theorem, you can assume 
      Gaussian errors and provide a much more sensitive analysis in terms of 
      classical inference. Furthermore, you could have used conventional F 
      statistics to test different hypotheses about the relative contribution of 
      dorsal and ventral predictors of the neuronal responses – or indeed their 
      interaction. I suspect you will choose to stay with the cross validation 
      scheme but it might be interesting to pursue standard statistical analyses 
      in future work? 4) On page 5 (line 250) I would say “A more 
      continuous and distributed representation scheme”. 5) On page 6 
      (line 299) what is “a matching function”? The notion of a snippet-matching 
      process is not described very clearly. Is this simply an instantaneous 
      mapping between some running average? Or is there something deeper going 
      on? 6) Page 5 (line 304), I would spell out that the dorsal and 
      ventral predictors both had a correlation coefficient of 0.33 – otherwise, 
      people will think this is the correlation coefficient for the combined 
      analysis. 7) You do not report any statistical inference or 
      p-values. It might be nice to associate p-values with your correlation 
      coefficients and use a Fishers transform when comparing the correlations 
      between different regression models (for example, the processed versus 
      non-processed principal components). I hope that these comments 
      help should any revision be required.