Mental speed is high until age sixty
Comment re-pasted from pubpeer [1], as this article contains claims which amy be disingenuous due to data handling.<p>&quot;This article was reviewed on the error bar podcast episode 25: <a href="https:&#x2F;&#x2F;theerrorbar.com&#x2F;?e=25" rel="nofollow">https:&#x2F;&#x2F;theerrorbar.com&#x2F;?e=25</a> &amp; story 106: <a href="https:&#x2F;&#x2F;theerrorbar.com&#x2F;?s=106" rel="nofollow">https:&#x2F;&#x2F;theerrorbar.com&#x2F;?s=106</a>. The review was done in the light of newspaper headlines about the article. The slightly-edited transcript is copied here:<p>headline: BRAIN DOESN&#x27;T SLOW DOWN UNTIL 60<p>tweet: analysis of 1.2 million responses on the &#x27;implicit association test&#x27; finds that adults&#x27; mental processing speed is totally good &amp; constant until age 60. that&#x27;s good news for all adults over age 35. until you look at the data.<p>description: in a story likely comforting to middle aged scientists &amp; journalists, the Guardian &amp; Independent reported last week that, Good News: the brain does not slow down until we are aged 60. what did they mean?<p>the result comes from a quite complicated analysis of an enormously huge set of data. a social psychology project in the USA has collected nearly 2 million responses from people who were asked to categorise things on a computer screen as, for example, &#x27;good&#x27; versus &#x27;bad&#x27; or as &#x27;black&#x27; versus &#x27;white&#x27;. they did this 120 times each.<p>in general, the results of this so-called &#x27;implicit association test&#x27; are able to reveal people&#x27;s unconscious attitudes - in this example, that white things are good &amp; black things are bad. this test is a classic &amp; it relies on people responding as quickly as possible.<p>in the new study, researchers in Germany downloaded 1.8 million test responses from the free online database. they analysed how people&#x27;s responses changed as they age, from 10 to 80 years old. using some sophisticated (&amp; unnecessarily complicated) analyses, the researchers reported that, while the total time spent doing the task changes a lot over life, one conclusion is that, in fact, our &#x27;mental processing speed&#x27; remains good &amp; strong until age 60.<p>phew!<p>evaluation: does mental speed stay high until age 60?<p>no.<p>this is not a hard one. with a total of nearly 1.2 million responses, all openly available online &amp; clearly-presented in this paper, it is a fantastically-simple task to fact-check this claim. all we need to do is look at the data.<p>i&#x27;ve annotated the image to accompany this story on the error bar website. what the image shows is a gradual &amp; continuous increase in mental speed between age 10 to a peak at about 35, then a gradual &amp; continuous decrease to age 80. it&#x27;s true that the speed of change seems to be different at different ages, but the strong claim that mental speed is &#x27;high &amp; constant until age 60&#x27; is contradicted by just looking at the data.<p>how did it come to this? the error bar dug deeper.<p>first, this is a massive dataset - which is great - but quite a lot of data were removed, for example anyone performing the task without making errors was excluded. that&#x27;s a bit odd, if the best responders are removed.<p>second, the task itself is quite subjective - is something good or bad? or is something black of white? - &amp; although a well-studied task, it&#x27;s just one task, so conclusions about mental speed in general must be extremely limited.<p>third, the data are not a good measure of mental processing speed. to test mental speed, we need volunteers to be fully-focussed &amp; making responses as quickly as possible. but the authors included &#x27;response times&#x27; as long as nearly 6 seconds, the average times were around 1 second &amp; there were some massive - and unexplained - differences between men&#x27;s &amp; women&#x27;s responses.<p>fourth, one claim in the report is that the &#x27;merely mechanical&#x27; processes of seeing the stimuli on screen &amp; making the keyboard press response take as long as half a second in the oldest adults. this is a pure fiction - the study provides no sensible measurement of these sensory &amp; motor processing speeds.<p>fifth, &amp; amusingly, the authors chose a particular kind of deep, machine-learning analysis because they said it was &quot;not possible&quot; to analyse all the data in a different, simpler, but more computationally-expensive way. but the new analysis only took 32 hours to run on a computer, which makes me wonder how long the full analysis would take &amp; why was there such a rush to get it done?<p>sixth &amp; critically, the conclusion that mental processing speed is high and constant until age 60 relies only on the authors&#x27; choice of how to put a best-fit line on the data. the analysis used requires all the people to be divided into two or more different age groups. for error bar nerds: the data were fit with three straight lines for three age groups. but age is a continuous variable, so why did the authors not just fit a single, continuous curve?<p>well, listeners, i downloaded the data, sharpened Occam&#x27;s razor &amp; fit that single curve. it provides a very slightly worse explanation of the data, but that&#x27;s what you would expected, as it is also a much less-complex explanation.&quot;<p>[1]