Introduction Goal of the current study was to compare the respective roles of domain general cognitive skills with domain specific quantitative understanding, as well as the effect of math anxiety, on the performance of different types of arithmetic tasks. Methods Fifty-eight adults performed a battery of tests. Results We found dissociations between domain general abilities that supported verbally or spatially mediated arithmetic tasks. The verbally mediated tasks were supported by the verbal central executive component of working memory, while the spatially mediated task, number line knowledge, was supported by the spatial central executive component of working memory. Conclusions Different tasks had differential relationships with math anxiety: math anxiety effected school-like math tasks more than verbally mediated tasks and number line task. Math anxiety was negatively influenced by the spatial central executive component of working memory, indicating that spatial working memory can be a source of vulnerability to math anxiety.