The validity of Baddeley's working memory model within the typically developing population, was tested. However, it is not clear if this model also holds in children and adolescents with mild to, borderline intellectual disabilities (ID; IQ score 55–85). The main purpose of this study was therefore, to explore the model's validity in this population. Several verbal and visuo-spatial STM and WM tasks, were administered to 115 children with mild to borderline ID (mean age 10 years) and to 98, adolescents with mild to borderline ID (mean age 15). Structural equation modeling (LISREL) shows, that Baddeley's working memory model does not fit the data of the 10-year and 15-year old, participants. Principal components analyses on the other hand show a hazy pattern with on the one, side an indication for a ‘general’ component with loadings of visuo-spatial short-term memory and, working memory tasks and a separate verbal short-term memory component. On the other hand there, is also an indication of a modality specific memory structure; a visuo-spatial- versus a verbal, component. A straight-forward dichotomy between STM and WM indicates apparently an, oversimplification, at least it is for children and adolescents with mild to borderline ID.