Studies on comprehensibility (e.g., Isaacs and Trofimovich, 2012; Saito et al., 2016) show that the main linguistic criteria explaining the effort deployed by native speakers (L1) to understand second language (Lx) speakers are grammar, vocabulary, phonology and fluency. These findings were established based on evaluations of speech samples produced in Lx English and, most often, assessed by expert judges. Therefore, this pilot study aims to assess whether the same trends can be observed in French Lx, based on the assessment of naive judges. In an oral stimulated recall task, six L1 French speakers commented on the comprehensibility of four 20-second speech samples produced by Lx French speakers. Their comments were transcribed and analyzed. Elements associated with cognitive effort during listening were coded according to the categories established by Isaacs and Trofimovich (2012). The results reveal that naive judges identified a wide variety of linguistic elements underlying their judgments, but fluency and pronunciation were the two major factors that appeared to impact their comprehensibility the most.