Panayiotopoulos syndrome (PS) is a common epilepsy syndrome associated with rare clinical seizures and unknown localization of the epileptogenic area. Despite findings of normal development in patients with PS, recent neuropsychological studies point to subtle and diverse cognitive impairments. No well-outlined hypothesis about the localization of the brain dysfunction responsible for these impairments has been proposed. We further explored the cognitive dysfunctions in PS and made inferences on the most likely anatomical localization of brain impairment. A group of 19 patients (aged 6–12) with PS was rated according to spike activity and lateralization. The patients were submitted to a neuropsychological evaluation to assess general intelligence, memory, language, visual–perceptual abilities, attention, and executive functions. Using 35-channel scalp EEG recordings, the N170 face-evoked event-related potential (ERP) was obtained to assess the functional integrity of the ventral pathway. All patients with PS showed normal IQ but subtle and consistent neurocognitive impairments. Namely, we found abnormalities in the copy task of the Rey–Osterrieth Complex Figure and in the Narrative Memory Test. There was no correlation between neuropsychological impairments with spike activity and hemispheric spike lateralization. The N170 ERP was normal in all patients except for one. Our neuropsychological findings demonstrate impairments in visual–perceptual abilities and in semantic processing. These findings, paired with the absence of occipital lobe dysfunction in all neuropsychological studies of PS performed to this date, support the existence of parietal lobe dysfunction.