Objective Cognitive deficits are common in epilepsy, though the impact of epilepsy on cognition in older adults is understudied. This study aimed to characterize cognition in older adults with epilepsy compared with healthy older adults and identify potential risk factors for impairment. Methods Thirty-eight older adults with epilepsy and 29 healthy controls completed a comprehensive neuropsychological battery, as well as measures of depression and anxiety. Chart review for current medications, seizure history, and neuroimaging was also completed. To compare cognitive performance between groups, ANOVA was used, and linear regression identified predictors of impairment among the group with epilepsy. Results Patients with epilepsy performed worse across nearly all cognitive domains, and were clinically impaired (i.e., ≥1.5 SD below mean) on more individual tests when compared with controls, including a subset of patients with epilepsy with normal MRIs. For all patients with epilepsy, taking a greater number of antiepileptic drugs was associated with poorer language and visuospatial abilities, and higher anxiety was associated with poorer visual memory. Conclusions Older adults with epilepsy demonstrated greater cognitive deficits than matched controls. Polytherapy and anxiety heightened the risk for cognitive impairment in some cognitive domains, but not in others. Understanding the nature of cognitive decline in this population, as well as associated risk factors, may assist in the differential diagnosis of cognitive complaints and improve the design of treatment studies for older patients with epilepsy. Replication in larger, longitudinal studies is warranted to generalize these findings.