Objectives Impairment of long-term recall may worsen everyday functioning of patients with epilepsy even if the standard short-term or delayed recall tests do not show significant abnormalities. We evaluated prospectively the decay of memory between delayed and long-term recall in patients with temporal lobe epilepsy (TLE) and controls with the aim of identifying the determinants of long-term memory impairment. Methods Seventy patients with TLE and 59 controls underwent neuropsychological assessment of verbal and nonverbal memory, attention, and executive functions at visit 1. Long-term verbal and nonverbal memory was tested with the same word list, verbal logical story, and Rey–Osterrieth complex figure test 4 weeks later at visit 2. The decay in memory was estimated as information recalled at visit 2 as a percentage of the delayed recall at visit 1. Results Frequent seizures (⩾4 per month) during the study period were related to poor long-term recall, even for those patients who did relatively well on delayed recall tests. On all long-term memory tests, patients with complex partial and/or secondary generalized seizures did significantly worse than patients with simple partial seizures. The presence of interictal generalized or focal temporal epileptiform activity was associated with more accelerated forgetting of the word list and complex figure. Multiple regression analysis confirmed that number of complex partial seizures, age of patient, and abnormal interictal EEG are significant predictors of accelerated forgetting. Conclusions Uncontrolled seizures, especially with ictal impairment of consciousness, can be a significant factor in the accelerated decay of memory, although subclinical interictal epileptiform EEG activity may also be relevant.