The objective of this study was to further explore proper name (PN) retrieval and conceptual knowledge in patients with left and right temporal lobe epilepsy (69 patients with LTLE and 62 patients with RTLE) using a refined assessment procedure. Based on the performance of a large group of age- and education-matched normals, a new test of famous faces and famous landmarks was designed. Recognition, naming, and semantic knowledge were assessed consecutively, allowing for a better characterization of deficient levels in the naming system. Impairment in PN retrieval was common in the cohort with TLE. Furthermore, side of seizure onset impaired stages of name retrieval differently: LTLE impaired the lexico-phonological processing, whereas RTLE mainly impaired the perceptual–semantic stage of object recognition. In addition to deficient PN retrieval, patients with TLE had reduced conceptual knowledge regarding famous persons and landmarks.