Presence of the apolipoprotein E (APOE) ε4 allele is linked to an increased risk to develop Alzheimer's dementia (AD). However, there are controversial data concerning the impact of the APOE genotype on cognitive functioning and brain activity in healthy subjects. We used event-related functional magnetic resonance imaging (fMRI) to investigate the effects of APOE genotype on spatial contextual memory encoding and retrieval success in healthy older adults. Eighteen subjects (eight APOE4 heterozygotes (ε4+) and 10 non-carriers (ε4−), mean age 60.0±5.0 years) were included in the present analysis. Behaviorally, ε4+ subjects performed significantly worse than ε4− subjects in item memory and spatial context retrieval. fMRI data revealed that ε4+ subjects, compared to ε4-subjects, predominantly showed an increase of neural activity specific to encoding of items and their spatial context in prefrontal, temporal and parietal regions. In contrast, ε4+ subjects showed activity decreases in the right amygdala during successful item recognition and in the prefrontal cortex bilaterally during spatial context retrieval when compared to ε4− subjects. While the activity increases during encoding may reflect compensatory activity in the attempt to maintain normal performance, the decreases during retrieval indicate incipient neural decline in ε4+ subjects. These data highlight that preclinical ApoE-related changes in neural activity are not unidirectional but dissociate depending on the memory phase, i.e., encoding or retrieval.