Age-related hearing loss is associated with a decrease in hearing abilities for high frequencies and therefore leads to impairments in understanding speech—in particular, under adverse listening conditions. Growing evidence suggests that age-related hearing loss is related to various neural changes, for instance, affecting auditory and frontal brain regions. How the decreased auditory input and the increased listening effort in daily life are associated with structural changes is less clear, since previous evidence is scarce and mostly involved low sample sizes. Hence, the aim of the current study was to investigate the impact of age-related untreated hearing loss and subjectively rated daily life listening effort on grey matter and white matter changes in a large sample of participants (n = 71). For that aim, we conducted anatomical MRI and diffusion tensor imaging (DTI) in elderly hard-of-hearing and age-matched normal-hearing participants. Our results showed significantly lower grey matter volume in the middle frontal cortex in hard-of-hearing compared to normal-hearing participants. Further, higher listening effort was associated with lower grey matter volume and cortical thickness in the orbitofrontal cortex and lower grey matter volume in the inferior frontal cortex. No significant relations between hearing abilities or listening effort were obtained for white matter integrity in tracts connecting auditory and prefrontal as well as visual areas. These findings provide evidence that hearing impairment as well as daily life listening effort seems to be associated with grey matter loss in prefrontal brain regions. We further conclude that alterations in cortical thickness seem to be linked to the increased listening effort rather than the hearing loss itself.