 In 1837, von Armansperg was dismissed from his post and he returned to the von Armansperg residence of "Schloss Egg" near Deggendorf in Lower Bavaria.He died in 1853 in Munich.Josef Ludwig had 4 daughters