ogle returned to maryland in 1770 to find tasker's daughters (his aunts) were occupying his father's estate, the belair mansion, so he resided in his father's city home at 247 king george street in annapolis (which is now alumni house for the united states naval academy). ogle took possession of the belair mansion in 1774 after a lawsuit to reclaim his father's home from the tasker family.