Reared in rural Virginia , one of seven children born to a tobacco-farming couple , Mr. Oliver was a quiet , courtly man who , according to family members , held a variety of jobs after moving to New York in the 1970 's : as a worker in a handbag factory , a salesman at a jewelry store and , in recent years , a housecleaner for affluent clients in Manhattan .