Downey mụrụ na Manhattan, New York, nke ọdụdụ n'ime ụmụ abụọ. Nna ya, Robert Downey Sr., bụ onye na-eme ihe nkiri na onye na-eme ihe nkiri, ebe nne ya, Elsie Ann (née Ford), bụ onye na-eme ihe nkiri nke pụtara na fim Downey Sr. Nna Downey bụ ọkara Lithuanian Jewish, otu ụzọ n'ụzọ anọ ndị Hungary, na otu ụzọ n'ụzọ anọ ndị Irish, ebe nne Downey nwere ndị Scotland, German, na Switzerland. Aha mbụ nke ezinụlọ Robert bụ Elias nke nna ya gbanwere ịbanye n'Òtù Ndị Agha. Downey na nwanne ya nwanyị nke okenye bụ Allyson tolitere na Greenwich Village. Dị ka nwatakịrị, Downey "gburugburu ọgwụ ọjọọ".
Elsie Ann