Rumney spent much of his life living as a wanderer, and was variously described as both a 'recluse' and a 'media whore', seeing his existence as a 'permanent adventure and endless experiment.' Rumney married Pegeen Guggenheim, the daughter of Peggy Guggenheim. He moved, as his friend Guy Atkins said, "between penury and almost absurd affluence.