Rufin was born in Bourges, Cher in 1952. An only child, he was raised by his grandparents as his father had left the family and his mother worked in Paris. His grandfather, a doctor and member of the French Resistance during World War II had been imprisoned for two years at Buchenwald.