Rafelson began dating Toby Carr in high school and they later married in the mid-1950s. The couple had two children: Peter Rafelson, born in 1960, and Julie Rafelson, born in 1962.[8] Toby Rafelson was a production designer on many films, including her husband's Five Easy Pieces, The King of Marvin Gardens, and Stay Hungry, as well as Martin Scorsese's Alice Doesn't Live Here Anymore and Jonathan Demme's Melvin and Howard.[9]