 Rufus was a grandson of the powerful Carteret County landowner and Revolutionary War veteran William Dennis Sr.

Pelletier moved to Onslow County and settled near Wantlands Ferry where he operated a turpentine distillery along with his brother William.In 1850, Rufus Pelletier constructed a one-roomed structure which survives today, known as Pelletier House.On August 2, 1863 he wed Joana Hines, and together they raised a daughter, Eliza.