After graduating from high school, Duffy spent five years on the rails before attending the University of California at Santa Barbara. He later wrote a rail riding guide called "Hopping Freight Trains in America."