Into the early 20th century, the village had an active baseball rivalry with the Dean College baseball team in the center of Franklin. Caryville gradually faded as a defined village with the growth of suburbs in Bellingham and the construction of Interstate 495 in 1965. A large mill, dated to 1918, remained standing in Caryville until its demolition in 2016. Simultaneously, the Bellingham government ordered the removal of the small dam adjacent to the mill.