Stony Brook was a remote area through the 18th century aside for a modest amount of commerce near the mill at the intersection of Main Street and Harbor Road. The community's development was stalled by its poorly accessible harbor relative to nearby Setauket and Port Jefferson.