 Stoughton resigned his regular commission in March 1861, and in September was appointed commander of the 4th Vermont Infantry with the rank of colonel.He was only 23 at the time of his appointment, and said to be the youngest colonel in the army.He led his command in the Peninsula Campaign, and his effective performance led to his selection for promotion and command of a brigade.