He made his debut against his former club Wimbledon on Boxing Day in a goalless draw. Brown scored his only goal of his second Watford stint on 5 April 2003 in their 7–4 away win against Burnley. After failing to make an appearance for Watford in the 2003–04 season, Brown was loaned out to Gillingham for one month on 19 September 2003. He made his debut on 20 September in a 2–0 win over West Ham United, and scored his only goal of his four-game stint on 29 September in a 2–1 away defeat by Walsall. Brown returned to the first-team at Watford and made regular appearances until Christmas 2003. He was then loaned to Colchester United for the second time on 12 February 2004 until the end of the season, making his second debut in the Essex derby in a 1–1 Football League Trophy Area Final 1–1 draw with Southend United. During his stint, he received the first red card of his career for a second bookable offence in Colchester's 2–1 Second Division defeat to Brighton & Hove Albion on 20 March. He made 17 appearances for the U's. Brown signed for Phil Parkinson's Colchester United on a permanent basis ahead of the 2004–05 season following his successful loan at the end of the previous season. He made what was his third debut for the club on 7 August 2004 in Colchester's opening day 3–0 win over Sheffield Wednesday at Hillsborough. He scored once in a 2–2 draw at Wrexham on 22 February 2005 as he went on to make 47 appearances in all competitions. During the 2005–06 season Brown helped his side to second position in the League One table, earning promotion to the Championship for the first time in the club's history. He made 45 appearances in all competitions. He had received only his second-ever dismissal on 3 September 2005 during Colchester's goalless draw at Bristol City. He scored three times during the season, with a goal in Colchester's record-equalling 9–1 victory over Leamington in the FA Cup first round, another in their 5–0 win at Layer Road against his former side Gillingham, and in a 1–0 home victory against Tranmere Rovers. He was named as Colchester United Player of the Year at the end of the season. Brown's good form continued into Colchester's inaugural season in English football's second tier, guiding the side to their highest-ever league finish of 10th position in the Championship.