Readfern was recalled for the next game however, meaning that Brown immediately returned to the reserves, although Brown did return to the first team in mid-October, when he scored on his home debut which Albion went on to win 4–3 against Aston Villa. He was selected sporadically by Albion manager Jimmy Hagan during the remainder of 1963–64 and finished the campaign on five goals in thirteen appearances. In the latter part of the season he missed four league games when he was called up to the England Youth squad for the UEFA European Youth Tournament in the Netherlands. England won the competition but Brown was not selected for any of the games. Brown became a regular first team player during the early part of the 1964–65 campaign, appearing in Albion's first 12 league games of the season. During this period he scored six goals, including his first professional hat-trick, which came in a 4–1 win against Sunderland. However he lost his place shortly after the signing of Jeff Astle and played just five more games that season. A desire to play first team football led him to submit a transfer request, which Jimmy Hagan turned down, but Brown continued his scoring run in the reserve team, hitting 22 goals in 20 games. At the start of the 1965–66 season, Brown was still playing in the reserves, but an injury to Clive Clark in September 1965 gave him the opportunity to reclaim his first team place. Brown scored against Everton on his return and remained a regular in the side for the rest of the season, scoring 17 goals in 35 league games. His tally in the league was just short of teammates John Kaye and Jeff Astle, who each scored 18 times, but Brown finished as the club's top scorer overall—with 27 goals—due to his goals in the Football League Cup. Albion were taking part in the competition for the first time, after the FA offered the winners a place in the Inter-Cities Fairs Cup. Brown became the first player to score in every round of the League Cup, including a hat-trick in the semi-final, second leg against Peterborough United. In the final, Brown's goal helped his team to a 4–1 second leg victory over West Ham United. Albion won 5–3 on aggregate to take the trophy and secure Brown his first major honour. He has since said of the game: Brown missed two months of the following season due to a back injury. He made his return in the second leg of the Inter-Cities Fairs Cup against DOS Utrecht, when he took—and scored—his first penalty kick in senior football; he went on to become Albion's regular penalty taker, scoring 51 of 61 spot-kicks during his time at the club.