He never intimated that he knew all the answers. He kept notes, and he learned." Red Smith described the relatively unknown Smith at the time: "He is tall and rangy, with graying brown hair and young, pleasant features smoothly tanned. There are crinkly lines of laughter around his blue eyes." After winning the National League pennant in 1950 (the year of the "Whiz Kids"), the Phillies had dropped into the middle of the National League, compiling a losing record in 1954. Smith became the team's fourth manager in three years. In Smith's first year with the Phillies, the team started poorly, dropping to 11 games below .500. The team rallied in the second half of the season and finished in fourth place with a 77–77 record, including 23 wins by pitcher Robin Roberts. At the end of the season, Smith received 30 of 99 votes by the Baseball Writers' Association of America as the 1955 National League Manager of the Year; he finished in second place behind Walter Alston. The Phillies slipped to 71–83 in 1956 with Richie Ashburn being the only regular player to bat over .289. After six consecutive 20-win seasons, Robin Roberts led the National League with 18 losses in 1956 while falling 1 win short of 20. In September 1956, and despite the losing record, the Phillies showed their confidence in Smith by signing him to a one-year contract extension through the 1957 season. In 1957, the Phillies returned to 77–77 despite pitcher Robin Roberts having a losing record of 10–22. Pitcher Jack Sanford, named Rookie of the Year in 1957, saved the Phillies from falling below .500 with his record of 19–8. In 1958, the team slipped below .500 with a record of 39–45 in late July. On July 22, 1958, Smith was fired and replaced by Eddie Sawyer. After Smith was fired, the 1958 Phillies dropped to last place under Sawyer. The Phillies finished in last place in the National League for four consecutive years from 1958 to 1961. In Smith's 3-1/2 years with the Phillies, he compiled a 264–281 record (). In the three-and-a-half years after he was fired, the Phillies fell to 200–332 (). In August 1958, shortly after Smith was fired by the Phillies, unconfirmed reports were published indicating that a deal was in the works for the Cincinnati Reds to hire Smith as their manager. Jimmy Dykes had been named interim manager after Birdie Tebbetts resigned the post. At the end of September, Smith signed a one-year contract to take over as manager for the 1959 season.