He was elected to start the All-Star Game as a second baseman that season despite being sidelined for a month by one of the more bizarre baseball injuries. While arguing with first base umpire Jim Duffy, Goodman was restrained by teammate Jim Piersall. Piersall pulled Goodman toward the dugout, and in doing so, strained Goodman's rib cartilage. In , Goodman returned to his "jack of all trades" role with the Bosox. After starting the season at second, he moved over to third when the Sox traded George Kell to the Chicago White Sox. He moved to left field when Ted Williams was sidelined by a virus infection in his right lung. Upon Williams' return, he began platooning at first with Harry Agganis (despite the fact that both were left-handed batters) until he was shifted back to second base in the beginning of August. He had a starting job at second base again in , and led the team with 100 runs scored while logging a team high 719 plate appearances. Both were career highs, as were his 176 hits and 99 walks. He began losing playing time to Ted Lepcio at second base toward the end of the season. He was relegated to pinch hitting duties early in the season before a mid-season trade sent him to the Baltimore Orioles for pitcher Mike Fornieles. During his time with the Red Sox, Goodman was one of the players featured in the 1957 Norman Rockwell painting "The Rookie". Goodman was immediately inserted into the starting line-up in Baltimore, and hit a home run in his first game as an Oriole. He mostly played third base, filling in for an injured George Kell, but also played first, second, short, left field and right field. He batted .308 with three home runs and 33 RBIs in 73 games for the Orioles. At the end of the season, he, Tito Francona and Ray Moore were dealt to the Chicago White Sox for Larry Doby, Jack Harshman and Jim Marshall. Chicago later sent pitcher Russ Heman to Baltimore as part of this deal when it was discovered by the Orioles that Harshman was suffering from a slipped disc. In 1958, with Nellie Fox at second base, Goodman shifted to third with the White Sox. He was sidelined by a knee injury for most of the month of May. Upon his return, he quickly shot up among the American League leaders in batting, with his average peaking at .336 in late July. At 33 years old at the start of the season, Goodman was used in a lefty-righty platoon with Bubba Phillips at third base.