Tries from Kotaro Matsushima and Yoshikazu Fujita helped them into an 18-8 half-time lead, with Takudzwa Ngwenya crossing for the US.
Amanaki Mafi burst through for Japan's third try before Chris Wyles crossed to give the United States hope.
But Goromaru's third penalty late in the game gave Japan a clear lead.
After their thrilling opening win against South Africa, Japan went on to add the scalps of Samoa and the US, but their heavy defeat by Scotland left them third in the group, and they miss out on a place in the last eight.
The Eagles finish having lost all four of their matches.
Japan, who left Kingsholm to a standing ovation after a lap of honour, host the competition in four years' time and will go into that tournament buoyed by an excellent World Cup on English soil.
They had only ever won one match at a World Cup before, and their three victories in England represent a significant improvement.
Japan are the first non-tier-one team to win three matches at a single Rugby World Cup.
Full-back Ayumu Goromaru is the first player to score more than 50 points in a single RWC campaign for Japan. At the end of the pool stage, only Greig Laidlaw (60) has scored more than him in 2015 (58).
Eric Fry became the seventh player to get a yellow card for the USA in a RWC match.
Japan's win brought to an end a highly-competitive and entertaining pool stage. The quarter-finalists are now known as the 2015 tournament reaches the business end.
USA: Chris Wyles; Takudzwa Ngwenya, Seamus Kelly, Thretton Palamo, Zach Test, AJ MacGinty, Mike Petri; Eric Fry, Zach Fenoglio, Titi Lamositele, Hayden Smith, Greg Peterson, Al McFarland, Andrew Durutalo, Samu Manoa.
Replacements: Phil Thiel Thiel for Fenoglio (63), Cam Dolan for Smith (31), John Quill for Peterson (77).
Japan: Ayumu Goromaru; Yoshikazu Fujita, Harumichi Tatekawa, Craig Wing, Kotaro Matsushima, Kosei Ono, Fumiaki Tanaka; Keita Inagaki, Shota Horie, Hiroshi Yamashita, Luke Thompson, Justin Ives, Michael Leitch, Michael Broadhurst, Ryu Koliniasi Holani.
Replacements: Karne Hesketh for H. Ono (73), Masataka Mikami for Inagaki (59), Takeshi Kizu for Horie (77), Kensuke Hatakeyama for Yamashita (41), Shinya Makabe for Ives (68), Hendrik Tui for Broadhurst (73), Lelei Mafi for Holani (41).
Attendance: 14,578
Referee: Glen Jackson (New Zealand)
For the latest rugby union news, follow @bbcrugbyunion on Twitter.
Japan made unwelcome history as the first team to exit the World Cup having won three matches after they beat the United States.