Andreas Loth scored the winning goal with barely a minute left after the Group A leaders squandered a 2-0 lead in the second period.
Klaus Kathan and Leonard Soccio had initially put Germany ahead in the first period, but Gerald Ressman and Gerhard Unterluggauer evened the scores for Austria.
A defensive mistake by Austria during a face-off allowed Loth to walk in alone on goalie Reinhard Divis and tip a weak backhand shot across the goal line.
The result leaves Germany riding high in Group A of the preliminaries with a 2-0 record, which includes their 3-0 upset of Slovakia on Saturday.
However, Austria became the first nation eliminated from medal contention, having lost both their matches so far.
In the other match on Sunday, Slovakia, which boasts the most NHL players in its squad, suffered another disappointing result by being held to a 6-6 draw against Latvia.
The tie means the Slovaks are out of the Olympics after just two games.
After losing 3-0 to Germany, Slovakia bolstered their offence by adding forwards Marian Hossa of the Ottawa, Jozef Stumpel of the Los Angeles and Pavol Demitra of St Louis.
The move paid off almost immediately as Stumpel scored just 11 seconds into the game on a blast from just inside the blue line.
In all the three combined for six points on the night.
But they could not make up for the shaky goaltending of Pavol Rybar, who was chased from the game five minutes into the second period after Aleksandrs Macijevskis made it 3-2 Latvia.
The moved spurred the Slovaks to four straight goals and a commanding 6-3 lead.
Replacement Jan Lasak, however, fared little better than Rybar, as Macijevskis scored his second, followed by goals by Aleksandrs Belavskis and Atvars Tribuncovs.
Florida Panthers defenceman Sandis Ozolins had a strong performance notching four assists.
Latvia, with one win and one tie, will face Germany in their final game on Tuesday needing a win to progress.