Question: When looking at old games like Mario64 or DukeNukem3D, all the mirrors in the game are essentially just holes in the wall with a mirrored copy of the geometry in front of the mirror put behind them. In the case of DukeNukem3D one can even activate no-clip and walk into that mirrored room.

In contrast modern games use a render-to-texture approach for mirrors. This leads to the mirrors getting noticeably pixelated when getting to close to them. One of the first games I noticed this approach was Luigi's Mansion, but it seems to be used in almost all modern games.

What change in the hardware or engines made the second approach become so dominant these days and what are the benefits to it? In terms of pure visuals the first approach seems superior, as it doesn't suffer from pixelation issues.

Answer: 
RTT would have been used if it was possible but the hardware rendering pipeline was one way.

Older hardware also had limitations that prevented render to texture. Writing to RAM means it can not be read at the same time. To improve rendering performance the destination buffer was locked to write only, only the display hardware could read from it. You could request reading, but that locked the RAM and the render had to wait for the lock to clear before it could start the next frame. RTT would cause a major bottleneck to the pipeline and thus other solutions were used.

You will find that befor Hardware rendering pipelines were the norm RTT was used as it provided a way to reduce the rendering load. 3D rendered to sprites to provide pseudo 3D content. Texture rendering was way too costly (CPU) to be used back then, apart from specialised machines that were outside the general consumer market.
Duke Nukem handle that by re-rendering the geometry behind the mirror, the other answers are partially correct. There are areas behind mirrors that actually contains no geometry (in game data files), the geometry is re-rendered at run time infact, the reason for those areas to exist is to avoid to place accidentally a piece of level there when editing the level:

since theres' an area marked you will not place accidentally any geometry in there.