How does Guy know how to perform the jutsu in the Eight Gates Released Formation? I know that if you open the 8th Gate, you will get enormously powerful but at the end you will die.  

In that case, how did Guy know Jutsu that can be performed in that state, given that against Madara was the first time he ever opened it? You can't just invent new Jutsu on the fly.
Okay, it's hard to explain, and i might not even be right, but I'll try my best. Sadly, there's no tldr.
What's different? Guy didn't invent the jutsu in the sense that Kakashi invented the Chidori. Kakashi spent years perfecting it. It's something he initially needed to use hand signs to perform every time.
Why?
The jutsus (more specifically taijutsu techniques) Guy used — Evening Elephant (Sekizo) and Night Guy — are byproducts of attacking with all eight gates opened. It's not something you have to learn; it's something you are just able to do with that much power flooding from your body.
The jutsu are just names he used to describe the phenomenon of having incredibly OP punches and kicks that can also generate a dragon-like aura (for some reason). He didn't actually know what would happen, unless he witnessed his dad do the same thing and he told Rock Lee what it was like.
Evidence From Narutopedia (Sekizo, Night Guy):

The Evening Elephant ... can be performed by those users who have
opened all Eight Gates.
The Night Guy ... can only be performed by
opening all Eight Gates.

^ They imply that it's something that you can naturally do with the eight gates opened.

When Guy initiates the technique, he emits an enormous amount of
chakra ... the blood vapour of the Eighth Gate manifests into a red,
flickering dragon-like aura ... He then dashes forward at such an
extreme speed that the space within the immediate vicinity of the
technique is distorted, making it impossible for the target to defend
against it. Guy then deals an immensely powerful kick which carries
such force that it is capable of obliterating the target's body...

^ Essentially, Guy is gathering power, running really fast, and kicking really hard. That's not really something that you have to practice and invent, like the Rasen-shuriken.
Examples: Kinda like when Obito and Madara became the Ten-tails Jinchuriki. They could do all this fancy truth-seeking ball stuff. Not because they spent time learning it, but because they were just able to, with the help of Ten-tails and huge sage power. Or when Naruto was able to heal people and Sasuke was able to teleport as soon as they got Hagoromo's chakra.
Sigh... Nevertheless, there are too many caveats and exceptions in the Naruto universe. Everything they say isn't necessarily true (e.g. Gaara's absolute defence is not absolute, Hashirama's unique wood release can be used by several other people). I mean, your first sentence is refuted by the fact that Guy actually survived (thanks to Naruto). So it's quite hard to make sense of anything.
More info - If you're not satisfied, look at this guy's explanation, which is good in the sense that it uses common sense: How do you train in a jutsu that kills you if used?