The answer depends somewhat on the interpretation of quantum mechanics you choose. However, if you take one of the standard ones, then a superposition of two states is not the same as being in both of these states simultaneously, nor as being in an unknown one of these states, nor as being in an unknown state "in between". Superposition is an entirely new idea that we don't have words in conversational English to express.
It might help to give a real-world example that mirrors the math. You can drive your car in any direction. Suppose your car also has a widget that indicates what compass direction you're moving in, but it's primitive and can only say "North", "East", "South", or "West". These are the only four things it can ever say. 28
If you drive northeast, it has a 50/50 chance of saying North or East. Now I'll reframe all your questions in this context and the answers should be clearer:
However, I'm wondering if before the measurement, i.e. during superposition, the quantum really is in those two different pure states ($|0\rangle$ and $|1\rangle$) - or maybe even in all states in between those two?
"I'm wondering if, when you drive northeast, the direction of your car really is in these two different pure directions (North and East) - or maybe even in all directions in between those two?"
or if it is always only in one "pure" state (or one state between those two)
"Or is the car always going one "pure" direction, like North or East?"
could the dice 1) show all numbers at the front side at once before measuring it and the measurement destroys this so that it only shows one of those sides then or 2) is there always only one number at the front side, but we talk of superposition because we don't know, which one until measuring it?
"1) Could the widget say North and East at the same time before looking at it? 2) Is the direction of the car always North or East, but we don't know which one until measuring it?"
Hopefully the answers to these questions are clear. The car can drive any way it wants to; the widget just doesn't tell us everything about the state, just like measuring if a quantum bit is 0 or 1 doesn't tell us everything about its state. Driving steadily northeast is just a fundamentally different thing from driving north and driving east and is not at all a probabilistic mixture of them. This is the way standard quantum mechanics thinks of superpositions. 680