The answer and discussions above don't really go beyond school physics course territory, and that does not provide the answer to my question at all.
I finally did find data that helped me formulate the answer I'm happy with in this youtube video. My explanation fits all the data I have. Here it is.
The energy is not transferred at all by the electrons moving around, the energy flow is created when there are both electric and magnetic fields present between two points, and it is the energy flow that we really mean when we talk about electricity. This is what's stipulated in that video that is different and contrary to the school physics explanations of electricity.
The energy flow needs both electric and magnetic fields to be present at the same time. The electric field by itself is not sufficient. E.g. if you take a well-charged AA battery - there is an electric field constantly present between its cathode and anode, but without a magnetic field to accompany it - the energy is not flowing, so the battery does not discharge and no work is done.
The magnetic field is created by electric charges moving within an electric field. The charges need to be moving to create a magnetic field, so once we create conditions where charges can move - we allow for energy to start flowing. I imagine that "charges" here can really be any kind of objects with a positive or negative electric charge, but the most common charged objects we use are electrons in a material that allows them to freely move - metal conductors.
When we apply an electric field to the conductor - the electrons in this conductor start moving. They start moving - and this creates a magnetic field around the conductor, and this magnetic field together with the electric field that was there already allows for power to flow from one point to another.
So in fact - the moving electrons themselves are not transferring any power whatsoever, the only reason we need them moving is to create a magnetic field. 395
Moving electrons are a means to an end, but not a vehicle for power transfer in itself.
Btw, this also nicely explains the alternating current - if electrons are not really transferring the power and they only have to move somehow - it doesn't matter in what direction they are actually moving, and they can just jiggle around back-n-forth in an AC circuit. They still move, the field is created, and power is flowing.
Now, as for the original question - why increasing voltage with constant power decreases current and decreases the side effects of the current on the conductor?
I'm making a further leap here to accommodate for that. Extending on the idea that the two fields are necessary and sufficient for transferring power - I think the fields don't have to be of the same intensity to achieve that.
Generally, the stronger the fields - the more power will be transferred. The voltage is directly measuring the intensity of the electrical field between two points. The current is measuring the amount and speed of moving charges, so it indirectly measures the magnetic field intensity.
Movement of the charges creates negative side-effects we want to minimise (energy loss through heat and conductors melting). So we have to minimise the movement of charges and thus - the magnetic field. We need some magnetic field still, but don't want much of it.
And to compensate for that - we at the same time increase the electric field. The huge electric field and small magnetic field will transfer the same amount of power overall as the average-size magnetic and electric fields together. But creating a small magnetic field will create fewer unwanted side effects.
This is why a cable at high voltage will heat up less than at low voltage. There is no relation to drift velocity at all in here. We will increase the voltage and decrease current by placing resistance on the way, and that will transfer all the power we need and not melt the cable. 395