Hydraulic disk brake stops working when hot; back to normal after cooling - is this expected? I have experienced a strange failure of my brake.

I was descending a steep smooth road, using only my front brake (my rear brake was broken; it was an unplanned bail from a route). I tried keeping the speed low, so i could jump off the bike at any moment.

After going this way for maybe 2 minutes, I stopped briefly to let some car pass (so released the brake for a few seconds). As I continued to descend, I pressed the brake, and surprisingly, its handle went all the way to the handlebar without doing any braking! So I immediately jumped off.

I didn't believe at first the bad luck I had, but then I pressed again and again on the brake (not while riding now), and gradually it returned to normal; after 1 minute it would give full braking force again.

So I wonder, is this an expected way a brake would fail while overheating? I heard that it would start to stink really bad before it became dangerous to ride, so I didn't worry about overheating.

The model of my brake is Hayes Nine.
The least you need to do is bleed the brakes to remove trapped air. It's best though if you completelly change the fluids and bleed. You can do that in an LBS, or yourself if you have a bleeding kit and know how to do it.

The reason for the behavior you described is that air is trapped in the system and makes things mushy and unpredictable. The fact that after a couple of lever presses it came back to "normal" is because hydraulic brakes are auto adjusting both on the pads end (when pads are worn) and on the lever end (when air is in the system).

Air bubbles exist in the system either because a proper bleed was not done in the first place or because the fluids absorbed moisture which vapourated and turned into air as soon as you overheated the fluids. Also note that as soon as the fluids absorb moisture (water) then the boiling point falls even more so having this issue is even easier.