Does radar altimeter work over the ocean? I'm wondering if radio waves are reflected by the sea. If so then the usual radar altimeter would work. If not, then is there any other fallback besides barometric altimeter?

According to the wiki article, for some reason radar altimeter uses radio waves (instead of microwaves like usual radar). This is why I put in the radio tag.
Yes.  In the Navy, our primary source of altitude indication under 5,000ft is the radar altimeter.  The AGL over the ocean conveniently also happens to be the MSL as well.