What type of media server software do I need to transcode in near real time? Can anyone help me find a software stack for doing near real time video encoding/transcoding?

I want to allow users to upload videos(wmv, avi, mov, flash video, other formats) to my website. As soon as the file is uploaded, I want to encode/transcode it into h.264 (preferably with an API so I can get progress percentage etc) so that I can then play it back via html5 or a flash player.

Anyone have any guidance for me on this?

Oh, and the server platform can be either windows or linux.
FFMPEG is what you're looking for.  You can use it to transcode most anything, it can easily be integrated via STDIO, and there are a handful of library versions of it you can use as well.

I'd recommend hopping over to StackOverflow.com, where this topic has been heavily discussed.