Question: Sounds like a silly question, but...

I'm adding the Facebook Javascript SDK to a site that appears to be in HTML5, it has a   

`<!doctype html>` and a `<meta http-equiv="Content-Type" content="text/html; charset=utf-8">` so I'm assuming this was the original authors intent. 

However, looking at the Google Analytics for the site, there might be quite a few people using IE8 to visit the site.

We don't want to use iFrames because the XFBML and HTML5 versions are more versatile.

So this is why I am asking if should I use HTML5 or XFBML for the page?

Answer: 
If your page is html5 then some browsers won't be supported, if you want to support all browsers don't go with html5, it's that simple.

With that said, I don't think it makes much difference which option you'll go with, and that's because in both cases you are loading the same facebook javascript sdk which then renders the plugin where the placeholder is placed.
The only difference between the two versions is how you define the plugin options in that placeholder, in html5 you're using *data-* attributes, and in the xfbml version you just define the facebook namespace and use custom attributes.
But both versions should work for all the browsers that are supported by facebook, the most that you'll get is invalid html in some browsers if you take the html5 version, but it should still work.
You should use XFBML as I see you get more IE users on your Google Analytics service, so I assume it is hard to ignore all those users .