Question: I'm looking for a way to check if a server is support xmpp protocol, i.e. xmpp-server is running on this domain.

Maybe special DNS check or something like that?

Answer: 
Following Joe's explanation, I personally would check SRV records and fall back to a quick (short timeout) connect check on port 5269. Then cache the result.
You can try IMtrends: <http://www.process-one.net/en/imtrends/>