Is their a file system that can read/write mac and win and do time machine? Is their a file system that can read/write mac and win and do time machine? 
I know that OS X can write to NTFS with the correct software installed, but Time Machine usually insists on formatting drives before using them. Is there a compromise?
You can't use Time Machine to back up Windows, if that's what you mean. 

The only way you could do that is if you're hosting the Win installation in a virtual machine like Parallels(Paid) or VirtualBox (free), then of course your entire VM would be backed up to TM along with everything else on your mac.

Still, rolling back to a particular state (or recoveing a particular file) that way would be like finding a needle in a haystack.

I'd suggest you'd be better off keeping a clone of your windows installation and a separate clone/or TM backup of your OS X installation (on a separate disk). 

Not only is this the only practical solution, more importantly its also the safest. Never keep all your eggs in one basket (hard drive) when it comes to backups, especially with Time Machine, which has a habit of getting itself corrupted just when you least want it.