Will Time Machine back up new data and bypass OS back up? Since Time Machine will back up new data as a higher priority, what if the Mac's hard drive is 800MB full, and the back up drive is only 500GB?

Since the OS system files are the "old data", will Time Machine back up the newer and newer data and therefore "forget about" the older data including the OS X system files, and that a "restore", if it is ever needed, will not be able to restore the Mac into a bootable condition?
First, I must say if you are running Lion you shouldn't worry about this, as all Lion machines have a "Restore Partition". That is they save all the system files required to restore the system on a special "section" of the disk unaccessible under normal conditions. If you ever do want to access this Restore Disk, simply hold down the option key while your Mac is booting up, and you should see be able to select it.

Time Machine, as far as I can tell, DOES backup system files. It full copy of your drive. Nonetheless, time machine backups are not bootable. You can save space by excluding system files, or any other big files you might not want/need to back up. Here are some articles that might help you with that: Time Machine tips and troubleshooting, Guide To Excluding Data from Time Machine Backups.

If you don't, time machine will leave them, no matter how old the data gets, and it would display an error/warning if you run out of space rather than deleting them. Time machine makes sure to always have at least a single full copy of your system. 

Regarding space, you can backup a larger drive with a smaller one, as long as your Time Machine Drive can hold all the space you are using on your larger drive. So if you have 800GB in your system, but have only used 200GB, you could back it up with a 500GB drive, though it is not recommended, because if one day you do fill the drive Time Machine wouldn't be able to have a single full backup. Nonetheless, if you keep your actual disk usage below your Time Machine Disk capacity, you should be fine, with at least one full backup. 

My final recommendation is you check those articles and save space for your backup, excluding the system files (Time Machine backups aren't bootable anyways. You'd need an installation disk or a Recovery Drive.) and any other large, unimportant files. Finally, keep an eye on your backup, and if you find out you do need more space I really recommend buying a larger hard drive! Time Machine can come up pretty handy in times of despair!