Way to limit how much RAM an arbitrary process can take up? Lets say we have a OS X machine running with 4 GB of RAM. Is it possible to set it up so no single process can take up more than say, 3 GB, so no matter how memory hogging a process is, there's always some memory left over to keep the system responsive?
A similar question to this is noted here How Can I Limit the Amount of RAM a Mac OS Application Can Use?

They recommend using the ulimit command, but they were unable to successfully limit the amount to what they desired. I think the problem that they ran into was the fact that the option being set -u or -v was not accounting for the memory paging system, eg you can only allocate X amount of space for you process in real RAM, but we can page you out to virtual memory all day long.  

Note that this may be Dangerous disabling Virtual Memory, it could result in your Mac just locking up, etc when it runs of of RAM.

Try running ulimit with arguments -l, -m, or -v however as noted in the ulimit man page it up to the OS if it will pay attention to these settings or not as noted.


  ulimit: ulimit [-SHacdefilmnpqrstuvx] [limit]
  
  Modify shell resource limits.
  
  Provides control over the resources available to the shell and processes
      it creates, on systems that allow such control.


Can't really find any definite source that confirms if OS X ignores ulimit or not, but after the referenced ulimit fail

If you can get ulimit to work for your needs by itself then give it a go, if not a combination of ulimit and or disabling virtual memory may be able to let you set up that 3 GB limit on select processes. FYI here is a good link for more information onCompletely Disable Virtual Memory in Snow Leopard System 10.6 

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist