title: Managing Scheduled Tasks in a Clustered Environment
body: <p>I have a small cluster of servers that are used for handling a fair amount of web traffic. It's a pretty standard setup; 3 web servers, load balancer, database server, warehouse server. </p>
  
  <p>The difficulty is that I have a number of different tasks that run on a scheduled basis, and those tasks vary drastically with respect to how often they run. The items are:</p>
  
  <ol>
  <li>Warehouse jobs. Obviously running on the single warehouse server. Might be a problem in the future, but for now we only have one warehouse. Runs nightly.</li>
  <li>Scheduled reports. Using Quartz.NET, fired in our ASP.NET environment. Runs whenever the users schedule them, typically daily.</li>
  <li>Stats snapshots. Running in ASP.NET via a thread fired up with the app pool, takes a snapshot of the overall system load for future analysis. Runs every 2 minutes.</li>
  <li>System events. We have lots of "stuff" in the system that need to change their status on a relatively high frequency (on any given check, ~5-10 items will need a status update) and broadcast that information out to the connected web clients. Currently set to run in a thread in the app pool as well. Runs every 2 seconds or so.</li>
  </ol>
  
  <p>All this works perfectly in a single-server setup, and even in a multi-server setup I can just turn things off on the non-primary web servers (which is my current solution).</p>
  
  <p>The real question is, what's the best way to set this up so there's a little more redundancy built into the system? Extract the whole thing into windows services on a different box (boxes)? Have the web servers communicate with each other so they can detect a failure on the primary and have a secondary take over?</p>
  
  <p>I'm interested in any thoughts/suggestions!</p>