Question: I am in the process of writing a message queue system.

My question is... Is it better to do this queue with files or in a database?

If I were to choose the database, it needs to check for new jobs every second and that just seems a bit of an overhead to me?

If it's files I guess you just constantly monitor the folder and execute based on that?

BR,

Answer: 
what are your criteria for deciding "Better"? performance? scalability? reliability? cost? 

The basic tradeoffs are:

Database - transactional support, integrity guarantees (via constraints), richer query support, *probably* will scale better (depending on implementation)

filesystem - cheaper, simpler, less moving parts (at least initially)

You may want to look at existing solutions first - MSMQ, SQL Server broker, open source libraries such as Rhino queues (<http://ayende.com/Blog/archive/2008/08/01/Rhino-Queues.aspx>)
Are you writing a message queue system because you want to write a message queue system; or is this just part of solving a bigger problem?

I would suggest not writing a new message queue system. You can use MSMQ or something else off the shelf.