Question: * What's the limitation?
* Is there a specific volume of data each can handle regardless of disk space?
* When to use what assuming licensing is not a problem?

Answer: 
This is a very nuanced question that really cannot be easily answered, as each situation can provide many pluses and minuses. Also, MySQL being owned by Oracle now and several branches off of the main functionality means that MySQL != MySQL anymore.

If you are looking for really really big data sets, then you will like have to break with the RDBMS sets and start to look at things like MapReduce and other large data set processing technologies.

I have personally worked with all three over the past decade or so from the application perspective. They all have their advantages, like MSSQL working will with the other Microsoft technologies like LINQ where as MySQL having a large open community support and Oracle being the workhorse of the commercial sector with lots of ability to embed application logic right into the database.

Again, it really depends on the application, the situation, the skills of the people who will maintain it after it is developed, commercial considerations, hardware and platform considerations, etc etc etc.
Perhaps one variable which people overlook in these cases is the availability of expert support. Okay, so currently there's an oversupply of people who can help you with db issues, efficiency issues, disaster recovery etc. However this may not be always the case in the future, and it's the applications you use it for that may be the defining issue. Are there people in your organization who have experience in one or more of the relevant databases? (as it happens I believe that someone's who's become proficient in say Oracle, can become fairly competent in Sql\*Server or Mysql in a fairly short space of time) You state it's going to be used for your financial systems - perhaps you really need input from a consultant who's worked on implementing and/or supporting financial systems - for example I understand that Sybase is popular in City type firms. Or perhaps there's an off-the-shelf package that utilises a preferred database? Try and define exactly what your system(s) needs to do first.