Frequent large, multi-record updates in MongoDB, Lucene, etc.
I am working on the high-level design of a web application with the following characteristics:

 - Millions of records
 - Heavily indexed/searchable by various criteria
 - Variable document schema
 - Regular updates in blocks of 10K - 200K records at a time
 - Data needs to remain highly available during updates
 - Must scale horizontally effectively

Today, this application exists in MySQL and we suffer from a few huge problems, particularly that it is challenging to adapt to flexible schema, and that large bulk updates lock the data for 10 - 15 seconds at a time, which is unacceptable. Some of these things can be tackled by better database design within the context of MySQL, however, I am looking for a better "next generation" solution.

I have never used MongoDB, but its feature set seemed to most closely match what I am looking for, so that was my first area of interest. It has some things I am excited about, such as data sharding, the ability to find-update-return in a single statement, and of course the schema flexibility of NoSQL.

There are two things I am not sure about, though, with MongoDB:

 1. I can't seem to find solid
    information about the concurrency of
    updates with large data sets (see my
    use case above) so I have a hard
    time understanding how it might
    perform.

 2. I do need open text search

That second requirement brought me to Lucene (or possibly to Solr if I kept it external) as a search store. I did read a few cases where Lucene was being used in place of a NoSQL database like MongoDB entirely, which made me wonder if I am over-complicating things by trying to use both in a single app -- perhaps I should just store everything directly in Lucene and run it like that?

Given the requirements above, does it seem like a combination of MongoDB and Lucene would make this work effectively? If not, might it be better to attempt to tackle it entirely in Lucene?

Frequent large, multi-record updates in MongoDB, Lucene, etc.
===

I am working on the high-level design of a web application with the following characteristics:

 - Millions of records
 - Heavily indexed/searchable by various criteria
 - Variable document schema
 - Regular updates in blocks of 10K - 200K records at a time
 - Data needs to remain highly available during updates
 - Must scale horizontally effectively

Today, this application exists in MySQL and we suffer from a few huge problems, particularly that it is challenging to adapt to flexible schema, and that large bulk updates lock the data for 10 - 15 seconds at a time, which is unacceptable. Some of these things can be tackled by better database design within the context of MySQL, however, I am looking for a better "next generation" solution.

I have never used MongoDB, but its feature set seemed to most closely match what I am looking for, so that was my first area of interest. It has some things I am excited about, such as data sharding, the ability to find-update-return in a single statement, and of course the schema flexibility of NoSQL.

There are two things I am not sure about, though, with MongoDB:

 1. I can't seem to find solid
    information about the concurrency of
    updates with large data sets (see my
    use case above) so I have a hard
    time understanding how it might
    perform.

 2. I do need open text search

That second requirement brought me to Lucene (or possibly to Solr if I kept it external) as a search store. I did read a few cases where Lucene was being used in place of a NoSQL database like MongoDB entirely, which made me wonder if I am over-complicating things by trying to use both in a single app -- perhaps I should just store everything directly in Lucene and run it like that?

Given the requirements above, does it seem like a combination of MongoDB and Lucene would make this work effectively? If not, might it be better to attempt to tackle it entirely in Lucene?
