How to estimate proper batch size number when processing DB records.
So let's say I have a processor that takes records from one table transforms and place them into another table. I'm trying to figure how many records I can load into memory and process at once.  

Obviously that would depend on a lot of factors: amount of data in the records, any BLOBS?, number of columns, database type, drivers, frameworks, how much memory available on a box, are there any other memory consuming process running in the same environment, etc.

Form tests I can see that it is able to process 10000 records at once but fails (with OOM) to query 100000. 

I guess I'm looking for a good strategy to figure out proper batch size number. 

Or should I just go with a very conservative and low batch size number like 100. And don't worry about any query overhead. 

Thanks.
How to estimate proper batch size number when processing DB records.
===

So let's say I have a processor that takes records from one table transforms and place them into another table. I'm trying to figure how many records I can load into memory and process at once.  

Obviously that would depend on a lot of factors: amount of data in the records, any BLOBS?, number of columns, database type, drivers, frameworks, how much memory available on a box, are there any other memory consuming process running in the same environment, etc.

Form tests I can see that it is able to process 10000 records at once but fails (with OOM) to query 100000. 

I guess I'm looking for a good strategy to figure out proper batch size number. 

Or should I just go with a very conservative and low batch size number like 100. And don't worry about any query overhead. 

Thanks.