I'm running a script to take a list of database tables, check the row count of each table, and append the results of each query to a dictionary. I'm using multiprocessing to speed it up a bit: Manager to create a sharable list and sharable dictionary that the processes can read from and append to, and Process to set up the processes.

```
from multiprocessing import Process, Manager

def main():
    mgr = Manager()
    # Function to get the list of tables
    table_list = mgr.list(get_table_list())

    counts = mgr.dict()
    for table in table_list:
        # get_table_count runs a 'SELECT COUNT(*) FROM <table>' and appends
        # the result to the counts dict
        p = Process(target=select_star, args=(table, counts, 'prod'))
        p.start()
        p.join()

```

My question is, how does Process manage the number of open processes? I'm accustomed to using Pool() with the apply\_async() method, where you specify the number of processes when instantiating the Pool.

Any additional commentary on if this is the best method to use for this application would also be appreciated.

--

Thanks to dano for helping implement the two solutions below:

1 - using Pool.apply\_async as I've been accustomed to:

```
from multiprocessing import Process, Manager, cpu_count
def main():
    mgr = Manager()

    table_list = get_table_list()
    pool = Pool(cpu_count() / 2)

    prod_counts = mgr.dict()

    for table in table_list:
        pool.apply_async(get_table_count, args=(table, prod_counts, 'prod'))
    pool.close()
    pool.join()

```

2 - using Pool.map() with itertools.partial()

```
from multiprocessing import Process, Manager, cpu_count
def main():
    mgr = Manager()

    table_list = get_table_list()
    pool = Pool(cpu_count() / 2)

    prod_counts = mgr.dict()
    func = partial(get_table_count, result_dict=prod_counts, env='prod')
    pool.map(func, table_list)

    print prod_counts

```