Slower Execution with Python multiprocessing
I may be in a little over my head here, but I am working on a little bioinformatics project in python. I am trying to parallelism a program that analyzes a large dictionary of sets of strings (~2-3GB in RAM). I find that the multiprocessing version is faster when I have smaller dictionaries but is of little benefit and mostly slower with the large ones. My first theory was that running out of memory just slowed everything and the bottleneck was from swapping into virtual memory. However, I ran the program on a cluster with 4*48GB of RAM and the same slowdown occurred. My second theory is that access to certain data was being locked. If one thread is trying to access a reference currently being accessed in another thread, will that thread have to wait? I have tried creating copies of the dictionaries I want to manipulate, but that seems terribly inefficient.  What else could be causing my problems?


My multiprocessing method is below:

    
	def worker(seqDict, oQueue):
		 #do stuff with the given partial dictionary
         oQueue.put(seqDict)
	oQueue = multiprocessing.Queue()
	chunksize = int(math.ceil(len(sdict)/4)) # 4 cores
	inDict = {}
	i=0
	dicts = list() 
	for key in sdict.keys():
		i+=1
		if len(sdict[key]) > 0:
			inDict[key] = sdict[key]
		if i%chunksize==0 or i==len(sdict.keys()):
			print(str(len(inDict.keys())) + ", size")
			dicts.append(copy(inDict))
			inDict.clear()
	 
	for pdict in dicts:
		p =multiprocessing.Process(target = worker,args = (pdict, oQueue)) 
		p.start()
	finalDict = {}

	for i in range(nprocs):
		finalDict.update(oQueue.get())
	return finalDict
Slower Execution with Python multiprocessing
===

I may be in a little over my head here, but I am working on a little bioinformatics project in python. I am trying to parallelism a program that analyzes a large dictionary of sets of strings (~2-3GB in RAM). I find that the multiprocessing version is faster when I have smaller dictionaries but is of little benefit and mostly slower with the large ones. My first theory was that running out of memory just slowed everything and the bottleneck was from swapping into virtual memory. However, I ran the program on a cluster with 4*48GB of RAM and the same slowdown occurred. My second theory is that access to certain data was being locked. If one thread is trying to access a reference currently being accessed in another thread, will that thread have to wait? I have tried creating copies of the dictionaries I want to manipulate, but that seems terribly inefficient.  What else could be causing my problems?


My multiprocessing method is below:

    
	def worker(seqDict, oQueue):
		 #do stuff with the given partial dictionary
         oQueue.put(seqDict)
	oQueue = multiprocessing.Queue()
	chunksize = int(math.ceil(len(sdict)/4)) # 4 cores
	inDict = {}
	i=0
	dicts = list() 
	for key in sdict.keys():
		i+=1
		if len(sdict[key]) > 0:
			inDict[key] = sdict[key]
		if i%chunksize==0 or i==len(sdict.keys()):
			print(str(len(inDict.keys())) + ", size")
			dicts.append(copy(inDict))
			inDict.clear()
	 
	for pdict in dicts:
		p =multiprocessing.Process(target = worker,args = (pdict, oQueue)) 
		p.start()
	finalDict = {}

	for i in range(nprocs):
		finalDict.update(oQueue.get())
	return finalDict