I am working on a proof of concept for an in-memory database. I do know about stuff like redis and love things like intersecting sets but that is not enough. I want to find out how to implement a service which is like the parstream database or the drawntoscale solutions. 
And therefore I like to have a daemon which keeps stuff in the memory.
I did a prototype with pyro which does what it should: generate a random bitmap index and accept an bitmap mask as input from a pyro client and return all primary keys which match the bitmap mask.
But iterating over the index is too slow by some magnitudes which is obvious since it is python. I would like to use an C extension which does the index scan. Therefore the bitmap index shoud be kept in memory inside the c extension.
The very next step would be to keep the same data in the gpu ram iterate over inside my gpu.
Having no experience writing c extensions and pyOpenCL my question is: can I keep data inside my C Extension and use python ( pyro daeomon or maybe even flask/tornado ) just for receiving and returning the queries without copying the whole bitmap index all the time?
Regards
Christian
Yes, C extensions can have internal structures allocated, but be sure to destroy your data properly when needed. 
You can also write your code as  a C/C++ DLL and use the ctypes python extension to access C functions directly and pass them C variables (ints, char *, etc.).
python c extension / opencl daemon