I have a python pandas dataframe on my local machine, and have access to a remote mongodb server that has additional data that I can query via pymongo.
If my local dataframe is large, say 40k rows with 3 columns in each row, what's the most efficient way to check for the intersection of my local dataframe's features and a remote collection containing millions of documents?
I'm looking for general advice here.  I thought I could just take a distinct list of values from each of the 3 features, and use each of these in an $or find statement, but if I have 90k distinct values for one of the 3 features it seems like a bad idea.
So any opinion would be very welcome.  I don't have access to insert my local dataframe into the remote server, I only have select/find access.
thanks very much!
As you already explained that you won't be able to insert data. So only thing is possible is first take the unique values to a list.df['column_name'].unique(). Then you can use the $in operator in .find() method and pass your list as a parameter. If it takes time or it is too much. Then break your list in equal chunks, I mean list of list [[id1, id2, id3], [id4, id5, id6] ... ] and do a for loop for sub-list in list: db.xyz.find({'key':{'$in': sublist}}, {'_id': 1}) and use the sub list as parameter in $in operator. Then for each iteration if the value exist in the db it will return the _id and we can easily store that in a empty list and append it and we will be able to get all the id's in such cases where the value exist in the collection.
So it's just the way I would do. Not necessarily the best possible.
Efficient Intersection of pandas dataframe with remote mongodb?