As a first step, disabling fingerprinting seems a good approach. Users should be aware that when they explicitly do so (non-default behaviour) that no automatic caching occurs.
I’m a bit confused about your annotation suggestion. So if I understand correctly, the current approach works as follows: every map (or filter) operation is hashed based on the dill dump of the whole function. You are suggesting to not hash the whole function, but to allow the user to assign a custom identifier to the mapping function. Instead of dill-dumping the whole function, you’d then just use the identifier in the fingerprint. Is that correct?
That would work of course, but it allows a lot room for easy errors. What I mean is that that solution is a lot less robust than the original implementation. For instance, if sent2vec_model is different between two runs (e.g. a different sent2vec model), then that will not be noticed because the fingerprint is the same (same ID of the function). That is of course expected behaviour if you think about it, but as I said it might lead to quick mistakes by users who just want to be able to use unpickable objects in their code but don’t realise what the consequences may be.
I prefer disabling fingerprinting all together and manually saving/loading the dataset. That way I always know what is going on.