I can find a load of information on the reverse, not so much this way around :)
So, the summary is that I want to write some Python code-completion stuff in C++, but I can't figure out the best way of tokenizing the Python code.
Are there any libraries out there that will do this? 
I'm leaning towards calling Python's tokenize.tokenize directly from C++... but whenever I look at calling Python code from C++ I go cross-eyed.
Using regular parser-generators to generate parsers from the grammar is usually complicated with Python (for example due to its significant whitespace and difficult line-continuation rules).
I am not sure about your experience with Python, but my recommendation would be to parse the Python file from Python, and do as much of the processing as possible in Python, then return the result to the C++ code using well-defined data types (such as the stdc++ ones) and using Boost.python for the bindings.
Parsing Python code from C++