How can I capture all of the python log records generated during the execution of a series of Celery tasks?
It sounds like some kind of 'watcher' would be ideal. If you can watch and consume the logs as a stream you could slurp the results as they come in. Since the watcher would be running seperately and therefore have no dependencies with respect to what it is watching I believe this would satisfy your requirements for a non-invasive solution.
I want to convert my homegrown task queue system into a Celery-based task queue, but one feature I currently have is causing me some distress.
Right now, my task queue operates very coarsely; I run the job (which generates data and uploads it to another server), collect the logging using a variant on Nose's log capture library, and then I store the logging for the task as a detailed result record in the application database.
I would like to break this down as three tasks:

collect data
upload data
report results (including all logging from the preceding two tasks)

The real kicker here is the logging collection. Right now, using the log capture, I have a series of log records for each log call made during the data generation and upload process. These are required for diagnostic purposes.  Given that the tasks are not even guaranteed to run in the same process, it's not clear how I would accomplish this in a Celery task queue.
My ideal solution to this problem will be a trivial and ideally minimally invasive method of capturing all logging during the predecessor tasks (1, 2) and making it available to the reporter task (3)
Am I best off remaining fairly coarse-grained with my task definition, and putting all of this work in one task? or is there a way to pass the existing captured logging around in order to collect it at the end?