I have an interesting problem where we need to document all the URLs that our massive Python project calls. It's not feasible to manually go through the code because it's too large and changes often.
Ideally, what I'd like is a piece of script that given a Python project folder can go through all the files in it and find where requests or urllib modules make calls and list the accompanying URL.
I think what you could do instead is to wrap requests or urllib modules with a wrapper that logs URLs your app is connecting and then just call real urllib or requests modules functions. So whenever you call import requests you actually import your wrapper.
List all urls called by requests/urllib in python project