@Sayak_Paul  thanks for asking.
By requesting external URLs automatically, my concerns are often, if the robots.txt allows the script to fetch/get that URL (even for the http status codes). Also the robots.txt rules may change over time, so the URL which is today accessible, is maybe on disallow tomorrow inside my data pipeline.
For example, I would like to also test the ../robots.txt of each URL,
depending on the user-agent in the first step, before I send a request:
IF https://findingblanche.files.wordpress.com/robots.txt
  GET https://findingblanche.files.wordpress.com/2013/07/photo4-1.jpg...

to ensure, that my data pipeline doesn’t request excluded URLs over time.
I’ve only found urllib.robotparser.can_fetch(useragent, url) so far, as a pre-condition before sending
HTTP Requests (e.g: request.get(url).response_code) in python.