How to scrape many HTML documents quickly using Python?
One of the idea which I can suggest is taking all the urls in Csv and keep few headings like path,title div,body div, image div as per your requirement and keep adding the particular div(div class=”title”).
Ex:
       PATH       TITLE DIV                IMAGE DIV              BODY DIV 
Similarly, you can give all links in one csv file nd read it through python script so that all data is pulled.
I have a list of about 100 HTML webpages (all have different structures, such as divs, anchors, classes, etc.) and I am trying to scrape the title of each page (where the title is under a certain div and class). To do this, I was using get requests and Beautifulsoup, however, this takes way to long (10 minutes every time I want to do it)!
I used a timer to see what is taking the most time: it's the get requests. Apparently Python (3.7) executes the code one after another, and since each get request takes about 5-6 seconds, it's taking approximately 500-600 seconds to complete the ~100 requests. 
I've searched for ways to make these requests work faster and came across many different solutions. However, a common theme seemed to be that making my requests asynchronous (so all requests start at the same time) will solve the problem (by making it faster).
There were many possible solutions for doing this that I read online including: multithreading, using grequest, using Scrapy, parsing lxml, etc. However, I'm new to programming and am not skilled enough to learn and experiment with each way (in fact, I tried following the answers to similar questions on SO, but wasn't successful), so I am unsure what is the best route for me to take. 
I don't need anything fancy; all I want to do is extract the titles from the HTML documents as text and then print them out. I don't need to download any CSS files, images, media, etc. Also, I'm hoping to keep the code as simple/bare as possible. How can I do this as fast as possible in Python? I'd appreciate it if someone could suggest the best path to take (i.e. using Scrapy), and give a short explanation of what I must do using that tool to get the results I'm hoping for. You don't have to write out the whole code for me. Thanks!