title: Crawling these specific sites and getting all internal links for Scraping
body: <p>Ok, so this is my third question on the same subject. I'm doing a project in which I will be analysing certain kinds of usages and word contexts in the Bengali language. I'm crawling and scraping multiple sites depending on the subject due to the lack of corpus availability the way I need.</p>
  <p>I have written a code for crawling and it has worked almost flawlessly for more than 20-25 different Bengali sites/ blogs/ news portals so far. But to get some data that deals with literary and story/fiction-like language, I would need to scrape some of such websites. I don't understand the exact issue as to why the code is not able to scrape all the &quot;internal&quot; links from the following five websites, which I will mention next. By <strong>internal</strong> I mean only the webpages which are of the website domain, within the website, not outside links.</p>
  <p><strong>Libraries used :</strong></p>
  <pre><code>import requests
  import urllib.parse
  from urllib.parse import urlparse, urljoin
  from bs4 import BeautifulSoup
  import time
  import random
  </code></pre>
  <p><strong>My Code for Crawling:</strong> (I have done a lot to handle errors and not lose data; made comments)</p>
  <pre><code>def crawl(url, max_urls=30):
      &quot;&quot;&quot;
      Crawls a web page and extracts all links. Iterative function
      You'll find all links in `internal_urls` global set variable.
      params:
          max_urls (int): number of max urls to crawl, default is 30.
      &quot;&quot;&quot;
      global total_urls_visited #Global crawled urls counter, initialized to zero
      global stack #Since it's an iterative function I maintain a stack
      global crawled #Track of all the links already crawled
      global stackLen #Length of Stack initialized to zero
  
      session = requests.Session() # I used Session because multiple links are being
                                                     # requested this kinda increased the speed
  
      #Certain website would not allow direct access without header.
      #This one seemed to work fine, with all.
      user_agent = &quot;Mozilla/5.0 (X11; CrOS x86_64 12871.102.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.141 Safari/537.36&quot;
  
      urlCur=url #Stores the current URL
  
      lastCount=0 #Error management, somehow some useless links kept returning
                          #links like &quot;.../%252525...&quot;&quot; something in loop, 
                          # I don't know how to avoid that. 
                          #So I did this: if the last link being crawled too many times, just return
      while True:
          sesCheck=0 #Counts number of times Session will be called again
  
          print(&quot;\nCrawled Count: \t\t&quot;, total_urls_visited) 
          #Just prints the number of sites already crawled,
           # helps to keep track of what's going on, it's time-taking
          
          total_urls_visited += 1
          if stackLen == 1: #Here, that issue of repeated useless links
              lastCount +=1
          req = ''
  
          #Multiple network failure checks, connection can be bad here :(,
          # I have lost too much data on this
          checkError1 = 0
          checkError2 = 0
          checkError3 = 0
          checkError4 = 0
  
          while req == '':
              try:
                  if (stackLen &gt; 0) and (checkError4 &gt; 15):
                      print(&quot;\n\nRepeated Error, endlessly. Abort.\n\n&quot;)
                      checkError4=0
                      return 0 #Value that tells that process couldn't be completed fully
  
                  if (stackLen &gt; 0) and (checkError3 &gt; 10):
                      print(&quot;\n\nRepeated Error, sleep two minutes. Then skip.\n\n&quot;)
                      time.sleep(120)
                      #Certain sites won't let me request anymore midway,
                      #something about max limit
  
                      random.shuffle(stack)
                       # I always shuffled the stack before popping,
                       #since no bias towards what link will show up
                       # Also, no intentional starvation
  
                      urlCur=stack.pop()
  
                      session = requests.Session() 
                      # I don't if I should do this, but to not overload the server
                      #(I don't know if that's how it works)
                      # I request a new Session
  
                      stackLen-=1
                      checkError3=0
                  
                  if (stackLen &gt; 0) and (checkError2 &gt; 8):
                      print(&quot;\n\nRepeated Error, sleep for a minute. Then skip.\n\n&quot;)
                      time.sleep(60) #Same reason
                      random.shuffle(stack)
                      urlCur=stack.pop()
                      session = requests.Session() #Should I be doing this?
                      stackLen-=1
                      checkError2=0
  
                  if (stackLen &gt; 0) and (checkError1 &gt; 3):
                      print(&quot;\n\nRepeated Error. Sleep for a minute.\n\n&quot;)
                      time.sleep(60) # It just gives me enough time to sometimes
                                              # switch off-on my modem,
                                              # to fix data issues 
                      session = requests.Session()
                      checkError1=0
                                      
                  startTime = time.time() #Checks how long it will take to make the request
  
                  req = session.get(urlCur, headers={'User-Agent': user_agent})
  
                  endTime = time.time()
                  
                  # If it took more than a minute and this isn't happening often,
                  # request new session (Is it required, I dealing with too many links)
                  if (endTime - startTime &gt; 60) and (sesCheck &lt;= 2):
                      session = requests.Session()
                      sesCheck +=1
  
              except Exception as e: 
                  print(e)
                  checkError1+=1
                  checkError2+=1
                  checkError3+=1
                  checkError4+=1
                  print(&quot;Connection refused :(&quot;)
                  print(&quot;5 seconds break&quot;)
                  time.sleep(5)
                  print(&quot;Let's go&quot;)
                  continue
  
          if stackLen &lt; 5: 
              print(f&quot;[*] Crawling: {urlCur}&quot;) #Prints last few links, in case some error is gonna show up
  
          links = get_all_website_links(urlCur, req) #Gets all internal links
  
          for link in links:
              if link not in (stack + crawled): #If the link has been crawled or is already in stack ignore
                  stack.append(link)
                  stackLen += 1
  
          if (total_urls_visited &gt; max_urls) or (stackLen==0) or (lastCount&gt;5):
              # lastCount will check useless repeated links in the end
              return 1 #Link crawled completely
              
          random.shuffle(stack) #Again no bias or intended starvation
          crawled.append(urlCur) #Hence crawled
  
          print(&quot;Stack Size: \t\t&quot;, stackLen)  #Helps keeping track of what the program's doing, that's it   
          urlCur = stack.pop()
          stackLen -= 1
  </code></pre>
  <p>This is the code for getting all internal links, used above:</p>
  <pre><code>def get_all_website_links(url, req):
      global internal_urls #Set of All Internal URLS that have collected 
      internal_urls.add(url)
      try:
          urls = set() #All internal links under the given link
          article = req.content #Get's the content
          soup = BeautifulSoup(article, &quot;lxml&quot;)
          domain_name = urlparse(url).netloc #Get the domain
          for a_tag in soup.findAll(&quot;a&quot;): #Looks for all a tags
              href = a_tag.attrs.get(&quot;href&quot;)
              if href == &quot;&quot; or href is None:
                  # href empty tag
                  continue
  
              #Avoids useless links, I keep changing these specific to sites ---
              if &quot;/tag/&quot; in href:
                  continue
              if &quot;/tags/&quot; in href:
                  continue
              if &quot;/wp-content/&quot; in href:
                  continue
              if &quot;/wp-contents/&quot; in href:
                  continue
              if &quot;/wp-admin/&quot; in href:
                  continue
              if &quot;/amp/&quot; in href:
                  continue
              if href.endswith((&quot;.png&quot;, &quot;.jpg&quot;, &quot;.css&quot;, &quot;.gif&quot;, &quot;.pdf&quot;, &quot;.ico&quot;, &quot;.feed&quot;, &quot;.json&quot;, &quot;.js&quot;, &quot;.svg&quot;, &quot;.php&quot;)):
                  continue
              if href.endswith((&quot;.png/&quot;, &quot;.jpg/&quot;, &quot;.css/&quot;, &quot;.gif/&quot;, &quot;.pdf/&quot;, &quot;.ico/&quot;, &quot;.feed/&quot;, &quot;.json/&quot;, &quot;.js/&quot;, &quot;.svg/&quot;, &quot;.php/&quot;)):
                  continue
             
              # join the URL if it's relative (not absolute link)
              href = urljoin(url, href)
              parsed_href = urlparse(href)
              # remove URL GET parameters, URL fragments, etc.
              href = parsed_href.scheme + &quot;://&quot; + parsed_href.netloc + parsed_href.path
              #If the joined link doesn't start with http skip
              if href.startswith(&quot;http&quot;) == False:
                  continue
              if not is_valid(href):
                  # not a valid URL
                  continue
              if href in internal_urls:
                  # already in the set
                  continue
              if domain_name not in href:
                  # external link
                  continue
              #Hence Internal Link, since they are sets, there's no repetition
              urls.add(href)
              internal_urls.add(href)
          return urls
      except Exception as e: 
          print(e)
          print(&quot;None returned&quot;)
          return set(stack) #Return the Stack itself if the link were problematic
  </code></pre>
  <p>The code to check if the link were valid, used above was this,</p>
  <pre><code>def is_valid(url):
      parsed = urlparse(url)
      return bool(parsed.netloc) and bool(parsed.scheme)
  </code></pre>
  <hr />
  <p>So, this code works absolutely fine for most domains, but somehow for these, it doesn't seem to get all links or any links. I used <code>max_urls = 5000</code>. Crawling and then later scraping these websites will be important for me to get an unbiased corpus for my analysis.</p>
  <p>So, the links are: (I printed the numbers to keep track)</p>
  <ol>
  <li><a href="https://www.kaliokalam.com/" rel="nofollow noreferrer">https://www.kaliokalam.com/</a></li>
  </ol>
  <pre><code>www_kaliokalam_com_.txt
  [+] Total Internal links: 79
  [+] Total URLs: 80
  [+] Total crawled URLs:79
  </code></pre>
  <p>And this way fewer than the number of links, the site seems to have. Even manually seeing it would suffice.</p>
  <ol start="2">
  <li><a href="http://www.rupalialo.com/" rel="nofollow noreferrer">http://www.rupalialo.com/</a></li>
  </ol>
  <pre><code>www_rupalialo_com_.txt
  [+] Total Internal links: 39
  [+] Total URLs: 45
  [+] Total crawled URLs:39
  </code></pre>
  <p>Same issue.</p>
  <ol start="3">
  <li><a href="https://www.maadhukari.com/" rel="nofollow noreferrer">https://www.maadhukari.com/</a></li>
  </ol>
  <pre><code>www_maadhukari_com_.txt
  [+] Total Internal links: 31
  [+] Total URLs: 75
  [+] Total crawled URLs:31
  </code></pre>
  <p>Yet again. Definitely way more hyperlinks within the site.</p>
  <ol start="4">
  <li><a href="https://www.tatkhanik.com/" rel="nofollow noreferrer">https://www.tatkhanik.com/</a></li>
  </ol>
  <pre><code>www_tatkhanik_com_.txt
  [+] Total Internal links: 25
  [+] Total URLs: 49
  [+] Total crawled URLs:25
  </code></pre>
  <ol start="5">
  <li><a href="https://www.sananda.in/" rel="nofollow noreferrer">https://www.sananda.in/</a></li>
  </ol>
  <p>This last one is more problematic and perhaps has a different problem. It doesn't give any links at all.</p>
  <pre><code>www_sananda_in_.txt
  [+] Total Internal links: 1
  [+] Total External links: 0
  [+] Total URLs: 1
  [+] Total crawled URLs:1
  </code></pre>
  <p>I would like to know if some changes in my code would help me crawl each of these sites. Why doesn't it work for them like all the other sites (many! many!) it worked with. I intend to get all the in-site links with articles/ story-like contents.</p>