title: Why Running Multiple Data Scraping Tasks Take More Time Asynchronsly In Comparsion To Individually?
body: <p>I am using puppeteer in Node.JS to run a data scraping task which tries to get data from multiple pages at once. Each page has a Tabbed structure which I need to get the data from 4 tabs. I click a tab and use &quot;await&quot; to get the data from the tab. At the end, I write the obtained data into an excel files. The code works correctly except the time it takes to be finished.
  If I run the code for a individual URL, it takes 7 to 10 seconds to get and write data into excel files for each URL. But if I run the code for 16 URLS in a &quot;for&quot; loop, it takes almost 30 to 33 seconds. I think it should take lower than 12 seconds to finish the job, because it is running asynchronously. During running 16 tasks in a for loop, all the threads in the CPU are occupied 100%. The code is too long, so I just write the code structure here:</p>
  <p>since I did not know how to repeat a task if it fails, I write 2 try-catch structure as they are nested.</p>
  <pre><code>for (i = 0; i &lt; URL.length; i++) {
      try {
          UpdateTable(URL[i], i, 1);
      } catch (e) {
          try {
              UpdateTable(URL[i], i, 2);
          } catch (e) {
              UpdateTable(URL[i], i, 3);
          }
      }
  }
  
  </code></pre>
  <pre><code>async function UpdateTable(URL, ii, Ind) {
      return new Promise((resolve, reject) =&gt; {
          var temp = [];
          temp = (async () =&gt; {
              console.time((&quot;i: &quot; + ii + &quot;: &quot; + Ind).toString());
              console.timeLog(&quot;i: &quot; + ii + &quot;: &quot; + Ind);
              try {
                  const browser = await puppeteer.launch({headless: true});
                  const page = await browser.newPage();
                  await page.goto(URL); //Bourse
  
  
                  //Tab 1:
                  await page.evaluate(() =&gt; {
                      document.getElementsByClassName('torq')[0].click();
                  })
                  await page.waitForSelector(&quot;tr.ev&quot;, {visible: true}).then(() =&gt; {
                  });
  
                  var temp = [];
                  var temp_1col = [];
                  var temp1_1col = [];
                  temp = await page.evaluate(() =&gt; {
                  //
                  //
                  //
                  //
                  //
                  }
                  
                  
                  //Tab 2:
                  await page.evaluate(() =&gt; {
                      document.getElementsByClassName('vio')[0].click();
                  })
                  await page.waitForSelector(&quot;div.CalMon&quot;, {visible: true}).then(() =&gt; {
                  });
  
                  var temp1 = [];
                  temp1 = await page.evaluate(() =&gt; {
                  //
                  //
                  //
                  //
                  //
                  }
                  
                  
                  //Tab 3:
                  await page.evaluate(() =&gt; {
                      document.querySelectorAll('a.re')[0].click();
                  })
                  var tt = await page.evaluate(() =&gt; {
                      return document.getElementById(&quot;d12&quot;).innerText;
                  })
  
                  while (tt === undefined || tt === &quot;&quot;) {
                      tt = await page.evaluate(() =&gt; {
                          return document.getElementById(&quot;d12&quot;).innerText;
                      })
                  }
  
                  var temp2 = [];
                  temp2 = await page.evaluate(() =&gt; {
                  //
                  //
                  //
                  //
                  //
                  }
                  
                  //Tab 4:
                  await page.evaluate(() =&gt; {
                      document.querySelectorAll('a.ye')[0].click();
                  })
                  await page.waitForXPath(&quot;//div[@class='Main']//span//tr[5]//td[2]&quot;, {visible: true}).then(() =&gt; {
                  });
  
                  var temp4 = await page.evaluate(() =&gt; {
                  //
                  //
                  //
                  //
                  //
                  }
                  
                  
                  //Writing to Excel file Using: &quot;exceljs&quot; module:
                  const sheet8 = workbook3.addWorksheet(&quot;Adjusted&quot;);
                  sheet8.views = [{rightToLeft: true}];
                  sheet8.addRows(temp_adj);
                  
                  await browser.close();
                  console.timeEnd((&quot;i: &quot; + ii + &quot;: &quot; + Ind).toString());
                  return temp;
                  } catch (e) {
                      console.log(&quot;Error In: &quot; + URL);
              }
          })();
      });
  }
                              
  </code></pre>
  <p>I measured the time that each loops take to complete:</p>
  <pre><code>i: 0: 1: 0.065ms
  i: 1: 1: 0.020ms
  i: 2: 1: 0.004ms
  i: 3: 1: 0.004ms
  i: 4: 1: 0.028ms
  i: 5: 1: 0.007ms
  i: 6: 1: 0.003ms
  i: 7: 1: 0.018ms
  i: 8: 1: 0.018ms
  i: 9: 1: 0.005ms
  i: 10: 1: 0.003ms
  i: 11: 1: 0.003ms
  i: 12: 1: 0.003ms
  i: 13: 1: 0.003ms
  i: 14: 1: 0.003ms
  i: 15: 1: 0.003ms
  i: 13: 1: 13728.609ms
  i: 0: 1: 14846.398ms
  i: 15: 1: 14857.476ms
  i: 4: 1: 18383.652ms
  i: 6: 1: 18504.466ms
  i: 10: 1: 22255.862ms
  i: 5: 1: 29188.227ms
  i: 7: 1: 29187.398ms
  i: 14: 1: 30093.722ms
  i: 8: 1: 30297.994ms
  i: 2: 1: 30381.256ms
  i: 1: 1: 30426.768ms
  i: 11: 1: 30644.432ms
  i: 12: 1: 30677.942ms
  i: 3: 1: 30689.566ms
  i: 9: 1: 30710.694ms
  
  </code></pre>
  <p>As you see in time log, the URL[13] takes 13728.609ms while if I run the code for this URL, it just takes  8000 ms. Why running 16 tasks together takes more time in comparison to individual task? and why it takes 30710.694ms to run asynchronously while the maximum time a task takes is lower than 10000ms?</p>
  <p>I would be thankful if you could help me about this challenge.</p>