title: fs.readFile is very slow, am I making too many request?
body: <p><em>node.js beginner here:</em></p>
  <p>A node.js applications scrapes an array of links (linkArray) from a list of ~30 urls.</p>
  <p>Each domain/url has a corresponding (name).json file that is used to check whether the scraped links are new or not.</p>
  <p>All pages are fetched, links are scraped into arrays, and then passed to:</p>
  <pre><code>function checkLinks(linkArray, name){
      console.log(name, &quot;checkLinks&quot;);
      fs.readFile('json/'+name+'.json', 'utf8', function readFileCallback(err, data){
          if(err &amp;&amp; err.errno != -4058) throw err;
          if(err &amp;&amp; err.errno == -4058){
              console.log(name+'.json', &quot; is NEW .json&quot;);
              compareAndAdd(linkArray, {linkArray: []}.linkArray, name);
          }
          else{
              //file EXISTS
              compareAndAdd(linkArray, JSON.parse(data).linkArray, name);
          }
      });
  }
  </code></pre>
  <p>compareAndAdd() reads:</p>
  <pre><code>function compareAndAdd(arrNew, arrOld, name){   
      console.log(name, &quot;compareAndAdd()&quot;);
      if(!arrOld) var arrOld = [];
      if(!arrNew) var arrNew = [];
      
      //compare and remove dups
      function hasDup(value) {
          for (var i = 0; i &lt; arrOld.length; i++)     
              if(value.href == arrOld[i].href)
                  if(value.text.length &lt;= arrOld[i].text.length) return false;
              arrOld.push(value);
              return true;
      }
      var rArr = arrNew.filter(hasDup);
  
      //update existing array;
      if(rArr.length &gt; 0){
          fs.writeFile('json/'+name+'.json', JSON.stringify({linkArray: arrOld}), function (err) {
              if (err) return console.log(err);
              console.log(&quot;  &quot;+name+'.json UPDATED');
          });    
      }
      else console.log(&quot;  &quot;+name, &quot;no changes, nothing to update&quot;);
      return;
  }
  </code></pre>
  <p>checkLinks() is where the program hangs, it's <em>unbelievably</em> slow. I understand that fs.readFile is being hit multiple times a second, but imo less than 30 hits seems pretty trivial: assuming this is a function meant to be used to serve data to (potentially) millions of users. Am I expecting too much from fs.readFile, or (more likely) is there another component (like writeFile, or something else entirely) that's locking everything up.</p>
  <p><em>supplemental</em>:</p>
  <p>using write/readFileSync creates a lot of problems: this program is inherently async because it begins with request to external websites with largely varied response times and read/write would frequently collide. the functions above insure that writing for a given file only happens after it's been read. (though it is very slow)</p>
  <p>Also, this program does not exit on its own, and I do not know why.</p>
  <h2>edit</h2>
  <p>I've reworked the program to read first then write synchronously last and the process is down to ~12 seconds. Apparently fs.readFile was getting hung when called multiple times. I don't understand when/how to use asynchronous fs, if multiple calls hangs the function.</p>