title: Building a basic PHP web-crawler
body: <p>Am fairly new to PHP but hear me out please. </p>
  
  <p>Would like to build a web-crawler that basically traverses all the links on a certain site -> gets specific content from it and returns link+specific info from each page.</p>
  
  <p>Got the link traverse function from a Youtube tutorial: <a href="https://www.youtube.com/watch?v=KBemN_bTnHU" rel="nofollow">https://www.youtube.com/watch?v=KBemN_bTnHU</a>, but can't seem to make the final part work though, when trying to follow the links nothing is outputed(noob alert).</p>
  
  <p>Here is the function to get the links out of a website(that doesn't fully work):</p>
  
  <pre><code>&lt;?php
  
  $to_crawl = "http://reteteculinare.ro";
  $c = array();
  
  function get_Links($to_crawl){
      global $c;
      $input = @file_get_contents($to_crawl);
      $base_url = parse_url($to_crawl, PHP_URL_HOST);
  
      $regexp = '&lt;a\s[^&gt;]*href=(\"??)([^\" &gt;]*?)\\1[^&gt;]*&gt;(.*)&lt;\/a&gt;';
      preg_match_all("/$regexp/siU", $input, $matches);
  
      $l = $matches[2];
  
      foreach ($l as $link) {
  
          if(strpos($link, "#")) {
              $link = substr($link, 0, strpos($link, "#"));
          }
  
          if(substr($link, 0, 1) == "."){
              $link = substr($link, 1);
          }
  
          if(substr($link, 0, 7) == "http://"){
              $link = $link;
          } else if (substr($link, 0, 8) == "https://"){
              $link = $link;
          } else if (substr($link, 0, 4) == "www."){
              $link = substr($link, 4);
          } else if (substr($link, 0, 6) == "//wwww."){
              $link = substr($link, 6);
          } else if (substr($link, 0, 2) == "//"){
              $link = substr($link, 2);
          } else if (substr($link, 0, 1) == "#"){
              $link = $to_crawl;
          } else if (substr($link, 0, 7) == "mailto:"){
              $link = "[".$link."]";
          } else {
              if(substr($link, 0, 1) != "/") {
                  $link = $base_url."/".$link;
              } else {
                  $link = $base_url.$link;
              }
          }
  
          if(substr($link, 0, 4) == "www."){
                  $link = substr($link, 4);
          } 
  
          if(substr($link, 0, 7) != "http://" &amp;&amp; substr($link, 0, 8) != "https://" &amp;&amp; substr($link, 0, 1) != "[") {
              $link = "http://".$link;
          } else {
              $link = "https://".$link;
          }
  
          if (!in_array($link, $c)) {
              array_push($c, $link);
          }
  
      }
  }
  get_links($to_crawl);
  
  foreach ($c as $page) {
      get_links($page);
  }
  
  foreach ($c as $page) {
      echo $page."&lt;br /&gt;";
  }
  
  ?&gt; 
  </code></pre>
  
  <p>Code works until it tries to follow the links. Any clues?:D In the videos it seems to be working fine for the guy....</p>
  
  <p>Finally here is my function for getting certain information from a page and saving it into an array:</p>
  
  <pre><code>&lt;?php
  include('simple_html_dom.php');
  
  header('Content-type: text/plain');
  
  $html = new simple_html_dom();
  
  $page = ('http://www.reteteculinare.ro/carte_de_bucate/dulciuri/gauffre-de-liege-1687/');
  
  $base_url = parse_url($page, PHP_URL_HOST);
  
  function getRecipe($page) {
      global $recipe, $page, $base_url;
  
      $html = new simple_html_dom();
  
      $html-&gt;load_file($page);
  
      $reteta = $html-&gt;getElementById('hrecipe');
  
      $r_title = $reteta-&gt;children(4)-&gt;outertext;
      $r_title = strip_tags($r_title);
      $r_title = trim($r_title);
  
      $r_poza = $reteta-&gt;getElementById('.div_photo_reteta')-&gt;children(0)-&gt;src;
      $r_poza = $base_url.$r_poza;
  
      $r_ingrediente = $reteta-&gt;getElementById('#ingrediente-lista')-&gt;outertext;
  
      $r_preparare = $reteta-&gt;getElementById('.instructions')-&gt;children(1)-&gt;outertext;
      $r_preparare = strip_tags($r_preparare);
  
      // $r_durata = $reteta-&gt;getElementById('.duration')-&gt;children(0)-&gt;outertext;
      // $r_durata = preg_replace('/\s/', '', $r_durata);
      // $r_durata = strip_tags($r_durata);
  
      $recipe = array(
              "Titlu: " =&gt; $r_title,
              // "Durata: " =&gt; $r_durata,
              "Link Poza: " =&gt; $r_poza,
              "Ingrediente: " =&gt; $r_ingrediente,
              "Preparare: " =&gt; $r_preparare
          );
      echo '&lt;pre&gt;';
      print_r($recipe);
      echo '&lt;/pre&gt;';
  }
  
  getRecipe($html);
  
  ?&gt;
  </code></pre>
  
  <p>This works fine, gets the info that I want into an array - noob method of data-mining I am sure but don't know better:)</p>
  
  <p>Finally, I would like to somehow connect these two functions so that when it traverses each link, to get the data from the second function and return an array that contains the link where the data was found+data.</p>
  
  <p>If any of you can throw a helping hand in my direction I would certainly appreciate it.</p>
  
  <p>Kinda out of my grasp what I am trying to achieve but really want to learn and expand my knowledge.</p>
  
  <p>Cheers!</p>