title: c# screen scraping project - webbrowser not changing url
body: <p>I'm working on a little automation project at the min and have hit a brick wall. Firstly i'd like to state the only reason i'm using webbrowser for this component of the project is the site being scraped has obfuscated code and requires a java enabled browser to display the code, i've got another app using webclient which works fine for other test sites but unfortunately can't be used on this target</p>
  
  <p>My problem arises when trying to programatically configure the webbrowser control</p>
  
  <p>First problem i've discovered is if i manually set the url in the controls properties it loads page 1 up and the scraper works for that page. <br>However, I proceeded to clear the url in the properties and set it manually in the Form1_Load method but it returns about:blank as the url despite the fact i've verified the automated parameter being pulled in is fine and should be getting set without issue</p>
  
  <p>Here's what i'm using:</p>
  
  <p>Note: <br>
  collection refers to an XML serialized array of definitions<br>
  definition refers to the active definition for this target,the idea being to configure this for multiple targets</p>
  
  <pre><code>    private void Form1_Load(object sender, EventArgs e)
      {
          PopulateScraperCollection();
          webBrowser1.Url = new Uri(collection.ElementAt(b).AccessUrl);
          NavigateToUrl(collection.ElementAt(b).AccessUrl);
      }
  
      public void PopulateScraperCollection()
      {
          string[] xmlFiles = Directory.GetFiles(@"E:\DealerConfigs\");
          foreach (string xmlFile in xmlFiles)
          {
              collection.Add(ScraperDefinition.Deserialize(xmlFile));
          }
      }
  
      public void NavigateToUrl(string url)
      {
          Console.WriteLine(collection.ElementAt(b).AccessUrl);
          webBrowser1.DocumentCompleted += webBrowser1_DocumentCompleted;
          webBrowser1.Navigate(webBrowser1.Url);
      }
  
      private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
      {
          WebBrowser wb = sender as WebBrowser;
          Process(collection.ElementAt(b), 0);
          b++;
      }
  </code></pre>
  
  <p>Consequently this causes another issue in using DocumentCompleted to navigate to the paginated results. On the first page load i use a DocumentCompleted event to trigger the link extraction. <br>When I attempt to set the url for the the next page,which is being picked out fine using xpath and again verified, using F10 to step over in debug indicates it hasnt been changed and the DocumentCompleted event isn't being triggered</p>
  
  <p>My code to change the url etc. is:</p>
  
  <pre><code>string nextPageUrl = string.Format(definition.NextPageUrlFormat, WebUtility.HtmlDecode(relativeUrl));
  webBrowser1.Url = new Uri(nextPageUrl);
  webBrowser1.DocumentCompleted += webBrowser1_DocumentCompleted;
  webBrowser1.Navigate(webBrowser1.Url);
  </code></pre>
  
  <p>Any help as always is greatly appreciated, this is proving to be a nightmare to automate, not only because WebBrowser is so much slower than WebClient, but its proving a pain to alter on the fly</p>
  
  <p>Regards</p>
  
  <p>Barry</p>