title: Download file from a dynamically generated link which lies in the source code of an HTML
body: <p>I am trying to get the Weather data from BOM Australia. The manual way is to go to <a href="http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&amp;p_display_type=dailyDataFile&amp;p_startYear=&amp;p_c=&amp;p_stn_num=2064" rel="nofollow">http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&amp;p_display_type=dailyDataFile&amp;p_startYear=&amp;p_c=&amp;p_stn_num=2064</a> and click 'All years of data', and it downloads the file!</p>
  
  <p>Here's what I have tried to automate this:</p>
  
  <pre><code>using (WebClient client = new WebClient())
              {
  
                  string html = client.DownloadString("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&amp;p_display_type=dailyDataFile&amp;p_startYear=&amp;p_c=&amp;p_stn_num=2064");
  
  
                  List&lt;string&gt; list = LinkExtractor.Extract(html);
                  foreach (var link in list)
                  {
                      if (link.StartsWith("/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile"))
                      {
  
                          string resource = "http://www.bom.gov.au" + link;
                          MessageBox.Show(resource);
  
  
                          client.DownloadFileAsync(new Uri(resource), Dts.Connections["data.zip"].ConnectionString);
                          break;
                      }
                  }
  
  
  
  
              }
  </code></pre>
  
  <p>Don't worry about the linkExtractor, it works as I am able to see the link that gives the file. The problem is that the 'DownloadFileAsync' creates a new request which does not let the file to get downloaded since the file needs the same session.</p>
  
  <p>Is there a way I can do this? Please reach out for more clarification.</p>
  
  <p>UPDATE:</p>
  
  <p>Here are the changes I made, utilising cookies from HttpWebRequest. However, I am still not able to download the file.</p>
  
  <pre><code>HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=136&amp;p_display_type=dailyDataFile&amp;p_startYear=&amp;p_c=&amp;p_stn_num=2064");
              request.CookieContainer = new CookieContainer();
  
              HttpWebResponse response = (HttpWebResponse)request.GetResponse();
  
              foreach (Cookie cook in response.Cookies)
              {
                  MessageBox.Show(cook.ToString());
              }
  
              if (response.StatusCode == HttpStatusCode.OK)
             {
                  Stream receiveStream = response.GetResponseStream();
                  StreamReader readStream = null;
  
                  if (response.CharacterSet == null)
                  {
                      readStream = new StreamReader(receiveStream);
                  }
                  else
                  {
                      readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
                  }
  
                  string data = readStream.ReadToEnd();
  
  
  
                  using (WebClient client = new WebClient())
                  {
                      foreach (Cookie cook in response.Cookies)
                      {
                          MessageBox.Show(cook.ToString());
                          client.Headers.Add(HttpRequestHeader.Cookie, cook.ToString());
                      }
  
                      List&lt;string&gt; list = LinkExtractor.Extract(data);
                      foreach (var link in list)
                      {
                          if (link.StartsWith("/jsp/ncc/cdio/weatherData/av?p_display_type=dailyZippedDataFile"))
                          {
  
                              string initial = "http://www.bom.gov.au" + link;
                              MessageBox.Show(initial);
  
                              //client.Headers.Add(HttpRequestHeader.Cookie, "JSESSIONID=2EBAFF7EFE2EEFE8140118CE5170B8F6");
                              client.DownloadFile(new Uri(initial), Dts.Connections["data.zip"].ConnectionString);
                              break;
                          }
                      }
  
  
  
  
                  }
  
                  response.Close();
                  readStream.Close();
              }
  </code></pre>