title: Python Requests - managing cookies
body: <p>I'm trying to get some content automatically from a site using requests (and bs4)</p>
  
  <p>I have a script that gets a cookie:</p>
  
  <pre><code>def getCookies(self):
      username = 'username'
      password = 'password'
      URL = 'logonURL'
      r = requests.get(URL, auth=('username', 'password'))
      cookies = r.cookies
  </code></pre>
  
  <p>dump of the cookies looks like:</p>
  
  <pre><code>&lt;&lt;class 'requests.cookies.RequestsCookieJar'&gt;[&lt;Cookie ASP.NET_SessionId=yqokjr55ezarqbijyrwnov45 for URL.com/&gt;, &lt;Cookie BIGipServerPE_Journals.lww.com_80=1440336906.20480.0000 for URL.com/&gt;, &lt;Cookie JournalsLockCookie=id=a5720750-3f20-4207-a500-93ae4389213c&amp;ip=IP address for URL.com/&gt;]&gt;
  </code></pre>
  
  <p>But when I pass the cookie object to the next URL:</p>
  
  <pre><code> soup = Soup(s.get(URL, cookies = cookies).content)
  </code></pre>
  
  <p>its not working out - I can see by dumping the soup that I'm not giving the webserver my credentials properly  </p>
  
  <p>I tried running a requests session:</p>
  
  <pre><code>def getCookies(self):
      self.s = requests.session()
      username = 'username'
      password = 'password'
      URL = 'logURL'
      r = self.s.get(URL, auth=('username', 'password'))
  </code></pre>
  
  <p>and I get the same no joy. </p>
  
  <p>I looked at the header via liveHttp in FF when I visit the 2nd page, and see a very different form: </p>
  
  <pre><code>Cookie: WT_FPC=id=264b0aa85e0247eb4f11355304127862:lv=1355317068013:ss=1355314918680; UserInfo=Username=username; BIGipServerPE_Journals.lww.com_80=1423559690.20480.0000; PlatformAuthCookie=true; Institution=ReferrerUrl=http://logonURL.com/?wa=wsignin1.0&amp;wtrealm=urn:adis&amp;wctx=http://URL.com/_layouts/Authenticate.aspx?Source=%252fpecnews%252ftoc%252f2012%252f06440&amp;token=method|ExpireAbsolute; counterSessionGuidId=6e2bd57f-b6da-4dd4-bcb0-742428e08b5e; MyListsRefresh=12/13/2012 12:59:04 AM; ASP.NET_SessionId=40a04p45zppozc45wbadah45; JournalsLockCookie=id=85d1f38f-dcbb-476a-bc2e-92f7ac1ae493&amp;ip=10.204.217.84; FedAuth=77u/PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48U2VjdXJpdHlDb250ZXh0VG9rZW4gcDE6SWQ9Il9mMGU5N2M3Zi1jNzQ5LTQ4ZjktYTUxNS1mODNlYjJiNGNlYzUtNEU1MDQzOEY0RTk5QURCNDFBQTA0Mjc0RDE5QzREMEEiIHhtbG5zOnAxPSJodHRwOi8vZG9jcy5vYXNpcy1vcGVuLm9yZy93c3MvMjAwNC8wMS9vYXNpcy0yMDA0MDEtd3NzLXdzc2VjdXJpdHktdXRpbGl0eS0xLjAueHNkIiB4bWxucz0iaHR0cDovL2RvY3Mub2FzaXMtb3Blbi5vcmcvd3Mtc3gvd3Mtc2VjdXJlY29udmVyc2F0aW9uLzIwMDUxMiI+PElkZW50aWZpZXI+dXJuOnV1aWQ6ZjJmNGY5MGItMmE4Yy00OTdlLTkwNzktY2EwYjM3MTBkN2I1PC9JZGVudGlmaWVyPjxJbnN0YW5jZT51cm46dXVpZDo2NzMxN2U5Ny1lMWQ3LTQ2YzUtOTg2OC05ZGJhYjA3NDkzOWY8L0luc3RhbmNlPjwvU2VjdXJpdHlDb250ZXh0VG9rZW4+
  </code></pre>
  
  <p>I have redacted the username, password, and URLS from the question for obvious reasons. </p>
  
  <p>Am I missing something obvious? is there a different / proper way to capture the cookie - the current method I'm using is not working. </p>
  
  <p>EDIT: </p>
  
  <p>This is a self standing version of the sessioned code: </p>
  
  <pre><code>s = requests.session()
  username = 'username'
  password = 'password'
  URL = 'logonURL.aspx'
  r = s.get(URL, auth=('username', 'password'))
  URL = r"URL.aspx"
  soup = Soup(s.get(URL).content)
  </code></pre>
  
  <p>reading a dump of the soup, I can see in the html that its telling me I don't have access - this string only appears via browser when you're not logged in. </p>