title: Login to a website via Python - how to deal with CSRF?
body: <p>I'm using Python 3 for a script that will monitor updates in a user's profile on a webpage. The login to this site is protected by CSRF countermeasures, which is a good thing. However, I can't get my script to login to this site.</p>
  
  <ul>
  <li><p>My approach using <code>mechanicalsoup</code>:</p>
  
  <pre><code>import mechanicalsoup
  
  browser = mechanicalsoup.Browser()
  login_page = browser.get(base_url)
  login_form = login_page.soup.select(".form-signin")[0]
  
  login_form.find(attrs={"name": "username"})['value'] = 'username'
  login_form.find(attrs={"name": "password"})['value'] = 'password'
  
  page2 = browser.submit(login_form, login_url)
  print(str(page2.text))
  </code></pre></li>
  <li><p>My approach using <code>robobrowser</code>:</p>
  
  <pre><code>import re
  from robobrowser import RoboBrowser
  
  browser = RoboBrowser(history=True)
  browser.open(base_url)
  form = browser.get_form(action='/login/')
  
  form["username"] = 'username'
  form["password"] = 'password'
  
  browser.submit_form(form)
  print(str(browser.select))
  </code></pre></li>
  </ul>
  
  <p>In both cases I end up with a HTTP status of <code>403</code> and a message saying <code>CSRF verification failed. Request aborted.</code></p>
  
  <ul>
  <li>Any ideas how to fix this?</li>
  <li>The form in question has a hidden input containing a CSRF token. I guess <code>mechanicalsoup</code> and <code>robobrowser</code> will submit this input as well. Am I right? Or do I have to treat it specially?</li>
  <li>I thought the session used by this two packages would handle everything like cookies and so on. Is there something I've missed?</li>
  </ul>