title: Scraping a site with python3.6. I can't progress past the login page
body: <p>The html form code for the site:</p>
  
  <pre><code>                &lt;form class="m-t" role="form" method="POST" action=""&gt;
  
                  &lt;div class="form-group text-left"&gt;
                      &lt;label for="username"&gt;Username:&lt;/label&gt;
                      &lt;input type="text" class="form-control" id="username" name="username" placeholder="" autocomplete="off" required /&gt;
                  &lt;/div&gt;
                  &lt;div class="form-group text-left"&gt;
                      &lt;label for="password"&gt;Password:&lt;/label&gt;
                      &lt;input type="password" class="form-control" id="pass" name="pass" placeholder="" autocomplete="off" required /&gt;
                  &lt;/div&gt;
  
                  &lt;input type="hidden" name="token" value="/bGbw4NKFT+Yk11t1bgXYg48G68oUeXcb9N4rQ6cEzE="&gt;
                  &lt;button type="submit" name="submit" class="btn btn-primary block full-width m-b"&gt;Login&lt;/button&gt;
  </code></pre>
  
  <p>Simple enough so far. I've scraped a number of sites in the past without issue.</p>
  
  <p>I have tried: selenium, mechanize(albeit had to drop back to earlier version of python), mechanicalsoup, requests.</p>
  
  <p>I have read: multiple posts here on SO as well as:
  <a href="https://kazuar.github.io/scraping-tutorial/" rel="nofollow noreferrer">https://kazuar.github.io/scraping-tutorial/</a>
  <a href="http://docs.python-requests.org/en/latest/user/advanced/#session-objects" rel="nofollow noreferrer">http://docs.python-requests.org/en/latest/user/advanced/#session-objects</a>
  and many many more.</p>
  
  <p>Sample code:</p>
  
  <pre><code>import requests
  from lxml import html
  session_requests = requests.session()
  result = session_requests.get(url)
  tree = html.fromstring(result.text)
  authenticity_token = list(set(tree.xpath("//input[@name='token']/@value")))[0]
  result = session_requests.post(
      url, 
      data = payload, 
      headers = dict(referer=url)
  )
  result = session_requests.get(url3)
  print(result.text)
  </code></pre>
  
  <p>and</p>
  
  <pre><code>import mechanicalsoup
  import requests
  from http import cookiejar
  
  c = cookiejar.CookieJar()
  s = requests.Session()
  s.cookies = c
  browser = mechanicalsoup.Browser(session=s)
  
  login_page = browser.get(url)
  
  login_form = login_page.soup.find('form', {'method':'POST'})
  
  login_form.find('input', {'name': 'username'})['value'] = username
  login_form.find('input', {'name': 'pass'})['value'] = password
  
  response = browser.submit(login_form, login_page.url)
  </code></pre>
  
  <p>Try as I might I just cannot return anything other than the html code for the login page and I don't know where to explore next to actually figure out what's not happening and why.</p>
  
  <p>url = variable that holds login page url, url3 = a page I want to scrape.</p>
  
  <p>Any help would be much appreciated!</p>