title: logging into a twitter using python3 and requests
body: <p>I have a project that I am working on, and the requirements are to login to a website using a username and password. I have to do it in python, and then be able to access a part of the site only accessible to people who are logged in. I have tried a few variations of coding to do this, and haven't been able to successfully log in yet. Here is my coding:</p>
  
  <p>the function to login to it:</p>
  
  <p>def session2(url):</p>
  
  <pre><code>#r = requests.get(url)
  #ckies = []
  
  #print("here are the cookies for twitter:\n")
  #for cky in r.cookies:
  #    print(cky.name, cky.value)
  #    ckies.append(cky)
  
  s = requests.Session()
  
  session = s.get(url, verify=False)
  print("\nheaders from site\n")
  print(session.headers)
  
  tree = html.fromstring(session.text)
  # extract the auth token needed to login along with username and password
  auth_token = list(set(tree.xpath("//input[@name='authenticity_token']/@value")))[0]
  uname = "username"
  pword = "password"
  username = 'session[username_or_email]'
  password = 'session[password]'
  # payload = {name of username variable : string you want, name of password variable:
  # string you want, name of auth token: string gotten from session
  payload = dict(username = uname, password = pword , authenticity_token = auth_token)
  header = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'}
  #do post request
  # might have to change headers to be a header for chrome
  response = s.post(
      url,
      data = payload,
      #headers = dict(referer = url)
      headers = header
  )
  print("\nheaders post\n")
  print(response.request.headers)
  session = s.get("http://www.twitter.com/username/followers", verify=False)
  print("\nheaders get\n")
  print(session.headers)
  print("\nhtml doc\n")
  print(session.text)
  return session
  </code></pre>
  
  <p>code to call it:</p>
  
  <pre><code>url = "http://www.twitter.com/login"
  sessions = session2(url)
  </code></pre>
  
  <p>the username on the site looks like this when you inspect it:</p>
  
  <pre><code>&lt;input class="js-username-field email-input js-initial-focus" type="text" name="session[username_or_email]" autocomplete="on" value="" placeholder="Phone, email or username"&gt;
  </code></pre>
  
  <p>and the password section/token section look like this:</p>
  
  <pre><code>&lt;input class="js-password-field" type="password" name="session[password]" placeholder="Password"&gt;
  
  &lt;input type="hidden" value="ef25cb09a8c7fe16c54e3df099e206e605b1170a" name="authenticity_token"&gt;
  </code></pre>
  
  <p>I know the auth token changes, which is why i have it get it from the function. When I try to run this, it just goes to the main page rather than the page i need.</p>
  
  <p>One problem I think is that when I print out the header that I send in the post, it says:</p>
  
  <p>{'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'Accept': '<em>/</em>', 'User-Agent': 'python-requests/2.9.1'}</p>
  
  <p>which I thought I changed to chrome's header, but it doesn't seem to stick.</p>
  
  <p>Also, I know there is a way if I use Oauth, but I'm not allowed to use that, i have to do it based on being able to login like I'm using a browser.</p>
  
  <p>Can you tell me if there is anything wrong with what I've done, as well as any hints on how to fix it? I've tried other stack overflow problems using requests and logging in, but those didn't work either.</p>
  
  <p>EDIT: ok, i did a response.request.headers, and it came out with the right header, i think, so i don't think that is the problem</p>
  
  <p>header it prints:</p>
  
  <pre><code> {'Accept': '*/*', 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36', 'Cookie': '_twitter_sess=some huge amount of number/letters; guest_id=v1%3A147509653977967101', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate'}
  </code></pre>