title: Python requests not redirecting
body: <p>I'm trying to scrape word definitions, but can't get python to redirect to the correct page. For example, I'm trying to get the definition for the word 'agenesia'. When you load that page in a browser with <a href="https://www.lexico.com/definition/agenesis" rel="noreferrer">https://www.lexico.com/definition/agenesia</a>, the page which loads is <a href="https://www.lexico.com/definition/agenesis" rel="noreferrer">https://www.lexico.com/definition/agenesis</a>, however in Python the page doesn't redirect and gives a 200 status code</p>
  <pre><code>URL = 'https://www.lexico.com/definition/agenesia'
  page = requests.head(URL, allow_redirects=True)
  </code></pre>
  <p>This is how I'm currently retrieving the page content, I've also tried using <code>requests.get</code>
  but that also doesn't work</p>
  <p>EDIT: Because it isn't clear, I'm aware that I could change the word to 'agenesis' in the URL to get the correct page, but I am scraping a list of words and would rather automatically follow the URL rather than searching in a browser for the redirect by hand first.</p>
  <p>EDIT 2: I realised it might be easier to check solutions with the rest of my code, so far this works with <code>agenesis</code> but not <code>agenesia</code>:</p>
  <pre><code>soup = BeautifulSoup(page.content, 'html.parser')
  
  print(soup.find(&quot;span&quot;, {&quot;class&quot;: &quot;ind&quot;}).get_text(), '\n')
  print(soup.find(&quot;span&quot;, {&quot;class&quot;: &quot;pos&quot;}).get_text())
  </code></pre>