title: How to use cssselect and lxml to get option values from select drop down?
body: <p>I want to get/scrape data from a select dropdown box using lxml and cssselect.</p>
  
  <p>I have tried a few things and searched the web with no luck so far:</p>
  
  <p>so the following or similar code is on a page I am scraping:</p>
  
  <pre><code>&lt;select id="attribute501" class="required-entry super-attribute-select" name="super_attribute[501]"&gt;
  &lt;option value=""&gt;Choose an Option...&lt;/option&gt;
  &lt;option value="33"&gt;5&lt;/option&gt;
  &lt;option value="31"&gt;10&lt;/option&gt;
  &lt;/select&gt;
  </code></pre>
  
  <p>using the following cssselect statement:</p>
  
  <pre><code>css_sel_price_dd = 'html body.catalog-product-view div.wrapper div.page div.main-container div.main div.col-main div.product-view div.product-essential form div.product-shop select'
  
  
  pack_size_obj = root.cssselect(css_sel_price_dd)
  print 'This strain has a drop down select box for different pack sizes: '
  print pack_size_obj
  pack_size = pack_size_obj[0].text_content().encode('utf-8').strip()
  print 'Pack Size: ' + pack_size
  </code></pre>
  
  <p>Yields the following result:</p>
  
  <pre><code>This strain has a drop down select box for different pack sizes: 
  [&lt;SelectElement 10e77d590 name='super_attribute[501]'&gt;]
  Pack Size: Choose an Option...
  </code></pre>
  
  <p>but so far I am unable to get access to the the text values from the other option, i.e. 5 and 10 from</p>
  
  <pre><code>&lt;option value="33"&gt;5&lt;/option&gt;
  &lt;option value="31"&gt;10&lt;/option&gt;
  </code></pre>
  
  <ul>
  <li>I have tried adding option to the css path and various things but can't seem to guess the syntax and can't find anything online...</li>
  </ul>
  
  <p>Still not working - here is actual code snippet with url etc.</p>
  
  <pre><code>import lxml.html
  import requests
  import csv
  
  
  user_agent = {'User-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17'}
  # Open a session 
  s = requests.session()
  
  s.get('http://www.seedsman.com/en/cannabis-seeds', headers = user_agent)
  
  css_sel_price_dd = '.product-shop select option'
  
  open_seed_url = 'http://www.seedsman.com/en/amsterdam-mist-feminised-seeds'
  print 'From the following URL: ' + open_seed_url
  
  r = s.get(open_seed_url, headers = user_agent)
  r_encoding = r.encoding
  print 'encoding type of r:'
  print r_encoding
  
  myhtml = r.text # gives us a unicode string work with unicode and encode at the end when have piece of data we want
  
  print myhtml.encode('utf-8')
  
  root = lxml.html.fromstring(myhtml)
  
  pack_size_obj = root.cssselect(css_sel_price_dd)
  print pack_size_obj
  for opt in pack_size_obj:
      print '{0} : {1}'.format(opt.get('value'), opt.text)
  </code></pre>
  
  <p>Hopefully somebody can see what is going wrong?</p>
  
  <p>returns:</p>
  
  <pre><code>From the following URL: http://www.seedsman.com/en/amsterdam-mist-feminised-seeds
  encoding type of r: UTF-8
  ** HTML removed for brevity ** 
  [&lt;Element option at 0x1108dcbf0&gt;] 
  None : Choose an Option... 
  [Finished in 3.9s]
  </code></pre>