title: How to use concurrent futures to web scrape faster? Selenium
body: <p>I'm trying to scrape a list of URLs with Selenium and concurrent futures to speed up the process. I've found that I get a StaleElementReferenceException when using concurrent futures, and also the job titles do not correspond to the URLs. For instance, I get repeated job titles. When using a normal &quot;for&quot; I do not get this error.</p>
  <p>I don't know what I'm doing wrong. Any help is welcomed.</p>
  <p>My simplified code is:</p>
  <pre class="lang-py prettyprint-override"><code>import concurrent.futures
  import time
  from selenium import webdriver
  
  options = webdriver.ChromeOptions()
  #options.add_argument('--headless')
  options.add_argument('--no-sandbox')
  options.add_argument('--disable-dev-shm-usage')
  
  PATH = &quot;C:\Program Files (x86)\chromedriver.exe&quot;
  wd = webdriver.Chrome(PATH, options=options)
  wd.maximize_window()
  
  vurl = ['https://www.bumeran.com.pe/empleos/asistente-contable-exp.-en-concar-ssp-1114585777.html',
   'https://www.bumeran.com.pe/empleos/asesor-a-comercial-digital-de-seguro-vehicular-1114584904.html',
   'https://www.bumeran.com.pe/empleos/mecanico-de-mantenimiento-arequipa-1114585709.html',
   'https://www.bumeran.com.pe/empleos/almacenero-l.o.-electronics-s.a.c.-1114585629.html',
   'https://www.bumeran.com.pe/empleos/analista-de-comunicaciones-ingles-avanzado-teleperformance-peru-s.a.c.-1114564863.html',
   'https://www.bumeran.com.pe/empleos/vendedores-adn-retail-s.a.c.-1114585422.html',
   'https://www.bumeran.com.pe/empleos/especialista-de-intervencion-de-proyectos-mondelez-international-1114585461.html',
   'https://www.bumeran.com.pe/empleos/desarrollador-java-senior-inetum-peru-1114584840.html',
   'https://www.bumeran.com.pe/empleos/practicante-legal-coes-sinac-1114584788.html',
   'https://www.bumeran.com.pe/empleos/concurso-publico-n-143-especialista-en-presupuesto-banco-central-de-reserva-del-peru-1114584538.html',
   'https://www.bumeran.com.pe/empleos/concurso-n-147-especialista-en-analisis-de-infraestructuras-financieras-banco-central-de-reserva-del-peru-1114584444.html',
   'https://www.bumeran.com.pe/empleos/asistente-legal-magdalena-del-mar-los-portales-1114584305.html',
   'https://www.bumeran.com.pe/empleos/asistente-de-nuevos-negocios-inmobiliarios-madrid-ingenieros-1114584269.html',
   'https://www.bumeran.com.pe/empleos/trabajo-desde-tres-horas-por-dia-ventas-ventas-por-internet-1114584205.html']
  
  
  vtitle = []
  
  def get_urls(url):
      wd.get(url)
      wd.implicitly_wait(20)
      try: 
          title = wd.find_element_by_xpath(&quot;//h1&quot;).text
          print('URL finished')
      except:
          title=''
          print('Exception!')
      vtitle.append(title)
      vurl2.append(url)
  
  # This throws an exception and does not scrape correctly
  with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
     executor.map(get_urls, vurl)
  
  # output is for example
  #['ALMACENERO', 'ALMACENERO', 'ALMACENERO', 'ALMACENERO', 'Desarrollador Java (Senior)', 'Desarrollador Java (Senior)', 'Desarrollador Java (Senior)']
  
  # when it should be:
  # ['ALMACENERO', 'Analista de Comunicaciones - Inglés Avanzado', 'Vendedores', 'Especialista de Intervención de Proyectos', 'Desarrollador Java (Senior)', 'Practicante Legal', 'Asistente Legal - Magdalena del Mar']
  
  
  # This works fine but is too slow
  for url in vurl:
      get_urls(url)
  
  </code></pre>