title: Scrape all youtube search results
body: <p>I am trying to collect data from youtube search result. Search term is <em>"border collie"</em> with a filter for videos that were uploaded <em>"Today"</em>.</p>
  
  <p>52 videos appear in the search result. However, when I try to parse the page, I only got 20 entries. How do I parse all 52 videos? Any suggestions is appreciated.</p>
  
  <p>P.S. I tried this <a href="https://stackoverflow.com/questions/20986631/how-can-i-scroll-a-web-page-using-selenium-webdriver-in-python">post</a> for the infinitive page, but it didn't work for youtube.</p>
  
  <p><strong>Current code:</strong></p>
  
  <pre><code>url = 'https://www.youtube.com/results?search_query=border+collie&amp;sp=EgIIAg%253D%253D'
  driver = webdriver.Chrome()
  driver.get(url)
  
  #waiting for the page to load
  sleep(3) 
  #repeat scrolling 10 times
  for i in range(10):
      #scroll 1000 px
      driver.execute_script('window.scrollTo(0,(window.pageYOffset+1000))')
      sleep(3) 
  
  response = requests.get(url)
  soup = bs(response.text,'html.parser',from_encoding="UTF-8")
  
  source_list = []
  duration_list = []
  
  #Scrape source of the video
  vids_source = soup.findAll('div',attrs={'class':'yt-lockup-byline'})
  for i in vids_source:
      source = i.text
      source_list.append(source)
  
  #Scrape video duration
  vids_badge = soup.findAll('span',attrs={'class':'video-time'})
  for i in vids_badge:
      duration = i.text
      duration_list.append(duration)
  </code></pre>