title: Clicking Google Scholar Button with Scrapy
body: <p>I am trying to scrape some data from Google Scholar with <code>scrapy</code>, my code is the following:</p>
  <pre><code>import scrapy
  class TryscraperSpider(scrapy.Spider):
      name = 'tryscraper'
      start_urls = ['https://scholar.google.com/citations?hl=en&amp;user=JUn8PgwAAAAJ&amp;pagesize=100&amp;view_op=list_works&amp;sortby=pubdate']
  
      def parse(self, response):
          for link in response.css('a.gsc_a_at::attr(href)'):
              yield response.follow(link.get(), callback=self.parse_scholar)
         
         
      def parse_scholar(self, response):
          try: 
              yield {
                 'authors': response.css('div.gsc_oci_value::text').get().strip(),
                 'journal': response.css('div.gsc_oci_value::text').extract()[2].strip(),
                 'date': response.css('div.gsc_oci_value::text').extract()[1].strip(),
                 'abstract': response.css('div.gsh_csp::text').get()
                   }
          except: 
              yield {
                  'authors': response.css('div.gsc_oci_value::text').get().strip(),
                  'journal': response.css('div.gsc_oci_value::text').extract()[2].strip(),
                  'date': response.css('div.gsc_oci_value::text').extract()[1].strip(),
                  'abstract': 'NA'
                   }
              
  </code></pre>
  <p>This code works well, but it only gives me the first 100 papers from the author, I would like to scrape them all, but I would need to code the spider to also press the button &quot;Show More&quot;. I have seen in related posts that <code>scrapy</code> does not have built in functions to do so, but that maybe you can incorporate functionalities from <code>selenium</code> to do the job. Unfortunately, I am a bit of a novice and therefore completely lost, any suggestions? Thanks in advance.</p>
  <p><a href="https://stackoverflow.com/questions/63318049/clicking-show-more-button-on-google-scholar-results">Here</a> there is the <code>selenium</code> code that should do the job, but I would like it to combine it with my <code>scrapy</code> spider, which works well and it's very fast.</p>