title: Integrate Scrapy with Django : How to
body: <p>I'm still very new in Django
  I am following <a href="https://medium.com/@ali_oguzhan/how-to-use-scrapy-with-django-application-c16fabd0e62e" rel="noreferrer">this</a> tutorial on how to integrate scrapy and django.</p>
  
  <p>the problem is when i am trying to use my own spider it's just wont work.
  I have tried the spider outside django and it's work just fine, some help will be very helpful.</p>
  
  <p>This is my spider.py file</p>
  
  <pre><code>import scrapy
  from scrapy_splash import SplashRequest
  
  class NewsSpider(scrapy.Spider):
     name = 'detik'
     allowed_domains = ['news.detik.com']
     start_urls = ['https://news.detik.com/indeks/all/?date=02/28/2018']
  
  def parse(self, response):  
      urls = response.xpath("//div/article/a/@href").extract()        
      for url in urls:
          url = response.urljoin(url)
          yield scrapy.Request(url=url, callback=self.parse_detail)
  
      # follow pagination link
      page_next =   response.xpath("//a[@class = 'last']/@href").extract_first()
      if page_next:
          page_next = response.urljoin(page_next)
          yield scrapy.Request(url=page_next, callback=self.parse)
  
  def parse_detail(self,response):
      x = {}
      x['breadcrumbs'] = response.xpath("//div[@class='breadcrumb']/a/text()").extract(),
      x['tanggal'] = response.xpath("//div[@class='date']/text()").extract_first(),
      x['penulis'] = response.xpath("//div[@class='author']/text()").extract_first(),
      x['judul'] = response.xpath("//h1/text()").extract_first(),
      x['berita'] = response.xpath("normalize-space(//div[@class='detail_text'])").extract_first(),
      x['tag'] = response.xpath("//div[@class='detail_tag']/a/text()").extract(),
      x['url'] = response.request.url,
      return x
  </code></pre>
  
  <p>this is my pipeline file </p>
  
  <pre><code>class DetikAppPipeline(object):
  
  def process_item(self, item, spider):
      item = detikNewsItem()
      self.items.append(item['breadcrumbs'])
      self.items.append(item['tanggal'])
      self.items.append(item['penulis'])
      self.items.append(item['judul'])
      self.items.append(item['berita'])
      self.items.append(item['tag'])
      self.items.append(item['url'])
      item.save()
  </code></pre>
  
  <p>And this is the models file in django </p>
  
  <pre><code>class detikNewsItem(models.Model):
      breadcrumbs = models.TextField()
      tanggal = models.TextField()
      penulis = models.TextField()
      judul = models.TextField()
      berita = models.TextField()
      tag = models.TextField()
      url = models.TextField()
  
      @property
      def to_dict(self):
      data = {
          'url': json.loads(self.url),
          'tanggal': self.tanggal
      }
      return data
  
      def __str__(self):
          return self.url
  </code></pre>