title: Python : Use PyQT4 + Soup to scrape SEVERAL pages
body: <p>I am trying to scrape several webpages using Python PyQT4 + Beautiful Soup.</p>
  
  <p>Due to the nature of my overal program, I use a main script "program.py" calling functions from other scripts, doing different analyses with beautiful Soup.</p>
  
  <p>Thus, the <strong>simplified architecture</strong> of my <strong>main program.py</strong> is as follows :</p>
  
  <pre><code>program.py :
  
  import script1
  import script2
  
  script1.function1(urlA)
  script2.function2(urlB)
  </code></pre>
  
  <p>With script1.py and script2.py as follows :</p>
  
  <pre><code>script1.py :
  
  import requests
  import re
  from bs4 import BeautifulSoup
  from PyQt4.QtGui import *
  from PyQt4.QtCore import *
  from PyQt4.QtWebKit import * 
  
  class Render(QWebPage):
      def __init__(self, url):
          self.app = QApplication(sys.argv)
          QWebPage.__init__(self)
          self.loadFinished.connect(self._loadFinished)
          self.mainFrame().load(QUrl(url))
          self.app.exec_()
      def _loadFinished(self, result):
          self.frame = self.mainFrame()
          self.app.quit()   
  
  
  def function1(url):
      r = Render(url)
      soup = BeautifulSoup(unicode(r.frame.toHtml()))
  
      #Do many things with soup.
      #Nothing related to PyQT4 further in this script
  </code></pre>
  
  <p>And my script 2 has exactly the same structure, but does other things on another url.</p>
  
  <pre><code>script2.py :
  
  import requests
  import re
  from bs4 import BeautifulSoup
  from PyQt4.QtGui import *
  from PyQt4.QtCore import *
  from PyQt4.QtWebKit import * 
  
  class Render(QWebPage):
      def __init__(self, url):
          self.app = QApplication(sys.argv)
          QWebPage.__init__(self)
          self.loadFinished.connect(self._loadFinished)
          self.mainFrame().load(QUrl(url))
          self.app.exec_()
      def _loadFinished(self, result):
          self.frame = self.mainFrame()
          self.app.quit()   
  
  
  def function2(url):
      r = Render(url)
      soup = BeautifulSoup(unicode(r.frame.toHtml()))
  
      #Do many other things with soup
      #Nothing related to PyQT4 further in this script
  </code></pre>
  
  <p>Everything works fine with script1.py. My function1 and analyses are run successfully.</p>
  
  <p>But script2.py bugs, and I have the following error :</p>
  
  <pre><code>QObject::connect: Cannot connect (null)::configurationAdded(QNetworkConfiguration) to QNetworkConfigurationManager::configurationAdded(QNetworkConfiguration)
  QObject::connect: Cannot connect (null)::configurationRemoved(QNetworkConfiguration) to QNetworkConfigurationManager::configurationRemoved(QNetworkConfiguration)
  QObject::connect: Cannot connect (null)::configurationChanged(QNetworkConfiguration) to QNetworkConfigurationManager::configurationChanged(QNetworkConfiguration)
  QObject::connect: Cannot connect (null)::onlineStateChanged(bool) to QNetworkConfigurationManager::onlineStateChanged(bool)
  QObject::connect: Cannot connect (null)::configurationUpdateComplete() to QNetworkConfigurationManager::updateCompleted()
  </code></pre>
  
  <p>I spent time searching for this problem, and I found that PyQT4 could not load several pages in the same instance.</p>
  
  <p>The problem is that I need PyQT4 to render Javascripts before loading the page content into Beautiful Soup.</p>
  
  <p>So I think I need to put some kind of "self.app.quit()" at the end of my function1 in script1, so that function2 in script2 can render a page with PyQT4 too. But I was not able to make it work.</p>