Problem accessing python wsgi server response running fastcgi protocol from urllib(python).
I am building an application where I have a web server set up running on fast-cgi.
This server is supposed to return results incrementally.

I want to query this server from another server which runs in python and I am making a urllib request from this server to fcgi-protocol wsgi server. 


Now what is happening is that I am able to get incremental results querying fcgi server that show up in the browser, but I am not able to get incremental results querying from urllib. Instead I am getting all results at once after all the results are completed processing on the fcgi-server.

The fcgi server, server.py

    from wsgiref.util import setup_testing_defaults
    from wsgiref.simple_server import make_server
    import re
    import time
    import random
    # A relatively simple WSGI application. It's going to print out the
    # environment dictionary after being updated by setup_testing_defaults
    
    port = 8011
    
    def process(query):
        r = re.compile("\s")
        query = "".join(r.split(query))
        params = query.split("&")
        rdict = {}
        for param in params:
            [key, value] = param.split("=")
            rdict[key] = value
        return rdict
    
    def simple_app(environ, start_response):
        setup_testing_defaults(environ)
        query = environ['QUERY_STRING']
        print query
        start_response('200 OK', [('Content-Type', 'text/plain')])
        ret = ["%s: %s\n" % (key, value) for key, value in process(query).iteritems()]
        print ret
        for item in ret:
            yield item
            time.sleep(2)
    
    httpd = make_server('127.0.0.1', port, simple_app)
    print "Serving on port %s...\nQuit by Ctrl+C." % (port)
    httpd.serve_forever()


I run it by python server.py and querying like

    http://127.0.0.1:8011/a.fcgi?a=b&b=x&q=p&r=b&l=u&x=h

urllib access, python code.

    import urllib, urllib2
    url = 'http://127.0.0.1:8011/a.fcgi?a=b&b=x&q=p&r=b&l=u&x=h&p=i&j=p'
    request_object = urllib2.Request(url)
    response = urllib2.urlopen(request_object)
    for item in response:
        print item

Moreover when I am trying to change the call to start_response with anything other than text/plain to text/html, text/json I stop getting incremental results on browser as well.
Problem accessing python wsgi server response running fastcgi protocol from urllib(python).
===

I am building an application where I have a web server set up running on fast-cgi.
This server is supposed to return results incrementally.

I want to query this server from another server which runs in python and I am making a urllib request from this server to fcgi-protocol wsgi server. 


Now what is happening is that I am able to get incremental results querying fcgi server that show up in the browser, but I am not able to get incremental results querying from urllib. Instead I am getting all results at once after all the results are completed processing on the fcgi-server.

The fcgi server, server.py

    from wsgiref.util import setup_testing_defaults
    from wsgiref.simple_server import make_server
    import re
    import time
    import random
    # A relatively simple WSGI application. It's going to print out the
    # environment dictionary after being updated by setup_testing_defaults
    
    port = 8011
    
    def process(query):
        r = re.compile("\s")
        query = "".join(r.split(query))
        params = query.split("&")
        rdict = {}
        for param in params:
            [key, value] = param.split("=")
            rdict[key] = value
        return rdict
    
    def simple_app(environ, start_response):
        setup_testing_defaults(environ)
        query = environ['QUERY_STRING']
        print query
        start_response('200 OK', [('Content-Type', 'text/plain')])
        ret = ["%s: %s\n" % (key, value) for key, value in process(query).iteritems()]
        print ret
        for item in ret:
            yield item
            time.sleep(2)
    
    httpd = make_server('127.0.0.1', port, simple_app)
    print "Serving on port %s...\nQuit by Ctrl+C." % (port)
    httpd.serve_forever()


I run it by python server.py and querying like

    http://127.0.0.1:8011/a.fcgi?a=b&b=x&q=p&r=b&l=u&x=h

urllib access, python code.

    import urllib, urllib2
    url = 'http://127.0.0.1:8011/a.fcgi?a=b&b=x&q=p&r=b&l=u&x=h&p=i&j=p'
    request_object = urllib2.Request(url)
    response = urllib2.urlopen(request_object)
    for item in response:
        print item

Moreover when I am trying to change the call to start_response with anything other than text/plain to text/html, text/json I stop getting incremental results on browser as well.