<p dir="auto">Hey, I am not sure if this is a duplicate or not, but I could not find this exact issue. I realize we are not running 3.1.7 so I will upgrade all machines to check</p>
<p dir="auto">Running:<br>
amqp==1.3.3<br>
billiard==3.3.0.10<br>
celery==3.1.6<br>
eventlet==0.14.0<br>
kombu==3.0.7</p>
<p dir="auto">Getting the following issue:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[2014-01-10 12:13:17,420: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/celery/worker/consumer.py&quot;, line 270, in start
    blueprint.start(self)
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/celery/bootsteps.py&quot;, line 123, in start
    step.start(parent)
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/celery/worker/consumer.py&quot;, line 790, in start
    c.loop(*c.loop_args())
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/celery/worker/loops.py&quot;, line 99, in synloop
    connection.drain_events(timeout=2.0)
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/kombu/connection.py&quot;, line 279, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py&quot;, line 90, in drain_events
    return connection.drain_events(**kwargs)
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/amqp/connection.py&quot;, line 282, in drain_events
    chanmap, None, timeout=timeout,
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/amqp/connection.py&quot;, line 345, in _wait_multiple
    channel, method_sig, args, content = read_timeout(timeout)
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/amqp/connection.py&quot;, line 316, in read_timeout
    return self.method_reader.read_method()
  File &quot;/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/amqp/method_framing.py&quot;, line 195, in read_method
    raise m
UnexpectedFrame: Received 0x2d while expecting 0xce"><pre class="notranslate">[2014-01-10 12:13:17,420: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/celery/worker/consumer.py<span class="pl-pds">"</span></span>, line 270, <span class="pl-k">in</span> start
    blueprint.start(self)
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/celery/bootsteps.py<span class="pl-pds">"</span></span>, line 123, <span class="pl-k">in</span> start
    step.start(parent)
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/celery/worker/consumer.py<span class="pl-pds">"</span></span>, line 790, <span class="pl-k">in</span> start
    <span class="pl-en">c.loop(*c.loop_args</span>())
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/celery/worker/loops.py<span class="pl-pds">"</span></span>, line 99, <span class="pl-k">in</span> synloop
    connection.drain_events(timeout=2.0)
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/kombu/connection.py<span class="pl-pds">"</span></span>, line 279, <span class="pl-k">in</span> drain_events
    <span class="pl-k">return</span> self.transport.drain_events(self.connection, <span class="pl-k">**</span>kwargs)
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py<span class="pl-pds">"</span></span>, line 90, <span class="pl-k">in</span> drain_events
    <span class="pl-k">return</span> connection.drain_events(<span class="pl-k">**</span>kwargs)
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/amqp/connection.py<span class="pl-pds">"</span></span>, line 282, <span class="pl-k">in</span> drain_events
    chanmap, None, timeout=timeout,
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/amqp/connection.py<span class="pl-pds">"</span></span>, line 345, <span class="pl-k">in</span> _wait_multiple
    channel, method_sig, args, content = read_timeout(timeout)
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/amqp/connection.py<span class="pl-pds">"</span></span>, line 316, <span class="pl-k">in</span> read_timeout
    <span class="pl-k">return</span> <span class="pl-en">self.method_reader.read_method</span>()
  File <span class="pl-s"><span class="pl-pds">"</span>/usr/local/virtualenvs/shapeup/local/lib/python2.7/site-packages/amqp/method_framing.py<span class="pl-pds">"</span></span>, line 195, <span class="pl-k">in</span> read_method
    raise m
UnexpectedFrame: Received 0x2d <span class="pl-k">while</span> expecting 0xce</pre></div>
<p dir="auto">I'm encountering an <a href="https://groups.google.com/forum/#!topic/consul-tool/U1GXnuNmMN8" rel="nofollow">issue</a> using the Consul result backend which I suspect is NOT a Celery issue as such but rather <a href="https://github.com/cablehead/python-consul/issues/144" data-hovercard-type="issue" data-hovercard-url="/cablehead/python-consul/issues/144/hovercard">an issue in the lack of thread safety in the library the backend uses</a>.</p>
<p dir="auto">The symptoms are consistent with confusion between the sorts of responses that Consul gives to the library when creating saved results, and expiring those saved results. So, if Celery both writes and expires saved results using threads and, the lack of thread-safety in the python-consul library could produce the results described.</p>
<p dir="auto">Can somebody confirm if Celery access to a results backend involves multiple threads?</p>