title: What is the best solution for sharing data between processes(workers) in Celery - Python?
body: <p>I have an application that reads float data from sensors every 100 milliseconds, appends it to list and every 5 minutes calculates some statistics from that list and insert to MongoDB Database. Then it cleans the list and so on.</p>
  
  <p>There are many of these list (as many as sensors) and I need to read the data periodic, so I set up Celery workers. It works pretty fine, but each Celery worker has its own specific global variables space, so lists during inserting to Database have different values, depends on which workers actually inserts data to Database.</p>
  
  <p>What is the solution for sharing data between workers and lock it somehow to prevent multiple workers inserting to database its own version of sensors data?</p>
  
  <p>I thought about Redis and appending sensors data direct to Redis dict and every 5 minutes reads the data from Redis, calculates the stats, cleans the Redis dict and so on. </p>
  
  <pre><code>import celery
  import my_data_reader
  import my_stats_calculator
  import my_mongo_manager
  
  app = celery.Celery('tasks', broker='redis://localhost')
  
  sensor_data = []
  
  data_reader = my_data_reader.TemperatureReader(1)
  mongo_writer = my_mongo_manager.DataWriter()
  stats_calculator = my_stats_calculator.Calculator()
  
  
  # Runs every 100 milliseconds
  @app.task
  def update_sensors():
  
      global sensor_data
      global data_reader
  
      sensor_data.append(data_reader.get_data())
  
  # Runs every 5 seconds
  @app.task
  def insert_to_database():
  
      global sensor_data
      global mongo_writer
      global stats_calculator
  
      stats_dict = stats_calculator.calculate_stats(sensor_data)
      mongo_writer.insert_data(stats_dict)
      del sensor_data[:]
  </code></pre>
  
  <p>After running this code using 1 process (--concurrency=1 celery flag) it works absolutely fine, however in actual project there are over 25 sensors and I would like to somehow do these operations efficiently. </p>
  
  <p>Does anybody know what is the proper way to share these objects between workers?</p>