Body: <p>I have finished implementing an Asynchronous Advantage Actor-Critic (A3C) agent for TensorFlow (gpu). By using a single RMSprop optimizer with shared statistics. To do so, a central controller holds both the Global Network (ActorCriticModel) and the Optimizer. Workers, to update the global network, need to communicate gradients to the Controller, while receiving back the updated model weights. It works decently: despite there might be some errors in the code, it still achieves the results pretty fast.
  Each Worker is a thread on its own.</p>
  <p><code>class Worker(Thread):</code></p>
  <p>Problem is Workers need to extend the Thread class while I would like to use heavy Processes (from multiprocessing) instead (Queues used in the following snippet of code are imported from the multiprocessing class).</p>
  <pre><code>class Controller:
  
      def __init__(self, **kwargs):
  
          self.terminated_workers = 0
          self.env_name = kwargs[&quot;env_name&quot;]
          self.available_actions = kwargs[&quot;available_actions&quot;]
          self.action_size = len(self.available_actions)
          self.env = make_env(kwargs, env_name=self.env_name)
          self.state_shape = self.env.observation_space.shape
  
          self.number_of_workers = kwargs[&quot;number_of_workers&quot;]
          self.workers = []
          self.controller_queue = Queue()
          self.worker_queues = []
  
          # Initialization of threads
          for i in range(self.number_of_workers):
              worker_queue = Queue()
              self.worker_queues.append(worker_queue)
  
              updater = GlobalUpdater(self.controller_queue, worker_queue)
              worker = Worker(i, updater, self.state_shape, kwargs)
              self.workers.append(worker)
  
          # Only import tensorflow after heavy Process initialization
          import tensorflow as tf
  
          self.global_model = ActorCriticModel(self.state_shape, self.action_size)
          self.opt = tf.keras.optimizers.RMSprop(
              learning_rate=kwargs[&quot;lr_schedule&quot;],
              rho=kwargs[&quot;rho&quot;],
              centered=kwargs[&quot;centered&quot;],
              clipvalue=kwargs[&quot;clipvalue&quot;]
          ) 
  </code></pre>
  <p>As soon as i change the Worker to extend Processes instead of threads, I get the following message for every worker instantiated:</p>
  <p><em>F tensorflow/stream_executor/cuda/cuda_driver.cc:152] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error</em></p>
  <p>Technically, since multiple methods of the Worker class need to use tensorflow, the library is imported statically, before declaring the Worker class, as follows:</p>
  <pre><code>import tensorflow as tf
  
  class Worker(Process):
     ...
  </code></pre>
  <p>I'm using Linux and the libraries' versions are:</p>
  <ul>
  <li>TensorFlow: 2.9.1</li>
  <li>CUDA: 11.2.2-1</li>
  <li>cuDNN: 8.1.1.33-1+cuda11.2</li>
  </ul>
  <p>I tried every other solution I found online, including setting <code>mp.set_start_method('spawn')</code> and trying to postpone as much as i can the import of the tensorflow library, so to postpone as much as possible the heavy process initialization but nothing seems to work. How would you approach this problem? Do you have any idea on how to solve it? Thanks in advance.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-12T18:58:41.777Z
Comments.Text: Firstly, this question is well suited for StackOverflow, but this forum is less about implementation issues - so this will probably get closed.
  However, this sounds to me like an issue with memory being not shared petween Processes. Have you tried to import tensorflow in the constructor and reference it as `self.tf` or use `global tf`?
  Also you should use [gpu memory growth](https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth), otherwise the first process will occupy the whole space on the GPU.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-12T21:44:44.623Z
Comments.Text: Thank you for your kind answer. That's pretty smart, but unfortunately the ```self.tf``` is not compatible with the code (as I'm using the @tf.function).
Comments.UserDisplayName: Lyn Cassidy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-12T21:52:02.227Z
Comments.Text: So, I gave it a try by commenting tf.function, setting ```multiprocessing.set_start_method('spawn', force=True)```, importing TensorFlow within the spawned process upon creation and putting ```global tf``` right after the import. 
  And it actually works. You're a beast! I've spent months trying to figure this out, it's a pity this post will be removed as I think a lot of people would actually need to see this.
Comments.UserDisplayName: Lyn Cassidy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-13T12:57:36.840Z
Comments.Text: Oh Really? :D That's amazing! I'm glad I was helpful
  Maybe the post stays I don't know, usually these post get removed but who reads the manual anyways^^
Comments.UserDisplayName: Chillston
Tags: <reinforcement-learning><tensorflow><actor-critic-methods><gpu><a3c>
Title: Tensorflow-gpu and multiprocessing
UserDisplayName: Lyn Cassidy
ThreadText: # Tensorflow-gpu and multiprocessing
  
  (Asked by: username_0 on 2022-07-12)
  
  I have finished implementing an Asynchronous Advantage Actor-Critic (A3C) agent for TensorFlow (gpu). By using a single RMSprop optimizer with shared statistics. To do so, a central controller holds both the Global Network (ActorCriticModel) and the Optimizer. Workers, to update the global network, need to communicate gradients to the Controller, while receiving back the updated model weights. It works decently: despite there might be some errors in the code, it still achieves the results pretty fast.
  Each Worker is a thread on its own.
  class Worker(Thread):
  Problem is Workers need to extend the Thread class while I would like to use heavy Processes (from multiprocessing) instead (Queues used in the following snippet of code are imported from the multiprocessing class).
  class Controller:
  
      def __init__(self, **kwargs):
  
          self.terminated_workers = 0
          self.env_name = kwargs["env_name"]
          self.available_actions = kwargs["available_actions"]
          self.action_size = len(self.available_actions)
          self.env = make_env(kwargs, env_name=self.env_name)
          self.state_shape = self.env.observation_space.shape
  
          self.number_of_workers = kwargs["number_of_workers"]
          self.workers = []
          self.controller_queue = Queue()
          self.worker_queues = []
  
          # Initialization of threads
          for i in range(self.number_of_workers):
              worker_queue = Queue()
              self.worker_queues.append(worker_queue)
  
              updater = GlobalUpdater(self.controller_queue, worker_queue)
              worker = Worker(i, updater, self.state_shape, kwargs)
              self.workers.append(worker)
  
          # Only import tensorflow after heavy Process initialization
          import tensorflow as tf
  
          self.global_model = ActorCriticModel(self.state_shape, self.action_size)
          self.opt = tf.keras.optimizers.RMSprop(
              learning_rate=kwargs["lr_schedule"],
              rho=kwargs["rho"],
              centered=kwargs["centered"],
              clipvalue=kwargs["clipvalue"]
          ) 
  
  As soon as i change the Worker to extend Processes instead of threads, I get the following message for every worker instantiated:
  F tensorflow/stream_executor/cuda/cuda_driver.cc:152] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error
  Technically, since multiple methods of the Worker class need to use tensorflow, the library is imported statically, before declaring the Worker class, as follows:
  import tensorflow as tf
  
  class Worker(Process):
     ...
  
  I'm using Linux and the libraries' versions are:
  
  TensorFlow: 2.9.1
  CUDA: 11.2.2-1
  cuDNN: 8.1.1.33-1+cuda11.2
  
  I tried every other solution I found online, including setting mp.set_start_method('spawn') and trying to postpone as much as i can the import of the tensorflow library, so to postpone as much as possible the heavy process initialization but nothing seems to work. How would you approach this problem? Do you have any idea on how to solve it? Thanks in advance.
  
  **Tags:** reinforcement-learning, tensorflow, actor-critic-methods, gpu, a3c
  > **Comments:**
  > **username_1**: Firstly, this question is well suited for StackOverflow, but this forum is less about implementation issues - so this will probably get closed. However, this sounds to me like an issue with memory being not shared petween Processes. Have you tried to import tensorflow in the constructor and reference it as `self.tf` or use `global tf`? Also you should use [gpu memory growth](https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth), otherwise the first process will occupy the whole space on the GPU.  
  > **username_0**: Thank you for your kind answer. That's pretty smart, but unfortunately the ```self.tf``` is not compatible with the code (as I'm using the @tf.function).  
  > **username_0**: So, I gave it a try by commenting tf.function, setting ```multiprocessing.set_start_method('spawn', force=True)```, importing TensorFlow within the spawned process upon creation and putting ```global tf``` right after the import.  And it actually works. You're a beast! I've spent months trying to figure this out, it's a pity this post will be removed as I think a lot of people would actually need to see this.  
  > **username_1**: Oh Really? :D That's amazing! I'm glad I was helpful Maybe the post stays I don't know, usually these post get removed but who reads the manual anyways^^  
  
  ---
  
  ## 0 answers
  
  