Answers: <p>Since you're talking about CI/CD I presume you have the possibility to automate the model trainings in those configurations. Let's call the scripts able to do that <code>train_model_config_A</code>, <code>train_model_config_B</code>, etc.</p>
  
  <p>Then you could have a wrapper script which checks an environment variable used to select which client environment you desire and invokes the corresponding <code>train_model_config_&lt;blah&gt;</code> script. Ideally translating the outcome of the training (whatever that is) into one or more results of the pass/fail type. Then such wrapper script can be integrated in a CI/CD pipeline like a custom test step/stage (or even a build one, if it produces any artifacts you might want to archive). Just like any test executed on a testbed incorporating some non-generic piece of hardware. In other words the CPU cluster makes no real difference.</p>
  
  <p>You might not need to install the slave directly on the GPU cluster if the <code>train_model_config_&lt;blah&gt;</code> scripts (and thus the wrapper script as well) can be executed on some other hosts and remotely control the GPUs - you can then have the slave on some other host, leaving the GPU cluster free to only do its stuff.</p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-25T08:12:43.343Z
Answers: 2018-02-25T08:12:43.343Z
Answers: Dan Cornilescu
Body: <p>With typical continuous integration environments, you configure an environment capable to execute compilation and test batches (agent, slave..) coordinated by a scheduler (master, server..)</p>
  
  <p>But what if your "client" environment is a Graphics Processing Unit (GPU) cluster used to perform model trainings in different configurations? Is there any difference or would you just for example let the head cluster node incorporate a Jenkins slave? (or Bamboo agent etc) </p>
  
Comments: CC BY-SA 3.0
Comments: 2017-09-14T07:52:32.413Z
Comments: You have a GPU cluster at the moment?
Comments: 030
Comments: CC BY-SA 3.0
Comments: 2017-09-14T08:00:28.213Z
Comments: please elaborate your question better - I tend to answer yes
Comments: Ta Mu
Comments: CC BY-SA 3.0
Comments: 2017-09-14T08:20:01.527Z
Comments: `But what if your "client" environment is a GPU cluster` Is it a hypothesis or is there a real GPU cluster?
Comments: 030
Comments: CC BY-SA 3.0
Comments: 2017-09-14T08:25:17.267Z
Comments: real, indeed it is
Comments: Ta Mu
Comments: CC BY-SA 3.0
Comments: 2017-09-14T08:38:04.287Z
Comments: GPU cluster = https://en.wikipedia.org/wiki/GPU_cluster?
Comments: 030
Comments: CC BY-SA 3.0
Comments: 2017-09-14T08:38:17.090Z
Comments: `model trainings in different configurations` Could you give examples?
Comments: 030
Comments: CC BY-SA 3.0
Comments: 2017-09-14T08:51:29.470Z
Comments: Yes. Many configurations means just using different tools/frameworks, or different sets of input data.
Comments: Ta Mu
Comments: CC BY-SA 3.0
Comments: 2017-09-14T08:56:01.687Z
Comments: But what exactly is the problem? It's just a resource, it shouldn't matter if it's a GPU cluster or a self-driving car you run tests on. You just have to schedule it's use.
Comments: Jakub Kania
Comments: CC BY-SA 3.0
Comments: 2017-09-20T06:23:11.253Z
Comments: @JakubKania: this should be an answer IMHO.
Comments: Dan Cornilescu
Comments: CC BY-SA 3.0
Comments: 2017-09-20T09:00:16.687Z
Comments: @DanCornilescu To qualify as a full asnwer it would have to be expanded much. That comment is only fishing for more details.
Comments: Jakub Kania
Tags: <continuous-integration><cloud>
Title: CI/CD with a GPU cluster
UserDisplayName: Ta Mu
ThreadText: # CI/CD with a GPU cluster
  
  (Asked by: username_0 on 2017-09-14)
  
  With typical continuous integration environments, you configure an environment capable to execute compilation and test batches (agent, slave..) coordinated by a scheduler (master, server..)
  
  But what if your "client" environment is a Graphics Processing Unit (GPU) cluster used to perform model trainings in different configurations? Is there any difference or would you just for example let the head cluster node incorporate a Jenkins slave? (or Bamboo agent etc)
  
  **Tags:** continuous-integration, cloud
  
  > **Comments:**
  > **username_1**: You have a GPU cluster at the moment?  
  > **username_0**: please elaborate your question better - I tend to answer yes  
  > **username_1**: `But what if your "client" environment is a GPU cluster` Is it a hypothesis or is there a real GPU cluster?  
  > **username_0**: real, indeed it is  
  > **username_1**: GPU cluster = https://en.wikipedia.org/wiki/GPU_cluster?  
  > **username_1**: `model trainings in different configurations` Could you give examples?  
  > **username_0**: Yes. Many configurations means just using different tools/frameworks, or different sets of input data.  
  > **username_2**: But what exactly is the problem? It's just a resource, it shouldn't matter if it's a GPU cluster or a self-driving car you run tests on. You just have to schedule it's use.  
  > **username_3**: @username_2: this should be an answer IMHO.  
  > **username_2**: @username_3 To qualify as a full asnwer it would have to be expanded much. That comment is only fishing for more details.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2018-02-25. Score: 2
  
  Since you're talking about CI/CD I presume you have the possibility to automate the model trainings in those configurations. Let's call the scripts able to do that `train_model_config_A`, `train_model_config_B`, etc.
  
  Then you could have a wrapper script which checks an environment variable used to select which client environment you desire and invokes the corresponding `train_model_config_<blah>` script. Ideally translating the outcome of the training (whatever that is) into one or more results of the pass/fail type. Then such wrapper script can be integrated in a CI/CD pipeline like a custom test step/stage (or even a build one, if it produces any artifacts you might want to archive). Just like any test executed on a testbed incorporating some non-generic piece of hardware. In other words the CPU cluster makes no real difference.
  
  You might not need to install the slave directly on the GPU cluster if the `train_model_config_<blah>` scripts (and thus the wrapper script as well) can be executed on some other hosts and remotely control the GPUs - you can then have the slave on some other host, leaving the GPU cluster free to only do its stuff.
  
  
  ---
  
  