Answers.Body: <p>Never mind, I figured it out. As it turned out I had to write to file like so and extra quotes <code>(&quot;\'&quot;)</code> around samples were not needed. Also shell=True does not work well with lists, only strings:</p>
  <pre class="lang-py prettyprint-override"><code>def remove_individuals(chr_samples):
      my_env = os.environ.copy()
      my_env['PATH'] = f'/home/aydar/bin/bcftools_1.11/bin/:{my_env[&quot;PATH&quot;]}'
      def remove_indiv(chr, indiv):
          in_vcf = f'data.filtered-updated-{chr}.vcf.gz'
          out_vcf = f'data.filtered-updated-{chr}-faulty-samp-rem.vcf.gz'
          drop_samples = &quot;^&quot;+&quot;,&quot;.join(indiv)
          out = subprocess.run(
              ['bcftools', 'view', in_vcf, 
               '--samples', drop_samples, '-Oz'],
              env=my_env,
              capture_output=True
          )
          with open(out_vcf, 'wb') as f:
              f.write(out.stdout)
      chr_samples.apply(lambda x: remove_indiv(x.index[0], list(x.values[0])))
  </code></pre>
  <p>Remember to use <code>'wb'</code> if you like me and use <code>Oz</code> output option, because it returns bgzipped file.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T12:59:45.103Z
Answers.Text: thanks for accepting your answer and please feel free to ask a question any time here.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-27T21:44:24.477Z
Answers.LastActivityDate: 2023-08-28T17:08:32.300Z
Answers.LastEditDate: 2023-08-28T17:08:32.300Z
Answers.UserDisplayName: YKY
Body: <p>I am trying to remove samples from a chromosome vcf file. I wrote a function that takes chromosome number and a list of samples to remove. When I try to run bcftools using subprocess module it only runs bcftools, as if I was running <code>subprocess.run([bcftools], shell=True, env=my_env)</code> instead of:</p>
  <pre class="lang-py prettyprint-override"><code>subprocess.run(['bcftools', 'view', in_vcf, '--samples', drop_samples, '-Oz', '&gt;', out_vcf], shell=True, env=my_env)
  </code></pre>
  <p>The function:</p>
  <pre class="lang-py prettyprint-override"><code>def remove_individuals(chr_samples):
      my_env = os.environ.copy()
      my_env['PATH'] = f'/home/aydar/bin/bcftools_1.11/bin/:{my_env[&quot;PATH&quot;]}'
      def remove_indiv(chr, indiv):
          in_vcf = f'data.filtered-updated-{chr}.vcf.gz'
          out_vcf = f'data.filtered-updated-{chr}-faulty-samp-rem.vcf.gz'
          drop_samples = &quot;\'^&quot;+&quot;,&quot;.join(indiv)+&quot;\'&quot;
          proc = subprocess.run(
              ['bcftools', 'view', in_vcf, 
               '--samples', drop_samples, '-Oz', '&gt;', out_vcf],
              shell=True,
              env=my_env
          )
          print(proc.args)
      chr_samples.apply(lambda x: remove_indiv(x.index[0], list(x.values[0])))
  </code></pre>
  <p>function input:</p>
  <pre><code>13,[2500000075_2500000075,2000000171_2000000171,1800000179_1800000179,330134048_330134048]
  </code></pre>
  <p>When I run the same command in the terminal everything works.</p>
  <p>The command I test with in terminal:</p>
  <pre><code>bcftools view data.filtered-updated-13.vcf.gz --samples '^2500000075_2500000075,2000000171_2000000171,1800000179_1800000179,330134048_330134048' -Oz &gt; out_test_vcf.gz
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-27T22:22:22.920Z
Comments.Text: Thanks for your question and answer. Using `popen` is the standard approach here than using `run`. Anyway could you very kindly mark your answer as "accepted"? This help site stats and prevents the question from being re-introduced as unresolved later.
Comments.UserDisplayName: M__
Tags: <python><bcftools>
Title: Can not launch bcftools using python's subprocess module, as it only accepts first command of commands list
ThreadText: # Can not launch bcftools using python's subprocess module, as it only accepts first command of commands list
  
  (Asked by: username_0 on 2023-08-27)
  
  I am trying to remove samples from a chromosome vcf file. I wrote a function that takes chromosome number and a list of samples to remove. When I try to run bcftools using subprocess module it only runs bcftools, as if I was running `subprocess.run([bcftools], shell=True, env=my_env)` instead of:
  
  ```py
  subprocess.run(['bcftools', 'view', in_vcf, '--samples', drop_samples, '-Oz', '>', out_vcf], shell=True, env=my_env)
  
  ```
  
  The function:
  
  ```py
  def remove_individuals(chr_samples):
      my_env = os.environ.copy()
      my_env['PATH'] = f'/home/aydar/bin/bcftools_1.11/bin/:{my_env["PATH"]}'
      def remove_indiv(chr, indiv):
          in_vcf = f'data.filtered-updated-{chr}.vcf.gz'
          out_vcf = f'data.filtered-updated-{chr}-faulty-samp-rem.vcf.gz'
          drop_samples = "\'^"+",".join(indiv)+"\'"
          proc = subprocess.run(
              ['bcftools', 'view', in_vcf, 
               '--samples', drop_samples, '-Oz', '>', out_vcf],
              shell=True,
              env=my_env
          )
          print(proc.args)
      chr_samples.apply(lambda x: remove_indiv(x.index[0], list(x.values[0])))
  
  ```
  
  function input:
  
  ```
  13,[2500000075_2500000075,2000000171_2000000171,1800000179_1800000179,330134048_330134048]
  
  ```
  
  When I run the same command in the terminal everything works.
  
  The command I test with in terminal:
  
  ```
  bcftools view data.filtered-updated-13.vcf.gz --samples '^2500000075_2500000075,2000000171_2000000171,1800000179_1800000179,330134048_330134048' -Oz > out_test_vcf.gz
  
  ```
  
  **Tags:** python, bcftools
  
  > **Comments:**
  > **username_1**: Thanks for your question and answer. Using `popen` is the standard approach here than using `run`. Anyway could you very kindly mark your answer as "accepted"? This help site stats and prevents the question from being re-introduced as unresolved later.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-08-27. Score: 3 | Accepted answer
  
  Never mind, I figured it out. As it turned out I had to write to file like so and extra quotes `("\'")` around samples were not needed. Also shell=True does not work well with lists, only strings:
  
  ```py
  def remove_individuals(chr_samples):
      my_env = os.environ.copy()
      my_env['PATH'] = f'/home/aydar/bin/bcftools_1.11/bin/:{my_env["PATH"]}'
      def remove_indiv(chr, indiv):
          in_vcf = f'data.filtered-updated-{chr}.vcf.gz'
          out_vcf = f'data.filtered-updated-{chr}-faulty-samp-rem.vcf.gz'
          drop_samples = "^"+",".join(indiv)
          out = subprocess.run(
              ['bcftools', 'view', in_vcf, 
               '--samples', drop_samples, '-Oz'],
              env=my_env,
              capture_output=True
          )
          with open(out_vcf, 'wb') as f:
              f.write(out.stdout)
      chr_samples.apply(lambda x: remove_indiv(x.index[0], list(x.values[0])))
  
  ```
  
  Remember to use `'wb'` if you like me and use `Oz` output option, because it returns bgzipped file.
  
  > **Comments:**
  > **username_1**: thanks for accepting your answer and please feel free to ask a question any time here.  
  
  ---
  
  