Answers.Body: <p>You will have to define the header either from an existing VCF file or hardcoded into you python script.  Then write the header the the output VCF file then write the dataframe to the same file with the mode options set to 'a' to append to the end of the file.  </p>
  
  <p>You will need to order the dataframe columns to match the VCF spec and insert the empty/ default <code>ID</code>, <code>FILTER</code>, <code>INFO</code> and <code>QUAL</code> columns</p>
  
  <pre><code>import pandas as pd 
  
  example_dict = {'CHROM':[20, 20, 20, 20, 20], 'POS':[14370, 17330, 1110696, 1230237, 1234567], 'REF':['G', 'T', 'A', 'T', 'GTCT'], 'ALT':['A', 'A', 'G', 'C', 'G']}
  
  df = pd.DataFrame(example_dict)
  
  df['ID'] = ""
  df['QUAL'] = 40
  df['FILTER'] = ""
  df['INFO'] = ""
  
  df = df[['CHROM', 'POS', 'ID', 'REF', 'ALT', 'QUAL', 'FILTER', 'INFO']] 
  
  header = """##fileformat=VCFv4.1
  ##fileDate=20090805
  ##source=myImputationProgramV3.1
  ##reference=file:///seq/references/
  #CHROM POS ID REF ALT QUAL FILTER INFO
  """
  
  output_VCF = "myfile.vcf"
  with open(output_VCF, 'w') as vcf:
      vcf.write(header)
  
  df.to_csv(output_VCF, sep="\t", mode='a', index=False)
  </code></pre>
  
  <p>Then <code>myfile.vcf</code> looks like </p>
  
  <pre><code>##fileformat=VCFv4.1
  ##fileDate=20090805
  ##source=myImputationProgramV3.1
  ##reference=file:///seq/references/
  #CHROM POS ID REF ALT QUAL FILTER INFO
  CHROM   POS ID  REF ALT QUAL    FILTER  INFO
  20  14370       G   A   40      
  20  17330       T   A   40      
  20  1110696     A   G   40      
  20  1230237     T   C   40      
  20  1234567     GTCT    G   40
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-15T20:01:31.183Z
Answers.Text: Thanks for the help. BTW, is fileDate year-month-day or year-day-month? I'm guessing there's no standard.
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-15T21:19:57.843Z
Answers.Text: fileDate is not a required header field you can remove it it you want
Answers.UserDisplayName: Bioathlete
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-15T19:43:38.043Z
Answers.LastActivityDate: 2017-12-15T19:43:38.043Z
Answers.UserDisplayName: Bioathlete
Body: <p>Let's say I have a pandas dataframe with fields <code>CHROM</code>, <code>POS</code>, <code>ALT</code>, <code>REF</code>. In this special case, I also wouldn't care about <code>ID</code>or <code>FILTER</code>, <code>INFO</code> could be blank (or meaningless) and we'll write <code>QUAL</code> as each 40. </p>
  
  <pre><code>import pandas as pd
  example_dict = {'CHROM':[20, 20, 20, 20, 20], 'POS':[14370, 17330, 1110696, 1230237, 1234567], 'REF':['G', 'T', 'A', 'T', 'GTCT'], 'ALT':['A', 'A', 'G', 'C', 'G']}
  
  df = pd.DataFrame(example_dict)
  print(df)
  
     CHROM      POS   REF ALT
  0     20    14370     G   A
  1     20    17330     T   A
  2     20  1110696     A   G
  3     20  1230237     T   C
  4     20  1234567  GTCT   G
  </code></pre>
  
  <p>In this case, saving the output with the extension <code>.vcf</code> via <code>df.to_csv("myfile.vcf")</code> with the other required fields would created the tab-delimited format of the VCF, but it wouldn't include the mandatory header. <a href="https://samtools.github.io/hts-specs/VCFv4.1.pdf" rel="noreferrer">https://samtools.github.io/hts-specs/VCFv4.1.pdf</a></p>
  
  <pre><code>##fileformat=VCFv4.1
  ##fileDate=20090805
  ##source=myImputationProgramV3.1
  ##reference=file:///seq/references/
  ...
  #CHROM POS ID REF ALT QUAL FILTER INFO
  </code></pre>
  
  <p>The most popular package to work with VCF's appears to be <code>PyVCF</code>: <a href="https://pypi.python.org/pypi/PyVCF" rel="noreferrer">https://pypi.python.org/pypi/PyVCF</a></p>
  
  <p>However, one needs to begin with a VCF as a template, and add to this (from the documentation):</p>
  
  <pre><code>vcf_reader = vcf.Reader(filename='vcf/test/tb.vcf.gz')
  vcf_writer = vcf.Writer(open('/dev/null', 'w'), vcf_reader)
  for record in vcf_reader:
       vcf_writer.write_record(record)
  </code></pre>
  
  <p>It appears the only information necessary from <code>vcf.Reader(filename='vcf/test/tb.vcf.gz')</code> is the metadata? </p>
  
  <pre><code>print(vcf_reader.metadata)
  OrderedDict([('fileformat', 'VCFv4.0'), ('fileDate', '20090805'), ('source', ['myImputationProgramV3.1']), ('reference', '1000GenomesPilot-NCBI36'), ('phasing', ['partial'])])
  </code></pre>
  
  <p>Otherwise, <code>tb.vcf.gz</code> already has variants, which I think means I would need to feed in a VCF with no variants?:</p>
  
  <pre><code>for record in vcf_reader:
       print(record)
  
  Record(CHROM=20, POS=14370, REF=G, ALT=[A])
  Record(CHROM=20, POS=17330, REF=T, ALT=[A])
  Record(CHROM=20, POS=1110696, REF=A, ALT=[G, T])
  Record(CHROM=20, POS=1230237, REF=T, ALT=[None])
  Record(CHROM=20, POS=1234567, REF=GTCT, ALT=[G, GTACT])
  </code></pre>
  
Tags: <python><vcf>
Title: Using Python, how to convert a pandas DataFrame into a VCF?
ThreadText: # Using Python, how to convert a pandas DataFrame into a VCF?
  
  (Asked by: username_0 on 2017-12-15)
  
  Let's say I have a pandas dataframe with fields `CHROM`, `POS`, `ALT`, `REF`. In this special case, I also wouldn't care about `ID`or `FILTER`, `INFO` could be blank (or meaningless) and we'll write `QUAL` as each 40.
  
  ```
  import pandas as pd
  example_dict = {'CHROM':[20, 20, 20, 20, 20], 'POS':[14370, 17330, 1110696, 1230237, 1234567], 'REF':['G', 'T', 'A', 'T', 'GTCT'], 'ALT':['A', 'A', 'G', 'C', 'G']}
  
  df = pd.DataFrame(example_dict)
  print(df)
  
     CHROM      POS   REF ALT
  0     20    14370     G   A
  1     20    17330     T   A
  2     20  1110696     A   G
  3     20  1230237     T   C
  4     20  1234567  GTCT   G
  
  ```
  
  In this case, saving the output with the extension `.vcf` via `df.to_csv("myfile.vcf")` with the other required fields would created the tab-delimited format of the VCF, but it wouldn't include the mandatory header. <https://samtools.github.io/hts-specs/VCFv4.1.pdf>
  
  ```
  ##fileformat=VCFv4.1
  ##fileDate=20090805
  ##source=myImputationProgramV3.1
  ##reference=file:///seq/references/
  ...
  #CHROM POS ID REF ALT QUAL FILTER INFO
  
  ```
  
  The most popular package to work with VCF's appears to be `PyVCF`: <https://pypi.python.org/pypi/PyVCF>
  
  However, one needs to begin with a VCF as a template, and add to this (from the documentation):
  
  ```
  vcf_reader = vcf.Reader(filename='vcf/test/tb.vcf.gz')
  vcf_writer = vcf.Writer(open('/dev/null', 'w'), vcf_reader)
  for record in vcf_reader:
       vcf_writer.write_record(record)
  
  ```
  
  It appears the only information necessary from `vcf.Reader(filename='vcf/test/tb.vcf.gz')` is the metadata?
  
  ```
  print(vcf_reader.metadata)
  OrderedDict([('fileformat', 'VCFv4.0'), ('fileDate', '20090805'), ('source', ['myImputationProgramV3.1']), ('reference', '1000GenomesPilot-NCBI36'), ('phasing', ['partial'])])
  
  ```
  
  Otherwise, `tb.vcf.gz` already has variants, which I think means I would need to feed in a VCF with no variants?:
  
  ```
  for record in vcf_reader:
       print(record)
  
  Record(CHROM=20, POS=14370, REF=G, ALT=[A])
  Record(CHROM=20, POS=17330, REF=T, ALT=[A])
  Record(CHROM=20, POS=1110696, REF=A, ALT=[G, T])
  Record(CHROM=20, POS=1230237, REF=T, ALT=[None])
  Record(CHROM=20, POS=1234567, REF=GTCT, ALT=[G, GTACT])
  
  ```
  
  **Tags:** python, vcf
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-12-15. Score: 8 | Accepted answer
  
  You will have to define the header either from an existing VCF file or hardcoded into you python script. Then write the header the the output VCF file then write the dataframe to the same file with the mode options set to 'a' to append to the end of the file.
  
  You will need to order the dataframe columns to match the VCF spec and insert the empty/ default `ID`, `FILTER`, `INFO` and `QUAL` columns
  
  ```
  import pandas as pd 
  
  example_dict = {'CHROM':[20, 20, 20, 20, 20], 'POS':[14370, 17330, 1110696, 1230237, 1234567], 'REF':['G', 'T', 'A', 'T', 'GTCT'], 'ALT':['A', 'A', 'G', 'C', 'G']}
  
  df = pd.DataFrame(example_dict)
  
  df['ID'] = ""
  df['QUAL'] = 40
  df['FILTER'] = ""
  df['INFO'] = ""
  
  df = df[['CHROM', 'POS', 'ID', 'REF', 'ALT', 'QUAL', 'FILTER', 'INFO']] 
  
  header = """##fileformat=VCFv4.1
  ##fileDate=20090805
  ##source=myImputationProgramV3.1
  ##reference=file:///seq/references/
  #CHROM POS ID REF ALT QUAL FILTER INFO
  """
  
  output_VCF = "myfile.vcf"
  with open(output_VCF, 'w') as vcf:
      vcf.write(header)
  
  df.to_csv(output_VCF, sep="\t", mode='a', index=False)
  
  ```
  
  Then `myfile.vcf` looks like
  
  ```
  ##fileformat=VCFv4.1
  ##fileDate=20090805
  ##source=myImputationProgramV3.1
  ##reference=file:///seq/references/
  #CHROM POS ID REF ALT QUAL FILTER INFO
  CHROM   POS ID  REF ALT QUAL    FILTER  INFO
  20  14370       G   A   40      
  20  17330       T   A   40      
  20  1110696     A   G   40      
  20  1230237     T   C   40      
  20  1234567     GTCT    G   40
  
  ```
  
  > **Comments:**
  > **username_0**: Thanks for the help. BTW, is fileDate year-month-day or year-day-month? I'm guessing there's no standard.  
  > **username_1**: fileDate is not a required header field you can remove it it you want  
  
  ---
  
  