Answers.Body: <p>To make disjoint intervals, you could use BEDOPS <code>bedops --partition</code>, piping to <code>bedmap --mean</code> to get the mean signal over disjoint regions.</p>
  
  <p>Starting with the input bedgraph file, convert it to five-column BED with GNU <code>awk</code>, putting the signal in the fifth column per UCSC convention:</p>
  
  <pre><code>$ awk -vOFS="\t" '{ print $1, $2, $3, ".", $4 }' /tmp/in.bedgraph | sort-bed - &gt; /tmp/in.bed
  $ cat in.bed
  chr9    8855925 8986425 .   -1
  chr9    8855965 8986325 .   0
  chr9    8893857 9022105 .   -1
  </code></pre>
  
  <p>The disjoint set from these intervals would look like this:</p>
  
  <pre><code>$ bedops --partition /tmp/in.bed
  chr9    8855925 8855965
  chr9    8855965 8893857
  chr9    8893857 8986325
  chr9    8986325 8986425
  chr9    8986425 9022105
  </code></pre>
  
  <p>We can pipe these intervals to <code>bedmap --mean</code> with the original five-column BED file as a "map" file, calculating the mean signal over the disjoint intervals:</p>
  
  <pre><code>$ bedops --partition /tmp/in.bed | bedmap --echo --mean --delim '\t' - /tmp/in.bed &gt; /tmp/answer.bedgraph
  $ cat /tmp/answer.bedgraph
  chr9    8855925 8855965 -1.000000
  chr9    8855965 8893857 -0.500000
  chr9    8893857 8986325 -0.666667
  chr9    8986325 8986425 -1.000000
  chr9    8986425 9022105 -1.000000
  </code></pre>
  
  <p>The <code>--mean</code> option calculates the arithmetic mean. Other aggregation functions are available to process the signal or score (fifth) column in map BED files, like <code>--stdev</code>, <code>--median</code>, <code>--mad</code>, etc. </p>
  
  <p>When calculating statistics, the default precision is six digits. The <code>--prec &lt;n&gt;</code> option can be used to specify desired precision in the score result.</p>
  
  <p>See <code>bedmap --help</code> or the online documentation for a full description.</p>
  
  <p>To avoid making and cleaning up intermediate files, <code>bash</code> process substitutions could be used to generate a one-liner:</p>
  
  <pre><code>$ bedops --partition &lt;(sort-bed in.bedgraph | awk -vOFS="\t" '{ print $1, $2, $3, ".", $4 }') | bedmap --echo --mean --delim '\t' - &lt;(sort-bed in.bedgraph | awk -vOFS="\t" '{ print $1, $2, $3, ".", $4 }') &gt; answer.bedgraph
  </code></pre>
  
  <p>This can be useful for cleaner pipelines or with processing very large, whole-genome scale inputs on systems with limited disk space.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-24T19:35:14.300Z
Answers.Text: Thanks for the detailed answer. I got confused with the numbers, I should have used very simple segment start and end. Your answer has the expected solution. I'll update my question.
Answers.UserDisplayName: gc5
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-24T19:49:17.303Z
Answers.Text: On macOS, use `gawk` instead of `awk` since the `-v` option is invalid. With homebrew `brew install gawk`.
Answers.UserDisplayName: gc5
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-24T18:46:42.430Z
Answers.LastActivityDate: 2018-05-24T20:52:17.630Z
Answers.LastEditDate: 2018-05-24T20:52:17.630Z
Answers.UserDisplayName: Alex Reynolds
Body: <p>I have a bed-like file in which I have several overlapping regions with associated features. I want to split the overlapping regions into disjoint regions in order to process the file (e.g. with pandas), group by regions and compute an aggregate function.</p>
  
  <p>The input data format is the following (overlapping are marked with <code>*</code>):</p>
  
  <pre><code>chr9    8855925    8986425    -1  *
  chr9    8855965    8986325     0  *
  chr9    8893857    9022105    -1
  </code></pre>
  
  <p>I want to split into disjoint intervals, then I want to groupby per-interval and compute an aggregation function (e.g. the mean of the last field):</p>
  
  <pre><code>chr9    8855925    8855965    -1
  chr9    8855965    8893857  -0.5
  chr9    8893857    8986325 -0.67
  chr9    8986325    8986425    -1
  chr9    8986425    9022105    -1
  </code></pre>
  
  <p>I was thinking of using <a href="https://bedtools.readthedocs.io/en/latest/" rel="nofollow noreferrer">bedtools</a> and <a href="https://pandas.pydata.org/" rel="nofollow noreferrer">pandas</a> to aggregate. However, I was unable to find the right tool to do this job in the bedtools/pybedtools documentation.</p>
  
Tags: <bed><bedtools><pybedtools>
Title: How to split bedfile into non-overlapping regions and compute aggregation function on duplicate segments
ThreadText: # How to split bedfile into non-overlapping regions and compute aggregation function on duplicate segments
  
  (Asked by: username_0 on 2018-05-24)
  
  I have a bed-like file in which I have several overlapping regions with associated features. I want to split the overlapping regions into disjoint regions in order to process the file (e.g. with pandas), group by regions and compute an aggregate function.
  
  The input data format is the following (overlapping are marked with `*`):
  
  ```
  chr9    8855925    8986425    -1  *
  chr9    8855965    8986325     0  *
  chr9    8893857    9022105    -1
  
  ```
  
  I want to split into disjoint intervals, then I want to groupby per-interval and compute an aggregation function (e.g. the mean of the last field):
  
  ```
  chr9    8855925    8855965    -1
  chr9    8855965    8893857  -0.5
  chr9    8893857    8986325 -0.67
  chr9    8986325    8986425    -1
  chr9    8986425    9022105    -1
  
  ```
  
  I was thinking of using [bedtools](https://bedtools.readthedocs.io/en/latest/) and [pandas](https://pandas.pydata.org/) to aggregate. However, I was unable to find the right tool to do this job in the bedtools/pybedtools documentation.
  
  **Tags:** bed, bedtools, pybedtools
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-05-24. Score: 3 | Accepted answer
  
  To make disjoint intervals, you could use BEDOPS `bedops --partition`, piping to `bedmap --mean` to get the mean signal over disjoint regions.
  
  Starting with the input bedgraph file, convert it to five-column BED with GNU `awk`, putting the signal in the fifth column per UCSC convention:
  
  ```
  $ awk -vOFS="\t" '{ print $1, $2, $3, ".", $4 }' /tmp/in.bedgraph | sort-bed - > /tmp/in.bed
  $ cat in.bed
  chr9    8855925 8986425 .   -1
  chr9    8855965 8986325 .   0
  chr9    8893857 9022105 .   -1
  
  ```
  
  The disjoint set from these intervals would look like this:
  
  ```
  $ bedops --partition /tmp/in.bed
  chr9    8855925 8855965
  chr9    8855965 8893857
  chr9    8893857 8986325
  chr9    8986325 8986425
  chr9    8986425 9022105
  
  ```
  
  We can pipe these intervals to `bedmap --mean` with the original five-column BED file as a "map" file, calculating the mean signal over the disjoint intervals:
  
  ```
  $ bedops --partition /tmp/in.bed | bedmap --echo --mean --delim '\t' - /tmp/in.bed > /tmp/answer.bedgraph
  $ cat /tmp/answer.bedgraph
  chr9    8855925 8855965 -1.000000
  chr9    8855965 8893857 -0.500000
  chr9    8893857 8986325 -0.666667
  chr9    8986325 8986425 -1.000000
  chr9    8986425 9022105 -1.000000
  
  ```
  
  The `--mean` option calculates the arithmetic mean. Other aggregation functions are available to process the signal or score (fifth) column in map BED files, like `--stdev`, `--median`, `--mad`, etc.
  
  When calculating statistics, the default precision is six digits. The `--prec <n>` option can be used to specify desired precision in the score result.
  
  See `bedmap --help` or the online documentation for a full description.
  
  To avoid making and cleaning up intermediate files, `bash` process substitutions could be used to generate a one-liner:
  
  ```
  $ bedops --partition <(sort-bed in.bedgraph | awk -vOFS="\t" '{ print $1, $2, $3, ".", $4 }') | bedmap --echo --mean --delim '\t' - <(sort-bed in.bedgraph | awk -vOFS="\t" '{ print $1, $2, $3, ".", $4 }') > answer.bedgraph
  
  ```
  
  This can be useful for cleaner pipelines or with processing very large, whole-genome scale inputs on systems with limited disk space.
  
  > **Comments:**
  > **username_0**: Thanks for the detailed answer. I got confused with the numbers, I should have used very simple segment start and end. Your answer has the expected solution. I'll update my question.  
  > **username_0**: On macOS, use `gawk` instead of `awk` since the `-v` option is invalid. With homebrew `brew install gawk`.  
  
  ---
  
  