Answers.Body: <p>You asked</p>
  <blockquote>
  <p>Does anyone know any tools that can intersect two bigbed files</p>
  </blockquote>
  <p>...but it appears you really wanted to subset a single very large bigbed file based on the contents of a much smaller bed file.</p>
  <p>I disagree with your summation &quot;there is not a very efficient way to work directly with the bigbed file&quot;.  I see two options.</p>
  <p>The first is to repeatedly call <code>bigBedTobed</code> once for each row in your .bed file.  If you have multiple cores, this can be parallelized using <a href="https://www.gnu.org/software/parallel/" rel="nofollow noreferrer">GNU parallel</a>.  The following does that, emitting each overlapping result with additional column being the index into your .bed file from which the result stems.</p>
  <pre><code>cat your.bed | parallel --lb -j --colsep '\t' bigBedToBed -chrom={1} -start={2} -end={3}  your.bb  /dev/stdout '|' sed -e 's/$/\\t${PARALLEL_SEQ}/'
  </code></pre>
  <p>Depending upon your application, there is a second approach which uses R/BioConductor's <code>rtracklayer::import.bb</code> which can be parallelized across (chunks of) your .bed file.</p>
  <p>I considered these approaches in what I expect is an almost identical situation:</p>
  <p>My bigbed holds genome-wide transcription factor binding site predictions from <a href="https://jaspar.genereg.net/genome-tracks/#ucsc_tracks" rel="nofollow noreferrer">JASPAR Genome Browser tracks</a> as produced by <a href="https://github.com/wassermanlab" rel="nofollow noreferrer">wassermanlab / JASPAR-UCSC-tracks</a>.</p>
  <p>My bed file holds the loci of ~23K putative enhancers as determined by an analysis of regions of accessible chromatin from ATAC-Seq and various ChIP marks.</p>
  <p>In my case I wished to produce a sparse Matrix with one row for each putative enhancer, and one column for each JASPAR motif (which will form the basis of downstream enrichment analyses).</p>
  <pre><code>bbf&lt;-BigBedFile('path/to/your.bb')
  gr&lt;-import.bed('path/to/your.bed')
  ## I found I had to harmonize seqlevels (e.g. chromosome identifiers) to make this work.  YMMV
  gr&lt;-renameSeqlevels(gr,paste0('chr',gsub('^(K.*)\\.(\\d+)$','Un_\\1v\\2',seqlevels(gr))))
  mm&lt;-motifMatrix(gr,bbf)
  </code></pre>
  <p>which results in a sparse Matrix, viz:</p>
  <pre><code>&gt; rownames(mm)&lt;-as.character(gr) # optional.
  &gt; mm[1:5,1:5]
  5 x 5 sparse Matrix of class &quot;dgCMatrix&quot;
                   MA0002.2 MA0003.4 MA0004.1 MA0006.1 MA0007.3
  chr1:11156-12180       14        .        2       28        2
  chr1:17836-20023       25        .       27       63        2
  chr1:22495-24367       22        1       22      137        2
  chr1:27533-28038        2        2        4        7        .
  chr1:36549-37079        6        .        2       41        .
  </code></pre>
  <p>This depends upon the following definition:</p>
  <pre><code>motifMatrix&lt;-function(gr,bbf,motifID=c('name','TFName')[[1]],mc.jobs=100) {
    ## produce a sparse Matrix with one row for each region in
    ## GenomicRanges gr, and one column for each JASPAR motif indicated
    ## as overlapping &lt;gr&gt; accorinding to BigBedFile &lt;bbf&gt;.  Default to
    ## using the 'name' column as the motif identifier, but allow for
    ## possibly using another, such as 'TFName' (as used by 2022 version
    ## of
    ## https://jaspar.genereg.net/genome-tracks/#ucsc_tracks). Parallelize
    ## into mc.jobs queries, defaulting to 100, which was found suitable
    ## for ~25000 loci in &lt;gr&gt; against JASPAR matches to zebrafish
    ## danRer11 genome running on a few dozen cores.  In general,
    ## knowing how &quot;big&quot; each parallel job *should* be depends on
    ## available RAM and cores, and number of loci in &lt;gr&gt;.
    ##
    ## AUTHOR: malcolm_cook@stowers.org
    ##
    ## NB: you must install a version of `rtracklayer` which addresses
    ## issue [import\.bb returning incorrect ranges &amp; results as
    ## compared with kent tools
    ## bigBedToBed](https://github.com/lawremi/rtracklayer/issues/59)
    ## (such as by `install_github('lawremi/rtracklayer')`)
    si&lt;-parallel::splitIndices(length(gr),mc.jobs)
    mm&lt;-mclapply(si,function(i) {
      gri&lt;-gr[i]
      m&lt;-import.bb(bbf,selection=BigBedSelection(gri, colnames = c(motifID)))
      fo&lt;-findOverlaps(gri,m)
      xt&lt;-
        xtabs(~.,data=list(locus=i[queryHits(fo)],motif=mcols(m[subjectHits(fo)])[[motifID]]),sparse=TRUE) 
      xt
    })
    mm&lt;-
      ## rBind.fill(mm) ## NOT!  It does NOT preserve dimnames and
      ## returns matrix when given Matrix.  Issue reported:
      ## [[https://github.com/cvarrichio/Matrix.utils/issues/5][rBind.fill
      ## on a list of Matrix arguably should return Matrix but returns
      ## matrix #5]]
      Reduce(rBind.fill,mm) # which does preserve dimnames and return Matrix
    mm
  }
  </code></pre>
  <p>NB: If you choose this approach, you MUST install a version of <code>rtracklayer</code> which addresses issue the <a href="https://github.com/lawremi/rtracklayer/issues/59" rel="nofollow noreferrer">import.bb returning incorrect ranges &amp; results as compared with kent tools bigBedToBed</a> (such as by <code>install_github('lawremi/rtracklayer')</code>)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-21T09:28:10.917Z
Answers.Text: Yes Malcook, this is exactly what I want at first! Thank you so much!
Answers.UserDisplayName: Binhuan Sun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-29T07:26:52.447Z
Answers.Text: glad to help - would you consider to change the accepted solution to mine?
Answers.UserDisplayName: malcook
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-30T09:09:41.387Z
Answers.Text: This is the accepted answer for sure @BinhuanSun
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-20T20:30:49.180Z
Answers.LastActivityDate: 2022-02-20T20:36:13.047Z
Answers.LastEditDate: 2022-02-20T20:36:13.047Z
Answers.UserDisplayName: malcook
Answers.Body: <p>No matter how you approach this problem 110+ GB is big:</p>
  <p>Your options are:</p>
  <ol>
  <li>Break up the file and perform the intersection on a chunk by chunk basis</li>
  <li>Ask your system administrator to install the program</li>
  <li>Load it into Docker, your systems administrator will understand this. Installing docker is a single line of code (which I forget). Docker is a container which protects the system from the program and no systems administrator in this day and age would refuse a request for Docker installation.</li>
  </ol>
  <p>Your.systrm administrator might be concerned about the file sizes involved, but I think a better dialogue with them is the right way forward here.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T07:43:32.317Z
Answers.Text: the first option is maybe a good one, you can select individual chromosomes at a time with bigBedToBed
Answers.UserDisplayName: Colin D
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T07:49:07.153Z
Answers.Text: Thank you for your suggestions! I will try the first one on my computer first :)
Answers.UserDisplayName: Binhuan Sun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T07:49:51.443Z
Answers.Text: @ColinD Yes, I will try. Thank you :)
Answers.UserDisplayName: Binhuan Sun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T15:42:38.807Z
Answers.Text: Hello M, because today is Sunday so I first tried the first method you suggested. I tried to break up the file using argument -Chrom of the bigBedtobed program: ```bigBedToBed -chrom=chr1 JASPAR2022_hg19.bb test.bed```. However, I still got the same error```Invalid argument, Error reading 9188986873 bytes```. I then tried a relatively smaller data (66GB) but got the same error. It seems that on my computer the bigBedtobed program only works on data that is smaller than 10 GB. I still have over 600 GB of drive memory, so I think this may due to the lack of RAM?
Answers.UserDisplayName: Binhuan Sun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T16:08:08.137Z
Answers.Text: If its been written in C (I honestly dunno) a core-memory dump is a common error. However, if you > 10 chromosomes each file should be smaller than 10 GB? If its Drosophila then you get the same error. Could you check the file size for each intersect?
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T16:28:44.613Z
Answers.Text: Hello M, Thank you for your quick reply! Actually, it is a Genomic track and TFBS prediction for human genome, and it is a whole file. I think after breaking up every file will < 10GB, but the thing is I don't know how to split the .bb file directly and then convert them to the .bed file :(. The website for the file I am working with is here: [link](http://expdata.cmmt.ubc.ca/JASPAR/downloads/UCSC_tracks/2022/). The file name is JASPAR2022_hg19.bb.
Answers.UserDisplayName: Binhuan Sun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-14T17:17:01.550Z
Answers.Text: I think this should be a separate question. It wouldn't be hard to modify the file to allow breakup the file given the chromosome tag is a recognised tag for this purpose as @ColinD has stated. Perl pie will modify massive files with ease (if it was simplistic).
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-15T10:09:19.377Z
Answers.Text: Yes M. I think you are right, and today I asked my supervisor to help me apply for an account on another server that installed those UCSC programs, and I hope it can work on the server. Thank you again for the patient reply :)
Answers.UserDisplayName: Binhuan Sun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T00:06:33.913Z
Answers.LastActivityDate: 2022-02-13T00:06:33.913Z
Answers.UserDisplayName: M__
Answers.Body: <p>The question I asked a couple of days ago has been solved. Here I want to make a summary of this and hope it can help people who also encounter such problems later.</p>
  <p>First, about the way to process the bigbed file. I didn't find a very good way to work directly with bigbed file at that time, so I still firstly converted the bigbed file to the bed file using <code>bigBedTobed</code> program and then apply bedtools. The <code>bigBedTobed</code> program can be downloaded through this link: <a href="https://hgdownload.soe.ucsc.edu/admin/exe/" rel="nofollow noreferrer">UCSC programs</a>, and it also contains many useful programs from  UCSC genome browser. I finally converted my data (117 GB bigbed file) on the server, and it took ~2.5h to convert it to bed file. After conversion, the converted bed file is about 550 GB. And it took ~4h to intersect with a 4 Mb bed file using bedtools.</p>
  <p>Second, the bug that I encountered when I tried to convert the bigbed file on my own laptop. The code I used is <code>bigBedToBed JASPAR2022_hg19.bb hg1 9.bed</code> and it has error <code>Invalid argument, Error reading 9188986873 bytes.</code> I reported this error to the UCSC genome browser group and they said it is a MacOS specific bug, reaching a default data limit. They have solved it in their development environment but it still needs some time to reach the public site. The current solution for this is to use a URL to the file instead of a local file allows the program to read appropriately sized pieces of data at a time<code>(-udcDir= in bigBedToBed)</code>.</p>
  <p>Finally, this is my first time asking questions here and I really appreciate those good suggestions from @M__ and @ColinD.</p>
  <hr />
  <p>Renew: Methods from @malcook can work directly with the bigbed file! For a very large bigbed file, it is indeed a more efficient way.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-19T21:05:50.850Z
Answers.Text: Hi nice to hear from you @BinhuanSun. Cool glad its worked out and thanks for the feedback.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-19T21:18:37.187Z
Answers.Text: MacOSX bug .. thanks thats interesting, I encountered something similar. They are nice machines, but it ain't a cool way to do things IMO.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-19T20:04:56.240Z
Answers.LastActivityDate: 2022-02-21T09:25:23.787Z
Answers.LastEditDate: 2022-02-21T09:25:23.787Z
Answers.UserDisplayName: Binhuan Sun
Answers.Body: <p>My 2p, a little late and maybe not too useful:</p>
  <blockquote>
  <p>I didn't run the bigbedtobed program on the server because I am not allowed to install the bigbedtobed program on our server.</p>
  </blockquote>
  <p>Presumably, you cannot install in <code>/usr/local/bin</code> or some other root directory because you don't have admin rights. This is the common setup on shared servers. However, many programs can be installed in the user's home directory without admin rights. Even better, install conda and mamba first (no need to be admin) and use that to install other programs.</p>
  <p>The problem with docker is that some system administrators are concerned about security and refuse to install it (and docker does need admin rights). Whether such concerns are founded, I don't know...</p>
  <blockquote>
  <p>I tried to convert bigbed file to bed file and then applied bedtools, but I found it was very slow and couldn't make it.</p>
  </blockquote>
  <p>bedtools intersect supports the <code>-sorted</code> options that should speed things up quite a bit - Have you applied it? Your huge bigbed files should be already sorted after you convert to bed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-21T09:35:36.723Z
Answers.Text: Hi Dariober, yes later I found I could install programs in my home directory. Sorry for this super easy problem because I am not familiar with the Linux server. I just checked my converted bed file, and it is indeed sorted, so I think it is just because the converted bed file is too big (560GB).
Answers.UserDisplayName: Binhuan Sun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-21T09:27:47.607Z
Answers.LastActivityDate: 2022-02-21T09:27:47.607Z
Answers.UserDisplayName: dariober
Body: <p>I am now trying to find intersect genes of two files: one is .bigbed file (very large, 117GB), and the other is a .bed file.</p>
  <p>At first, I tried to convert bigbed file to bed file and then applied bedtools. I didn't run the bigbedtobed program on the server because I am not allowed to install the bigbedtobed program on our server. Thus I planned to convert the .bb file on my own computer and then upload the converted .bed file to the server, and then apply bedtools. Firstly I tried to convert some smaller .bb data sets (&lt; 10 GB) to .bed and it worked slowly but well on my computer. However, when I tried the 117GB data there was an error:</p>
  <p><code>Invalid argument, Error reading 9188986873 bytes.</code></p>
  <p>My code is :</p>
  <p><code>bigBedToBed JASPAR2022_hg19.bb hg1 9.bed</code></p>
  <p>Thank you in advance!</p>
  <hr />
  <p>Now the bug is solved, and I got many other good suggestions. Below you can find many methods to work with the bigbed file, directly working with it or converting it to bed file.</p>
  
Tags: <bedtools>
Title: Methods to work directly with bigbed file?
ThreadText: # Methods to work directly with bigbed file?
  
  (Asked by: username_0 on 2022-02-11)
  
  I am now trying to find intersect genes of two files: one is .bigbed file (very large, 117GB), and the other is a .bed file.
  
  At first, I tried to convert bigbed file to bed file and then applied bedtools. I didn't run the bigbedtobed program on the server because I am not allowed to install the bigbedtobed program on our server. Thus I planned to convert the .bb file on my own computer and then upload the converted .bed file to the server, and then apply bedtools. Firstly I tried to convert some smaller .bb data sets (< 10 GB) to .bed and it worked slowly but well on my computer. However, when I tried the 117GB data there was an error:
  
  `Invalid argument, Error reading 9188986873 bytes.`
  
  My code is :
  
  `bigBedToBed JASPAR2022_hg19.bb hg1 9.bed`
  
  Thank you in advance!
  
  ---
  
  Now the bug is solved, and I got many other good suggestions. Below you can find many methods to work with the bigbed file, directly working with it or converting it to bed file.
  
  **Tags:** bedtools
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2022-02-20. Score: 1 | Accepted answer
  
  You asked
  
  > Does anyone know any tools that can intersect two bigbed files
  
  ...but it appears you really wanted to subset a single very large bigbed file based on the contents of a much smaller bed file.
  
  I disagree with your summation "there is not a very efficient way to work directly with the bigbed file". I see two options.
  
  The first is to repeatedly call `bigBedTobed` once for each row in your .bed file. If you have multiple cores, this can be parallelized using [GNU parallel](https://www.gnu.org/software/parallel/). The following does that, emitting each overlapping result with additional column being the index into your .bed file from which the result stems.
  
  ```
  cat your.bed | parallel --lb -j --colsep '\t' bigBedToBed -chrom={1} -start={2} -end={3}  your.bb  /dev/stdout '|' sed -e 's/$/\\t${PARALLEL_SEQ}/'
  
  ```
  
  Depending upon your application, there is a second approach which uses R/BioConductor's `rtracklayer::import.bb` which can be parallelized across (chunks of) your .bed file.
  
  I considered these approaches in what I expect is an almost identical situation:
  
  My bigbed holds genome-wide transcription factor binding site predictions from [JASPAR Genome Browser tracks](https://jaspar.genereg.net/genome-tracks/#ucsc_tracks) as produced by [wassermanlab / JASPAR-UCSC-tracks](https://github.com/wassermanlab).
  
  My bed file holds the loci of ~23K putative enhancers as determined by an analysis of regions of accessible chromatin from ATAC-Seq and various ChIP marks.
  
  In my case I wished to produce a sparse Matrix with one row for each putative enhancer, and one column for each JASPAR motif (which will form the basis of downstream enrichment analyses).
  
  ```
  bbf<-BigBedFile('path/to/your.bb')
  gr<-import.bed('path/to/your.bed')
  ## I found I had to harmonize seqlevels (e.g. chromosome identifiers) to make this work.  YMMV
  gr<-renameSeqlevels(gr,paste0('chr',gsub('^(K.*)\\.(\\d+)$','Un_\\1v\\2',seqlevels(gr))))
  mm<-motifMatrix(gr,bbf)
  
  ```
  
  which results in a sparse Matrix, viz:
  
  ```
  > rownames(mm)<-as.character(gr) # optional.
  > mm[1:5,1:5]
  5 x 5 sparse Matrix of class "dgCMatrix"
                   MA0002.2 MA0003.4 MA0004.1 MA0006.1 MA0007.3
  chr1:11156-12180       14        .        2       28        2
  chr1:17836-20023       25        .       27       63        2
  chr1:22495-24367       22        1       22      137        2
  chr1:27533-28038        2        2        4        7        .
  chr1:36549-37079        6        .        2       41        .
  
  ```
  
  This depends upon the following definition:
  
  ```
  motifMatrix<-function(gr,bbf,motifID=c('name','TFName')[[1]],mc.jobs=100) {
    ## produce a sparse Matrix with one row for each region in
    ## GenomicRanges gr, and one column for each JASPAR motif indicated
    ## as overlapping <gr> accorinding to BigBedFile <bbf>.  Default to
    ## using the 'name' column as the motif identifier, but allow for
    ## possibly using another, such as 'TFName' (as used by 2022 version
    ## of
    ## https://jaspar.genereg.net/genome-tracks/#ucsc_tracks). Parallelize
    ## into mc.jobs queries, defaulting to 100, which was found suitable
    ## for ~25000 loci in <gr> against JASPAR matches to zebrafish
    ## danRer11 genome running on a few dozen cores.  In general,
    ## knowing how "big" each parallel job *should* be depends on
    ## available RAM and cores, and number of loci in <gr>.
    ##
    ## AUTHOR: malcolm_cook@stowers.org
    ##
    ## NB: you must install a version of `rtracklayer` which addresses
    ## issue [import\.bb returning incorrect ranges & results as
    ## compared with kent tools
    ## bigBedToBed](https://github.com/lawremi/rtracklayer/issues/59)
    ## (such as by `install_github('lawremi/rtracklayer')`)
    si<-parallel::splitIndices(length(gr),mc.jobs)
    mm<-mclapply(si,function(i) {
      gri<-gr[i]
      m<-import.bb(bbf,selection=BigBedSelection(gri, colnames = c(motifID)))
      fo<-findOverlaps(gri,m)
      xt<-
        xtabs(~.,data=list(locus=i[queryHits(fo)],motif=mcols(m[subjectHits(fo)])[[motifID]]),sparse=TRUE) 
      xt
    })
    mm<-
      ## rBind.fill(mm) ## NOT!  It does NOT preserve dimnames and
      ## returns matrix when given Matrix.  Issue reported:
      ## [[https://github.com/cvarrichio/Matrix.utils/issues/5][rBind.fill
      ## on a list of Matrix arguably should return Matrix but returns
      ## matrix #5]]
      Reduce(rBind.fill,mm) # which does preserve dimnames and return Matrix
    mm
  }
  
  ```
  
  NB: If you choose this approach, you MUST install a version of `rtracklayer` which addresses issue the [import.bb returning incorrect ranges & results as compared with kent tools bigBedToBed](https://github.com/lawremi/rtracklayer/issues/59) (such as by `install_github('lawremi/rtracklayer')`)
  
  > **Comments:**
  > **username_0**: Yes username_1, this is exactly what I want at first! Thank you so much!  
  > **username_1**: glad to help - would you consider to change the accepted solution to mine?  
  > **username_2**: This is the accepted answer for sure @username_0  
  
  ---
  
  ### Answer by username_2 on 2022-02-13. Score: 1
  
  No matter how you approach this problem 110+ GB is big:
  
  Your options are:
  
  1. Break up the file and perform the intersection on a chunk by chunk basis
  2. Ask your system administrator to install the program
  3. Load it into Docker, your systems administrator will understand this. Installing docker is a single line of code (which I forget). Docker is a container which protects the system from the program and no systems administrator in this day and age would refuse a request for Docker installation.
  
  Your.systrm administrator might be concerned about the file sizes involved, but I think a better dialogue with them is the right way forward here.
  
  > **Comments:**
  > **username_3**: the first option is maybe a good one, you can select individual chromosomes at a time with bigBedToBed  
  > **username_0**: Thank you for your suggestions! I will try the first one on my computer first :)  
  > **username_0**: @username_3 Yes, I will try. Thank you :)  
  > **username_0**: Hello M, because today is Sunday so I first tried the first method you suggested. I tried to break up the file using argument -Chrom of the bigBedtobed program: ```bigBedToBed -chrom=chr1 JASPAR2022_hg19.bb test.bed```. However, I still got the same error```Invalid argument, Error reading 9188986873 bytes```. I then tried a relatively smaller data (66GB) but got the same error. It seems that on my computer the bigBedtobed program only works on data that is smaller than 10 GB. I still have over 600 GB of drive memory, so I think this may due to the lack of RAM?  
  > **username_2**: If its been written in C (I honestly dunno) a core-memory dump is a common error. However, if you > 10 chromosomes each file should be smaller than 10 GB? If its Drosophila then you get the same error. Could you check the file size for each intersect?  
  > **username_0**: Hello M, Thank you for your quick reply! Actually, it is a Genomic track and TFBS prediction for human genome, and it is a whole file. I think after breaking up every file will < 10GB, but the thing is I don't know how to split the .bb file directly and then convert them to the .bed file :(. The website for the file I am working with is here: [link](http://expdata.cmmt.ubc.ca/JASPAR/downloads/UCSC_tracks/2022/). The file name is JASPAR2022_hg19.bb.  
  > **username_2**: I think this should be a separate question. It wouldn't be hard to modify the file to allow breakup the file given the chromosome tag is a recognised tag for this purpose as @username_3 has stated. Perl pie will modify massive files with ease (if it was simplistic).  
  > **username_0**: Yes M. I think you are right, and today I asked my supervisor to help me apply for an account on another server that installed those UCSC programs, and I hope it can work on the server. Thank you again for the patient reply :)  
  
  ---
  
  ### Answer by username_0 on 2022-02-19. Score: 1
  
  The question I asked a couple of days ago has been solved. Here I want to make a summary of this and hope it can help people who also encounter such problems later.
  
  First, about the way to process the bigbed file. I didn't find a very good way to work directly with bigbed file at that time, so I still firstly converted the bigbed file to the bed file using `bigBedTobed` program and then apply bedtools. The `bigBedTobed` program can be downloaded through this link: [UCSC programs](https://hgdownload.soe.ucsc.edu/admin/exe/), and it also contains many useful programs from UCSC genome browser. I finally converted my data (117 GB bigbed file) on the server, and it took ~2.5h to convert it to bed file. After conversion, the converted bed file is about 550 GB. And it took ~4h to intersect with a 4 Mb bed file using bedtools.
  
  Second, the bug that I encountered when I tried to convert the bigbed file on my own laptop. The code I used is `bigBedToBed JASPAR2022_hg19.bb hg1 9.bed` and it has error `Invalid argument, Error reading 9188986873 bytes.` I reported this error to the UCSC genome browser group and they said it is a MacOS specific bug, reaching a default data limit. They have solved it in their development environment but it still needs some time to reach the public site. The current solution for this is to use a URL to the file instead of a local file allows the program to read appropriately sized pieces of data at a time`(-udcDir= in bigBedToBed)`.
  
  Finally, this is my first time asking questions here and I really appreciate those good suggestions from @M\_\_ and @username_3.
  
  ---
  
  Renew: Methods from @username_1 can work directly with the bigbed file! For a very large bigbed file, it is indeed a more efficient way.
  
  > **Comments:**
  > **username_2**: Hi nice to hear from you @username_0. Cool glad its worked out and thanks for the feedback.  
  > **username_2**: MacOSX bug .. thanks thats interesting, I encountered something similar. They are nice machines, but it ain't a cool way to do things IMO.  
  
  ---
  
  ### Answer by username_4 on 2022-02-21. Score: 1
  
  My 2p, a little late and maybe not too useful:
  
  > I didn't run the bigbedtobed program on the server because I am not allowed to install the bigbedtobed program on our server.
  
  Presumably, you cannot install in `/usr/local/bin` or some other root directory because you don't have admin rights. This is the common setup on shared servers. However, many programs can be installed in the user's home directory without admin rights. Even better, install conda and mamba first (no need to be admin) and use that to install other programs.
  
  The problem with docker is that some system administrators are concerned about security and refuse to install it (and docker does need admin rights). Whether such concerns are founded, I don't know...
  
  > I tried to convert bigbed file to bed file and then applied bedtools, but I found it was very slow and couldn't make it.
  
  bedtools intersect supports the `-sorted` options that should speed things up quite a bit - Have you applied it? Your huge bigbed files should be already sorted after you convert to bed.
  
  > **Comments:**
  > **username_0**: Hi username_4, yes later I found I could install programs in my home directory. Sorry for this super easy problem because I am not familiar with the Linux server. I just checked my converted bed file, and it is indeed sorted, so I think it is just because the converted bed file is too big (560GB).  
  
  ---
  
  