<p>I need to extract information from a few columns in ~20k different fits files with python. Each file is relatively small, ~0.2MB. I have been doing this so far with a loop and <code>astropy</code> like this:</p>
<pre><code>from astropy.io import fits

data = []
for file_name in fits_files_list:
    with fits.open(file_name, memmap=False) as hdulist:
        lam = np.around(10**hdulist[1].data['loglam'], 4)
        flux = np.around(hdulist[1].data['flux'], 4)
        z = np.around(hdulist[2].data['z'], 4)
    data.append([lam, flux, z])
</code></pre>
<p>This takes for the 20k fits files ~2.5 hours and from time to time I need to loop through the files for other reasons.</p>
<p>So my question is: Can I minimize the time for looping? Do you know of other packages besides <code>astropy</code> that would help me? Or do you know if I can change my algorithm to make it run faster, e.g. somehow vectorize the loop? Or some software to stack quickly 20k fits files into one fits-file (TOPCAT has no function that does this for more than 2 files)? Thanks for any ideas and comments!</p>

CC BY-SA 4.0
2021-11-10T02:19:08.090Z
Does the situation improve with `memmap=True` ? Could you use `from astropy.table import Table ; table = Table.open(file_name)` to read them in as Astropy Tables instead and then join them together into one more efficient FITS table ? (Astropy [Table docs](https://docs.astropy.org/en/stable/table/index.html))
astrosnapper
CC BY-SA 4.0
2021-11-10T12:39:12.333Z
@astrosnapper Thanks for answering! unfortunately `memmap=True` doesn't improve the speed. I think I tried `astropy.table` and I decided to not use it, because I couldn't figure out how to read with it `hdulist[0]`, `hdulist[1]`, `hdulist[2]` a.s.o. I think it was reading only one of the tables, but my fits files contain 3 tables. I don't know if it matters, but I am running the code on a google colab notebook without utilizing the provided GPU or TPU (they make the code even slower)
NeStack
CC BY-SA 4.0
2021-11-10T17:41:57.913Z
Try using multiprocessing and queues - https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue - `put()` each item in your file list in a queue (call it "inbox"), have a function which works on a single item which it `get()` out of the inbox, and have it `put()` the result in another queue ("outbox"). Use multiple processes (see the multiprocessing.Pool class) and you should be able to spread the load across your CPU cores.
Aaron F
CC BY-SA 4.0
2021-11-10T19:17:16.560Z
@AaronF Thanks! I will give it a try. Do you think it will also work on google colab - this is the environment in which I am coding this loop. And the data is stored in my google drive
NeStack
CC BY-SA 4.0
2021-11-10T19:47:23.333Z
@NeStack you are dominated by file I/O overhead for this many files which is why more CPU, GPU etc isn't helping. Opening and closing many files isn't quick on regular filesystems and while I have no experience of Google Collab, our experience of AWS EC2 with FITS files on S3 is that it's quite a bit slower than even NFS. So effort should be in reducing the amount of I/O needed, particularly if you are re-running this code many times. Tables with multiple HDUs can be read by adding `hdu=2` to the `Table.open()` call [doc link](https://docs.astropy.org/en/stable/io/unified.html#table-io-fits)
astrosnapper
CC BY-SA 4.0
2021-11-10T22:56:27.843Z
@astrosnapper Thank you for the informative answer! But would I escape with `from astropy.table import Table ; table = Table.open(file_name)` the I/O problem? Will it make a difference to `with fits.open(file_name)`. I know I can check this by just running it, and I will, I am asking you more about the theoretical justification why it would make a difference. Thanks!
NeStack
CC BY-SA 4.0
2021-11-11T00:00:47.473Z
My thinking was you could use the Astropy Table read and write routines to reformat your 20k small FITS files with 3 short tables into 1 large FITS file with 3 large FITS tables of 20k rows (not sure how many rows you have in the original small files but you get the idea) so the I/O of opening and closing all the small files is gone and operating on the (hopefully memmapped) large Table/arrays is much faster. Easier faster and cheaper to get more RAM and operate on things there than to get more IO bandwidth, particularly for GET/PUT-based API cloud storage like GoogleDrive/S3
astrosnapper
CC BY-SA 4.0
2021-11-11T17:55:07.747Z
@NeStack i'm afraid i don't know anything about google colab. astrosnapper's advice looks good to me. if you have trouble concatenating the files with astropy, you could check out [STILTS](http://www.star.bristol.ac.uk/~mbt/stilts/) and its 'tcat' command, and use it to pre-process your files before running your python code.
Aaron F
<data-analysis><python><fits>
How to quickly loop through many small fits files for data analysis?
NeStack
# How to quickly loop through many small fits files for data analysis?

(Asked by: username_0 on 2021-11-09)

I need to extract information from a few columns in ~20k different fits files with python. Each file is relatively small, ~0.2MB. I have been doing this so far with a loop and astropy like this:
from astropy.io import fits

data = []
for file_name in fits_files_list:
    with fits.open(file_name, memmap=False) as hdulist:
        lam = np.around(10**hdulist[1].data['loglam'], 4)
        flux = np.around(hdulist[1].data['flux'], 4)
        z = np.around(hdulist[2].data['z'], 4)
    data.append([lam, flux, z])

This takes for the 20k fits files ~2.5 hours and from time to time I need to loop through the files for other reasons.
So my question is: Can I minimize the time for looping? Do you know of other packages besides astropy that would help me? Or do you know if I can change my algorithm to make it run faster, e.g. somehow vectorize the loop? Or some software to stack quickly 20k fits files into one fits-file (TOPCAT has no function that does this for more than 2 files)? Thanks for any ideas and comments!

**Tags:** data-analysis, python, fits
> **Comments:**
> **username_1**: Does the situation improve with `memmap=True` ? Could you use `from astropy.table import Table ; table = Table.open(file_name)` to read them in as Astropy Tables instead and then join them together into one more efficient FITS table ? (Astropy [Table docs](https://docs.astropy.org/en/stable/table/index.html))  
> **username_0**: @astrosnapper Thanks for answering! unfortunately `memmap=True` doesn't improve the speed. I think I tried `astropy.table` and I decided to not use it, because I couldn't figure out how to read with it `hdulist[0]`, `hdulist[1]`, `hdulist[2]` a.s.o. I think it was reading only one of the tables, but my fits files contain 3 tables. I don't know if it matters, but I am running the code on a google colab notebook without utilizing the provided GPU or TPU (they make the code even slower)  
> **username_2**: Try using multiprocessing and queues - https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue - `put()` each item in your file list in a queue (call it "inbox"), have a function which works on a single item which it `get()` out of the inbox, and have it `put()` the result in another queue ("outbox"). Use multiple processes (see the multiprocessing.Pool class) and you should be able to spread the load across your CPU cores.  
> **username_0**: @AaronF Thanks! I will give it a try. Do you think it will also work on google colab - this is the environment in which I am coding this loop. And the data is stored in my google drive  
> **username_1**: @NeStack you are dominated by file I/O overhead for this many files which is why more CPU, GPU etc isn't helping. Opening and closing many files isn't quick on regular filesystems and while I have no experience of Google Collab, our experience of AWS EC2 with FITS files on S3 is that it's quite a bit slower than even NFS. So effort should be in reducing the amount of I/O needed, particularly if you are re-running this code many times. Tables with multiple HDUs can be read by adding `hdu=2` to the `Table.open()` call [doc link](https://docs.astropy.org/en/stable/io/unified.html#table-io-fits)  
> **username_0**: @astrosnapper Thank you for the informative answer! But would I escape with `from astropy.table import Table ; table = Table.open(file_name)` the I/O problem? Will it make a difference to `with fits.open(file_name)`. I know I can check this by just running it, and I will, I am asking you more about the theoretical justification why it would make a difference. Thanks!  
> **username_1**: My thinking was you could use the Astropy Table read and write routines to reformat your 20k small FITS files with 3 short tables into 1 large FITS file with 3 large FITS tables of 20k rows (not sure how many rows you have in the original small files but you get the idea) so the I/O of opening and closing all the small files is gone and operating on the (hopefully memmapped) large Table/arrays is much faster. Easier faster and cheaper to get more RAM and operate on things there than to get more IO bandwidth, particularly for GET/PUT-based API cloud storage like GoogleDrive/S3  
> **username_2**: @NeStack i'm afraid i don't know anything about google colab. astrosnapper's advice looks good to me. if you have trouble concatenating the files with astropy, you could check out [STILTS](http://www.star.bristol.ac.uk/~mbt/stilts/) and its 'tcat' command, and use it to pre-process your files before running your python code.  

---

## 0 answers

