<p>I'm a programmer who'd like to process some of the smaller Gaia DR3 datasets. I ultimately want to use the <a href="https://www.cosmos.esa.int/web/gaia/gaiadr2_gaiasky" rel="nofollow noreferrer">Gaia Sky</a> 15M <a href="https://gaia.ari.uni-heidelberg.de/gaiasky/files/repository/catalog/dr3/000-default/v02_20221117/" rel="nofollow noreferrer">default dataset</a>, but decided to start with a <a href="https://gaia.ari.uni-heidelberg.de/gaiasky/files/repository/catalog/dr3/012-weeny/v01_20230330/" rel="nofollow noreferrer">smaller 1.9M dataset</a> as a prototype.</p>
<p>I'm currently unable to find a spec on how these binary files are encoded and structured.</p>
<p>I'm aware there are <a href="http://cdn.gea.esac.esa.int/Gaia/gdr3/gaia_source/" rel="nofollow noreferrer">CSV files</a> available for download, but they appear to be massive dumps of <em>all</em> Gaia data. I'm really only interested in stars this side of the galaxy, particularly to complement the <a href="https://heasarc.gsfc.nasa.gov/W3Browse/star-catalog/bsc5p.html" rel="nofollow noreferrer">BSC5P</a> and fainter stars in-between BSC5P stars we cannot see with the naked eye. These pre-prepared binary datasets are therefore really attractive.</p>
<p>Does anyone know where I can find either a specification doc for the pre-prepared binary files, or CSV files that have been similarly culled?</p>

CC BY-SA 4.0
2023-08-07T15:24:33.230Z
Consider that by the time you write a program to use those binary files, you could have downloaded all of the CSV data and extracted what you needed.  I'm not sure if I've worked with ESA binaries, but NASA does a horrible job of documenting their binary formats.
Greg Miller
CC BY-SA 4.0
2023-08-07T15:32:38.973Z
@GregMiller The value here is in using a dataset curated by teams who know more about astronomy than I do. Sure, I can bite the 10 terabytes and write a data extraction tool myself. But 1) whether I write a CSV extraction tool or binary extraction tool is the same thing to me, and 2) the huge amount of revisions on those datasets alone show just how many mistakes even the well-educated make; flying completely blind with no peer review on my dataset filtering is immensely intimidating.
aggregate1166877
CC BY-SA 4.0
2023-08-07T15:54:59.093Z
Where did you access those links to the datafiles? The first link you give is to something called GaiaDR2_GaiaSky? I presume the tar file contains no documentation files at all?
ProfRob
CC BY-SA 4.0
2023-08-08T01:51:04.123Z
@ProfRob Those links are provided for download from within the Gaia Sky application. They do not contain instructions. My initial assumption was that you could use Gaia Sky as a previewer to decide which datasets you want to download and use. I've since managed to find Gaia Sky's source code, and within it the reader code. It seems now these aren't for general use; the source code reveals these datasets are [octrees](https://en.wikipedia.org/wiki/Octree) apparently made specifically for Gaia Sky itself. I'm unsure why they offer download links for these. It seems somewhat pointless now..
aggregate1166877
CC BY-SA 4.0
2023-08-12T07:08:36.247Z
Your best bet is to learn enough SQL to do a simple query of the DR3 catalogue based on apparent Gaia G magnitude and then download a fits table of everything with say $G<7$ (would not be a large catalogue) and then merge it with the BSC (since DR3 is incomplete at the bright end). Or probably easier, just use the astroquery python library https://www.cosmos.esa.int/web/gaia-users/archive/use-cases#ClusterAnalysisPythonTutorial
ProfRob
CC BY-SA 4.0
2023-08-17T21:59:47.297Z
@ProfRob Thank you! I will give it a shot. Out of interest, what is does the G magnitude represent? Is it analogous to apparent magnitude?
aggregate1166877
<gaia><star-catalogues>
Filtered Gaia DR3 datasets
aggregate1166877
# Filtered Gaia DR3 datasets

(Asked by: username_0 on 2023-08-07)

I'm a programmer who'd like to process some of the smaller Gaia DR3 datasets. I ultimately want to use the [Gaia Sky](https://www.cosmos.esa.int/web/gaia/gaiadr2_gaiasky) 15M [default dataset](https://gaia.ari.uni-heidelberg.de/gaiasky/files/repository/catalog/dr3/000-default/v02_20221117/), but decided to start with a [smaller 1.9M dataset](https://gaia.ari.uni-heidelberg.de/gaiasky/files/repository/catalog/dr3/012-weeny/v01_20230330/) as a prototype.

I'm currently unable to find a spec on how these binary files are encoded and structured.

I'm aware there are [CSV files](http://cdn.gea.esac.esa.int/Gaia/gdr3/gaia_source/) available for download, but they appear to be massive dumps of *all* Gaia data. I'm really only interested in stars this side of the galaxy, particularly to complement the [BSC5P](https://heasarc.gsfc.nasa.gov/W3Browse/star-catalog/bsc5p.html) and fainter stars in-between BSC5P stars we cannot see with the naked eye. These pre-prepared binary datasets are therefore really attractive.

Does anyone know where I can find either a specification doc for the pre-prepared binary files, or CSV files that have been similarly culled?

**Tags:** gaia, star-catalogues
> **Comments:**
> **username_1**: Consider that by the time you write a program to use those binary files, you could have downloaded all of the CSV data and extracted what you needed.  I'm not sure if I've worked with ESA binaries, but NASA does a horrible job of documenting their binary formats.  
> **username_0**: @GregMiller The value here is in using a dataset curated by teams who know more about astronomy than I do. Sure, I can bite the 10 terabytes and write a data extraction tool myself. But 1) whether I write a CSV extraction tool or binary extraction tool is the same thing to me, and 2) the huge amount of revisions on those datasets alone show just how many mistakes even the well-educated make; flying completely blind with no peer review on my dataset filtering is immensely intimidating.  
> **username_2**: Where did you access those links to the datafiles? The first link you give is to something called GaiaDR2_GaiaSky? I presume the tar file contains no documentation files at all?  
> **username_0**: @ProfRob Those links are provided for download from within the Gaia Sky application. They do not contain instructions. My initial assumption was that you could use Gaia Sky as a previewer to decide which datasets you want to download and use. I've since managed to find Gaia Sky's source code, and within it the reader code. It seems now these aren't for general use; the source code reveals these datasets are [octrees](https://en.wikipedia.org/wiki/Octree) apparently made specifically for Gaia Sky itself. I'm unsure why they offer download links for these. It seems somewhat pointless now..  
> **username_2**: Your best bet is to learn enough SQL to do a simple query of the DR3 catalogue based on apparent Gaia G magnitude and then download a fits table of everything with say $G<7$ (would not be a large catalogue) and then merge it with the BSC (since DR3 is incomplete at the bright end). Or probably easier, just use the astroquery python library https://www.cosmos.esa.int/web/gaia-users/archive/use-cases#ClusterAnalysisPythonTutorial  
> **username_0**: @ProfRob Thank you! I will give it a shot. Out of interest, what is does the G magnitude represent? Is it analogous to apparent magnitude?  

---

## 0 answers

