<p><strong>Disclaimer:</strong> I have a Machine Learning/Computer Vision background, <em>not</em> Astrophysics, so please be a bit patient with me. If you feel like this question is too far out there it would be great if you could direct me to meta or some bulletin board/mailing list that you think is a better fit. :)</p>

<p><strong>Important:</strong> I am talking about "datasets" in a machine learning / computer vision manner. I do <em>not</em> mean "the entirety of SDSS data" but rather a common set of inputs and (for supervised learning) associated outputs for a specific problem.</p>

<p>I have been looking for a while now to tinker with some problems in Astronomy/Astrophysics and I keep finding the same kind of advice when it comes to datasets:</p>

<p>Go to SDSS/GalaxyZoo, they provide downloads for their data.</p>

<p>Now, while that is true, none of these datasets are anywhere near as accessible as datasets in Computer Vision. "Over here" a dataset is mostly a bunch of (huge) tar files with jpeg images and some <code>.mat</code> or <code>.csv</code> for annotations. The process of getting started is pretty straigth-forward: unzip, load annotations into memory, start sciencing (or usually, applying wild heuristics). Obviously I simplify slightly, but you get the idea.</p>

<p>Now, to stay with the example of GalaxyZoo, the object coordinates, annotations and computed features are conveniently <a href="https://data.galaxyzoo.org/" rel="nofollow noreferrer">presented</a> as a <code>csv</code>, <code>fits</code> or <code>voTable</code> file. But to get the actual visual data (if I understand this correctly) I have to, for each annotated galaxy...</p>

<ul>
<li>determine a bunch of <code>.fits</code> urls from the coordinates in the annotations (one per band and galaxy) and download them (~12MB/file)</li>
<li>download the containing "tiles" from the SDSS servers (in each of the bands I'd like to work with)</li>
<li>extract the patch containing the galaxies (with the aptly named sExtractor)</li>
<li>possibly apply some non-trivial corrections/transforms to the image patch</li>
</ul>

<p>I tried following along this procedure with a semi-randomly chosen paper from arXiv for which there is code available <a href="https://github.com/EdwardJKim/dl4astro" rel="nofollow noreferrer">here</a> and without wanting to make any statement about said paper/code, this process is not comparable to the simple workflow I described above, and I would probably download some TB of data, were I to use the entire dataset, of which only a tiny fraction would actually be useful.</p>

<p>The resulting dataset only occupies a few GB of space, and has a very simple nature: <strong>ca 50000 images, with associated labels -1 or 1, for stars or galaxies</strong>. The data preparation pipeline, while one might also want to optimize it for this task, has next to nothing to do with the "real" work being done in the paper, and so having the reader go through those steps makes little sense to me.</p>

<p>So my questions are:</p>

<ol>
<li><p><strong>Did I understand the workflow correctly?</strong> Maybe I'm just following the wrong kind of paper, or I really just overlooked a big red "Download Here!" button? Otherwise:</p></li>
<li><p><strong>Did anyone go through these steps and make the resulting dataset available?</strong> Or are there good reasons not to do so? From the size of the sExtractor manual I gather that the choice of parameters in this processing step might significantly influence the outcome of any downstream method. So...</p></li>
<li><p><strong>Are there any sane/agreed-upon default settings for the preprocessing pipeline?</strong> While for any given method, applying extra "magic" in the preprocessing might make sense, having a default set of parameters would make dataset creation and a bunch of other things much easier, no?</p></li>
</ol>

<p>I am sorry if this is a naive newcomer question. And for the long post. :P
As a meta, I would add tags "dataset" and "programming" or some such, but I lack the reputation.</p>

CC BY-SA 3.0
2017-05-13T14:44:04.563Z
Could you clarify what kind of data you are looking for? I suspect the problem stems from the fact that you are looking for a dataset that astronomers don't usually use (i.e. selection by "unusual" criteria), so there's no need, from an astronomer's perspective, to have the access mechanism you seek. I tried to replicate your problem and went to SDSS website, but found it very straightforward to find FITS images for the first object that came to mind, the Coma cluster. How to proceed from there depends strongly on the kind of analysis one wants to do, so there's no universal answer for that.
Alex
CC BY-SA 3.0
2017-05-13T15:57:26.867Z
As I come from Machine Learning/Computer Vision, I'm mostly looking for entire datasets, not individual images. The paper I linked to above does a decent job (IMHO) of making it possible to replicate the results. Crucially, it gives step-by-step instructions how to create the dataset. BUt following these steps takes days of computing time, and hundreds of GB of downloads. So it's not the kind of data that is the problem. It's the amount, and the know-how to preprocess it, that makes it a hurdle for newcomers.
black_puppydog
CC BY-SA 3.0
2017-05-13T16:33:31.437Z
@black_puppydog: Please explain what you mean by "entire datasets". I'm sure you have something in mind, and I'm sure it's very different from what I am thinking of. If you want to work on every image that SDSS has ever observed, of course you'll have to download every image they have ever observed. If you want, for example, all galaxies that are members of a galaxy cluster beyond redshift of 0.1, that requires detailed analysis that noone has done yet, and you'll have some objects missing as well as false positives. So, please clarify what you mean by "entire datasets".
Alex
CC BY-SA 3.0
2017-05-14T03:19:10.643Z
Question title and content do not match.
ProfRob
CC BY-SA 3.0
2017-05-14T10:21:02.737Z
Wow, thanks for the feedback here guys. I knew this was a bit tricky, and I added a note at the top. In ML/CV, "dataset" has a very different meaning from astronomy. Over there, a dataset is relatively small (in most cases) and a standard task that everyone can "benchmark" their methods on. Common example: http://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database These datasets have their limits when it comes to representing real data. Like the datasets I'm after here, they take time and work to acquire, but then they get published and anyone can use them.
black_puppydog
CC BY-SA 3.0
2017-05-11T12:38:42.543Z
Looks like TRQ (the real question) is "how do I properly process the datasets I found?'  Do you want to edit the title to reflect this?
Carl Witthoft
CC BY-SA 3.0
2017-05-11T13:56:03.513Z
Assuming that the answer to 1 and 2 are "yes" and "no", respectively, that would be true and question 3 is going in that direction. But given that, I would probably rephrase it more like "How come the dataset distribution in astronomy is so complicated on the receiving side?" It puts an awful lot of work on the person trying to replicate an experiment, and on the data hosting servers, no?
black_puppydog
CC BY-SA 3.0
2017-05-11T13:59:36.067Z
By the way I do realize that for an insider, this question may sound like a lazy newcomer whining, and in that case I'd really like to hear why this is not happening. I can imagine that people have good reasons to not simply use a dataset from somewhere else, instead making sure the preprocessing steps best match their method. But for replication specifically, a big download would likely be the best, especially for datasets of the "we use a random sample of size N from catalog C" kind.
black_puppydog
<data-analysis>
Are there any ready-made machine-learning datasets of photometric data that are easy to use?
black_puppydog
# Are there any ready-made machine-learning datasets of photometric data that are easy to use?

(Asked by: username_0 on 2017-05-10)

**Disclaimer:** I have a Machine Learning/Computer Vision background, *not* Astrophysics, so please be a bit patient with me. If you feel like this question is too far out there it would be great if you could direct me to meta or some bulletin board/mailing list that you think is a better fit. :)

**Important:** I am talking about "datasets" in a machine learning / computer vision manner. I do *not* mean "the entirety of SDSS data" but rather a common set of inputs and (for supervised learning) associated outputs for a specific problem.

I have been looking for a while now to tinker with some problems in Astronomy/Astrophysics and I keep finding the same kind of advice when it comes to datasets:

Go to SDSS/GalaxyZoo, they provide downloads for their data.

Now, while that is true, none of these datasets are anywhere near as accessible as datasets in Computer Vision. "Over here" a dataset is mostly a bunch of (huge) tar files with jpeg images and some `.mat` or `.csv` for annotations. The process of getting started is pretty straigth-forward: unzip, load annotations into memory, start sciencing (or usually, applying wild heuristics). Obviously I simplify slightly, but you get the idea.

Now, to stay with the example of GalaxyZoo, the object coordinates, annotations and computed features are conveniently [presented](https://data.galaxyzoo.org/) as a `csv`, `fits` or `voTable` file. But to get the actual visual data (if I understand this correctly) I have to, for each annotated galaxy...

* determine a bunch of `.fits` urls from the coordinates in the annotations (one per band and galaxy) and download them (~12MB/file)
* download the containing "tiles" from the SDSS servers (in each of the bands I'd like to work with)
* extract the patch containing the galaxies (with the aptly named sExtractor)
* possibly apply some non-trivial corrections/transforms to the image patch

I tried following along this procedure with a semi-randomly chosen paper from arXiv for which there is code available [here](https://github.com/EdwardJKim/dl4astro) and without wanting to make any statement about said paper/code, this process is not comparable to the simple workflow I described above, and I would probably download some TB of data, were I to use the entire dataset, of which only a tiny fraction would actually be useful.

The resulting dataset only occupies a few GB of space, and has a very simple nature: **ca 50000 images, with associated labels -1 or 1, for stars or galaxies**. The data preparation pipeline, while one might also want to optimize it for this task, has next to nothing to do with the "real" work being done in the paper, and so having the reader go through those steps makes little sense to me.

So my questions are:

1. **Did I understand the workflow correctly?** Maybe I'm just following the wrong kind of paper, or I really just overlooked a big red "Download Here!" button? Otherwise:
2. **Did anyone go through these steps and make the resulting dataset available?** Or are there good reasons not to do so? From the size of the sExtractor manual I gather that the choice of parameters in this processing step might significantly influence the outcome of any downstream method. So...
3. **Are there any sane/agreed-upon default settings for the preprocessing pipeline?** While for any given method, applying extra "magic" in the preprocessing might make sense, having a default set of parameters would make dataset creation and a bunch of other things much easier, no?

I am sorry if this is a naive newcomer question. And for the long post. :P
As a meta, I would add tags "dataset" and "programming" or some such, but I lack the reputation.

**Tags:** data-analysis
> **Comments:**
> **username_3**: Looks like TRQ (the real question) is "how do I properly process the datasets I found?'  Do you want to edit the title to reflect this?  
> **username_0**: Assuming that the answer to 1 and 2 are "yes" and "no", respectively, that would be true and question 3 is going in that direction. But given that, I would probably rephrase it more like "How come the dataset distribution in astronomy is so complicated on the receiving side?" It puts an awful lot of work on the person trying to replicate an experiment, and on the data hosting servers, no?  
> **username_0**: By the way I do realize that for an insider, this question may sound like a lazy newcomer whining, and in that case I'd really like to hear why this is not happening. I can imagine that people have good reasons to not simply use a dataset from somewhere else, instead making sure the preprocessing steps best match their method. But for replication specifically, a big download would likely be the best, especially for datasets of the "we use a random sample of size N from catalog C" kind.  
> **username_1**: Could you clarify what kind of data you are looking for? I suspect the problem stems from the fact that you are looking for a dataset that astronomers don't usually use (i.e. selection by "unusual" criteria), so there's no need, from an astronomer's perspective, to have the access mechanism you seek. I tried to replicate your problem and went to SDSS website, but found it very straightforward to find FITS images for the first object that came to mind, the Coma cluster. How to proceed from there depends strongly on the kind of analysis one wants to do, so there's no universal answer for that.  
> **username_0**: As I come from Machine Learning/Computer Vision, I'm mostly looking for entire datasets, not individual images. The paper I linked to above does a decent job (IMHO) of making it possible to replicate the results. Crucially, it gives step-by-step instructions how to create the dataset. BUt following these steps takes days of computing time, and hundreds of GB of downloads. So it's not the kind of data that is the problem. It's the amount, and the know-how to preprocess it, that makes it a hurdle for newcomers.  
> **username_1**: @black_puppydog: Please explain what you mean by "entire datasets". I'm sure you have something in mind, and I'm sure it's very different from what I am thinking of. If you want to work on every image that SDSS has ever observed, of course you'll have to download every image they have ever observed. If you want, for example, all galaxies that are members of a galaxy cluster beyond redshift of 0.1, that requires detailed analysis that noone has done yet, and you'll have some objects missing as well as false positives. So, please clarify what you mean by "entire datasets".  
> **username_2**: Question title and content do not match.  
> **username_0**: Wow, thanks for the feedback here guys. I knew this was a bit tricky, and I added a note at the top. In ML/CV, "dataset" has a very different meaning from astronomy. Over there, a dataset is relatively small (in most cases) and a standard task that everyone can "benchmark" their methods on. Common example: http://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database These datasets have their limits when it comes to representing real data. Like the datasets I'm after here, they take time and work to acquire, but then they get published and anyone can use them.  

---

## 0 answers

