<p>Yes, astronomical objects are contained in multiple catalogues. These catalogues will be observations made at different times, with different instruments and at possibly different wavelengths.</p>
<p>Cross-matching the catalogues isn't foolproof and there is no universal recipe for cross-matching or universal identifier for astronomical objects. Real objects are brighter at some wavelengths than others; are variable; and they move!</p>
<p>Very often one matches the catalogues based on similarity of the catalogued objects' positions (RA and Dec). That requires knowledge of how precise the coordinates are and possibly how fast objects are moving on the sky (proper motions).</p>
<p>An excellent (but non-trivial to use/learn) resource is <a href="https://www.ascl.net/1101.010" rel="nofollow noreferrer">Topcat</a> - a Java-based astronomical catalogue and display tool. It includes a handy interface to query and cross-match <em>many</em> different catalogues.</p>

CC BY-SA 4.0
2023-03-30T10:43:05.967Z
Thanks for your answer and hinting to cross-matching. I have searched a little bit on this topic and found that it is very similar to my use case. I am wondering if something like a groundtruth or a source of truth exist? So something for which cross-matching for a dataset (or a subset of it) has been checked (manually) to allow evaluation of newly developed cross-matching methods.
Lasklu
CC BY-SA 4.0
2023-04-04T16:31:47.757Z
That link is broken for me - perhaps because I'm based in Europe / The Netherlands? Otherwise it's rather unique for an answer younger than a week old ...
Glorfindel
CC BY-SA 4.0
2023-04-04T16:35:32.750Z
@Glorfindel their site appears to be down. The address is correct.
ProfRob
CC BY-SA 4.0
2023-03-29T21:09:29.523Z
2023-04-04T16:27:38.300Z
2023-04-04T16:27:38.300Z
ProfRob
<p>I am currently doing my PhD in computer science, and I have almost no astronomical knowledge. I am currently researching on duplicate detection in databases (so detection of multiple entries referring to the same real world object).</p>
<p>For that, I put my focus on datasets consisting mainly of numerical data (e.g. measurement data). However, there are hardly any data sets in the computer science domain. An idea from me was to take data sets from the astronomy domain. I have seen that there are, for example, star catalogs (such as Gaia GDR-3 or USNO-B1.0), but the interpretation of these catalogs is difficult for me.</p>
<p>Therefore, my question: Are there data sets in the field of astronomy that contain duplicates (for example, that the same star appears several times in the same database)? Or are there multiple datasets that contain/refer to the same star? If so, could you please point me to such datasets?</p>

CC BY-SA 4.0
2023-03-29T17:45:50.737Z
I'm not quite sure I understand. Are you asking for sources of badly-vetted data? The answer certainly is 'yes', and it will also be 'yes' to "are there errors in current databases. The question is: are they known where these errors are. Because... if they are known, they likely will be corrected... You might go through the revision history of e.g. the gaia catalogues, or also of others, e.g. the hubble guide star catalogue, compare revision 1.0 and 1.1
planetmaker
CC BY-SA 4.0
2023-03-29T21:03:01.583Z
You just listed two.
ProfRob
CC BY-SA 4.0
2023-03-29T21:04:20.500Z
Hi, exactly, I mean such dirty data. My goal is to develop a method, which is able to automatically detect (and remove) such duplicates (the records do not need to be exactly the same but similar). To develop and evaluate a method like this, I need a groundtruth. So records for which I know whether they are a duplicate or not. Do you know if something like this exist? Another possibility might be to use two data sources which contain the same stars, but then I need to know, which star in source A is the same as a specific star in source B. Thank you!
Lasklu
CC BY-SA 4.0
2023-03-29T21:05:39.250Z
Hi @ProfRob, but do I get the information in these sources which star in source A relates to which star in source B? Thanks for your help
Lasklu
CC BY-SA 4.0
2023-03-30T07:55:07.190Z
Asteroids. But detecting duplicate asteroids requires calculating their orbits.
PM 2Ring
CC BY-SA 4.0
2023-03-30T15:12:30.273Z
@uhoh Understood! Will keep a note :) [flag the comment once acknowledged]
Nilay Ghosh
<star><data-analysis><astrometry>
Do astronomical datasets exist that receive duplicates or are there multiple datasets that contain the same entities (stars)?
Lasklu
# Do astronomical datasets exist that receive duplicates or are there multiple datasets that contain the same entities (stars)?

(Asked by: username_0 on 2023-03-29)

I am currently doing my PhD in computer science, and I have almost no astronomical knowledge. I am currently researching on duplicate detection in databases (so detection of multiple entries referring to the same real world object).

For that, I put my focus on datasets consisting mainly of numerical data (e.g. measurement data). However, there are hardly any data sets in the computer science domain. An idea from me was to take data sets from the astronomy domain. I have seen that there are, for example, star catalogs (such as Gaia GDR-3 or USNO-B1.0), but the interpretation of these catalogs is difficult for me.

Therefore, my question: Are there data sets in the field of astronomy that contain duplicates (for example, that the same star appears several times in the same database)? Or are there multiple datasets that contain/refer to the same star? If so, could you please point me to such datasets?

**Tags:** star, data-analysis, astrometry
> **Comments:**
> **username_1**: I'm not quite sure I understand. Are you asking for sources of badly-vetted data? The answer certainly is 'yes', and it will also be 'yes' to "are there errors in current databases. The question is: are they known where these errors are. Because... if they are known, they likely will be corrected... You might go through the revision history of e.g. the gaia catalogues, or also of others, e.g. the hubble guide star catalogue, compare revision 1.0 and 1.1  
> **username_2**: You just listed two.  
> **username_0**: Hi, exactly, I mean such dirty data. My goal is to develop a method, which is able to automatically detect (and remove) such duplicates (the records do not need to be exactly the same but similar). To develop and evaluate a method like this, I need a groundtruth. So records for which I know whether they are a duplicate or not. Do you know if something like this exist? Another possibility might be to use two data sources which contain the same stars, but then I need to know, which star in source A is the same as a specific star in source B. Thank you!  
> **username_0**: Hi @ProfRob, but do I get the information in these sources which star in source A relates to which star in source B? Thanks for your help  
> **username_3**: Asteroids. But detecting duplicate asteroids requires calculating their orbits.  
> **username_4**: @uhoh Understood! Will keep a note :) [flag the comment once acknowledged]  

---

## 1 answers

### Answer by username_2 on 2023-03-29. Score: 3

Yes, astronomical objects are contained in multiple catalogues. These catalogues will be observations made at different times, with different instruments and at possibly different wavelengths.

Cross-matching the catalogues isn't foolproof and there is no universal recipe for cross-matching or universal identifier for astronomical objects. Real objects are brighter at some wavelengths than others; are variable; and they move!

Very often one matches the catalogues based on similarity of the catalogued objects' positions (RA and Dec). That requires knowledge of how precise the coordinates are and possibly how fast objects are moving on the sky (proper motions).

An excellent (but non-trivial to use/learn) resource is [Topcat](https://www.ascl.net/1101.010) - a Java-based astronomical catalogue and display tool. It includes a handy interface to query and cross-match *many* different catalogues.

> **Comments:**
> **username_0**: Thanks for your answer and hinting to cross-matching. I have searched a little bit on this topic and found that it is very similar to my use case. I am wondering if something like a groundtruth or a source of truth exist? So something for which cross-matching for a dataset (or a subset of it) has been checked (manually) to allow evaluation of newly developed cross-matching methods.  
> **username_5**: That link is broken for me - perhaps because I'm based in Europe / The Netherlands? Otherwise it's rather unique for an answer younger than a week old ...  
> **username_2**: @Glorfindel their site appears to be down. The address is correct.  

---

