<p>I finally got communication with someone who would know the answer to this and here's the answer they gave me:</p>
<blockquote>
<p>Yes, I've seen this issue before when they fit the data to a finite number of templates. Suppose you have template stellar model atmospheres at effective temperatures in 100 K bins, e.g., 5000 K, 5100 K, 5200 K, etc. For a population of observed systems with their best-fit templates, you'll only get temperatures with those finite intervals. A cursory glance at the paper indicates they also have finite template sampling of both redshift and log ssfr, leading to those aliasing issues. Hope this helps.</p>
</blockquote>
<p>Or basically, values are rounded up or down to the nearest bin size, meaning that we lose a lot of data due to say... a ssfr of -14.588777927423 or something being rounded to -14.58880 or something.</p>
<p>Long story short... if anyone in the future wants to solve this problem, you'll probably have to just make your own catalog honestly...</p>
<p>Unfortunately I can't really figure out how to filter out these problems past hard cutoffs but hopefully this allows future researchers to not spend multiple weeks trying to fix something that's unfixable.</p>

CC BY-SA 4.0
2024-03-28T06:14:30.527Z
That's great news! And thank you, it's great when a (relatively) new user takes the time to stop by and add a complete "problem solved!" answer for the benefit of future readers.
uhoh
CC BY-SA 4.0
2024-03-28T04:27:05.083Z
2024-03-28T06:00:57.870Z
2024-03-28T06:00:57.870Z
Ataaamic
<p>This question has been completely rewritten as I have found a bit more information and been helped enough to hopefully ask this question better.</p>
<p>Hi everyone. I am an undergraduate pursuing an honors research project for one of my astronomy classes. I have chosen to simply determine how the specific star formation rates (SSFRs) of galaxies have changed over time by plotting SSFR over redshift or 'z.' However, I have ran into a problem as two of my catalogs have these strange data spikes that I am confident are faulty and skewing the data I already have.</p>
<p><a href="https://i.sstatic.net/57i2k.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/57i2k.png" alt="Data from C3, a catalog identifier explained in the next paragraph" /></a></p>
<p>I am using 3 catalogs. First used is <a href="https://ui.adsabs.harvard.edu/abs/2019ApJS..243...22B/abstract" rel="nofollow noreferrer">The CANDELS/SHARDS Multiwavelength Catalog in GOODS-N</a> (specifically its J/ApJS/243/22/fast &amp; J/ApJS/243/22/phot1 tables) which we'll call C1; the second, with the best data, is the <a href="https://ui.adsabs.harvard.edu/abs/2013MNRAS.428.1958R/abstract" rel="nofollow noreferrer">Revised SWIRE photometric redshifts catalog</a> which we'll call C2; and finally. the one I need the most help with, <a href="https://ui.adsabs.harvard.edu/abs/2013ApJS..206....8M/abstract" rel="nofollow noreferrer">the Public <span class="math-container">$K_s$</span>-selected Catalog in the COSMOS/ULTRAVISTA Field</a> (notably using data from the J/ApJS/206/8/m05 &amp; J/ApJS/206/8/catalog tables, also the image above is using data from this catalog) which we'll call C3.</p>
<p>I have somewhat read the articles linked to the data I've pulled from VizieR from these catalogs however, I've yet to find anything that could be causing these spikes in the methods and data provided by these papers. I've used Vizier to already do some filtering for me related to NaN values and object type (no stars), and I've plotted the data on an Aitoff projection to ensure there's no double values between the separate catalogs.</p>
<p>The strange data spikes appear to be intrinsic to the actual data found in the tables and unrelated to anything of my own doing. I have determined this by looking at my maximum and minimum values in the data and noticing some crazy numbers that simply cannot be real (stuff like log(SFR) having values in the -30's).</p>
<p>Below is the code I used to plot the three catalogs side by side with the exact same axes. The order from left to right is: C3, C2, C1.</p>
<pre><code>fig, axs = plt.subplots(1, 3, figsize=(15, 5), tight_layout=True)

h = axs[0].hist2d(z5, logsSFR5, bins=100, norm=colors.LogNorm(), cmap='twilight')
axs[0].set(title=r'LogSSFR Over Redshift of 239,102 galaxies',
       xlabel='Redshift (z)', ylabel=r'Log SSFR   <span class="math-container">$(\log_{10}(yr^{-1}))$</span>',
       ylim=(-15,3))

h = axs[1].hist2d(zph0, logsSFR2, bins=100, norm=colors.LogNorm(), cmap='twilight')
axs[1].set(title=r'LogSSFR Over Redshift of 233,700 galaxies',
       xlabel='Redshift (z)', ylabel=r'Log SSFR   <span class="math-container">$(\log_{10}(yr^{-1}))$</span>')

h = axs[2].hist2d(newpd['zbest3'], logsSFR3, bins=100, norm=colors.LogNorm(), cmap='twilight')
axs[2].set(title=r'LogSSFR Over Redshift of 34,924 galaxies',
       xlabel='Redshift (z)', ylabel=r'Log SSFR   <span class="math-container">$(\log_{10}(yr^{-1}))$</span>',
       ylim=(-15, 3))

plt.show()
</code></pre>
<p><a href="https://i.sstatic.net/05IdF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/05IdF.png" alt="3 graphs of different data with various strange errors" /></a></p>
<p>From graphing all of these we can see two notable and annoying problems with the data from C1 and C3:</p>
<p>1.) For some reason, the data has a hard cutoff somewhere between SSFR values -8 and -5. This is also potentially a problem of false values.</p>
<p>... And 2.) There exists those data spikes at strange SSFR values undoubtedly skewing any data analysis I wish to do with all three catalogs combined.</p>
<p>So, my question to the community is, how do I filter out these data spikes from my data without ruining the <em>good</em> data I have from C2 (assuming I combine all three catalogs) or otherwise... what <em>are</em> these data spikes, as if I know perhaps I can better determine how to either filter them out OR fix their values. If that can also not be achieved I'd at least like some advice on how to smooth out whatever data remains after I am likely forced to create a hard cutoff point for the data.</p>
<p>I have a couple theories related to <em>what</em> is causing these spikes... either these are some weird default values from the data/methods the team who made these catalogs created. These could also be repeats of specific galaxies but with different redshifts (I don't know how or why that'd happen but it's an idea). My only problem is that the sheer volume of data that is... wrong, seems way too high for a &quot;minor errors in every catalog&quot; explanation.</p>
<p>The data, for those interested, is as follows:</p>
<p><a href="https://vizier.cds.unistra.fr/viz-bin/VizieR-4?-ref=VIZ65e6a38133b9e&amp;-to=-4b&amp;-from=-3&amp;-this=-4&amp;%2F%2Fsource=J%2FApJS%2F243%2F22%2Ffast%2B%2B%2BJ%2FApJS%2F243%2F22%2Fphot1&amp;%2F%2Ftables=J%2FApJS%2F243%2F22%2Fphot1&amp;%2F%2Ftables=J%2FApJS%2F243%2F22%2Ffast&amp;-out.max=50&amp;%2F%2FCDSportal=http%3A%2F%2Fcdsportal.u-strasbg.fr%2FStoreVizierData.html&amp;-out.form=HTML%20Table&amp;%2F%2Foutaddvalue=default&amp;-sort=_r&amp;-order=I&amp;-oc.form=sexa&amp;-out.src=J%2FApJS%2F243%2F22%2Ffast%2CJ%2FApJS%2F243%2F22%2Fphot1&amp;-nav=cat%3AJ%2FApJS%2F243%2F22%26tab%3A%7BJ%2FApJS%2F243%2F22%2Fphot1%7D%26tab%3A%7BJ%2FApJS%2F243%2F22%2Ffast%7D%26key%3Asource%3DJ%2FApJS%2F243%2F22%2Ffast%2B%2B%2BJ%2FApJS%2F243%2F22%2Fphot1%26HTTPPRM%3A&amp;-c=&amp;-c.eq=J2000&amp;-c.r=%20%202&amp;-c.u=arcmin&amp;-c.geom=r&amp;-source=&amp;-source=J%2FApJS%2F243%2F22%2Ffast%20J%2FApJS%2F243%2F22%2Fphot1&amp;-out.orig=standard&amp;-joincol=%7BJ%2FApJS%2F243%2F22%2Ffast%7D.%7BID%7D&amp;-joincol=ID&amp;-joincol=%7BJ%2FApJS%2F243%2F22%2Fphot1%7D.%7BID%7D&amp;-out=zbest&amp;zbest=%21%3D&amp;-out=logMass&amp;logMass=%21%3D&amp;-out=logSFR&amp;logSFR=%21%3D&amp;-out=logsSFR&amp;logsSFR=%21%3D&amp;-out=RAJ2000&amp;-out=DEJ2000&amp;-meta.ucd=2&amp;-meta=1&amp;-meta.foot=1&amp;-usenav=5&amp;-bmark=GET" rel="nofollow noreferrer">C1</a>, <a href="https://vizier.cds.unistra.fr/viz-bin/VizieR-4?-ref=VIZ65e6a29b28d15&amp;-to=-4b&amp;-from=-3&amp;-this=-4&amp;%2F%2Fsource=II%2F326&amp;%2F%2Ftables=II%2F326%2Fzcatrev&amp;-out.max=50&amp;%2F%2FCDSportal=http%3A%2F%2Fcdsportal.u-strasbg.fr%2FStoreVizierData.html&amp;-out.form=HTML%20Table&amp;%2F%2Foutaddvalue=default&amp;-sort=_r&amp;-order=I&amp;-oc.form=sexa&amp;-out.src=II%2F326%2Fzcatrev&amp;-nav=cat%3AII%2F326%26tab%3A%7BII%2F326%2Fzcatrev%7D%26key%3Asource%3DII%2F326%26HTTPPRM%3A&amp;-c=&amp;-c.eq=J2000&amp;-c.r=%20%202&amp;-c.u=arcmin&amp;-c.geom=r&amp;-source=&amp;-source=II%2F326%2Fzcatrev&amp;-out=RAJ2000&amp;-out=DEJ2000&amp;-out=logM*&amp;logM*=%21%3D&amp;-out=logSFR&amp;logSFR=%21%3D&amp;-out=zph0&amp;zph0=%21%3D&amp;-meta.ucd=2&amp;-meta=1&amp;-meta.foot=1&amp;-usenav=1&amp;-bmark=GET" rel="nofollow noreferrer">C2</a>, <a href="https://vizier.cds.unistra.fr/viz-bin/VizieR-4?-ref=VIZ65e6a13213ba2&amp;-to=-4b&amp;-from=-3&amp;-this=-4&amp;%2F%2Fsource=J%2FApJS%2F206%2F8%2Fm05%2B%2B%2BJ%2FApJS%2F206%2F8%2Fcatalog&amp;%2F%2Ftables=J%2FApJS%2F206%2F8%2Fcatalog&amp;%2F%2Ftables=J%2FApJS%2F206%2F8%2Fm05&amp;-out.max=50&amp;%2F%2FCDSportal=http%3A%2F%2Fcdsportal.u-strasbg.fr%2FStoreVizierData.html&amp;-out.form=HTML%20Table&amp;%2F%2Foutaddvalue=default&amp;-sort=_r&amp;-order=I&amp;-oc.form=sexa&amp;-nav=cat%3AJ%2FApJS%2F206%2F8%26tab%3A%7BJ%2FApJS%2F206%2F8%2Fcatalog%7D%26tab%3A%7BJ%2FApJS%2F206%2F8%2Fm05%7D%26key%3Asource%3DJ%2FApJS%2F206%2F8%2Fm05%2B%2B%2BJ%2FApJS%2F206%2F8%2Fcatalog%26HTTPPRM%3A&amp;-c=&amp;-c.eq=J2000&amp;-c.r=%20%202&amp;-c.u=arcmin&amp;-c.geom=r&amp;-source=&amp;-out.src=J%2FApJS%2F206%2F8%2Fm05%2CJ%2FApJS%2F206%2F8%2Fcatalog&amp;-source=J%2FApJS%2F206%2F8%2Fm05%20J%2FApJS%2F206%2F8%2Fcatalog&amp;-joincol=Seq&amp;-out=z&amp;z=%21%3D&amp;-out=lmass&amp;lmass=%21%3D&amp;-out=lsfr&amp;lsfr=%21%3D&amp;-out=lssfr&amp;lssfr=%21%3D&amp;-out=RAJ2000&amp;-out=DEJ2000&amp;Star=%21%3D1&amp;-meta.ucd=2&amp;-meta=1&amp;-meta.foot=1&amp;-usenav=1&amp;-bmark=GET" rel="nofollow noreferrer">C3</a></p>
<p>(NOTE: you <em>will</em> need to look at the &quot;preferences&quot; box on the right side and select your choice of file format and set the number of returns to unlimited to get the full scope of the data on all of these, you can otherwise truncate the data down to about 1000 rows for an HTML table before things become unbearably slow)</p>
<p>Thank you for any help and patience you provide.</p>

CC BY-SA 4.0
2024-02-09T08:40:16.800Z
How are star formation rates below $10^{-15}$ per year estimated? Have you read the papers associated with the catalogues to see whether they say anything about the extremely low values?
ProfRob
CC BY-SA 4.0
2024-02-09T22:53:02.767Z
Yeah I thought those values were strange as well. I have just been taking the data in these catalogs and converting it into SSFR by subtracting logM* from logSFR to 'divide' them. That's a very good idea to check out the papers more thoroughly thank you! I am very new to this so forgive me for making such a simple mistake.
Ataaamic
CC BY-SA 4.0
2024-02-10T07:40:06.963Z
Can you cite and link to the sources of your catalogues? (And all the data used to produce your plots).
ProfRob
CC BY-SA 4.0
2024-02-23T03:07:51.443Z
Hi again, sorry for the late response, I got hit with COVID and have been catching back up, the post has been updated with notably the problem catalogs. Definitely don't kill yourself over it but thanks for the help thus far, your initial comment was very helpful in at least narrowing my data.
Ataaamic
CC BY-SA 4.0
2024-03-05T05:46:42.440Z
@ProfRob there's a new comment and update to the post.
uhoh
<data-analysis><visualization>
What are These Strange Data Spikes in my Catalog Data?
Ataaamic
# What are These Strange Data Spikes in my Catalog Data?

(Asked by: username_0 on 2024-02-09)

This question has been completely rewritten as I have found a bit more information and been helped enough to hopefully ask this question better.
Hi everyone. I am an undergraduate pursuing an honors research project for one of my astronomy classes. I have chosen to simply determine how the specific star formation rates (SSFRs) of galaxies have changed over time by plotting SSFR over redshift or 'z.' However, I have ran into a problem as two of my catalogs have these strange data spikes that I am confident are faulty and skewing the data I already have.

I am using 3 catalogs. First used is The CANDELS/SHARDS Multiwavelength Catalog in GOODS-N (specifically its J/ApJS/243/22/fast & J/ApJS/243/22/phot1 tables) which we'll call C1; the second, with the best data, is the Revised SWIRE photometric redshifts catalog which we'll call C2; and finally. the one I need the most help with, the Public $K_s$-selected Catalog in the COSMOS/ULTRAVISTA Field (notably using data from the J/ApJS/206/8/m05 & J/ApJS/206/8/catalog tables, also the image above is using data from this catalog) which we'll call C3.
I have somewhat read the articles linked to the data I've pulled from VizieR from these catalogs however, I've yet to find anything that could be causing these spikes in the methods and data provided by these papers. I've used Vizier to already do some filtering for me related to NaN values and object type (no stars), and I've plotted the data on an Aitoff projection to ensure there's no double values between the separate catalogs.
The strange data spikes appear to be intrinsic to the actual data found in the tables and unrelated to anything of my own doing. I have determined this by looking at my maximum and minimum values in the data and noticing some crazy numbers that simply cannot be real (stuff like log(SFR) having values in the -30's).
Below is the code I used to plot the three catalogs side by side with the exact same axes. The order from left to right is: C3, C2, C1.
fig, axs = plt.subplots(1, 3, figsize=(15, 5), tight_layout=True)

h = axs[0].hist2d(z5, logsSFR5, bins=100, norm=colors.LogNorm(), cmap='twilight')
axs[0].set(title=r'LogSSFR Over Redshift of 239,102 galaxies',
       xlabel='Redshift (z)', ylabel=r'Log SSFR   $(\log_{10}(yr^{-1}))$',
       ylim=(-15,3))

h = axs[1].hist2d(zph0, logsSFR2, bins=100, norm=colors.LogNorm(), cmap='twilight')
axs[1].set(title=r'LogSSFR Over Redshift of 233,700 galaxies',
       xlabel='Redshift (z)', ylabel=r'Log SSFR   $(\log_{10}(yr^{-1}))$')

h = axs[2].hist2d(newpd['zbest3'], logsSFR3, bins=100, norm=colors.LogNorm(), cmap='twilight')
axs[2].set(title=r'LogSSFR Over Redshift of 34,924 galaxies',
       xlabel='Redshift (z)', ylabel=r'Log SSFR   $(\log_{10}(yr^{-1}))$',
       ylim=(-15, 3))

plt.show()


From graphing all of these we can see two notable and annoying problems with the data from C1 and C3:
1.) For some reason, the data has a hard cutoff somewhere between SSFR values -8 and -5. This is also potentially a problem of false values.
... And 2.) There exists those data spikes at strange SSFR values undoubtedly skewing any data analysis I wish to do with all three catalogs combined.
So, my question to the community is, how do I filter out these data spikes from my data without ruining the good data I have from C2 (assuming I combine all three catalogs) or otherwise... what are these data spikes, as if I know perhaps I can better determine how to either filter them out OR fix their values. If that can also not be achieved I'd at least like some advice on how to smooth out whatever data remains after I am likely forced to create a hard cutoff point for the data.
I have a couple theories related to what is causing these spikes... either these are some weird default values from the data/methods the team who made these catalogs created. These could also be repeats of specific galaxies but with different redshifts (I don't know how or why that'd happen but it's an idea). My only problem is that the sheer volume of data that is... wrong, seems way too high for a "minor errors in every catalog" explanation.
The data, for those interested, is as follows:
C1, C2, C3
(NOTE: you will need to look at the "preferences" box on the right side and select your choice of file format and set the number of returns to unlimited to get the full scope of the data on all of these, you can otherwise truncate the data down to about 1000 rows for an HTML table before things become unbearably slow)
Thank you for any help and patience you provide.

**Tags:** data-analysis, visualization
> **Comments:**
> **username_1**: How are star formation rates below $10^{-15}$ per year estimated? Have you read the papers associated with the catalogues to see whether they say anything about the extremely low values?  
> **username_0**: Yeah I thought those values were strange as well. I have just been taking the data in these catalogs and converting it into SSFR by subtracting logM* from logSFR to 'divide' them. That's a very good idea to check out the papers more thoroughly thank you! I am very new to this so forgive me for making such a simple mistake.  
> **username_1**: Can you cite and link to the sources of your catalogues? (And all the data used to produce your plots).  
> **username_0**: Hi again, sorry for the late response, I got hit with COVID and have been catching back up, the post has been updated with notably the problem catalogs. Definitely don't kill yourself over it but thanks for the help thus far, your initial comment was very helpful in at least narrowing my data.  
> **username_2**: @ProfRob there's a new comment and update to the post.  

---

## 1 answers

### Answer by username_0 on 2024-03-28. Score: 2 | Accepted answer

I finally got communication with someone who would know the answer to this and here's the answer they gave me:

> Yes, I've seen this issue before when they fit the data to a finite number of templates. Suppose you have template stellar model atmospheres at effective temperatures in 100 K bins, e.g., 5000 K, 5100 K, 5200 K, etc. For a population of observed systems with their best-fit templates, you'll only get temperatures with those finite intervals. A cursory glance at the paper indicates they also have finite template sampling of both redshift and log ssfr, leading to those aliasing issues. Hope this helps.

Or basically, values are rounded up or down to the nearest bin size, meaning that we lose a lot of data due to say... a ssfr of -14.588777927423 or something being rounded to -14.58880 or something.

Long story short... if anyone in the future wants to solve this problem, you'll probably have to just make your own catalog honestly...

Unfortunately I can't really figure out how to filter out these problems past hard cutoffs but hopefully this allows future researchers to not spend multiple weeks trying to fix something that's unfixable.

> **Comments:**
> **username_2**: That's great news! And thank you, it's great when a (relatively) new user takes the time to stop by and add a complete "problem solved!" answer for the benefit of future readers.  

---

