## Enhancing weak lensing redshift distribution characterization by optimizing the Dark Energy Survey Self-Organizing Map Photoz method

- A. Campos 1 2 , ⋆ , B. Yin , S. Dodelson 3 1 2 , , A. Amon 4 5 , , A. Alarcon 6 7 , , C. Sánchez , G. M. Bernstein , 8 8 G. Giannini 9 10 , , J. Myles 11 , S. Samuroff 12 , O. Alves 13 , F. Andrade-Oliveira 13 , K. Bechtol 14 , M. R. Becker , 6 J. Blazek 12 , H. Camacho 15 16 17 , , , A. Carnero Rosell 15 18 19 , , , M. Carrasco Kind 20 21 , , R. Cawthon 22 , C. Chang 23 9 , , R. Chen , A. Choi 3 24 , J. Cordero 25 , C. Davis 26 , J. DeRose 27 , H. T. Diehl 28 , C. Doux 8 29 , , A. Drlica-Wagner 9 23 28 , , , K. Eckert , 8 T. F. Eifler 30 31 , , J. Elvin-Poole 32 , S. Everett 30 , X. Fang 31 33 , , A. Ferté 34 , O. Friedrich , 5 M. Gatti , 8 D. Gruen 35 , R. A. Gruendl 20 21 , , I. Harrison 36 , W. G. Hartley 37 , K. Herner 28 , H. Huang 31 38 , , E. M. Huff 30 , M. Jarvis , 8 E. Krause 31 , N. Kuropatkin 28 , P.-F. Leget 26 , N. MacCrann 39 , J. McCullough 26 , A. Navarro-Alsina 40 , S. Pandey , 8 J. Prat 41 23 , , M. Raveri 42 , R. P. Rollins 25 , A. Roodman 26 34 , , R. Rosenfeld 43 15 , , A. J. Ross 44 , E. S. Rykoff 34 26 , , J. Sanchez 45 , L. F. Secco , I. Sevilla-Noarbe 9 46 , E. Sheldon 17 , T. Shin 47 , M. A. Troxel , I. Tutusaus 3 48 , T. N. Varga 49 50 51 , , , R. H. Wechsler 26 52 34 , , , B. Yanny 28 , Y. Zhang 53 , J. Zuntz 54 , M. Aguena 15 , J. Annis 28 , D. Bacon 55 , S. Bocquet 35 , D. Brooks 56 , D. L. Burke 26 34 , , J. Carretero 10 , F. J. Castander 7 57 , , M. Costanzi 58 59 60 , , , L. N. da Costa 15 , J. De Vicente 46 , P. Doel 56 , I. Ferrero 61 , B. Flaugher 28 , J. Frieman 9 28 , , J. GarcíaBellido 62 , E. Gaztanaga 7 55 57 , , , G. Gutierrez 28 , S. R. Hinton 63 , D. L. Hollowood 64 , K. Honscheid 44 65 , , D. J. James 66 , K. Kuehn 67 68 , , M. Lima 15 69 , , H. Lin 28 , J. L. Marshall 70 , J. Mena-Fernández 71 , F. Menanteau 20 21 , , R. Miquel 10 72 , , R. L. C. Ogando 73 , M. Paterno 28 , M. E. S. Pereira 74 A. Pieres 15 73 , , A. A. Plazas Malagón 26 34 , , A. Porredon 46 75 , , E. Sanchez 46 , D. Sanchez Cid 46 , M. Smith 76 , E. Suchyta 77 , M. E. C. Swanson 21 , G. Tarle 13 , C. To 44 , V. Vikram 78 , and N. Weaverdyck 27 33 ,

## DES Collaboration

5 August 2024

## ABSTRACT

Characterization of the redshift distribution of ensembles of galaxies is pivotal for large scale structure cosmological studies. In this work, we focus on improving the Self-Organizing Map (SOM) methodology for photometric redshift estimation (SOMPZ), specifically in anticipation of the Dark Energy Survey Year 6 (DES Y6) data. This data set, featuring deeper and fainter galaxies than DES Year 3 (DES Y3), demands adapted techniques to ensure accurate recovery of the underlying redshift distribution. We investigate three strategies for enhancing the existing SOM-based approach used in DES Y3: 1) Replacing the Y3 SOM algorithm with one tailored for redshift estimation challenges; 2) Incorporating g -band flux information to refine redshift estimates (i.e. using griz fluxes as opposed to only riz ); 3) Augmenting redshift data for galaxies where available. These methods are applied to DES Y3 data, and results are compared to the Y3 fiducial ones. Our analysis indicates significant improvements with the first two strategies, notably reducing the overlap between redshift bins. By combining strategies 1 and 2, we have successfully managed to reduce redshift bin overlap in DES Y3 by up to 66 % . Conversely, the third strategy, involving the addition of redshift data for selected galaxies as an additional feature in the method, yields inferior results and is abandoned. Our findings contribute to the advancement of weak lensing redshift characterization and lay the groundwork for better redshift characterization in DES Year 6 and future stage IV surveys, like the Rubin Observatory.

Key words: gravitational lensing: weak - methods: statistical - techniques: photometric - galaxies: distances and redshifts cosmology: observations.

## 1 INTRODUCTION

Large galaxy surveys afford us promising opportunities to learn about the constituents of the universe and the way they are dis-

tributed in space. This in turn can help us connect fundamental physics - for example of dark energy and dark matter - to observations and to learn about the nature of the most mysterious substances postulated to exist. Photometric surveys can capture images of many more galaxies than spectroscopic surveys but are hindered by the inability to measure accurate distances to the objects they image. Photometric redshifts , or distances inferred from the observed galaxy properties such as colors, have become essential in extracting information about cosmology from these large surveys.

One of the observables for which photometric redshifts play a major role in is weak gravitational lensing. Weak gravitational lensing is a fundamental cosmological probe that enables the investigation of the large-scale structure of the universe and has been employed in many contemporary analyses (see, e.g., Heymans et al., 2013; Dark Energy Survey Collaboration 2016; Jee, Tyson, Hilbert, Schneider, Schmidt &amp; Wittman 2016; Hildebrandt et al., 2017; Troxel et al., 2018; Hikage et al., 2019; Hamana et al., 2020; Asgari et al., 2021; Loureiro et al., 2022; Amon et al. 2022; Secco, Samuroff et al. 2022; Doux et al., 2022; Dalal et al., 2023; Li et al., 2023).

In photometric surveys, while galaxy positions serve as tracers of matter density, it is by measuring the distortions in the shapes and orientations of background galaxies induced by the gravitational influence of intervening mass distributions that we can obtain a direct connection to the underlying density field. However, to extract precise cosmological information from weak lensing, it is imperative to have a robust characterization of the redshift distribution, n z ( ) , of the observed galaxies. Measuring the spectrum of each galaxy in a large optical imaging survey, though, is unfeasible, and therefore spectra are available only for small subsets of galaxies. As a result, photometric surveys heavily rely on limited, noisy photometric bands to estimate redshifts. The main challenge arises from degeneracies in the color-redshift relation, which prevent the unique determination of redshifts from wide-band photometry. In addition, since lensing samples tend to be fainter/deeper/bluer, they can not be characterised as accurately as the sort of bright red galaxies typically used as lens samples. Therefore, the accurate characterization of the redshift distribution thus becomes one of the main challenges, and yet a crucial aspect, for interpreting gravitational lensing measurements, including cosmic shear and galaxy-galaxy lensing correlation functions.

Techniques to estimate photometric redshifts date back several decades. Template-fitting methods compare the observed photometric data of galaxies with a library of template spectra, allowing for redshift estimation (Benítez 2000; Ilbert et al., 2009). However, this approach can be sensitive to template choices and might not capture all spectral features accurately, leading to biases in redshift predictions, particularly for poorly represented galaxy populations. Empirical approaches exploit statistical correlations between observable features (e.g., color-redshift relations) to estimate photometric redshifts (Blake &amp; Bridle 2005; Mandelbaum et al., 2008). However, these methods necessitate accurate and extensive spectroscopic data for calibration. Machine learning techniques, such as artificial neural networks or random forests, have gained popularity due to their ability to learn complex photometry-redshift relationships from training datasets (Collister &amp; Lahav 2004; Carrasco Kind &amp; Brunner 2013). Nonetheless, these methods heavily rely on the quality and representativeness of the training data, and their performance can degrade when extrapolating to redshift regimes not adequately covered by the training set. Most recently, unsupervised machine learning methods that compress data embedded in a Bayesian approach have emerged as a promising direction (see for example, Buchs et al. 2019).

A Self-Organizing Map (SOM), also known as a Kohonen map

(Kohonen 1982), is an unsupervised machine learning algorithm and neural network architecture used for dimensionality reduction and data mining. It allows for complex and high-dimensional data to be represented in a lower-dimensional space while preserving the topological relationships between data points. For the purposes of redshift estimation, when assigning each galaxy to a cell in a Self Organizing Map (SOM), galaxies with similar redshifts are grouped in the same cell, or 'nearby' cells if the grouping is in a 2D grid, and the redshift distribution for those galaxies can be determined fairly accurately. DES used this technique in its analyses of the data from the first three years of observations, i.e., the Y3 weak lensing cosmological analyses (Myles, Alarcon et al. 2021; DES Collaboration 2022; Amon et al. 2022; Secco, Samuroff et al. 2022), and also as an additional validation follow-up of the Y3 lens sample calibration (Giannini et al. 2023). The KiDS collaboration has also used it (Wright, Hildebrandt, van den Busch &amp; Heymans 2020) to achieve few-percent level constaints on the mean of the redshift distribution for each redshift bin. It has emerged as a viable candidate for upcoming surveys such as Rubin and Euclid (Ivezi´ c et al., 2019; Laureijs et al., 2011), but improvements are required to achieve subpercent level constraints (The LSST Dark Energy Science Collaboration (DESC) 2021; Euclid Collaboration 2020).

Here, we explore several improvements to the SOM methodology used in DES-Y3, ahead of the final DES Year-6 (Y6) analysis. This serves two primary purposes: (i) allowing for the potential of improving on the Y3 implementation and (ii) stress-testing the robustness of the cosmological conclusions. The latter point is particularly important in the context of more stringent requirements that come with more statistically powerful data, as well as applying this methodology to deeper photometric data. If different implementations of the SOM framework give the same answer, we will become more confident applying it moving forward as statistical errors continue to decrease.

First, we test replacing the SOM algorithm used in Y3 by the one proposed in Sánchez, Raveri, Alarcon &amp; Bernstein (2020). This new algorithm implements a Self-Organizing Map with a distance metric specific for the problem of photometric redshift estimation. Although it was shown in the Year 3 analysis that the generic SOM algorithm is already successful at estimating redshifts at the percentage level (Myles, Alarcon et al. 2021), we hope that by introducing a SOM that is tailored for the problem of redshifts, we can achieve even better precision. Second, we show the impact that including an extra flux band, the g -band, has on our ability to obtain welldefined redshift bins, motivating the importance of well calibrated point spread functions in those limits, such that we do not lose this very crucial piece of information. Finally, we try adding the redshift information of the spectroscopic galaxies in our sample as an additional feature in the SOM. This is an unconventional approach to an unsupervised machine learning method, since the norm is for the quantity being estimated to not be included among the SOM features; we indeed find that it is not beneficial, but we present our attempts for the sake of completeness.

Section 2 details the DES Year 3 data that we re-analyse with the proposed modifications to our redshift estimation method. Section 3 presents a summary of the Self-Organizing Map algorithm and the SOMPZ method for redshift estimation used by DES. Section 4 presents the proposed modifications to the SOMPZ method used in DES Y3. Section 5 discusses the results of implementing these different modifications, and their impact on the redshift bins. Finally, Section 6 shows the impact on cosmological parameter constraints.

Table 1. Summary of the catalogs used in DES Year 3 for redshift estimation of the weak lensing source galaxies, including the area covered and the number of galaxies.

| Sample      | Area (sq. deg.)   | Number of Galaxies   |
|-------------|-------------------|----------------------|
| Wide        | 4143              | 100,208,944          |
| Deep        | 5.88              | 2.8M                 |
| Redshift    | -                 | 57,000               |
| Deep/Balrog | -                 | 2,417,437            |

## 2 THE DARK ENERGY SURVEY

We summarize the samples used in this work in Table 1. These are the same ones used for redshift characterization of the weak lensing source galaxies in DES Year 3. The strategy employs SelfOrganizing maps (SOMs), which we detail in Section 3, and leverages the information present in three catalogs - wide, deep, and redshift - as well as Balrog injections:

Wide: The weak lensing source catalog is described in detail in Gatti, Sheldon et al. (2021). After the applied selections ˆ s in magnitudes and colors, the wide sample is composed of 100 208 944 , , galaxies, spread over 4143 square degrees. DES has made flux measurements for all of these galaxies in the griz bands of the electromagnetic spectrum (although the g -band was not used in DES-Y3).

Deep: The deep sample refers to the DES deep field galaxies, which have measured fluxes in additional bands ugrizJHK . There are four deep-fields mapped in DES Y3, see Hartley, Choi et al. (2021), that added cover an area of 5 88 . square degrees. Notice that Y -band data in the deep fields had large offsets between the constituent exposures, and therefore could not be used.

Redshift: Asubset of the deep field galaxies have accurate redshifts obtained from a variety of external data sets (for details of the sample and a list of external data included, see Myles, Alarcon et al. 2021). We call this set, containing approximately 57 000 , galaxies, the redshift sample .

Balrog: In order to connect the information in our samples, we use Balrog injections. The Balrog software, developed by Suchyta et al., (2016), enables the creation of simulated galaxies, or Balrog injections, which are inserted into authentic images. These synthetic galaxies are designed based on the DES deep field photometry and are placed multiple times at various positions across the broader wide-field footprint, as specified in Everett et al., (2022). The resulting catalogue, called the deep/Balrog sample, includes 2 417 437 , , injection-realization pairs, each of which has both deep and wide photometric data. This sample is a crucial element of our redshift calibration inference technique.

In what follows, we denote the wide data by ˆ x with covariance matrix ˆ Σ and the deep data by x with covariance Σ , and the selection by ˆ s , following the notation in Buchs, Davis et al. (2019); Myles, Alarcon et al. (2021). The wide field data vector has three components, ˆ x = [ r,i,z ] . For the deep fields, there are 3 infrared bands available - J,H, and K - and the DES u,g bands are also used, such that x = [ u, g, r, i, z, J,H,K ] has eight components in total. Since the redshift galaxies are a subset of the deep galaxies, they too have the 8 components x ; Balrog galaxies typically have approximately 15 realizations ˆ x (corresponding to the number of wide field injections) for a single x (corresponding to a single deep field galaxy).

## 3 SELF-ORGANIZING MAPS FOR PHOTOMETRIC REDSHIFTS

In what follows, we review the SOM standard algorithm, and describe the SOMPZ method, i.e., how SOMs are used in practice for redshift estimation in DES-Y3.

## 3.1 The SOM Algorithm

A Self-Organizing Map (SOM) is a type of Artificial Neural Network (ANN) that produces a discretized, lower dimensional, representation of the input space, while preserving its topology. Proposed by Kohonen (1982), it is an unsupervised Machine Learning method that uses soft competitive learning: the cells of the map (also known as nodes or neurons) compete to most closely resemble each training example until the best matching unit (BMU) is found, then the winner and its neighborhood are adapted.

Consider a set of n training samples, each with m features, i.e., for each sample we have an input vector x ∈ R m . In our case, for instance, each sample is a single galaxy and the features are its fluxes (or colors or magnitudes) in m bands. The SOM can be understood as collection of C cells arranged in a l -dimensional grid that has a specified topology. Each cell is associated with a weight vector ω k ∈ R m , where k = 1 , · · · , C . Both the input and weight vectors live in the input space, while the cells live in the output, or lattice, space.

The training of a SOM is relatively straightforward. The weights are initialized to random or from data samples and the learning happens in three stages: Competition, cooperation, and weight adaptation.

- · Competition: at each step, a random sample of the training set is presented to the self-organizing map. The cell whose weight vector is the closest to the sample vector is identified as the best matching unit (BMU):

$$c _ { b } = \arg \min _ { k } d ( \boldsymbol x, \omega _ { k } ). \quad \quad \quad ( 1 )$$

The degree of "closeness" is determined by evaluating a distance metric d x,ω ( k ) between the sample and the cell in the map. Typically the Euclidian distance is used, but in Buchs, Davis et al. (2019) the chi-square distance was chosen:

$$d ^ { 2 } ( x, \omega _ { k } ) = ( x - \omega _ { k } ) ^ { \top } \Sigma ^ { - 1 } ( x - \omega _ { k } ), \quad \quad ( 2 )$$

where Σ is the covariance matrix for the training vector, x . The cell minimizing this distance is identified as the BMU, and the sample is then assigned to it.

- · Cooperation: to preserve the topology of the input space, not only the BMU is identified and updated, but also its neighborhood. The neighborhood function H b,k ( ) t creates the connection between the input space and the cells in the map, responsible for the selforganizing property of the map. The size of the BMU's neighborhood decreases as a function of time steps, t . In addition, H b,k should decrease as the distance from the BMU increases. It must also satisfy the properties that it is maximum in the winning cell b and is symmetric about it. A Gaussian neighborhood function attend those requirements:

$$H _ { b, k } ( t ) = \exp [ - D _ { b, k } ^ { 2 } / \sigma ^ { 2 } ( t ) ]. \text{ \quad \ \ } ( 3 )$$

The distance between the BMU, c b , and any cell on the map, c k ,

is the Euclidian distance in the l -dimensional map:

$$D _ { b, k } ^ { 2 } = \sum _ { i = 1 } ^ { l } ( c _ { b, i } - c _ { k, i } ) ^ { 2 }. \quad \quad ( 4 ) \quad,$$

The width of the Gaussian kernel is given by

$$\sigma ( t ) = \sigma _ { s } ^ { 1 - t / t _ { m a x } }. \quad \quad \ ( 5 ) \quad \ \ \ \ c e$$

At the beginning of the training, σ s should be large enough that most of the map is initially affected. As the training progresses, the width shrinks until only the BMU and its closest neighbours are significantly affected by new data.

- · Weight adaptation: once the BMU is computed, we can calculate the updated value of the weight vectors for the t +1 -th iteration through the following relation:

$$\omega _ { k } ( t + 1 ) = \omega _ { k } ( t ) + a ( t ) H _ { b, k } ( t ) [ x ( t ) - \omega _ { k } ( t ) ], \quad ( 6 ) \quad \text{(i)}$$

where t is the current time step in training, a t ( ) is the learning rate:

$$a ( t ) = a _ { 0 } ^ { t / t _ { m a x } }, \quad \ \ \ ( 7 ) \quad \ \ s e.$$

where a 0 ∈ [0 1] , . In each iteration, this update function is applied to each of the cells in the map.

Different galaxy samples can be assigned to a trained SOM using the same methodology: for each object, the distance is computed, and the galaxy is assigned to the BMU. However, it is important to note that problems may arise when assigning a sample to a SOM that has been trained with a sample that is not representative of the color space of the samples to be assigned.

These steps describe the standard SOM algorithm, which has been applied for the purpose of redshift estimation in previous works (see e.g. Masters et al., (2015); Speagle et al., (2019); Buchs, Davis et al. (2019)), including the DES-Y3 analyses (Myles, Alarcon et al. 2021).

## 3.2 Dark Energy Survey SOMPZ

In order to estimate the redshift distribution of the wide sample, we construct two SOMs:

## (i) Deep SOM:

- (a) Training sample: deep data
- (b) Assignment sample: deep data (including the sub-sample with redshifts)

## (ii) Wide SOM:

- (a) Training sample: wide data (or sufficiently large subsample thereof)
- (b) Assignment sample: wide data

Figure 1 shows the number of objects, the redshift distribution and the standard deviation in the deep (top) and wide (bottom) SOMs used in the DES Y3 analysis. As we can see, and the name suggests, the deep SOM has deeper, and fainter, galaxies, going to higher z values compared to the wide SOM.

Each SOM cell acts as a way of discretizing the continuous color and color-magnitude spaces spanned by x (ˆ) x and Σ ( ˆ ) Σ into discrete categories c (ˆ) c . Therefore, the probability distribution function for the redshift of an ensemble of galaxies, conditioned on being observed in a particular cell ˆ c , and on passing a selection function ˆ s , can be written by marginalizing over the deep-field information:

$$p ( z | \hat { c }, \hat { s } ) = \sum _ { c } p ( z | c, \hat { c }, \hat { s } ) p ( c | \hat { c }, \hat { s } ).$$

We then assign each cell ˆ c to a tomographic bin (see Myles, Alarcon et al. (2021) for the details on the assignment algorithm) and construct the n z ( ) of each tomographic bin ( ˆ b ) by summing over the cells belonging to the bin:

$$\text{n} \\ \text{are} \quad & n _ { \tilde { b } } ( z ) \equiv p ( z | \hat { b }, \hat { s } ) = \sum _ { \hat { c } \in \tilde { b } } p ( z | \hat { c }, \hat { s } ) p ( \hat { c } | \hat { s }, \hat { b } ) \\ = \sum _ { \hat { c } \in \tilde { b } } \sum _ { c } p ( z | c, \hat { c }, \hat { s } ) p ( c | \hat { c }, \hat { s } ) p ( \hat { c } | \hat { s }, \hat { b } ). \quad ( 1 0 ) \\ \text{:u-} \quad & \quad \cdot \. \. \. \. \. \. \. \. \. \. \. \. \. \. \.$$

In the equation above, each term is obtained from one of the galaxy samples we are using:

(i) p z c,c,s ( | ˆ ˆ) is computed from the redshift sample subset of the deep sample, which contains spectroscopic redshifts, deep photometry, and wide-field Balrog realisations. It tells us the probability of getting a redshift z , given the deep cell c , the wide cell ˆ c , and the selection ˆ s .

(ii) p c c, s ( | ˆ ˆ) is computed from the Balrog injections of the entire deep sample. It tells us the probability of ending up in the deep cell c , given the wide cell ˆ c and the selection ˆ s . We call this term the transfer function, because it connects the deep and wide cells. It is computed from Balrog realisations, because it requires both widefield and deep-field photometry to be available.

(iii) p c s,b (ˆ ˆ ˆ ) | is computed from the wide sample. It tells us the probability that a galaxy in bin ˆ b is in the wide SOM cell ˆ c . Therefore, cells with very few galaxies in them are down-weighted when determining the redshift distribution of the bin 1 .

Assuming that the p z ( ) in the deep cells (with high quality photometry) do not depend on the wide (noisy) photometry of those galaxies, we can remove the conditions on ˆ c and ˆ b in the first and last terms of Equation 10, and approximate it to

$$p ( z | \hat { b }, \hat { s } ) \approx \sum _ { \hat { c } \in \hat { b } } \sum _ { c } p ( z | c, \hat { s } ) p ( c | \hat { c }, \hat { s } ) p ( \hat { c } | \hat { s } ). \quad ( 1 1 )$$

The transfer function, p c c, s ( | ˆ ˆ) , connecting the deep and wide samples, is computed from Balrog realisations, not the full wide galaxy sample. Re-writing it as

$$p ( c | \hat { c }, \hat { s } ) = \frac { p ( c, \hat { c } | \hat { s } ) } { p ( \hat { c } | \hat { s } ) }, \text{ \quad \quad } ( 1 2 )$$

and replacing it in the equation above, we can write each term highlighting the sample from which it is obtained

$$\underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \undersET { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { - } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } \ { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { - } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } { \underset { \substack { w l s \\ t s, } } \ { \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$$\text{$\text{$\text$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text$\text$\text{$\text{$\text{$\text{$\text{$\text$\text$\text$\text{$\text{$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$\text$} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } _ } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } \ } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } { } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }, } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } ; } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } ( } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } \\ } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } | } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }. } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }$$

We would like to emphasize that solving Equation 13 is not the final result for the Y3 n z ( ) 's, as two other pieces of information were added in: clustering redshifts and shear ratios (see Gatti, Giannini et al. (2022), Sanchez, Prat et al. (2022)). However, this is the main

1 In order to address the concern relative to overand under-weighting galaxies when using raw fluxes, luptitudes were used for the input vectors.

Figure 1. Visualization of the self-organizing maps constructed using the fiducial Y3 SOM algorithm described in Section 3.1. Top: Deep field self-organizing map composed of 4096 cells. Bottom: Wide field self-organizing map composed of 1024 cells. The left-hand panels show the total number of galaxies assigned to each SOM, the middle panels show the mean redshift for each cell, and the right panels show the standard deviation of the redshift distribution in each cell of the map. The white cells found in the deep SOM are due to the lack of spectroscopic information in those regions of the color space, i.e., there are no galaxies in the COSMOS2015 sample that were assigned to those cells.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-f029d0ad-f689-4f8f-b720-14270022223b.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-4988f002-e368-4b96-aa2d-f940fbaaf337.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-d7bbb9c8-274a-4a56-9545-11f29abb95e5.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-2381b079-47bc-499e-aa5c-10b3aa68f4c5.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-157cb645-00f2-4a1d-a2ff-f25306347b8f.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-71eccec2-690b-4b56-b47a-0c92e40fad71.jpg)

result to which we are interested in comparing in this work. In what follows we will compare this fiducial Y3 n z ( ) to the one obtained by each SOM modification proposed in this paper.

In addition, the sample features ( x ) and cell weights ( ω k ) are converted into units of signal-to-noise ratio (SNR), specifying a maximum for the sample SNR as a means of softening the specificity of the cells:

## 4 TESTING IMPROVED SOM METHODOLOGY

In this section we describe the three modifications to the standard method, and assess the impact on the DES Y3 redshift distributions: replacing the SOM algorithm used in Year 3 (see Buchs, Davis et al. 2019) by the one proposed in Sánchez, Raveri, Alarcon &amp; Bernstein (2020); including an extra band ( g -band), even though it has low SNR; including redshifts, when available, as an additional feature to train and assign galaxies to the SOM.

## 4.1 SOMfor faint galaxies - SOMF

A characteristic of the majority of machine learning methods, selforganizing maps included, is the assumption that the training data is ideal, i.e., does not contain errors. This assumption is not true in general, especially when working with empirical data. This point is addressed for the case of SOMs in Sánchez, Raveri, Alarcon &amp; Bernstein (2020), where the authors propose a modification of the standard SOM algorithm that accounts for measurement uncertainties in the training set, with the problem of faint galaxies in mind. The basic idea is to take the errors into account such that, examples with larger measurement uncertainties will result in less change to the weights than examples with smaller uncertainties. The main modifications to the standard algorithm consist in redefining the distance measure between a training sample and a cell on the map, and the training shift through which the weights are updated.

$$s _ { i b } \equiv \max \left ( \Sigma _ { i b }, \frac { x _ { i b } } { S N R _ { \max } } \right ) \quad \quad ( 1 4 )$$

$$\nu _ { i b } \equiv \frac { x _ { i b } } { s _ { i b } } \, \begin{array} { c } \cdots \, \end{array} \,,$$

$$\nu _ { c b } \equiv \frac { \lambda _ { c b } } { s _ { i b } }$$

Here, the index i represents each individual galaxy, the index c represents each cell, and the quantities ν ib and ν cb are the galaxy fluxes and cell weights (indexed by the photometric band b). This helps to ensure that the SOM does not overreact to outliers or extreme values in the data, and makes the SOM more robust or, less sensitive, to variations in the SNR of the data samples.

The SOM algorithm presented in Section 3 uses the chi-square distance, defined in Equation 2, as the metric between the sample and the cell in the map. In Sánchez, Raveri, Alarcon &amp; Bernstein (2020), however, the authors define:

$$d ( x, \Sigma, \omega _ { k } ) = \inf _ { s } \left [ \tilde { d } ( x, \Sigma, e ^ { s } \omega _ { k } ) + \frac { s ^ { 2 } } { \sigma _ { s } ^ { 2 } } \right ], \quad ( 1 7 )$$

where

$$\underset { \text{$me$} } { \text{$\hat{d}$} } \underset { \text{$\hat{d}$} } { \text{$\hat{d}$} } ( x, \Sigma, \omega _ { k } ) = & \sum _ { b } \left [ \frac { \sinh \nu _ { c b } + W _ { i b } \log 2 \nu _ { c b } } { 1 + W _ { i b } } - \underset { \text{$\hat{d}$} } { \sinh \nu _ { i b } } \right ] ^ { 2 } \\ \text{$\hat{d}$} } \underset { \text{$\hat{d}$} } { \times } ( 1 + \nu _ { i b } ^ { 2 } ),$$

approaches the Euclidean distance in log-flux at high SNR, and is

Figure 2. Visualization of the self-organizing maps constructed using the SOMF algorithm described in Section 4.1. Top: Deep field self-organizing map composed of 4096 cells. Bottom: Wide field self-organizing map composed of 1024 cells. The left-hand panels show the total number of galaxies assigned to each SOM, the middle panels show the mean redshift for each cell, and the right panels show the standard deviation of the redshift distribution in each cell of the map. The white cells found in the deep SOM are due to the lack of spectroscopic information in those regions of the color space, i.e., there are no galaxies in the COSMOS2015 sample that were assigned to those cells.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-a874e97e-a2bc-4c52-ab61-c8b5cc002443.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-4636a738-b78e-4282-95af-438c99afa66e.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-36579336-b47b-4784-865b-9e5c76f2fefc.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-3fe2520c-065e-439f-90f9-30c169a41064.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-1349a4c4-c1a7-429a-b6d2-867bc431e029.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-5a41e3ba-c31e-48fb-8c25-608dc84561dd.jpg)

also Euclidean in linear flux at low SNR, while weighting each band by its SNR (up to a maximum). As a result this metric is better suited to the wide dynamic range of galaxy fluxes. The weighting function is defined as the variation of redshift in both the 8-band and 3-band space topology, shown in Figures A1 and A2, are reasonably well traced by the 2D SOM. The right-most panels show the standard deviation of the redshift distribution in each SOM cell.

$$W _ { i b } = e ^ { 2 ( \nu _ { i b } - 4 ) }, \quad \quad \ \ ( 1 9 ) \quad.$$

such that it is possible to transition from the highto low-SNR regimes. Equation 17 includes an overall scale constant e s which allows the cells to be 'fuzzy' in overall flux level. As pointed out in Sánchez, Raveri, Alarcon &amp; Bernstein (2020), there is no natural periodicity in the feature space of galaxy colors and magnitudes. Therefore, the assumption of periodic boundary conditions, usual to the standard algorithm, is not adopted here.

For SOMF, the input vector consists of fluxes. This choice addresses the challenge of color stability in low signal-to-noise (S/N) scenarios, as lupti-colors of faint galaxies tend not to be very reliable. Furthermore, the distance estimator in Equation 17 ensures that, for galaxies with significant noise in certain bands, the SOM assignment does not rely on noisy color information, while still finely distinguishing galaxy types through their color profiles in high S/N situations.

## 4.1.1 Application to DES Y3

We test the impact of using this modified SOM methodology with the DES Y3 data. Figure 2 shows a deep (top) and a wide (bottom) SOMconstructed using the DES Y3 data described in Section 2 and this modified SOM algorithm. The left panels show the number of objects distributed in each of the two SOMs. The smooth behavior of redshift across the SOM, as seen in the middle panels, shows that

We emphasize that the smoothness present in the middle panel of the SOMs is evidence that the redshift space is being well mapped by the flux space. We see that the transitions between low- and highredshifts do not happen abruptly, in general, as we expect in a successful compression of this high dimensional space. The white cells in the top panels represents cells that ended up without galaxies from the redshift sample. Therefore we could not estimate the redshift distribution in those cells, and they were not used. Comparing Figure 1 and Figure 2 we see that, even though the fiducial Y3 Deep SOM also had white cells, there were fewer than using the SOMF algorithm. This is not an issue, given that the amount of information in the transfer function is still the same, however it further emphasizes that the two SOM algorithms group galaxies from the same catalog in a different way. In a recent paper, Sánchez, Alarcon et al. (2023) argue that the reason for this could be related to the SOMF algorithm being better at anomaly detection, and this difference could come from strange, misdetected objects in our catalog. Notice that the map initialization when running SOM and SOMF is not the same, i.e., it is not possible to do a cell-to-cell comparison between the SOMs in Figure 1 and Figure 2.

An additional feature to notice is that comparing the lower-right panels in both figures, showing the standard deviation of the redshift distribution in each wide SOM cell, the SOMF exhibits a significantly larger fraction of blue, i.e. low-standard deviation, cells. Figure 3 compares the standard deviation of the redshift distribution in each cell of the wide SOM σ z c ( | ˆ) for the SOMF and the Y3 SOM,

Figure 3. Standard deviation σ z c ( | ˆ) of the redshift distribution in each wide SOM cell, versus the mean redshift z of each cell, for the standard Y3 SOM (blue), and the SOMF (orange). The horizontal lines represent the 25 (solid), 50 (dashed) and 75 (dotted) percentiles of σ z c ( | ˆ) . We observe that the SOMF presents an overall reduction in the uncertainty per wide cell.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-40673581-a5cf-45c4-9b36-f5f2378ec3fc.jpg)

showing an overall improvement when using the SOMF algorithm. The horizontal lines represent the 25 (solid), 50 (dashed) and 75 (dotted) percentiles of σ z c ( | ˆ) . We can observe that for the SOMF we have about 50% of the wide cells with σ z c ( | ˆ) &lt; 0 2 . , while that is true for only about 25% of the Y3 SOM wide cells. This represents a ∼ 20% reduction in the median of the standard deviation for SOMF when compared to the Y3 SOM.

## 4.2 Regaining blue bands for redshift estimation griz

Although measured in wide field photometry, the g -band did not have an accurate enough point spread function to measure the shapes of galaxies. In particular, the g -band rho statistics (see Jarvis et al., (2020) Figure 13) were considered unacceptably large, which led to the exclusion of g -band data from the Y3 weak lensing analysis. Here we perform the exercise of including g -band information (obtained from Sevilla-Noarbe et al. 2021, and matched to Gatti, Sheldon et al. 2021), in addition to r, i, z bands, to create and assign galaxies to the wide SOM. Notice that, since the Metacal convolution and deconvolution (Gatti, Sheldon et al. 2021) could not be carried out, we do not have shape measurements with g -band for Y3, therefore this exercise is purely at the photometric redshift level.

## 4.2.1 Application to DES Y3

The samples used here are exactly the same as used in obtaining the fiducial Y3 weak lensing redshift measurements (see Section 2), the only difference is the inclusion of the g -band in training the wide SOM and, therefore, including g -band fluxes when assigning the wide and balrog samples to the wide SOM. Our purpose is to quantify the improvement in our redshift constraints, in the hypothetical case that the g -band measurements had been considered good enough to use in the DES Y3. This is particularly timely because we expect that for the Y6 analysis the g -band PSF solution will be sufficiently improved by the addition of color dependence in the PSF

model, allowing it to be used for the weak lensing analysis (Jarvis et al., 2020).

We test the impact of the addition of the g -band using the fiducial Y3SOMalgorithm (see Section 3), and the modified SOM described in Section 4. Adding the g -band impacts only the wide SOM part of the SOMPZ method. Figure 4 shows wide SOM constructed using the fiducial Y3 SOM described in Section 3 (top), and the one described in Section 4 (bottom), adding the g -band information to the train the wide SOM and assigning data to it. The left panels show the number of objects distributed in each of the two SOMs. The middle panels show the mean redshift for each cell. Notice that the inclusion of the g -band creates more cells at higher redshifts in the wide SOM, when compared to either the Y3 fiducial SOM or the SOMF, indicating that the additional g -band information results in fewer cells mistakenly assigned to low redshifts. The right-most panels show the standard deviation of the redshift distribution in each SOM cell, which is compatible with what we see for the Y3 fiducial SOM and the SOMF.

## 4.3 Including redshift

The original deep SOM is trained using deep field galaxies and assigned using redshift sample galaxies and deep field galaxies. This enables us to infer the deep field galaxy p z c,s ( | ˆ) distribution from the redshift sample galaxies for each cell. The redshift itself is not used as a feature; only the photometric fluxes in each band are in the feature vector. Here, we investigate the impact of including redshifts of galaxies (when available) as an extra feature, such that for each sample we have an input vector x :

$$x = \left [ u, g, r, i, z, J, H, K, \text{redshift} \right ], \text{ \quad \ \ } ( 2 0 )$$

containing the 8 fiducial bands plus the redshift information. For deep field galaxies that do not have redshift information, their cells are still determined based only on their fluxes.

To quantify the contribution of redshift in the training and assigning process, we use a weighting factor λ to modulate how much it contributes relative to the photometric bands. When λ = 1 , the redshift information is normalized to have the same contribution as a flux. We also consider λ = 0 1 . and 0.05, in which cases redshift contributes only 10% and 5% that of a flux, so that the redshift information plays a smaller role in constructing the SOM and assigning galaxies to cells.

The methodology and samples used here are exactly the same as used in obtaining the fiducial Y3 weak lensing redshift measurements (see Sections 2 and 3). The only difference is the inclusion of the redshift of galaxies as an extra feature in training the deep SOM. We test two possibilities (i) including redshift information both in training and assigning galaxies to the SOM and (ii) in the SOM training process only. The results relative to these tests are shown in Figures B1 and B2.

## 5 REDSHIFT BINS AND BIN OVERLAP

Having well-defined redshift bins is essential for weak lensing analysis, in order to ensure accurate and unbiased measurements of the gravitational lensing effect. Minimizing the overlap of redshift distributions reduces the contamination of signals between bins, which is crucial to probe the lensing signal as a function of source redshift, control systematic errors, and enable precise cosmological parameter constraints. For instance, having narrower/less-overlapping bins

Figure 4. Visualization of the self-organizing maps constructed adding the g -band to train and assign the wide data, as described in Section 4.2. Top: Wide field self-organizing map obtained using the DES Y3 SOM algorithm, but adding the g -band information. As discussed in Section 4.1.1, the smoother map of mean redshift in wide SOM cells when using the SOMF reflects an improved compression of the color-space. Bottom: Wide field self-organizing map obtained using the SOMF algorithm, but adding the g -band information. Notice that in this case, the deep SOM is exactly the same as the one in Figure 2. The left-hand panels show the total number of galaxies assigned to each SOM, the middle panels show the mean redshift for each cell, and the right panels show the standard deviation of the redshift distribution in each cell of the map.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-0c4d3ea8-355b-4c30-8c5d-c6c70a9c3d41.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-2c7e6cd4-b697-4616-b20b-3d917011592f.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-ab24f810-8ccb-4dc5-bda7-1aa732c092fc.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-058a8da6-72a5-4518-b887-63e57af80224.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-6346fd01-2371-4514-aed1-100e62c6e3d6.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-ca9683e0-0958-48a9-b7d5-9ea81e8f8724.jpg)

reduces the IA signal (specially the II component in cross-bin pairs), reducing the impact of any potential mismodelling. In addition, having finer definition along the line of sight helps constrain the evolution of structure, so it is also better from a purely cosmological perspective.

Using the DES Y3 data, described in Section 2, we assessed the ability of the modifications to the fiducial Y3 SOMPZ method (see Section 3 and Myles, Alarcon et al. 2021), detailed in Section 4. In particular, replacing the fiducial Y3 SOM by the SOMF and including the g -band information improves our redshift constraining power and reduces the bin overlap. We detail our findings regarding those modifications in what follows. The third possibility that we described in Section 4.3, adding the redshift information, when available, to train and assign galaxies to the SOM, does not provide any improvements in comparison to the Y3 results. Therefore, we move our findings on that to Appendix B.

Samples, the use of Balrog, and variations in the Photometric Calibration across deep fields are exactly the same as those estimated for DES Y3 (see Myles, Alarcon et al. 2021 for details on each uncertainty and how they were estimated). The inherent SOMPZ Method uncertainty is the only one affected by the change of method, but given the good agreement in mean redshift and shape of the distributions, we decided to not recompute it, and assume it is the same as for Y3 as well. This is further justified by the fact the contribution of the SOMPZ uncertainty to the total error budget per tomographic bin was minor. Notice that the SOMF method produces bins seem slightly better defined, with higher peaks, even though the two distributions follow each other very closely.

## 5.1 N(z) distributions

In Figure 5, we compare the result of applying the redshift schema described in Section 3.2, in particular the solution of Equation 10, using both the Y3 SOM and the SOMF. We can see that the two methods agree very well, both in mean redshift and shape of the n z ( ) . In particular, once we apply the uncertainties due to each component of the method, shown in Table 2, all bins agree well inside the uncertainty level, with the exception of bin 2 that is slightly off the uncertainty bound. Notice that we can safely assume that the uncertainties due to Shot Noise and Sample Variance (caused by the size and area of the deep fields), irreducible biases in the Redshift

Figure 6 shows a similar comparison, but now including the g -band information, for for the fiducial Y3 SOM and the SOMF. We can see that two SOM algorithms again agree very well, both in mean redshifts and shape of the n z ( ) . The mean redshift in each bin agree within the uncertainty level in Table 2. The difference in the peak heights, is even more pronounced now with the addition of the g -band information, showing that the SOMF algorithm leverages the g -band information to get even better defined redshift bins. Notice that the means and shapes of the distributions in Figure 5 and Figure 6 differ from each other, which is a expected consequence of the addition of the g -band information.

## 5.2 Bin overlap

We aim for minimal overlap between bins, indicating distinct redshift ranges that have been well separated. This results in a higher likelihood that a galaxy is correctly assigned to its designated bin

| z PZ range                               | Bin 0 0.0-0.358   | Bin 1 0.358-0.631   | Bin 2 0.631-0.872   | Bin 3 0.872-2.0   |
|------------------------------------------|-------------------|---------------------|---------------------|-------------------|
| ⟨ z ⟩ Y3 SOM                             | 0.335             | 0.518               | 0.750               | 0.936             |
| ⟨ z ⟩ SOMF                               | 0.327             | 0.510               | 0.735               | 0.928             |
| ⟨ z ⟩ Y3 SOM griz                        | 0.328             | 0.473               | 0.729               | 0.968             |
| ⟨ z ⟩ SOMF griz                          | 0.312             | 0.467               | 0.725               | 0.976             |
| Uncertainty*                             |                   |                     |                     |                   |
| Shot Noise & Sample Variance             | 0.006             | 0.005               | 0.004               | 0.006             |
| Redshift Sample Uncertainty              | 0.003             | 0.004               | 0.006               | 0.006             |
| Balrog Uncertainty                       | < 0.001           | < 0.001             | < 0.001             | < 0.001           |
| Photometric Calibration Uncertainty      | 0.010             | 0.005               | 0.002               | 0.002             |
| Inherent SOMPZ Method Uncertainty        | 0.003             | 0.003               | 0.003               | 0.003             |
| Combined Uncertainty: SOMPZ (from 3sDir) | 0.012             | 0.008               | 0.006               | 0.009             |

* We refer to Myles, Alarcon et al. (2021) for the definition of each uncertainty.

Table 2. Values of and approximate error contributions to the mean redshift of each tomographic bin. Given that the the only difference between the redshift distributions estimated using the Y3 SOM and the SOMF comes from the SOM recipe (all the samples are the same in both cases), we can safely assume that the uncertainties due to Shot Noise &amp; Sample Variance, Redshift Sample, Balrog and Photometric Calibration are exactly the same ones estimated for DES Y3 (Myles, Alarcon et al. 2021). The only uncertainty affected by the change in our method is the inherent SOMPZ Method uncertainty. Figure 3 suggests that uncertainty to be even smaller for the SOMF, therefore we decided to not re-compute the SOMPZ uncertainty, and assume its upper bound to be the same as the Y3 SOM.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-ea23698a-a5b8-4cc1-8091-7af05f928a2b.jpg)

Figure 5. Photometric redshift distribution obtained from the riz bands, using the Y3 SOM (dot-dashed line) and the SOMF algorithm (filled line). The two methods show good agreement regarding the shape of each bin, and their mean redshifts values, shown in the legend on the top right. The SOMF method, however, presents better defined bins.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-06190c85-9950-4dae-804d-1b1960d05c43.jpg)

rather than to a neighboring one. Figure 7 compares the amount of bin overlap obtained with each method. The amount of bin overlap when using the Y3 SOM and the riz bands is shown in blue, the SOMF with riz bands is shown in green, Y3 SOM with griz bands in yellow, and the SOMF with griz bands in red. We can immediately see that the Y3 SOM riz presents the highest overlap among all methods, and the greatest reduction in bin overlap is obtained when we combine the SOMF recipe and the griz bands.

The bin overlap is calculated by first computing an overlap matrix and then normalizing it. The overlap matrix is essentially a matrix of dot products between all pairs of bins in the n z ( ) distribution. Mathematically, this can be represented as

$$G = X X ^ { \top }, \quad \quad \ ( 2 1 ) \quad w$$

Figure 6. Photometric redshift distribution obtained from the griz bands, using the Y3 SOM (dot-dashed line) and the SOMF algorithm (filled line). The two methods show good agreement regarding the shape of each bin, and their mean redshifts values, shown in the legend on the top right. However, the addition of the g -band further emphasizes the ability of SOMF to produce better defined bins.

where X is an n × m matrix representing n z ( ) , with n = 4 bins (assuming X has 4 rows and m columns). This matrix G is akin to a Gram matrix in linear algebra, which contains the dot products of vectors in a set. We then normalize the overlap matrix by dividing it element-wise by the product of the square roots of its diagonal elements. In other words, we compute a normalization matrix D where each element D ij = √ G G ii jj . The final normalized overlap matrix is obtained by

$$G _ { \text{normalized} } = \frac { G } { D }. \quad \quad \quad ( 2 2 )$$

This step adjusts the overlap values based on the scale of the data, which is crucial to ensure that the results are not skewed by differ-

Figure 7. Redshfit bin overlap between bins 0 -1 , 0 -2 , 0 -3 , 1 -2 , 1 -3 and 2 -3 for each SOM recipe. The Y3 SOM riz is shown in blue, the SOMF riz in green, the Y3 SOM griz in yellow and the SOMF griz in red. We can see that all proposed modifications reduce the bin overlap with respect to the fiducial Y3 SOM using riz bands, with the best result obtained for SOMF griz .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-28ebcae8-132d-4aab-a728-628434b3dd61.jpg)

ences in variance among datasets. The normalized Gram matrix is used as a measure of similarity in various applications, including clustering methods in machine learning (see e.g. Rahman, Johnson &amp;Rao 2022).

The numerical values corresponding to the bin overlap between bin pairs in shown in Table 3, where we also show the percent decrease in bin overlap relative to the Y3 SOM riz . The SOMF riz , Y3 SOM riz , SOMF griz present decreasing amount in overlap, having a reduction of 3 % , 23 % , and 25 % respectively for bins 0-1; 5 % , 14 % , and 33 % for bins 1-2; 0 % , 52 % , and 66 % for bins 1-3; 6 % , 14 % , and 31 % for bins 2-3. In the case of the overlap between bins 0-2 and 0-3, the amount of overlap is already small when compared to the other bin pairs. For those two pairs all methods yield similar results, with the Y3 SOM riz having the best performance by a few percent for bins 0-2, and the SOMF griz again having the best performance for bins 0-3.

In summary, using the same data and same methodology, described in Section 3.2, we can reduce the amount of bin overlap in our wide sample just by replacing the fiducial Y3 SOM algorithm used in the Y3 analysis (see Myles, Alarcon et al. 2021, Buchs, Davis et al. 2019), described in Section 3, by the SOMF algorithm (see Sánchez, Raveri, Alarcon &amp; Bernstein 2020) described in Section 4, and adding the g -band information. By adding the g -band, the overlap between redshift bins undergoes significant improvement. The fiducial Y3 SOM already shows significant reduction in bin overlap when the g -band is added, but it is by combining the SOMF with the g -band information that we obtain a substantial reduction in bin overlap and the best defined redshift bins. In the next section, we show the impact of this reduction in bin overlap on the cosmological parameters.

## 6 IMPACT ON COSMOLOGICAL PARAMETERS

In this Section we quantify how the changes in the SOMPZ method proposed in this paper impact the final cosmological constraints. In particular, we want to see how these changes impact the S 8 -Ω m plane, i.e., the main cosmology results for DES Y3.

Changing the redshift estimation of the source catalog impacts

Table 3. Amount of bin overlap between each redshift bin pair, for each method, together if the percentage overlap reduction with respect to the Y3 SOM riz (the fiducial method used in DES Y3).

|           | Method                                   | Method                                   | Method                                   | Method                                   |
|-----------|------------------------------------------|------------------------------------------|------------------------------------------|------------------------------------------|
| Bin Pairs | Y3 SOM riz                               | SOMF riz                                 | Y3 SOM griz                              | SOMF griz                                |
| 0-1       | 0.6974                                   | 0.6767                                   | 0.5399                                   | 0.5207                                   |
| 0-2       | 0.1354                                   | 0.1407                                   | 0.1634                                   | 0.1393                                   |
| 0-3       | 0.1283                                   | 0.1283                                   | 0.1306                                   | 0.1112                                   |
| 1-2       | 0.4536                                   | 0.4329                                   | 0.3880                                   | 0.3053                                   |
| 1-3       | 0.3258                                   | 0.3253                                   | 0.1574                                   | 0.1113                                   |
| 2-3       | 0.7216                                   | 0.6812                                   | 0.6210                                   | 0.4986                                   |
|           | Overlap Reduction Relative to Y3 SOM riz | Overlap Reduction Relative to Y3 SOM riz | Overlap Reduction Relative to Y3 SOM riz | Overlap Reduction Relative to Y3 SOM riz |
| 0-1       | -                                        | 3 %                                      | 23 %                                     | 25 %                                     |
| 0-2       | -                                        | -4 %                                     | -21 %                                    | -3 %                                     |
| 0-3       | -                                        | 0 %                                      | -2 %                                     | 13 %                                     |
| 1-2       | -                                        | 5 %                                      | 14 %                                     | 33 %                                     |
| 1-3       | -                                        | 0 %                                      | 52 %                                     | 66 %                                     |
| 2-3       | -                                        | 6 %                                      | 14 %                                     | 31 %                                     |

all the following steps in the cosmology estimation pipeline. In the SOMPZ method (see Section 3), the galaxy bin assignment is based on the wide SOM assignment. Therefore, when we train a new wide SOM, the galaxies are re-assigned and it is necessary to re-compute the two-point statistics measurements and the covariance matrix. In the case of the SOMF method with the riz bands, it is possible to perform all those steps using the Y3 data, and get a direct comparison between the Y3 SOM and the SOMF in the Y3 cosmology. In the cases when we add the g -band information, both for the Y3 SOM and the SOMF, it is not possible to carry the comparison all the way to the cosmological parameters, given that we do not have shape measurements for the g -band. Instead, we generated simulated datavectors, based on the Y3 cosmology, and compare the contours obtained in this simulated data.

Since in this paper we are exploring modifications on the method for redshift estimation for the weak lensing source catalog, we will be focusing on the cosmic shear measurement. Notice however, that the changes discussed here also impact galaxy-galaxy lensing and, naturally, the 3 × 2 pt statistics.

## 6.1 Cosmological Constraints - Y3 Data

We tested the impact of replacing the SOM algorithm all the way from the SOM creation and assignment, to the cosmological parameter estimation. We use the redshift estimation schema and data described in Section 3.2, but replace the SOM algorithm outline in Section 3.1 (see also Buchs, Davis et al. 2019), with the one outlined in Section 4.1 (see also Sánchez, Raveri, Alarcon &amp; Bernstein 2020).

Creating a new wide SOM and assigning the wide sample to it has a significant impact on bin assignments, influencing which galaxies are sorted into specific redshift bins. Consequently, to derive cosmological parameters, it becomes necessary to recalibrate various components of the analysis. This entails the reassessment of 2-point statistics, and the subsequent re-calculation of the covariance matrix. In essence, this process entails a complete reconstruction of the

data-vector. Details on each step can be found in Appendix C. Subsequently, we initiate a parameter estimation chain using the updated data-vector, adhering to the methodology outlined in the DES Y3 pipeline, as comprehensively expounded in Krause et al., (2021), and concisely summarized in Appendix D. It's worth emphasizing that we follow the same methodology as Amon et al. (2022) and Secco, Samuroff et al. (2022).

Figure 8 compares the 1 σ and 2 σ contours in the Ω m -σ 8 plane. The blue contour uses the Y3 SOM algorithm, however, containing only the SOMPZ information when constraining the redshift (as opposed to the complete redshift information used in Y3 that contains SOMPZ + Clustering Redshifts (WZ) + Blending + Shear Ratios information, as detailed in Myles, Alarcon et al. 2021). The green contour shows the chain for which the data vector was constructed with the redshift information from the SOMF algorithm.

The contours agree at the level of chain variance, and we do not observe any gain in constraining power on the cosmological parameters, due to the small reduction in bin overlap obtained when using the SOMF riz (see Figure 7). The marginalized mean S 8 and Ω m values in Λ CDMare:

$$S _ { 8 } \ & = \ \ 0. 7 6 1 ^ { + 0. 0 3 7 } _ { - 0. 0 2 7 } \quad \ \ ( Y 3 \, S O M ) \\ \Omega _ { m } \ & = \ \ 0. 2 9 8 ^ { + 0. 0 4 6 } _ { - 0. 0 6 1 } \quad \ \ ( Y 3 \, S O M )$$

and

$$S _ { 8 } \ & = \ 0. 7 5 6 ^ { + 0. 0 3 5 } _ { - 0. 0 3 0 } \quad \ ( S O M F ) \\ \Omega _ { m } \ & = \ 0. 3 0 1 ^ { + 0. 0 4 1 } _ { - 0. 0 6 6 } \quad \ ( S O M F )$$

where uncertainties are 68% confidence intervals. We can see that the values agree well within the uncertainty level, and the confidence intervals are also equivalent. Notice that the mean S 8 and Ω m for the Y3 SOM are not the same as the ones quoted in Amon et al. (2022) and Secco, Samuroff et al. (2022), given that here our chains includes redshifts estimated only with the SOMPZ method, but again they are in perfect agreement.

The good level of agreement of the two chains, and to the Y3 fiducial results, demonstrates the robustness of our method, one of the main results of this paper. This result, combined to the agreement in mean redshift and shape of the distribution, demonstrates that the SOMF algorithm is compatible with the SOMPZ pipeline, and robust against the cosmology results, validating it and making it a viable option for DES Year 6.

We also emphasize that the improvements on cosmology due to the enhanced redshift methodology described in this paper could be more significant for a cosmic shear analysis more limited by redshift uncertainty than DES Y3.

## 6.2 Cosmological Constraints - Simulations

To assess the potential impact of incorporating g -band information into redshift estimation on cosmological parameters, we constructed simulated data-vectors based on the DES Y3 setup, as detailed in Appendix D0.3. Subsequently, we analyzed these simulations using the Y3 pipeline. We generated simulated data for four distinct cases, each employing different methods: the Y3 SOM method with the riz bands, the SOMF method with the riz bands, the Y3 SOM method with the griz bands, and the SOMF method with the griz bands.

At the level of simulated data vectors we can already see differences. Switching from using the riz SOM n z ( ) s to griz SOMF, we see a rougly 5-10 % increase in the lensing ( κκ ) signal in the uppermost 4 4 , bin correlation. This is likely due to the small upwards

Figure 8. Cosmological constraints on the clustering amplitude, S 8 with the matter density, Ω m in Λ CDM, using the DES Y3 data. The marginalised posterior contours (inner 68% and outer 95% confidence levels) are shown for the Y3 SOM in blue and SOMF in green.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-b36b16cb-f9e4-4e15-9c7a-57b4485d0117.jpg)

Figure 9. Cosmological constraints on the clustering amplitude, S 8 with the matter density, Ω m in Λ CDM, using the simulated data described in Section D0.3, in order to include the g -band information. The marginalised posterior contours (inner 68% and outer 95% confidence levels) are shown for the Y3 SOM in blue and SOMF in green, for the riz bands, and Y3 SOM in yellow and SOMF in red for the griz bands. Notice that the parameters have been centered to zero, in order to focus on the relative error reduction.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-813699d7-d8ff-454e-ac15-d45bfc0fa2e4.jpg)

shift in the mean, and the reduction in the weight of the low redshift tail. Much of the signal-to-noise of cosmic shear comes from these upper bin correlations, and so boosting the signal here is useful for optimising our cosmological constraint. We also see an overall reduction of the intrinsic alignment II contribution in the cross-bin correlations, as well as a decrease of the GI component in the auto bin pairs. This is again expected from a reduction in the width of the source bins (we can see this by considering, for example, Eq 16-17 of Secco, Samuroff et al. 2022). The cleaner separation of IA signals is helpful as it can break degeneracies and allow the data to constrain IAs more effectively (Campos et al. 2023).

In Figure 9, we present a comparison of the 1 σ and 2 σ contours

Table 4. Values of the 68 % confidence intervals, shown in Figure 9, for the Y3 SOM and SOMF using riz bands, and Y3 SOM and SOMF using griz bands.

|             | ∆ S 8     | ∆Ω m               |
|-------------|-----------|--------------------|
| Y3 SOM riz  | ± 0 . 024 | +0 . 044 - 0 . 056 |
| SOMF riz    | ± 0 . 025 | +0 . 041 - 0 . 061 |
| Y3 SOM griz | ± 0 . 022 | +0 . 042 - 0 . 058 |
| SOMF griz   | ± 0 . 022 | +0 . 039 - 0 . 054 |

in the Ω m -σ 8 plane for each case. Importantly, the cosmological parameters have been centered to zero in these plots, enabling us to concentrate on the gain in signal-to-noise (SN) or the reduction in errors relative to one another. This centered approach helps highlight how the various methods improve in comparison to their counterparts. Table 4 shows the numerical values of the 1 σ uncertainty intervals for each case shown in Figure 9.

Upon examination of Figure 9 and Table 4, we conclude that the SOMF method with griz bands exhibits the most significant improvement in constraining both Ω m and S 8 , reducing the error bars by ∼ 10% , in each parameter, over Y3 SOM riz . Moreover, even though both the Y3 SOM and SOMF methods using griz bands demonstrate comparable constraining power for S 8 , the SOMF shows better constraints in Ω m , highlighting the effectiveness of the SOMF algorithm. Additionally, Figure 10 presents the constraints on Intrinsic Alignment parameters, further underscoring the benefits of adding g -band data and using the SOMF algorithm, as we expected from the datavector.

Consequently, we recommend incorporating the SOMF griz method in the upcoming DES Year 6 analysis. While the primary gain in constraining power currently stems from the addition of g -band data, this trend might evolve in future surveys as we observe more faint objects.

## 7 CONCLUSIONS

In this paper, we explore three modifications to the SOMPZ method (Buchs, Davis et al. 2019, Myles, Alarcon et al. 2021) employed in the DES Year 3 analysis: 1) changing the SOM algorithm; 2) including the photometry from the g -band; 3) adding the redshift information, when available, to train and assign galaxies to the SOM. Our goal is to optimize our redshift estimation pipeline for the DES Year 6 data, with especial focus on the weak lensing source galaxies. This is an important problem, given that those galaxies will be deeper and fainter compared to DES Y3, and we want to be able to treat them properly, minimizing cuts in our catalogs. That said, the findings of this paper are applicable to ensemble redshift estimation in general, and can be used as a guide for the lens redshifts in DES Year 6, and the redshift analysis of other surveys.

Using the DES Y3 weak lensing data, we tested each of the three modifications, and compared their impact relative to the Year 3 fiducial results. The main conclusions of our study are as follows:

- · We showed that the SOMF successfully compresses the highdimensional flux space into the redshift space in a smooth way, i.e., transitions between low- and high-redshift happen gradually. Given that Self-Organizing Map is a unsupervised clustering algorithm, neighbor cells should present similar properties. This property should definitely be observed in the flux space (the features the SOM was trained on, Figures A1 and A2 ), and the fact that its also present

Figure 10. Intrinsic Alignment parameters constraints in Λ CDM, using the simulated data described in Section D0.3, in order to include the g -band information. The marginalised posterior contours (inner 68% and outer 95% confidence levels) are shown for the Y3 SOM in blue and SOMF in green, for the riz bands, and Y3 SOM in yellow and SOMF in red for the griz bands. Notice that the parameters have been centered to zero, in order to focus on the relative error reduction.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-b6f1f069-add3-4c50-ba3f-ff340c155c52.jpg)

in redshift space, Figure 2, is evidence of the successful mapping of redshifts.

- · By using the SOMF algorithm, tailored for the problem of photometric redshifts, we were able to reduce the standard deviation in the redshift distribution in each cell. Figure 3 shows a ∼ 20% reduction in the median of the standard deviation for SOMF when compared to the Y3 SOM. That means that, within each cell of the SOM, we have a better estimation of the redshift of those galaxies.
- · We are able to reduce bin-overlap even further, and therefore have better defined ensemble redshift distributions. Using the SOMF algorithm helps, however including an additional photometric band, the g -band, plays a major role in reducing the overlap between redshift bins.
- · Two-point measurements and cosmology results are robust. Changing the galaxy assignment in the wide SOM affects all the subsequent stages of the pipeline, meaning that all measurements need to be repeated. We tested the robustness of the 2pt cosmic shear measurements, and the final cosmology results, verifying that they are completely consistent with our Y3 findings. Any observed shift of Ω m and S 8 was within previous systematic uncertainty. In addition, SOMF griz reduces the error bars in each of these parameters by ∼ 10% over Y3 SOM riz .
- · We considered adding the redshift measurements that we had available for galaxies in our spectroscopic sample as an additional feature to train the SOM and assign galaxies to it. This path did not lead to improvements in the method, as shown in Appendix B, since the inclusion of the redshift feature seem to dominate over the other features, and create a very sparse SOM.

Photometric redshift estimation is an important topic in cosmology. There have been several proposals on how to improve redshift

estimation, given the limitations imposed by photometric data. SelfOrganizing Maps, in particular the SOMPZ pipeline proposed for DES Year 3 showed promising results, constraining the redshift of the weak lensing source galaxies to the 2% level. That was a groundbreaking work, that leverages on the deep fields high-quality photometric data to connect the spectroscopic information available for a small group of galaxies, to the main wide data set.

This paper sets out an recommendation for improvement of the redshift estimation pipeline, SOMPZ, for weak leasing source galaxies in DES Year 6. Although the fiducial SOMPZ method employed for Y3 is perfectly suitable for Y6 as well, by switching the Y3 SOM recipe to the SOMF and including the g -band photometry (even if only at the redshifts level) we can obtain even better redshift estimates for the Y3 wide sample. We expect these effects to be even more accentuated for Y6, given the increased depth of the wide sample, therefore the changes proposed here will have an even greater impact.

The implications of our results extend beyond the DES Y6 project. They provide a valuable foundation for the improvement and refinement of redshift characterization in future Stage IV surveys, such as those conducted by the Rubin Observatory, Euclid Collaboration and Roman Space Telescope. By reducing redshift bin overlap and enhancing the accuracy of redshift estimates, we are poised to unlock new possibilities for advancing our understanding of the universe's dark components, and to achieve more precise and robust cosmological parameter estimates in the years to come.

It is also worth emphasising that the code for the SOMPZ pipeline including the SOMF algorithm was further revised, simplified and documented, making it easier to use. Given its generality in the redshift estimation context, simplicity, and open source nature, we foresee the use of this method in future analyses, as a ensemble photometric redshift estimation pipeline option for many data sets.

## CODE AND DATA AVAILABILITY

The photometric redshift ensemble estimation code used in this work is publicly available at https://github.com/ AndresaCampos/sompz\_y6 .

The DES Y3 data products used in this work, are publicly available at https://des.ncsa.illinois.edu/releases. As cosmology likelihood sampling software we use cosmosis , available at https://github.com/joezuntz/cosmosis.

## CONTRIBUTION STATEMENT

AC: Designed the project, developed the code, led the analysis and the writing of the manuscript. BY: Performed the analysis of SOMz, merged g-band data and edited the manuscript. SD: Supervised the project, assisted with result interpretation and edited the manuscript. AA: Provided suggestions and feedback on analyses, assisted with result interpretation and edited the manuscript. AA: Assisted with project design, code testing and review, and result interpretation. CS: Assisted with project design and result interpretation. GB: Proposed the project, provided updated SOM algorithm, assisted with result interpretation. GG, JM, SS: DES internal reviewers of the paper. Production of the DES gold catalog: ISN, KB, MCK, ACR, MB, ADW, RG, ER, ES, BY. Production of the DES shape catalog: MG, ES, AA, MB, MT, AC, CD, NM, ANA, IH, DG, GB, MJ, LFS, AF, JM, RPR, RC, CC, SP, IT, JP, JEP, CS. Production of the DES PSF: MJ, GB, AA, CD, PFL, KB, IH, MG, AR. Production of the DES

deep fields: WH, AC, AA, RG, ES, IH, GB, ISN, BY, KE, TD. Production of the DES Balrog: SE,BY, NK, EH, YZ. Production of the DES redshift distribution: JM, AA, AA, CS, SE, JD, JM, DG, GB, MT, SD, AC, NM, BY, MR. Production of the DES image simulations: NM, MB, JM, AA, DG, MJ, AC, MT, ES, BY, KH, SD, JZ, KE, RR, TV. Production of the DES covariance: OF, FAO, HC, OA, RR, JS, XF, TE, EK. Production of the DES methods: EK, XF, SP, LS, OA, HH, JB, JP, JZ, TE, JD. Production of the sims: JD, RW, RB, ER, SP, NM, AA, JM. Builders: The remaining authors have made contributions to this paper that include, but are not limited to, the construction of DECam and other aspects of collecting the data; data processing and calibration; developing broadly used methods, codes, and simulations; running the pipelines and validation tests; and promoting the science analysis.

## ACKNOWLEDGEMENTS

Andresa Campos thanks the support from the U.S. Department of Energy grant DE-SC0010118 and the NSF AI Institute: Physics of the Future, NSF PHY-2020295.

Funding for the DES Projects has been provided by the U.S. Department of Energy, the U.S. National Science Foundation, the Ministry of Science and Education of Spain, the Science and Technology Facilities Council of the United Kingdom, the Higher Education Funding Council for England, the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign, the Kavli Institute of Cosmological Physics at the University of Chicago, the Center for Cosmology and Astro-Particle Physics at the Ohio State University, the Mitchell Institute for Fundamental Physics and Astronomy at Texas A&amp;M University, Financiadora de Estudos e Projetos, Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro, Conselho Nacional de Desenvolvimento Científico e Tecnológico and the Ministério da Ciência, Tecnologia e Inovação, the Deutsche Forschungsgemeinschaft and the Collaborating Institutions in the Dark Energy Survey.

The Collaborating Institutions are Argonne National Laboratory, the University of California at Santa Cruz, the University of Cambridge, Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas-Madrid, the University of Chicago, University College London, the DES-Brazil Consortium, the University of Edinburgh, the Eidgenössische Technische Hochschule (ETH) Zürich, Fermi National Accelerator Laboratory, the University of Illinois at Urbana-Champaign, the Institut de Ciències de l'Espai (IEEC/CSIC), the Institut de Física d'Altes Energies, Lawrence Berkeley National Laboratory, the Ludwig-Maximilians Universität München and the associated Excellence Cluster Universe, the University of Michigan, the National Optical Astronomy Observatory, the University of Nottingham, The Ohio State University, the University of Pennsylvania, the University of Portsmouth, SLAC National Accelerator Laboratory, Stanford University, the University of Sussex, Texas A&amp;M University, and the OzDES Membership Consortium.

Based in part on observations at Cerro Tololo Inter-American Observatory at NSF's NOIRLab (NOIRLab Prop. ID 2012B-0001; PI: J. Frieman), which is managed by the Association of Universities for Research in Astronomy (AURA) under a cooperative agreement with the National Science Foundation.

The DES data management system is supported by the National Science Foundation under Grant Numbers AST-1138766 and AST-1536171. The DES participants from Spanish institutions are partially supported by MINECO under grants AYA2015-

71825, ESP2015-66861, FPA2015-68048, SEV-2016-0588, SEV2016-0597, and MDM-2015-0509, some of which include ERDF funds from the European Union. IFAE is partially funded by the CERCA program of the Generalitat de Catalunya. Research leading to these results has received funding from the European Research Council under the European Union's Seventh Framework Program (FP7/2007-2013) including ERC grant agreements 240672, 291329, and 306478. We acknowledge support from the Brazilian Instituto Nacional de Ciência e Tecnologia (INCT) e-Universe (CNPq grant 465376/2014-2).

This manuscript has been authored by Fermi Research Alliance, LLC under Contract No. DE-AC02-07CH11359 with the U.S. Department of Energy, Office of Science, Office of High Energy Physics.

## REFERENCES

| Amon A., et al., 2022, Phys. Rev. D, 105, 023514 Asgari M., et al., 2021, A&A, 645, A104                                 |
|--------------------------------------------------------------------------------------------------------------------------|
| Benítez N., 2000, ApJ, 536, 571                                                                                          |
| Blake C., Bridle S., 2005, Monthly Notices of the Royal Astronomical Soci- ety, 363, 1329                                |
| Blazek J., Vlah Z., Seljak U., 2015, Journal of Cosmology and Astroparticle Physics, 2015, 015                           |
| Blazek J. A., MacCrann N., Troxel M. A., Fang X., 2019, Phys. Rev. D, 100, 103506                                        |
| Buchs R., Davis C., et al., 2019, Mon. Not. Roy. Astron. Soc., 489, 820                                                  |
| Campos A., Samuroff S., Mandelbaum R., 2023, MNRAS, 525, 1885                                                            |
| Carrasco Kind M., Brunner R. J., 2013, Monthly Notices of the Royal Astro- nomical Society, 432, 1483                    |
| Collister A. A., Lahav O., 2004, PASP, 116, 345                                                                          |
| Crittenden R. G., Natarajan P., Pen U.-L., Theuns T., 2002, ApJ, 568, 20                                                 |
| DES Collaboration 2022, Phys. Rev. D, 105, 023520                                                                        |
| Dalal R., et al., 2023, Hyper Suprime-Cam Year 3 Results: Cosmology from Cosmic Shear Power Spectra ( arXiv:2304.00701 ) |
| Dark Energy Survey Collaboration 2016, Phys. Rev. D, 94, 022001                                                          |
| Doux C., et al., 2022, MNRAS,                                                                                            |
| Euclid Collaboration 2020, A&A, 644, A31                                                                                 |
| Everett S., et al., 2022, The Astrophysical Journal Supplement Series, 258, 15                                           |
| Fang X., Eifler T., Krause E., 2020, Monthly Notices of the Royal Astronom- ical Society, 497, 2699                      |
| Friedrich O., et al., 2021, Monthly Notices of the Royal Astronomical Soci- ety, 508, 3125                               |
| Gatti M., Sheldon E., et al., 2021, Mon. Not. Roy. Astron. Soc., 504, 4312                                               |
| Gatti M., Giannini G., et al., 2022, MNRAS, 510, 1223                                                                    |
| Giannini G., et al., 2023, Monthly Notices of the Royal Astronomical Soci- ety, 527, 2010                                |
| Hamana T., et al., 2020, PASJ, 72, 16                                                                                    |
| Handley W. J., Hobson M. P., Lasenby A. N., 2015a, MNRAS, 450, L61                                                       |
| Handley W. J., Hobson M. P., Lasenby A. N., 2015b, MNRAS, 453, 4384                                                      |
| Hartley W. G., Choi A., et al., 2021, Monthly Notices of the Royal Astro- nomical Society, 509, 3547                     |
| Hikage C., et al., 2019, PASJ, 71, 43                                                                                    |
| Hildebrandt H., et al., 2017, MNRAS, 465, 1454 Ilbert O., et al., 2009, ApJ, 690, 1236                                   |
| Jarvis M., et al., 2020, Monthly Notices of the Royal                                                                    |
| Ivezi´ c Ž., et al., 2019, ApJ, 873, 111                                                                                 |
| Astronomical Society, 501, 1282                                                                                          |
| Jee M. J., Tyson J. A., Hilbert S., Schneider M. D., Schmidt S., Wittman D., 2016, ApJ, 824, 77                          |
| Kohonen T., 1982, Biological Cybernetics, 43, 59                                                                         |
| Krause E., et al., 2021, arXiv e-prints, p.                                                                              |
| arXiv:2105.13548                                                                                                         |

| Laureijs R., et al., 2011, arXiv e-prints, p. arXiv:1110.3193 Lemos P., Weaverdyck N., et al., 2022, MNRAS, Lewis A., Challinor A., Lasenby A., 2000, ApJ, 538, 473                                                   |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Li X., et al., 2023, Hyper Suprime-Cam Year 3 Results: Cos- mology from Cosmic Shear Two-point Correlation Functions ( arXiv:2304.00702 )                                                                             |
| cal Society, 505, 4249 Rahman S., Johnson V. E., Rao S. S., 2022, Using the left Gram matrix cluster high dimensional data ( arXiv:2202.08236 ) Sánchez C., Raveri M., Alarcon A., Bernstein G. M., 2020, MNRAS, 498, |
| to                                                                                                                                                                                                                    |
| Sanchez C., Prat J., et al., 2022, Phys. Rev. D, 105, 083529 C., Alarcon A., et al., 2023, Monthly Notices of the Royal Astro- nomical Society, 525, 3896                                                             |
| Sánchez                                                                                                                                                                                                               |
| Schneider P., van Waerbeke L., Mellier Y., 2002, A&A, 389, 729                                                                                                                                                        |
| Secco L. F., Samuroff S., et al., 2022, Phys. Rev. D, 105, 023515 Sevilla-Noarbe I., et al., 2021, ApJS, 254, 24                                                                                                      |
| Speagle J. S., et al., 2019, MNRAS, 490, 5658 Suchyta E., et al., 2016, Monthly Notices of the Royal Astronomical                                                                                                     |
| Society, 457, 786                                                                                                                                                                                                     |
| Takahashi R., Sato M., Nishimichi T., Taruya A., Oguri M., 2012, ApJ, 761, 152                                                                                                                                        |
| LSST Dark Energy Science Collaboration (DESC) 2021, The LSST Dark Energy Science Collaboration (DESC) Science Requirements Doc- ument ( arXiv:1809.01669 )                                                            |
| The                                                                                                                                                                                                                   |
| Wright A. H., Hildebrandt H., van den Busch J. L., Heymans C., 2020, tronomy &Astrophysics, 637, A100 Zuntz J., et al., 2015, Astronomy and Computing, 12, 45                                                         |
| Troxel M. A., et al., 2018, Phys. Rev. D, 98, 043528                                                                                                                                                                  |
| As-                                                                                                                                                                                                                   |

## APPENDIX A: MAGNITUDE AND COLORS - SOMF

Figures A1 and A2 provide visual representations of the i -band magnitude and colors for each cell within the wide and deep SelfOrganizing Map (SOM), respectively. These maps were created using the SOMF algorithm, as outlined in Section 4.1. The SOM is designed to create a smooth map encompassing the entire parameter space derived from the training inputs. By drawing a comparison with Figure 2, we can appreciate how effectively the SOMF method maps the color-redshift relationship. This effectiveness is evident through the creation of a smooth redshift map that corresponds with the observed color patterns. The correlation between color and redshift in the SOM effectively illustrates its capacity to capture and represent this intricate relationship. Any abrupt differences observed between adjacent cells can be interpreted as indirect indications of potential degeneracies within the color-redshift relationship.

## APPENDIX B: SOM-Z

In this section, we delve into the details of incorporating redshift information, when available, as an extra feature in the training and assignment of the deep SOM used in the SOMPZ method. This is an unconventional way of applying an unsupervised learning method, such as the SOM, however we were motivated to investigate whether this approach would maximize the use of the available information, leading to enhancements in our ability to estimate redshifts. We applied this strategy to the Y3 SOM, described in Section 3. In light of what we observed, described in what follows, we did not try applying this strategy to the SOMF.

An interesting phenomenon arises when we include redshift information in the training and assignment process. The redshift sample galaxies tend to cluster in a few cells, leaving most cells sparsely occupied. This behavior can be observed in Figure B1 (top), where we show the number of spectroscopic galaxies assigned to each cell in the deep SOM (top row), the mean redshift per cell (middle row), and the standard deviation in each cell (bottom row). The first column shows the DES Y3 SOM, without including the redshift information, and λ is a scaling factor for the contribution of redshift information relative to flux data, i.e., λ = 0 means no redshift contribution, while λ = 1 means the contribution of the redshift is the same as a flux. As the contribution of redshift information increases, the clustering effect intensifies.

This clustering of galaxies on the deep SOM negatively impacts the photometric redshift calibration. Figure B3 presents the wide data photometric distribution for the DES Y3 method and the variants with added redshift information. The solid line represents the DES Y3 n z ( ) , while the dot-dashed and dashed lines represent λ = 0 05 . and λ = 0 1 . respectively, and the dotted line represents the most "extreme" case, where λ = 1 . As redshift information gains more weight, the n z ( ) distribution in each redshift bin spreads further. This results in increased bin overlap, as illustrated in Figure B4, which is the opposite effect of what we are looking for.

Given that adding the redshift information the training and assigning phases of the deep SOM seems to impact negatively our n z ( ) bins, we conducted a final test before abandoning the concept: adding the redshift information only during the training phase of the deep SOM. That still has a clustering effect upon the deep SOM, as we can see in Figure B1 (bottom), but to a lesser degree. In this case, we observe that the n z ( ) distribution in each bin, shown in Figure B5, and the bin overlap, shown in Figure B6, are very similar to those of the fiducial Y3 method, but still slightly worse given that the Y3 SOM still presents the smallest overlap.

Based on these findings, we infer that incorporating the redshift of individual galaxies as an additional feature alongside fluxes in the estimation of the n z ( ) distribution using the SOMPZ method is not a viable approach.

## APPENDIX C: COSMIC SHEAR MEASUREMENT

The small distortions in the observed shapes of galaxies due to weak gravitational lensing by the intervening large scale structure of the Universe are called cosmic shear. Considering two redshift bins i and j , the shear correlation function estimator can be written in terms of a galaxy measured tangential, ϵ t , and radial, ϵ × , ellipticities as

$$\xi _ { \pm } ^ { i j } ( \theta ) = \langle \epsilon _ { t } \epsilon _ { t } \pm \epsilon _ { \times } \epsilon _ { \times } \rangle ( \theta ). \text{ \quad \ \ } ( C 1 )$$

We can determine the shear-shear statistics by averaging over all galaxy pairs ( a,b ) separated by and angle θ

$$\xi _ { \pm } ^ { i j } ( \theta ) = \frac { \sum _ { a b } w _ { a } w _ { b } [ \epsilon _ { t } ^ { i } \epsilon _ { t } ^ { j } \pm \epsilon _ { \times } ^ { i } \epsilon _ { \times } ^ { j } ] } { \sum _ { a b } w _ { a } w _ { b } R _ { a } R _ { b } } \quad \quad ( C 2 )$$

where w represents the the per-galaxy inverse-variance weight, which is taken over galaxy pairs whose angular separation is within an interval ∆ θ around θ , and R is the shear response correction from Metacalibration.

The tomographic DES Y3 cosmic shear data vector, D , is computed using Equation C2 and the code TreeCorr 2 . It includes four auto-correlations and six cross-correlations between redshift bins for both positive and negative angular scales, spanning 2 5 . to 250 0 . arcmin. The impact of baryonic effects is mitigated by excluding small angular scales, leaving a total of 167 (60) data points for ξ + ( ξ -) correlations.

The covariance matrix, C , is a function of the redshift distributions, cosmological parameters and nuisance parameters. We assume a multivariate Gaussian distribution to model the statistical uncertainties in our cosmic shear data vector. The complete modelling of the disconnected 4-point function part of the covariance matrix is described in Friedrich et al., (2021). We compute the connected 4-point function part of the covariance matrix and the contribution from super-sample covariance using the public code CosmoCov 3 Fang, Eifler &amp; Krause (2020), which is part of the CosmoLike framework Krause &amp; Eifler (2017).

Following Amon et al. (2022), Secco, Samuroff et al. (2022) and previous cosmic shear analysis, we use a iteratively fixed covariance matrix. This means that we start with a set of fiducial input parameters, in our case we use the DES Y3 best fit parameters. Then the covariance is recomputed at the best fit from this first iteration, and the final chains are run.This update had negligible effects on the cosmic shear constraints that we present in this paper.

## APPENDIX D: MODELLING AND ANALYSIS CHOICES

We carry out our analysis in the context of the flat Λ CDM cosmological model. The cosmological parameters are { Ω m , Ω b , h 0 , A , n , s s Ω ν h 2 } , where Ω m is the density parameter for matter, and Ω b the equivalent for baryons; h 0 is the dimensionless Hubble constant; A s and n s are the amplitude and slope of the primordial curvature power spectrum at a scale of

2 https://github.com/rmjarvis/TreeCorr

3 https://github.com/CosmoLike/CosmoCov.

Figure A1. Wide Self-Organizing Map constructed using the SOMF algorithm and data from the riz bands. The visualization depicts the mean i -band magnitude (on the left), the mean r -i color (in the middle), and the mean i -z color (on the right) for each cell within the wide SOM.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-31de223d-abdb-49a6-ad6a-6942f886016a.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-66046395-0378-46b7-93b9-03455bb7625c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-c5b7132c-62ed-4f37-a176-71e86399b345.jpg)

Figure A2. Deep field Self-Organizing Map constructed using the SOMF algorithm with data from the ugrizJHK bands. In the upper left, we have the mean i -band magnitude for each cell within the deep SOM. Additionally, the various colors utilized in the deep SOM training are shown.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-a157118a-9fa2-4780-8c13-e55da5c614fe.jpg)

k = 0 05 . Mpc -1 respectively; and Ω ν h 2 is the neutrino mass density parameter. We assume three degenerate massive neutrino species, following Krause et al., (2021).

## D0.1 Modelling Cosmic Shear

For two redshift bins, i and j , the two-point cosmic shear correlations ξ ij ± ( θ ) can be obtained by decomposing the convergence power spectrum C κ ( ℓ ) , at an angular wavenumber ℓ , into E - and B -mode components (Crittenden, Natarajan, Pen &amp; Theuns 2002; Schneider, van Waerbeke &amp; Mellier 2002)

$$\xi _ { + } ^ { i j } ( \theta ) = \sum _ { \ell } \frac { 2 \ell + 1 } { 4 \pi } G _ { \ell } ^ { \pm } \left ( \cos \theta \right ) \left [ C _ { \kappa, E E } ^ { i j } ( \ell ) + C _ { \kappa, B B } ^ { i j } ( \ell ) \right ], \, \left ( \mathbf D 1 \right )$$

$$\xi _ { - } ^ { i j } ( \theta ) = \sum _ { \ell } \frac { 2 \ell + 1 } { 4 \pi } G _ { \ell } ^ { \pm } \left ( \cos \theta \right ) \left [ C _ { \kappa, E E } ^ { i j } ( \ell ) - C _ { \kappa, B B } ^ { i j } ( \ell ) \right ], \text{ (D2)}$$

where the functions G ± ℓ ( x ) are calculated from Legendre polynomials P ℓ ( x ) and averaged over angular bins (see Eqs. 19 and 20 in Krause et al., 2021).

The 2D convergence power spectrum C ij κ ( ℓ ) can be written in terms of the 3D matter power spectrum, assuming the Limber ap-

Figure B2. Adding redshift information in training deep SOM only, using Y3 SOM.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-3d632ecd-b5cb-4c39-a6c2-6675996bfb55.jpg)

Figure B3. Comparison of the n(z) bins obtained using the fiducial DES Y3 SOM (solid line), and adding redshift in training and assigning deep SOM. The dotted line represents the most "extreme" case, where λ = 1 and the contribution of the redshift in training and assigning is the same as the fluxes, while the dashed and dot-dashed lines represent λ = 0 1 . and λ = 0 05 . respectively. The vertical lines are the mean redshift in each bin, shown in the legend for the fiducial method, or λ = 0 , and the λ = 1 case.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-72675261-846e-4f6e-b992-30ed79c34c75.jpg)

Figure B4. Redshfit bin overlap plot for fiducial DES Y3 (blue) and adding redshift in both training and assigning deep SOM. The bin overlap increases as the contribution of the redshift, represented by λ , increases. The green line represents λ = 0 05 . or 5% contribution, the yellow λ = 0 1 . , contributing 10% , and the red line λ = 1 , contributing the same as flux.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-e1a0d1cd-4fb2-4ea3-858d-0054a6c739bf.jpg)

proximation (Limber 1953; LoVerde &amp; Afshordi 2008), as:

$$C _ { \kappa } ^ { i j } ( \ell ) = \int _ { 0 } ^ { \chi ( z _ { \max } ) } d \chi \frac { W ^ { i } ( \chi ) W ^ { j } ( \chi ) } { \chi ^ { 2 } } P _ { \delta } \left ( \frac { \ell + 0. 5 } { \chi }, z ( \chi ) \right ), \ ( D 3 ) \quad \text{$\text{$opt all}$} \, ($$

where P δ ( k,z ) is the nonlinear matter power spectrum and the lensing weight is:

$$W ^ { i } ( \chi ) = \frac { 3 H _ { 0 } ^ { 2 } \Omega _ { m } } { 2 c ^ { 2 } } \frac { \chi } { a ( \chi ) } \int _ { \chi } ^ { \chi _ { H } } \mathrm d \chi ^ { \prime } \, n ^ { i } \left ( z ( \chi ^ { \prime } ) \right ) \frac { \mathrm d z } { \mathrm d \chi ^ { \prime } } \frac { \chi ^ { \prime } - \chi } { \chi ^ { \prime } }, \, \mathrm ( \mathrm D 4 ) \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

with the source galaxy redshift distribution n i ( z ) normalised to integrate to 1, and χ H the horizon distance. The effect of intrinsic alignments is modelled using the tidal alignment and tidal torquing model (TATT; Blazek, Vlah &amp; Seljak 2015, Blazek, MacCrann, Troxel &amp; Fang 2019) We follow Krause et al., (2021), and model P δ using a combination of CAMB (Lewis, Challinor &amp; Lasenby 2000) for the

Figure B5. Comparison of the n(z) bins obtained using the fiducial DES Y3 SOM (solid line), and adding redshift only in the in training phase of the deep SOM. The dotted line represents the most "extreme" case, where λ = 1 and the contribution of the redshift in training and assigning is the same as the fluxes, while the dashed and dot-dashed lines represent λ = 0 1 . and λ = 0 05 . respectively. The vertical lines are the mean redshift in each bin, shown in the legend for the fiducial method, or λ = 0 , and the λ = 1 case.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-75733c86-796e-41c3-bd37-08413770722e.jpg)

Figure B6. Redshfit bin overlap plot for fiducial DES Y3 (blue) and adding redshift in only training the deep SOM. The bin overlap increases as the contribution of the redshift, represented by λ , increases. The green line represents λ = 0 05 . or 5% contribution, the yellow λ = 0 1 . , contributing 10% , and the red line λ = 1 , contributing the same as flux.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00922v1-figure-1e2bdcd5-5d0c-47b8-af8e-589c9fa25e8e.jpg)

linear part, and HALOFIT (Takahashi, Sato, Nishimichi, Taruya &amp; Oguri 2012) for nonlinear modifications. As highlighted in Amon et al. (2022) and Secco, Samuroff et al. (2022), the impact of higher order contributions to the observed two-point statistics is verified to be negligible for the scales covered in this work.

## D0.2 Nuisance Parameters &amp; Scale Cuts

Our setup matches the fiducial choices of the DES Y3 cosmic shear analysis. The only significant difference is that, for the sake of simplicity, we choose not to use the additional shear ratio likelihood included by Amon et al. (2022); Secco, Samuroff et al. (2022) (a similar decision was made for validating the analysis choices preunblinding; see Krause et al., 2021). As a result, our model space is slightly smaller, since we do not need to vary parameters for galaxy

Table D1. A summary of the central values and priors used in our analysis. The top seven rows are cosmological parameters, while those in the lower sections are nuisance parameters corresponding to astrophysics and data calibration. Priors are either uniform (U) or normally-distributed, N ( µ,σ ) .

| Parameter                      | Fiducial Value                 | Prior                          |
|--------------------------------|--------------------------------|--------------------------------|
| Cosmological Parameters        | Cosmological Parameters        | Cosmological Parameters        |
| Ω m                            | 0 . 29                         | U[0 . 1 , 0 . 9]               |
| Ω b                            | 0 . 052                        | U[0 . 03 , 0 . 07]             |
| h                              | 0 . 75                         | U[0 . 55 , 0 . 91]             |
| A s                            | 2 . 38 × 10 - 9                | U[0 . 5 , 5 . 0] × 10 - 9      |
| n s                            | 0 . 99                         | U[0 . 87 , 1 . 07]             |
| Ω ν h 2                        | 0 . 00053                      | U[0 . 6 , 6 . 44] × 10 - 3     |
| Calibration Parameters         | Calibration Parameters         | Calibration Parameters         |
| m 1                            | 0 . 0                          | N (0 . 0 , 0 . 0059)           |
| m 2                            | 0 . 0                          | N (0 . 0 , 0 . 0042)           |
| m 3                            | 0 . 0                          | N (0 . 0 , 0 . 0054)           |
| m 4                            | 0 . 0                          | N (0 . 0 , 0 . 0072)           |
| ∆ z 1                          | 0 . 0                          | N (0 . 0 , 0 . 018)            |
| ∆ z 2                          | 0 . 0                          | N (0 . 0 , 0 . 015)            |
| ∆ z 3                          | 0 . 0                          | N (0 . 0 , 0 . 011)            |
| ∆ z 4                          | 0 . 0                          | N (0 . 0 , 0 . 017)            |
| Intrinsic Alignment Parameters | Intrinsic Alignment Parameters | Intrinsic Alignment Parameters |
| A 1                            | 0 . 7                          | U[ - 5 , 5]                    |
| A 2                            | - 1 . 36                       | U[ - 5 , 5]                    |
| η 1                            | - 1 . 7                        | U[ - 5 , 5]                    |
| η 2                            | - 2 . 5                        | U[ - 5 , 5]                    |
| b TA                           | 1 . 0                          | U[0 , 2]                       |

bias or lens photo -z error. The corresponding parameters and their priors are shown in Table D1. Note that these are identical to the priors used in the Y3 analysis. We also adopt the fiducial DES Y3 cosmic shear scale cuts (see Krause et al., 2021 for an explanation of how these were derived).

## D0.3 Generating Mock Data

In this section, we outline the process of generating mock data, which serves as a means to assess the impact of including the g -band in the DES Y3 setup. For a set of input parameters, we generate four noiseless DES Y3-like cosmic shear data vectors denoted as D . These data vectors are produced using the theoretical pipeline described in Section D and are centered around the central values outlined in Table D1.

All four data vectors share the same input flat Λ CDMcosmological model, with parameters set as follows: Ω m = 0 29 . , A s = 2 38 . × 10 -9 , Ω b = 0 052 . , h = 0 75 . , n s = 0 99 . , and Ω ν h 2 = 0 00053 . . This configuration corresponds to σ 8 = 0 79 . and S 8 = 0 77 . , where S 8 ≡ σ 8 √ Ω m / . 0 3 . However, each data vector is distinct in terms of the redshift distribution of the source galaxies, determined using one of the methods employed in this study: 1) One data vector utilizes the redshift distribution obtained by the Y3 SOM, using the riz bands. 2) Another data vector adopts the redshift distribution obtained by the

SOMF method, utilizing the riz bands. 3) A third data vector relies on the redshift distribution derived from the Y3 SOM, employing the griz bands. 4) The final data vector is constructed with the redshift distribution acquired through the SOMF method, employing the griz bands. Our analysis framework and mock data generation follow the choices made in the DES Y3, ensuring that our assessments are consistent with the established DES Year 3 standards.

## D0.4 Bayesian Inference

For the purpose of parameter estimation, the likelihood function of the data vector, D , given the model, T , characterized by parameters, p , can be represented as L ( D | p ) . This probability distribution is presumed to follow a multivariate Gaussian distribution

$$\ln \mathcal { L } ( D | \mathbf p ) = - \frac { 1 } { 2 } \sum _ { i j } \left ( D _ { i } - T _ { i } ( \mathbf p ) \right ) [ C ] _ { i j } ^ { - 1 } \left ( D _ { j } - T _ { j } ( \mathbf p ) \right )$$

D i represents the i th component within the data vector ξ ± , together with its covariance matrix, C (see Section C). Initially, this vector incorporates 20 angular data points, each spanning across the intersections of 4 redshift bins and 2 correlation functions, leading to a total of 227 data points after constraining the angular scales. The corresponding theoretical predictions for these statistical quantities, represented as T i ( p ) , are elaborated upon in this section. The Bayesian posterior probability distributions of the cosmological parameters, denoted as P ( p | D ) , are derived by combining the likelihood with the prior probabilities , P ( p ) , as outlined in Table D1, following the principles of Bayes' theorem

$$\mathcal { P } ( \mathbf p | D ) = \frac { P ( \mathbf p ) \mathcal { L } ( D | \mathbf p ) } { P ( D ) } \, \text{ \quad \ \ } ( \mathbf D 6 )$$

where P D ( ) is the evidence of the data.

The posterior distribution is sampled using the Polychord (Handley, Hobson &amp; Lasenby 2015a,b). The analysis framework is based on CosmoSIS (Zuntz et al., 2015), a modular tool for estimating cosmological parameters. We use the fiducial sampler settings (500 live points, tolerance 0.01) that have been verified to showcase the precision of the posterior distributions and Bayesian evidence estimations (as discussed in Lemos, Weaverdyck et al. 2022).

## AFFILIATIONS

- 1 Department of Physics, Carnegie Mellon University, Pittsburgh, Pennsylvania 15312, USA
- 2 NSF AI Planning Institute for Physics of the Future, Carnegie Mellon University, Pittsburgh, PA 15213, USA
- 3 Department of Physics, Duke University Durham, NC 27708, USA
- 4 Institute of Astronomy, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK
- 5 Kavli Institute for Cosmology, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK
- 6 Argonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USA
- 7 Institute of Space Sciences (ICE, CSIC), Campus UAB, Carrer de Can Magrans, s/n, 08193 Barcelona, Spain
- 8 Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA 19104, USA
- 9 Kavli Institute for Cosmological Physics, University of Chicago, Chicago, IL 60637, USA

- 10 Institut de Física d'Altes Energies (IFAE), The Barcelona Institute of Science and Technology, Campus UAB, 08193 Bellaterra (Barcelona) Spain
- 11 Department of Astrophysical Sciences, Princeton University, Peyton Hall, Princeton, NJ 08544, USA
- 12 Department of Physics, Northeastern University, Boston, MA 02115, USA
- 13 Department of Physics, University of Michigan, Ann Arbor, MI 48109, USA
- 14 Physics Department, 2320 Chamberlin Hall, University of Wisconsin-Madison, 1150 University Avenue Madison, WI 537061390
- 15 Laboratório Interinstitucional de e-Astronomia -LIneA, Rua Gal. José Cristino 77, Rio de Janeiro, RJ - 20921-400, Brazil
- 16 Instituto de Física Teórica, Universidade Estadual Paulista, São Paulo, Brazil
- 17 Brookhaven National Laboratory, Bldg 510, Upton, NY 11973, USA
- 18 Universidad de La Laguna, Dpto. Astrofísica, E-38206 La Laguna, Tenerife, Spain
- 19 Instituto de Astrofisica de Canarias, E-38205 La Laguna, Tenerife, Spain
- 20 Department of Astronomy, University of Illinois at UrbanaChampaign, 1002 W. Green Street, Urbana, IL 61801, USA
- 21 Center for Astrophysical Surveys, National Center for Supercomputing Applications, 1205 West Clark St., Urbana, IL 61801, USA
- 22 Physics Department, William Jewell College, Liberty, MO, 64068
- 23 Department of Astronomy and Astrophysics, University of Chicago, Chicago, IL 60637, USA
- 24 NASA Goddard Space Flight Center, 8800 Greenbelt Rd, Greenbelt, MD 20771, USA
- 25 Jodrell Bank Center for Astrophysics, School of Physics and Astronomy, University of Manchester, Oxford Road, Manchester, M13 9PL, UK
- 26 Kavli Institute for Particle Astrophysics &amp; Cosmology, P. O. Box 2450, Stanford University, Stanford, CA 94305, USA
- 27 Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA 94720, USA
- 28 Fermi National Accelerator Laboratory, P. O. Box 500, Batavia, IL 60510, USA
- 29 Université Grenoble Alpes, CNRS, LPSC-IN2P3, 38000 Grenoble, France
- 30 Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Dr., Pasadena, CA 91109, USA
- 31 Department of Astronomy/Steward Observatory, University of Arizona, 933 North Cherry Avenue, Tucson, AZ 85721-0065, USA
- 32 Department of Physics and Astronomy, University of Waterloo, 200 University Ave W, Waterloo, ON N2L 3G1, Canada
- 33 Department of Astronomy, University of California, Berkeley, 501 Campbell Hall, Berkeley, CA 94720, USA
- 34 SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA
- 35 University Observatory, Faculty of Physics, LudwigMaximilians-Universität, Scheinerstr. 1, 81679 Munich, Germany
- 36 School of Physics and Astronomy, Cardiff University, CF24 3AA, UK
- 37 Department of Astronomy, University of Geneva, ch. d'Écogia 16, CH-1290 Versoix, Switzerland
- 38 Department of Physics, University of Arizona, Tucson, AZ 85721, USA
- 39 Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge CB3 0WA, UK
- 40 Instituto de Física Gleb Wataghin, Universidade Estadual de Campinas, 13083-859, Campinas, SP, Brazil
- 41 Nordita, KTH Royal Institute of Technology and Stockholm University, Hannes Alfvéns väg 12, SE-10691 Stockholm, Sweden
- 42 Department of Physics, University of Genova and INFN, Via Dodecaneso 33, 16146, Genova, Italy
- 43 ICTP South American Institute for Fundamental Research
- Instituto de Física Teórica, Universidade Estadual Paulista, São Paulo, Brazil
- 44 Center for Cosmology and Astro-Particle Physics, The Ohio State University, Columbus, OH 43210, USA
- 45 Space Telescope Science Institute, 3700 San Martin Drive, Baltimore, MD 21218, USA
- 46 Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas (CIEMAT), Madrid, Spain
- 47 Department of Physics and Astronomy, Stony Brook University, Stony Brook, NY 11794, USA
- 48 Institut de Recherche en Astrophysique et Planétologie (IRAP), Université de Toulouse, CNRS, UPS, CNES, 14 Av. Edouard Belin, 31400 Toulouse, France
- 49 Excellence Cluster Origins, Boltzmannstr. 2, 85748 Garching, Germany
- 50 Max Planck Institute for Extraterrestrial Physics, Giessenbachstrasse, 85748 Garching, Germany
- 51 Universitäts-Sternwarte, Fakultät für Physik, LudwigMaximilians Universität München, Scheinerstr. 1, 81679 München, Germany
- 52 Department of Physics, Stanford University, 382 Via Pueblo Mall, Stanford, CA 94305, USA
- 53 Cerro Tololo Inter-American Observatory, NSF's National Optical-Infrared Astronomy Research Laboratory, Casilla 603, La Serena, Chile
- 54 Institute for Astronomy, University of Edinburgh, Edinburgh EH9 3HJ, UK
- 55 Institute of Cosmology and Gravitation, University of Portsmouth, Portsmouth, PO1 3FX, UK
- 56 Department of Physics &amp; Astronomy, University College London, Gower Street, London, WC1E 6BT, UK
- 57 Institut d'Estudis Espacials de Catalunya (IEEC), 08034 Barcelona, Spain
- 58 Institute for Fundamental Physics of the Universe, Via Beirut 2, 34014 Trieste, Italy
- 59 Astronomy Unit, Department of Physics, University of Trieste, via Tiepolo 11, I-34131 Trieste, Italy
- 60 INAF-Osservatorio Astronomico di Trieste, via G. B. Tiepolo 11, I-34143 Trieste, Italy
- 61 Institute of Theoretical Astrophysics, University of Oslo. P.O. Box 1029 Blindern, NO-0315 Oslo, Norway
- 62 Instituto de Fisica Teorica UAM/CSIC, Universidad Autonoma
- de Madrid, 28049 Madrid, Spain
- 63 School of Mathematics and Physics, University of Queensland, Brisbane, QLD 4072, Australia
- 64 Santa Cruz Institute for Particle Physics, Santa Cruz, CA 95064, USA
- 65 Department of Physics, The Ohio State University, Columbus, OH 43210, USA
- 66 Center for Astrophysics | Harvard &amp; Smithsonian, 60 Garden Street, Cambridge, MA 02138, USA
- 67 Australian Astronomical Optics, Macquarie University, North Ryde, NSW 2113, Australia

- 68 Lowell Observatory, 1400 Mars Hill Rd, Flagstaff, AZ 86001, USA
- 69 Departamento de Física Matemática, Instituto de Física, Universidade de São Paulo, CP 66318, São Paulo, SP, 05314-970, Brazil 70 George P. and Cynthia Woods Mitchell Institute for Fundamental Physics and Astronomy, and Department of Physics and Astronomy, Texas A&amp;M University, College Station, TX 77843, USA
- 71 LPSC Grenoble 53, Avenue des Martyrs 38026 Grenoble, France
- 72 Institució Catalana de Recerca i Estudis Avançats, E-08010 Barcelona, Spain
- 73 Observatório Nacional, Rua Gal. José Cristino 77, Rio de Janeiro, RJ - 20921-400, Brazil
- 74 Hamburger Sternwarte, Universität Hamburg, Gojenbergsweg 112, 21029 Hamburg, Germany
- 75 Ruhr University Bochum, Faculty of Physics and Astronomy, Astronomical Institute, German Centre for Cosmological Lensing, 44780 Bochum, Germany
- 76 Physics Department, Lancaster University, Lancaster, LA1 4YB, UK
- 77 Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN 37831
- 78 Argonne National Laboratory, 9700 S Cass Ave, Lemont, IL 60439, USA
A. Campos B. Yin S. Dodelson A. Amon A. Alarcon C. Sánchez G. M. Bernstein G. Giannini J. Myles S. Samuroff O. Alves F. Andrade-Oliveira K. Bechtol M. R. Becker J. Blazek H. Camacho A. Carnero Rosell M. Carrasco Kind R. Cawthon C. Chang R. Chen A. Choi J. Cordero C. Davis J. DeRose H. T. Diehl C. Doux A. Drlica-Wagner K. Eckert T. F. Eifler J. Elvin-Poole S. Everett X. Fang A. Ferté O. Friedrich M. Gatti D. Gruen R. A. Gruendl I. Harrison W. G. Hartley K. Herner H. Huang E. M. Huff M. Jarvis E. Krause N. Kuropatkin P. -F. Leget N. MacCrann J. McCullough A. Navarro-Alsina S. Pandey J. Prat M. Raveri R. P. Rollins A. Roodman R. Rosenfeld A. J. Ross E. S. Rykoff J. Sanchez L. F. Secco I. Sevilla-Noarbe E. Sheldon T. Shin M. A. Troxel I. Tutusaus T. N. Varga R. H. Wechsler B. Yanny Y. Zhang J. Zuntz M. Aguena J. Annis D. Bacon S. Bocquet D. Brooks D. L. Burke J. Carretero F. J. Castander M. Costanzi L. N. da Costa J. De Vicente P. Doel I. Ferrero B. Flaugher J. Frieman J. García-Bellido E. Gaztanaga G. Gutierrez S. R. Hinton D. L. Hollowood K. Honscheid D. J. James K. Kuehn M. Lima H. Lin J. L. Marshall J. Mena-Fernández F. Menanteau R. Miquel R. L. C. Ogando M. Paterno M. E. S. Pereira A. Pieres A. A. Plazas Malagón A. Porredon E. Sanchez D. Sanchez Cid M. Smith E. Suchyta M. E. C. Swanson G. Tarle C. To V. Vikram N. Weaverdyck
Enhancing weak lensing redshift distribution characterization by optimizing the Dark Energy Survey Self-Organizing Map Photo-z method
Characterization of the redshift distribution of ensembles of galaxies is
pivotal for large scale structure cosmological studies. In this work, we focus
on improving the Self-Organizing Map (SOM) methodology for photometric redshift
estimation (SOMPZ), specifically in anticipation of the Dark Energy Survey Year
6 (DES Y6) data. This data set, featuring deeper and fainter galaxies than DES
Year 3 (DES Y3), demands adapted techniques to ensure accurate recovery of the
underlying redshift distribution. We investigate three strategies for enhancing
the existing SOM-based approach used in DES Y3: 1) Replacing the Y3 SOM
algorithm with one tailored for redshift estimation challenges; 2)
Incorporating $\textit{g}$-band flux information to refine redshift estimates
(i.e. using $\textit{griz}$ fluxes as opposed to only $\textit{riz}$); 3)
Augmenting redshift data for galaxies where available. These methods are
applied to DES Y3 data, and results are compared to the Y3 fiducial ones. Our
analysis indicates significant improvements with the first two strategies,
notably reducing the overlap between redshift bins. By combining strategies 1
and 2, we have successfully managed to reduce redshift bin overlap in DES Y3 by
up to 66$\%$. Conversely, the third strategy, involving the addition of
redshift data for selected galaxies as an additional feature in the method,
yields inferior results and is abandoned. Our findings contribute to the
advancement of weak lensing redshift characterization and lay the groundwork
for better redshift characterization in DES Year 6 and future stage IV surveys,
like the Rubin Observatory.