<p>I think there are two ways to answer your question.</p>
<p>The first is to point out that GADGET was developed in the late 1990s and early 2000s, when GPUs were still a bit primitive, programming support for them not very portable, and most supercomputers did not include GPUs.</p>
<p>The second is to note that there <em>are</em> updated versions of GADGET that use GPUs. <a href="https://arxiv.org/abs/2003.10850v1" rel="nofollow noreferrer">Ragagnin et al. 2020, &quot;Gadget3 on GPUs with OpenACC&quot;,</a> discusses progress being made in rewriting GADGET-3 to take advantage of GPUs (while still being usable on systems without GPUs). I suspect a number of your questions might be answerable by reading it; I'll note some interesting comments about why GPUs are not perfect for cosmological simulations:</p>
<blockquote>
<p>Here below we list various limitations that prevent an easy porting of
the whole code Gadget3 to the GPUs:</p>
<ul>
<li>The code do not benefit from vectorisation because it stores data in arrays of large data structures (≈ 500B each) that do not fit
modern architecture caches. Chang- ing the data layout to a structure
of arrays would require a massive refactoring ef- fort and introduce
additional memory movement (of packing and unpacking data) in the
domain decomposition.</li>
<li>The use of blocking MPI communications (to exchange neighbouring particles between MPI ranks) poses a limit in fully utilising GPUs and
CPUs. [...]</li>
<li>GPUs memories have less capacity than their host memories, thus simulations that keeps all data in GPUs will require more computing
nodes than CPU only runs.</li>
<li>Gadget3 has been built over a decennial effort of developers who implemented various flavours of gravity, SPH solvers, and
sub-resolution models that have been extensively tested; rewriting
these modules using CUDA/OpenCL languages would imply a massive
rewrite of portions of such modules with associated risks of adding
mistakes.</li>
</ul>
<p>For these reasons, a directive-based approach that uses OpenACC [10]
has been adopted. This reduces modifications of the ongoing
development of Gadget3 and further- more makes it possible to still
run the code on CPU-only systems.</p>
</blockquote>
<p>(<a href="https://en.wikipedia.org/wiki/Message_Passing_Interface" rel="nofollow noreferrer">MPI</a> is used for communication between different nodes.)</p>
<p>As an example of the memory-limitation problem, they note that in a typical simulation</p>
<blockquote>
<p>&quot;each node was allocating 4GB for the Barnes Hut tree, 22GB for the
basic quantities used in gravity (e.g. position, mass, acceleration
ecc..), and additional 14GB for the SPH-only part (that is split in
den- sity computation and hydro-force computation), 0.6GB for the
metal evolution and an additional amount of 4GB for the active
particle list and to store the Hilbert space-filling-curve keys, for a
grand total of 40GB per node.&quot;</p>
</blockquote>
<p>and then go on to point out, &quot;It is clear that a 16GB GPU system (as for instance, the ones in <a href="https://www.cscs.ch/computers/piz-daint/" rel="nofollow noreferrer">Piz Daint</a>) would not be able to store the same number of particles of its underlying host.&quot;</p>

CC BY-SA 4.0
2024-08-19T09:55:49.363Z
*Peter Erwin to the rescue!*
uhoh
CC BY-SA 4.0
2024-08-13T15:09:51.917Z
2024-08-13T19:08:20.027Z
2024-08-13T19:08:20.027Z
Peter Erwin
<p>I noticed that most of the tools for cosmological simulation (e.g. large scale structure or single galaxy evolution) are based on GADGET. It only allows to use CPU.</p>
<p>Why not GPU? What is the main obstacle? Is it computationally ineffective for this type of simulations, and if so, why?</p>
<p>Are there any other tools for cosmological simulations that make use of GPU?</p>

CC BY-SA 4.0
2024-08-13T09:09:25.880Z
This doesn't answer your question, but there's lots of good info & links related to large-scale structure sims: https://astronomy.stackexchange.com/q/55010/16685
PM 2Ring
CC BY-SA 4.0
2024-08-13T13:31:26.250Z
I think that although they're faster, GPUs don't offer the same amount of memory, which is an issue for large-scale simulations. There are indeed simulations carried out on GPUs, but I think a more practical issue is that the codes are developed by astrophysicists, not computer scientists, so often are perhaps not the most optimal solutions.
pela
CC BY-SA 4.0
2024-08-13T14:25:48.333Z
There's maybe another issue in that GPUs are often optimized for single-precision (or even half-precision) floating-point operations, and are slower when doing double-precision calculations.
Peter Erwin
<galaxy><cosmology><software><n-body-simulations><physics-simulations>
Large scale structure/Single galaxy simulation on GPU
dust
# Large scale structure/Single galaxy simulation on GPU

(Asked by: username_0 on 2024-08-13)

I noticed that most of the tools for cosmological simulation (e.g. large scale structure or single galaxy evolution) are based on GADGET. It only allows to use CPU.

Why not GPU? What is the main obstacle? Is it computationally ineffective for this type of simulations, and if so, why?

Are there any other tools for cosmological simulations that make use of GPU?

**Tags:** galaxy, cosmology, software, n-body-simulations, physics-simulations
> **Comments:**
> **username_1**: This doesn't answer your question, but there's lots of good info & links related to large-scale structure sims: https://astronomy.stackexchange.com/q/55010/16685  
> **username_2**: I think that although they're faster, GPUs don't offer the same amount of memory, which is an issue for large-scale simulations. There are indeed simulations carried out on GPUs, but I think a more practical issue is that the codes are developed by astrophysicists, not computer scientists, so often are perhaps not the most optimal solutions.  
> **username_3**: There's maybe another issue in that GPUs are often optimized for single-precision (or even half-precision) floating-point operations, and are slower when doing double-precision calculations.  

---

## 1 answers

### Answer by username_3 on 2024-08-13. Score: 8 | Accepted answer

I think there are two ways to answer your question.

The first is to point out that GADGET was developed in the late 1990s and early 2000s, when GPUs were still a bit primitive, programming support for them not very portable, and most supercomputers did not include GPUs.

The second is to note that there *are* updated versions of GADGET that use GPUs. [Ragagnin et al. 2020, "Gadget3 on GPUs with OpenACC",](https://arxiv.org/abs/2003.10850v1) discusses progress being made in rewriting GADGET-3 to take advantage of GPUs (while still being usable on systems without GPUs). I suspect a number of your questions might be answerable by reading it; I'll note some interesting comments about why GPUs are not perfect for cosmological simulations:

> Here below we list various limitations that prevent an easy porting of
> the whole code Gadget3 to the GPUs:
>
> * The code do not benefit from vectorisation because it stores data in arrays of large data structures (≈ 500B each) that do not fit
>   modern architecture caches. Chang- ing the data layout to a structure
>   of arrays would require a massive refactoring ef- fort and introduce
>   additional memory movement (of packing and unpacking data) in the
>   domain decomposition.
> * The use of blocking MPI communications (to exchange neighbouring particles between MPI ranks) poses a limit in fully utilising GPUs and
>   CPUs. [...]
> * GPUs memories have less capacity than their host memories, thus simulations that keeps all data in GPUs will require more computing
>   nodes than CPU only runs.
> * Gadget3 has been built over a decennial effort of developers who implemented various flavours of gravity, SPH solvers, and
>   sub-resolution models that have been extensively tested; rewriting
>   these modules using CUDA/OpenCL languages would imply a massive
>   rewrite of portions of such modules with associated risks of adding
>   mistakes.
>
> For these reasons, a directive-based approach that uses OpenACC [10]
> has been adopted. This reduces modifications of the ongoing
> development of Gadget3 and further- more makes it possible to still
> run the code on CPU-only systems.

([MPI](https://en.wikipedia.org/wiki/Message_Passing_Interface) is used for communication between different nodes.)

As an example of the memory-limitation problem, they note that in a typical simulation

> "each node was allocating 4GB for the Barnes Hut tree, 22GB for the
> basic quantities used in gravity (e.g. position, mass, acceleration
> ecc..), and additional 14GB for the SPH-only part (that is split in
> den- sity computation and hydro-force computation), 0.6GB for the
> metal evolution and an additional amount of 4GB for the active
> particle list and to store the Hilbert space-filling-curve keys, for a
> grand total of 40GB per node."

and then go on to point out, "It is clear that a 16GB GPU system (as for instance, the ones in [Piz Daint](https://www.cscs.ch/computers/piz-daint/)) would not be able to store the same number of particles of its underlying host."

> **Comments:**
> **username_4**: *Peter Erwin to the rescue!*  

---

