Partnership

Web site:

• IRFU. Institut de recherche sur les lois fondamentales de l'Univers (CEA)
• CRAL. Centre de recherche Astrophysique de Lyon
  
  

Context

Figure 1: Re-simulation on a supercluster of galaxies to increase the resolution

RAMSES is a typical computational intensive application used by astrophysicists to study the formation of galaxies. RAMSES is used, among other things, to simulate the evolution of a collisionless, self-gravitating fluid called "dark matter" through cosmic time. Individual trajectories of macro-particles are integrated using a state-of-the-art "N body solver", coupled to a finite volume Euler solver, based on the Adaptive Mesh Refinement technics. The computational space is decomposed among the available processors using a mesh partitionning strategy based on the Peano—Hilbert cell ordering. Cosmological simulations are usually divided into two main categories. Large scale periodic boxes requiring massively parallel computers are performed on very long elapsed time (usually several months.) The second category stands for much faster small scale "zoom simulations". One of the particularity of the HORIZON project is that it allows the re-simulation of some areas of interest for astronomers.

Use case

Performing a zoom simulation requires two steps: the first step consists of using RAMSES on a low resolution set of initial conditions i.e. with a small number of particles) to obtain at the end of the simulation a catalog of "dark matter halos", seen in Figure 2 as high-density peaks, containing each halo position, mass and velocity.

Figure 2: Time sequence (from left to right) of the projected density field in a cosmological simulation (large scale periodic box).

A small region is selected around each halo of the catalog, for which we can start the second step of the "zoom" method. This idea is to resimulate this specific halo at a much better resolution. For that, we add in the Lagrangian volume of the chosen halo a lot more particles, in order to obtain more accurate results. Similar "zoom simulations" are performed in parallel for each entry of the halo catalog and represent the main resource consuming part of the project.

RAMSES simulations are started from specific initial conditions, containing the initial particle masses, positions and velocities. These initial conditions are read from Fortran binary files, generated using a modified version of the Grafic code. This application generates Gaussian random fields at different resolution levels, consistent with current observational data obtained by the WMAP satellite observing the cosmic microwave background radiation. Two types of initial conditions can be generated with Grafic:

• Single level: this is the "standard" way of generating initial conditions. The resulting files are used to perform the first,low-resolution simulation, from which the halo catalog is extracted.
• Multiple levels: this initial conditions are used for the "zoom simulation" The resulting files consist of multiple, nested boxes of smaller and smaller dimensions, as for Russian dolls. The smallest box is centered around the halo region, for which we have locally a very high accuracy thanks to a much larger number of particles.

The result of the simulation is a set of "snaphots". Given a list of time steps (or expansion factor), RAMSES outputs the current state of the universe (i.e. the different parameters of each particules) in Fortran binary files.

Contribution

• Deployment of a SysFera-DS platform on 7 sites on Grid'5000 (12 clusters), for 48 hours. On the whole 1824 processors were dedicated to the experiment.
• We studied the possibility of computing a lot of medium-resolution simulations. The client requests as many 256³ particles 100Mpc.h^-1 simulations as possible.

The experiment permitted to compute 33 complete simulations, and 26 partial simulations, producing a total of 193GB of data.

• The benefit of running the simulation in parallel on different clusters is clearly visible: it would have taken more than 368 days to run the simulations sequentially.

Technical Form

References

"Thanks to SysFera-DS, we went from a single-cluster usage to a "grid", multi-cluster usage. Access to that grid coupled to a finer-grained management of data and execution workflows has caused a dramatic increase in performance, with simulations no longer taking three days, but less than one hour." - Hélène Courtois, IPNL, IN2P3

• Y. Caniou, E. Caron, H. Courtois, B. Depardon, and R. Teyssier. Cosmological simulations using grid middleware. In Fourth High-Performance Grid Computing Workshop (HPGC’07), Long Beach, California, USA, Mar. 2007. IEEE.
• B. Depardon, E. Caron, F. Desprez, J. Blaizot, and H. Courtois. Cos- mological simulations on a grid of computers. In J.M. Alimi and A. Füzfa, editors, INVISIBLE UNIVERSE: Proceedings of the Conference, volume 1241, pages 816–825, Paris, France, 2009. AIP.