The National Institute for Computational Sciences

The Illustris Project

XSEDE Supercomputers Laid the Foundation for an Unprecedented Simulation of Cosmological Evolution

An international team of researchers used resources from the Extreme Science and Engineering Discovery Environment (XSEDE) to develop components that would serve as the basis for "Illustris," the most ambitious simulation of galaxy formation ever done. The project is described in a paper published on May 8, 2014, in the journal Nature.

This work has received worldwide news media coverage, including The New York Times, Los Angeles Times, The Washington Post, CNN, BBC News, Der Spiegel (in Germany), Le Figaro (in France), The Times of India, and many others.

The Illustris simulation project was led by Mark Vogelsberger of the Massachusetts Institute of Technology (MIT) and initiated while he was still a postdoctoral researcher at the Harvard-Smithsonian Center for Astrophysics (CfA). "Illustris represents the most detailed model of the universe," he says. "It took our international team, including major contributions from Volker Springel at the Heidelberg Institute for Theoretical Studies (HITS) in Germany, about five years to prepare and execute the simulation."

An excerpt from a news release issued by CfA conveys the commitment and computational power necessary for an undertaking of this magnitude: "The actual calculations took 3 months of 'run time,' using a total of 8,000 CPUs running in parallel. If they had used an average desktop computer, the calculations would have taken more than 2,000 years to complete."

Illustris allows one to journey back and see in high detail our Universe 12 million years after the Big Bang and then watch the cosmos evolve over a period of 13.8 billion years. The simulated volume captures tens of thousands of galaxies with 12 billion resolution elements (pixels) in a cube running 350 million light-years (106.5 megaparsecs) across.

This simulation is distinguished not only by how comprehensive it is but also by the fact that it creates a mixed population of elliptical and spiral galaxies, as well as shows small-scale evolution of gas and stars.

Vogelsberger highlights the importance of this type of simulation: "It is crucial to have these self-consistent models which include the dark components and baryonic physics in one complete simulation," he says.

The Role of XSEDE

The Illustris project used the XSEDE resources—Kraken (now decommissioned) at the National Institute for Computational Sciences (NICS), and Ranger and Stampede at the Texas Advanced Computing Center, under XSEDE project number TG-AST110016—to run three sets of simulations described by project collaborator Shy Genel of CfA as "crucial."

The first contribution of XSEDE, he explains, came in the form of a series of small-volume simulations that were run during the development phase of the physical model, and were critical to understanding different aspects of how the researchers' formulations of astrophysical processes affect the resulting simulated galaxy populations. In the end, the fiducial physical model—that is, the fixed basis of reference or comparison—that would be used for the production simulations was developed based on these tests.

The second aspect of assistance of the XSEDE resources was a series of simulations that are identical to the 'flagship' one—which was accomplished with resources from the Partnership for Advanced Computing in Europe, known as PRACE—except employing a smaller number of pixels, and thus resulting in coarser details. However, those simulations were completed about six months before the flagship, and produced very similar results, Genel says. He adds that this similarity allowed the researchers to develop and test post-processing analysis tools, and by applying them on these lower-resolution simulations, to already have a good idea of what the results of the flagship simulation would look like, and thereby "move the project in the most exciting directions."

Genel explains that the third area of support consisted of simulations that follow the same large cosmological volume as the other production runs, but with physics that were less complete, namely excluding the formation of stars and black holes, or normal matter altogether—keeping only dark matter, the mysterious and invisible substance that comprises an estimated 23 percent of the cosmos. He adds that through a direct comparison of this set of simulations with the others, the researchers were able to gain insight into the effects of the components.

"Each of these sets of early-phase simulations was very important to our project as a whole," Genel says. "Without having that first and second set of simulations, our project would have advanced significantly slower—probably on the order of a year or so."

He says that according to computer run logs for the project, 3 million central processing unit (CPU) hours were consumed on Ranger, with a maximum use of 4096 message passing interface (MPI) tasks and a maximum of 8192 processor cores (in MPI and OpenMP modes); approximately 1.5 million CPU hours on Kraken, with a maximum use of 4096 MPI tasks and a maximum of 8192 processor cores (in MPI and OpenMP modes); and approximately 2 million CPU hours on Stampede, with a maximum use of 4096 MPI tasks.

Genel explains the project's use of MPI and OpenMP (an application program interface): "The memory consumption of our code that runs the simulation changes during the run as cosmic structures become more and more complex. When our simulation approaches the current age of the universe, we sometimes need to increase the total memory available to it by increasing the number of CPUs. Using a combination of MPI and OpenMP, we are able to also efficiently use the compute power that comes with those additional CPUs, not only their internal memory."

The integrated nature of XSEDE's advanced digital services was advantageous to the Illustris project. "On top of using the machines for the calculations themselves, we also used the tape system, as well as the fast connection of the XSEDE network, which was very handy for us," Genel says. "Also, the flexibility of transferring allocated CPU time from one machine to the other is a feature we used several times, which has contributed to the global efficiency of using the CPU time we were granted."

Illustris: "Like a Time Machine"

The CfA news release expresses the unique perspective that Illustris offers: "Since light travels at a fixed speed, the farther away astronomers look, the farther back in time they can see. A galaxy one billion light-years away is seen as it was a billion years ago. Telescopes like Hubble can give us views of the early Universe by looking to greater distances. However, astronomers can't use Hubble to follow the evolution of a single galaxy over time."

Genel adds, "Illustris is like a time machine. We can go forward and backward in time. We can pause the simulation and zoom into a single galaxy or galaxy cluster to see what's really going on."

The Illustris Collaboration

The project's international team provides code development and simulation planning and analysis. The members are: Mark Vogelsberger (MIT), Shy Genel (Harvard University), Volker Springel (HITS), Paul Torrey (Harvard University), Deborah Sijaki (Cambridge University), Dandan Xu (HITS), Greg Snyder (Space Telescope Science Institute), Simeon Bird (Institute for Advanced Study, Princeton University), Dylan Nelson (Harvard University), and Lars Hernquist (Harvard University).

Scott Gibson, science writer, NICS

Article posting date: 21 July 2014

About JICS and NICS: The Joint Institute for Computational Sciences (JICS) was established by the University of Tennessee and Oak Ridge National Laboratory (ORNL) to advance scientific discovery and state-of-the-art engineering, and to further knowledge of computational modeling and simulation. JICS realizes its vision by taking full advantage of petascale-and-beyond computers housed at ORNL and by educating a new generation of scientists and engineers well versed in the application of computational modeling and simulation for solving the most challenging scientific and engineering problems. JICS runs the National Institute for Computational Sciences (NICS), which had the distinction of deploying and managing the Kraken supercomputer. NICS is a leading academic supercomputing center and a major partner in the National Science Foundation's eXtreme Science and Engineering Discovery Environment, known as XSEDE. In November 2012, JICS sited the Beacon system, which set a record for power efficiency and captured the number one position on the Green500 list of the most energy-efficient computers.