Applications
Applications
To support the Grand Challenge project: "Three Dimensional Spherical Simulations of the Earth's Core and Mantle Dynamics Project" (PI: Professor Peter Olson, Johns Hopkins University) in achieving the NASA HPCC/ESS project's milestones by providing technical assistance in parallelization and optimization of their high performance applications (working with Tom Clune and Spencer Swift, SGI), assuring that the projects can succeed in their goals, and verifying that their milestones have been met.
The Simulations of the Earth's Core and Mantle Dynamics project's performance and software submission milestones focus on two codes.
The first simulates the generation and evolution of the Earth's magnetic field. It is named DYNAMO (lead author: Dr. Gary Glatzmaier, Los Alamos National Laboratory.) The changes that have been made to DYNAMO include new Fourier and Chebyshev transform routines added by Dan Katz and Tom Clune, optimized Legendre transforms and a large amount of loop and index reordering implemented by Tom Clune, and a change from treating the Coriolis term implicitly to treating it explicitly performed by Gary Glatzmaier. However, the 1-D data distribution greatly limits the problem size that may be considered when using large numbers of processors. Therefore, Tom Clune is currently implementing a 2-D data distribution that should have minimal effect on overall speed, but allow more realistic problem sizes to be run.
The second code is a mantle convection code named TERRA (lead author: Dr. John Baumgardner, Los Alamos National Laboratory). Changes that have been made to TERRA include addition of more realistic plate routines by John Baumgardner, and a number of small optimizations by Spencer Swift and Dan Katz.
At the end of FY97, DYNAMO ran on 54 CRAY T3E processors at 1.8 gigaFLOPS. Using a 1-D data decomposition, it currently runs on up to 427 CRAY T3E-600 (jsimpson) PE's at 104 GFLOPS, and on 512 T3E-1200 PE's, it has obtained a speed of 180 gigaFLOPS. The code can read in boundary data which was written by TERRA. Clearly, this should be able to attain the third milestone at any time.
At the end of FY98, TERRA ran on 512 CRAY T3E-600 PE's at 38.0 gigaFLOPS. Currently, it runs on the same machine at 40.1 gigaFLOPS. and on 256 CRAY T3E-1200 PE's at 28.7 gigaFLOPS. The code current writes out data on its boundary for use by DYNAMO. This code should be able to attain the performance required for the 3rd milestone when a large CRAY T3E-1200 is available (most likely, in October 1998).
Most of the computational significant changes to TERRA were made in the previous FY to meet the second performance milestone. Meeting the third milestone is a matter on ensure proper scaling throughout the complete code, and adding a realistic plate model without affecting the performance or scaling. The changes to DYNAMO were much more significant, changing the code from a parallel version of a vector code to a truly parallel, scalable code that will perform at the limits of any application on massively parallel machines.
TERRA must be run on a large CRAY T3E-1200 or equivalent machine, and the run must be verified and documented. The 2-D decomposition of DYNAMO must be completed, and the code must also be run on a large machine after TERRA is run to produce boundary conditions. These should both be complete in October 1998.
Daniel S. Katz
NASA Jet Propulsion Laboratory
Daniel.S.Katz@jpl.nasa.gov
818-354-7359