CT Project: Ten Years of Advances

FEATURE

Ten Years of Advances in Computing Earth and Space

Investigators tackle the universe's big questions

By Jarrett Cohen

Computer models of climate must be able to capture what happens in the real world. On that score, the UCLA Earth System Model is doing well. Its Pacific Ocean alternately warms and cools along the equator to produce realistic El Niño and La Niña events over a half-century simulation. Its atmosphere transmits the information from the equatorial Pacific to alter the modeled climate in remote places.

UCLA team leader C. Roberto Mechoso is especially proud of the model's record in anticipating impacts of the 1997–98 El Niño. In the early fall of 1997, the model predicted above average precipitation for California in November through December and more than double the average precipitation in the following February through March. The prediction materialized, encouraging further model development.

Visualization of precipitation deviations on globe

The UCLA Earth System Model predicted climate impacts of the 1997–98 El Niño. Colors show deviations from normal precipitation in millimeters per day.

Mechoso's team is but one of 17 Grand Challenge investigations chosen in the first two rounds (1992–2000) of NASA HPCC's Earth and Space Sciences (ESS) Project. To further its science goals, NASA competitively selected teams with world-class expertise in these disciplines to develop software technology that either simulates natural phenomena or analyzes observational data. The ESS Project has supported all of its investigators with access to large parallel supercomputers, including one of the earliest 512-processor Cray T3E machines, which was placed at Goddard Space Flight Center. ESS staff assisted the teams in optimizing the software codes for the latest architectures, resulting in performance as high as 630 billion floating-point operations per second.

Propelled by such performance gains, ESS investigators have been expanding the nation's ability to apply high-performance computing to thorny problems that science could not solve otherwise, such as:

How do the parts of the Earth influence each other?
What are the dynamics of magnetic fields in the sun and the solar wind?
How does matter behave in extreme environments?
What can we learn from NASA data using new analysis techniques?

By developing applications software to unravel these mysteries, ESS Project teams have pushed the boundaries of knowledge about our planet and the greater cosmos.

How do the parts of the Earth influence each other?

In representing the Earth as an interrelated system, the UCLA group couples atmospheric and oceanic circulation models with atmospheric and oceanic chemistry models. NASA HPCC has supported model development since 1992. A recent advance was a 35-year (1965–2000) global simulation of chlorofluorocarbons—human-made chemicals linked to ozone depletion in the middle atmosphere. Results suggest that those harmful chemicals continue to leak into the atmosphere, despite a 1996 ban on their production.

Another early ESS climate team, led by Max Suarez at Goddard Space Flight Center, evolved into NASA's Seasonal to Interannual Prediction Project. This effort is working to improve predictions of seasonal changes in circulation patterns, surface temperatures and precipitation, including those wrought by El Niño. Besides oceanic and atmospheric circulation models, they add a land surface model with soil moisture statistics, which heighten precipitation forecasting capabilities for some regions. A system to assimilate ocean observations is under development to drive model runs with more precise initial conditions.

A team directed by Peter Olson of Johns Hopkins University made huge software development strides in representing tectonic plates, whose motions cause earthquakes, trigger volcanoes and build mountains. Modeled with rock that can vary greatly in strength, simulated plates interact with the underlying mantle to produce a virtual Earth interior that matches observations. Model runs on the ESS Project's Cray T3E supercomputer reproduced plate behavior that likely built the Rocky Mountains. Olson's team also duplicated reversals in the Earth's magnetic field stemming from molten iron that twists and shears in the planetary core. Computations showed that heat-loss patterns between the core and the mantle affect the frequency of these reversals.

What are the dynamics of magnetic fields in the sun and the solar wind?

The Earth's magnetic field can become dangerous, at least to satellites and power grids, when it gets slammed by billion-ton gas bubbles carrying magnetic fields and charged particles from the sun. NASA better understands these coronal mass ejections (CMEs) thanks to new software from Tamas Gombosi's University of Michigan team. In a first-of-its-kind simulation, their software followed a CME riding the solar wind all the way to the Earth and interacting with the planet's magnetic field and atmosphere. The simulation ran 50 percent faster than reality, showing promise for predicting CME arrivals on Earth.

Visualization of magnetic field lines erupting from sun globe

In a Navy simulation enabled by NASA software, magnetic field lines reconnect so that one billion tons of gas can escape the sun as a coronal mass ejection.

Solar phenomena such as CMEs arise from magnetic reconnection, as first modeled by John Gardner and his Naval Research Laboratory group under ESS support. During reconnection, stretching magnetic field lines causes them to snap apart and reconnect to form new lines. Gardner's calculations identified the roles reconnection plays in sunspots, flares, prominences (gas arches suspended over the sun) and CMEs. Recently, a collaboration adding adaptive mesh refinement software from the ESS Project to the Navy's model code proved that reconnection can cut the magnetic field lines holding down a prominence and release it as a CME.

Moving deeper into the sun with their latest high-performance computing software, Andrea Malagoli's multi-university team found two possible origins of solar magnetic fields. In University of Colorado simulations, portions of the sun rotating at varying speeds pump magnetic fields with energy from the core. In University of Chicago models that consumed NASA HPCC's Cray T3E for three weeks, turbulent motions in gas near the surface create moving magnetic fields all over the star. These fields bump heat into the sun's atmosphere to make it nearly one million degrees hotter than the surface, a paradox this finding appears to resolve.

How does matter behave in extreme environments?

When a supernova squeezes the mass of a sun into a city-sized sphere, the result is one of the strangest objects in the universe—a neutron star. From the University of Illinois, Paul Saylor orchestrated development of software to study these creatures. It combines a new way of solving Einstein's General Relativity equations with gas dynamics. Armed with full relativity, computations revealed for the first time that two neutron stars could merge to form a black hole.

Visualization of microgravity flow using ribbons

University of Texas microgravity flow software follows warmer fluid rising from the bottom and cooler fluid descending from the top. Temperatures increase as they move toward red.

Closer to home is the microgravity environment on board the International Space Station. To learn how fluids behave in the near absence of gravity, Graham Carey at the University of Texas organized a two-pronged approach combining laboratory experiments with microgravity flow software built from scratch for the NASA HPCC research. Both techniques showed that fluid gets pulled by a surface tension gradient brought about by temperature and pressure differences. This fresh understanding may enhance station safety, especially in fire prevention. A related phenomenon is a dry spot, which forms when fluid drains away. Dry spots are common in microgravity, and the Texas research points to the possibility of such spots occurring on the ground in the increasingly thinner silicon films used to manufacture computer chips.

What can we learn from NASA data using new analysis techniques?

While casual observers think supercomputers just run simulations, the machines also handle data. That function is crucial for NASA, which is among the busiest data collectors in the world. Two recent ESS teams worked on methods for analyzing Earth science data taken from space.

Dave Curkendall and Jet Propulsion Laboratory (JPL) colleagues focused on synthetic aperture radar (SAR), exploiting its anytime, all-weather capabilities in three applications. By combining 3,000 data scenes into continental-scale mosaics, JPL scientists exposed environmental differences between the Amazon rain forest's wet and dry seasons. Notably, they were able to measure deforestation, most of which results from farmers burning trees to clear land. Considering several of the world's mountain ranges, Curkendall's collaborators from the University of California, Santa Barbara used SAR along with ground measurements to determine how much water melted-snow produces. They mapped the extent of snow, discriminating between it and other ground features. Analysis routines then estimated snow density and depth to project water runoff.

Repeat passes of a SAR satellite can detect even very small ground displacements, especially those occurring during and just after earthquakes. With software developed under this NASA HPCC investigation, Scripps Institution of Oceanography scientists reduced data processing time from three hours to a few minutes, displaying a near-real-time capability for earthquake monitoring.

Peter Lyster spearheaded efforts at NASA's Data Assimilation Office and the University of Maryland to advance techniques for incorporating atmospheric observations into climate models. His team reengineered the 100,000-lines-of-code production system that must assimilate NASA satellite data and provide accurate maps of the entire atmosphere on a daily basis. They also reprogrammed the software using the Message Passing Interface for interprocessor communications. To improve future production software, the researchers explored optimal Kalman filters, which evolve uncertainties in the mapped fields through space and time. Their efforts produced the first Kalman filter based on mapping chemical compounds using trajectories that move with the wind flow. The filters clarified how methane and other gases mix in the middle atmosphere.

NASA's Data Assimilation Office (DAO) produced Kalman filters to explore how chemical compounds mix in the middle atmosphere. A Kalman filter based on a fixed computational grid produces a rather smooth distribution of methane gas. The red balls represent the lowest concentrations, and the dark blue balls the highest.

A Kalman filter based on trajectories that move with the wind is more accurate because in nature the gas concentration tends to be constant along the flow. DAO researchers developed such a Lagrangian filter and used it to make a better map of the sharp contours in the distribution of methane.

Using data is a two-way street: simulations can sometimes aid in planning observation campaigns. For example, the University of Chicago simulations reproduced the granular pattern observed on the sun's surface but also resolved structures smaller than present observatories can see. NASA is using these results to plan its Solar-B mission, a satellite slated for launch in 2004.

Moving into Round 3

Creating software technology to help NASA support its missions is the heart of ESS Round 3. This fall, newly selected research teams begin designing high-performance software that others—ultimately non-developers—can use.

By itself, even best-in-class performance is not enough to make software usable for the broader community. To interpret exploding data volumes from fleets of new spacecraft, scientists and policy makers need high-fidelity, validated modeling and analysis tools that can be adapted inexpensively to changing needs. Moreover, selecting new software often involves comparing many candidates. For instance, there are more than 30 atmospheric circulation models to consider. The ESS Project thus requested its Round 3 teams to produce software that interoperates with offerings from other institutions while continuing to improve performance. Interoperability is most evident in the proposed Earth System Modeling Framework.

The Framework will be a software infrastructure for exchanging one model component, such as an atmosphere or ocean model, for another. It seeks to unite the nation's major climate modeling groups through software compatibility. An exchange capability will help these developers address a complaint of the modeling skeptic: two climate models produce different results when they start with the same conditions, so neither can be trusted. Being able to compare similar software components and pick the best one will allow systematic identification and removal of model weaknesses, thus improving fidelity to the real world.

This new software infrastructure is one tool to work towards predicting those changes that will occur in the next decade to century, both naturally and in response to human activity. Policy makers will then have a firm basis for decisions that affect the destiny of the Earth and those who live on it.

Performance and resolution gains

Note: *Team began work in ESS Project Round 1 (1992).

The Earth system

Principal Investigator	Phenomenon (Software Name)	Performance (gigaflops) Start -> End	Resolution (°/layers/km) Start -> End
*Roberto Mechoso UCLA	Atmosphere (UCLA AGCM)	0.2 -> 50	4° x 5°, 9 layers -> 2° x 2.5°, 29 layers
	Ocean (POP)	0.2 -> 39	1/6° x 1/6°, 37 layers -> 1/4° x 1/2°, 60 layers
	Coupled Atmosphere-Ocean	10 -> 37	Atmosphere same as above Ocean at 1/6° x 1/6°, 37 layers for coupling
Peter Olson Johns Hopkins Univ.	Mantle (TERRA)	0.96 -> 121	120 km -> 30 km
Peter Olson Johns Hopkins Univ.	Core (DYNAMO)	2 -> 630	100 km -> 20 km

Solar magnetic fields

Principal Investigator	Phenomenon (Software Name)	Performance (gigaflops) Start -> End	Resolution (levels/grid boxes) Start -> End
Tamas Gombosi Univ. of Michigan	Solar Wind (BATS-R-US)	New code -> 345	New code -> Adaptive-13 levels
*John Gardner NRL	Sun Atmosphere (CRUNCH3D)	0.40 -> 57	64³ -> 256³
	Sun Atmosphere (FCTMHD3D)	0.10 -> 56	64³ -> 512³
	Sun Atmosphere (AMRMHD3D)	0.05 -> 22	48³ -> 256³
*Andrea Malagoli Univ. of Chicago	Sun Interior (PPMC)	12 -> 196	128³ -> 1024 x 512²
	Sun Interior (MPS)	9 -> 160	64³ -> 1024²x 512
	Sun Interior (HPS)	0.25 -> 167	96³ -> 1024² x 3069

Extreme environments

Principal Investigator	Discipline (Software Name)	Performance (gigaflops) Start -> End	Resolution (grid boxes) Start -> End
Paul Saylor Univ. of Illinois	General Relativity (GR3D)	5 -> 142	160³ -> 500²x 996
Graham Carey Univ. of Texas	Microgravity (MGFLO)	10 -> 118.7	32² x 16 -> 320²x 10

NASA data analysis

Principal Investigator	Discipline (Software Name)	Performance (gigaflops/timing) Start -> End
Dave Curkendall Jet Propulsion Lab	Synthetic Aperture Radar (S⁴)	2 -> 102.6
*Peter Lyster Univ. of Maryland	Kalman Filter (Eulerian -> Lagrangian)	147 min. for 8-day assimilation -> 42 sec. for 8-day assimilation