UCLA Climate Models

A Paradigm for The Real World
This article is the second in a series on the nine NASA HPCC Earth and Space Sciences (ESS) Project Science Team II Grand Challenge Investigations.

The NASA Jet Propulsion Laboratory's Yi Chao explains the results of a 30-year North Atlantic Ocean model run on the JPL Cray T3D.

by
Jarrett Cohen

Issue 2, May 1997

Insights was published by the High Performance Computing and Communications (HPCC) Program Office.

Program Manager:
Dr. William Feiereisen
Editor, Photography, Design:
Ms. Judy Conlon
WWW Design and Conversion:
Mr. Lawrence Picha

While many are not fully aware of it, Americans see and hear supercomputer output every day -- in the form of the weather forecast. Meteorologists base their predictions on National Weather Service atmospheric models that are very accurate up to five days in advance and often useful out to 10 days.

Longer-term trends enter the realm of climate, where modeling the atmosphere alone is not enough. "The climate system evolves through complicated interactions among its components," said C. Roberto Mechoso, professor of atmospheric sciences at the University of California, Los Angeles (UCLA). "In phenomena such as the El Niño-Southern Oscillation, the atmosphere and ocean influence each other, while the Antarctic ozone hole involves chemical reactions enabled by stratospheric dynamics."

Circulation of a moist atmosphere
At the base of the Earth System Model is the UCLA Atmospheric General Circulation Model (AGCM). General circulation models are the most advanced climate models in that the variables (e.g., temperature, wind velocities, relative humidity) progress through three dimensions. Begun in the early 1970s under Akio Arakawa, now professor emeritus of atmospheric dynamics and research professor, the UCLA AGCM was among the first and has been adapted by climate groups internationally.

This model divides the atmosphere into a rectangular grid of layered vertical columns. As supercomputer speed increases and algorithms improve, the grid's resolution becomes finer and finer to cover more climate phenomena in greater detail, or the simulations can cover longer periods of time. In 1992, Mechoso's Science Team I Grand Challenge team started with a resolution of five degrees longitude by four degrees latitude with nine vertical layers. "We are already more than double that spatially to 2.5 by two degrees with 29 vertical layers," Mechoso said. "We have even experimented with 1.25 by one degrees with 29 layers, which is close to the maximum we will get to with the AGCM in the current project, although I still hope for 57 layers."

Even with 1.25 by one-degree resolution, critical phenomena occur on scales smaller than the grid. The phase changes of water are particularly complicated events of this type. "Don't forget that the atmosphere is moist," said co-investigator Arakawa. "The problem is that condensation takes place in small clouds, which have the size of only a few kilometers. It's too much to model the individual, tiny clouds. Their collective effects are very important, however."

To incorporate such processes climate modelers use parameterizations, which formulate the statistical effect of sub-grid scale processes in terms of grid-scale prognostic variables. Arakawa developed a groundbreaking cumulus cloud convection scheme with Wayne Schubert, a former graduate student at UCLA and current professor at Colorado State University. In addition, the UCLA model's parameterized processes include transfer of longwave and shortwave radiation and heat exchange with the land and oceans.

Along with higher resolution and more efficient parameterizations, the team has seen nearly 50-fold speed-up on the AGCM. This year the 2.5 by two-degree, 29-layer version achieved nearly 10 billion floating-point operations per second (gigaflops) on NASA Goddard Space Flight Center's (GSFC) 512-node CRAY T3D. "When we started the HPCC project, the model we had could not run on multiple processors," Mechoso said. A collaboration with Lawrence Livermore National Laboratory, the optimized message passing code run on the CRAY T3D has been successful despite major portions requiring substantial interprocessor communication and thus not scaling well.

Ripples in the ocean
The primary oceanic general circulation model (OGCM) component is the Parallel Ocean Program (POP), which grew out of a Geophysical Fluid Dynamics Laboratory model. OGCM/POP also uses a rectangular grid, with nearly global coverage at one-fourth degree resolution with 16 vertical layers. Mechoso said they expect to get to one-tenth degree resolution with 100 layers. Solutions ensue through simultaneously determining a "vertical average" of velocity, temperature and salinity fields and then a "deviation from average," said UCLA's John Farrara, assistant researcher in atmospheric sciences.

Late last year, the team made a breakthrough with a North Atlantic rendition of OGCM/POP. They ran a one-sixth degree, 37-layer model for an unprecedented simulation of 30 years on the NASA Jet Propulsion Laboratory (JPL) 256-processor CRAY T3D. "We get Gulf Stream separation from the North Atlantic coast," said co-investigator Yi Chao of JPL's oceanography group. "This is a mesoscale feature that has never been produced before!" As the Gulf Stream flows eastward, it produces eddies narrower than 100 kilometers. Eddies such as these contain most of the ocean's kinetic energy, and "any climate model with an aim to produce the right ocean heat transport has to explicitly resolve them," Chao said.

Chemical journeys
In concert with physical dynamics, "the Earth's climate system is now known to be controlled to a significant extent by the chemical composition of the atmosphere," said co-investigator Richard Turco, UCLA professor of atmospheric sciences. "Greenhouse gases, for example, provide a direct radiative forcing of the climate. Human emissions of carbon dioxide, methane, and chlorofluorocarbons have already perturbed the global atmosphere. Indeed, ozone, another greenhouse gas, is photochemically affected by the chlorofluorocarbons as well."

Although largely in the test phase, chemistry tracer models follow these transformations of chemicals over global pathways. The UCLA Atmospheric Chemistry Tracer Model considers 50 chemical constituents and 100 photochemical processes in the troposphere, the atmosphere's lowest layer, ranging from eight to 18 kilometers.

"We are also developing a unique global aerosol tracer simulation," Turco explained. It will probe how aerosols, subvisible particles generated by chemical and mechanical processes, interact with clouds and cool the planet by reflecting sunlight, among other roles. Many of these aerosols have industrial sources (e.g., hydrocarbons, sulfur oxides). Initial runs showed that "tracers can be carried from the surface and dispersed globally," Turco said. "They are pumped into the upper troposphere and bumped around at very fast rates. The whole globe is connected very intimately."

Biology becomes the focus of the currently more primitive oceanic chemistry tracer model. Simulating the long-term carbon cycle will allow studies of detrital sedimentation, which involves a food chain of microscopic species settling nutrients on the ocean floor. By removing "the nutrients from the surface water," Turco said, the detrital process can "limit how much ocean productivity you can have."

A meeting of models
Because of their considerable technical and scientific challenges, Mechoso said that parallel coupled models are mainly developed by a few national laboratories. They include the National Center for Atmospheric Research and NASA's GSFC. "This NASA grant came at a critical stage in our project. Although we were making big progress in science issues, we desperately needed technical support to prepare the code for testing our ideas and addressing new challenges," he said. "Thus, we can contribute to the development of better models for prediction of either natural or human-caused changes of the Earth system."

Among his team's advances is a data broker for the models, "which have different resolutions and need to exchange information on different time steps," said UCLA postdoc Tony Drummond. The time-scale mismatch is especially difficult, for the atmosphere is like a flitting hummingbird, changing in as little as one hour, compared to the hibernating ocean taking one month to 1,000 years. Realistically, "the ocean needs surface winds, heat flux and fresh water flux," Chao said. "The atmosphere needs sea surface temperature (SST). The models exchange fields at specified intervals, typically every day."

With some model components better-suited to distinct architectures, the group also has emphasized heterogeneous computing. They demonstrated conceptually that a five-fold speed-up could be achieved by breaking the AGCM into two pieces and distributing them onto a CRAY C90 and CRAY T3D. "In our field, a combination of shared memory and distributed memory would be ideal," Mechoso said. "The speed-up could be superlinear," or greater than would be expected from the increased number of processors. Whether on one or multiple machines, algorithmic improvements are expected to bring the fully coupled Earth System Model to 100 gigaflops sustained by January 1999.

Validated model use will center on seasonal to interdecadal variation, chiefly the El Niño-Southern Oscillation (ENSO), a quasi-periodic warming of eastern Pacific Ocean SSTs and associated atmospheric events. As only the seasons have greater influence on climate, "we also plan to study the remote effects of ENSO," Mechoso said. "For example, we have found that rainfall in southeastern South America is linked to ENSO and with SSTs in the western Atlantic Ocean." To probe if the SST anomalies are connected, "you take the model with the Atlantic Ocean coupled to the atmosphere, and you run it with the SST in the Pacific with and without ENSO, and then you compare."

Direct policy input will occur through the Campus-Laboratory Collaboration Project, which aims to produce better predictions of precipitation over California for water reservoir decisions. One of several three-year grants was awarded to UCLA and Lawrence Livermore. "We will nest smaller-scale, regional models within the same global-scale model like the Atmospheric General Circulation Model and possibly the coupled general circulation model," Farrara said. The regional model has much higher resolution, about three kilometers, and covers roughly the West Coast.

"With two petaflops we might model the whole world like this!" Mechoso said. ( A petaflop is one million billion floating-point operations per second.) "The result would be an unprecedented level of detail on the evolution of the global climate system, including local impacts of extreme weather conditions such as floods and the relationships between precipitation runoff and water quality in specific areas."

Other Earth System Model co-investigators include James Demmel and Michael Stonebraker, University of California, Berkeley; David Halpern, JPL; Richard Muntz, UCLA; George Philander, Princeton University; and David Randall, Colorado State University.