Coronal mass ejections (CMEs) consist of ionized hydrogen and electrons that burst out from the sun at irregular intervals. Shown are 3D magnetic field lines (white and magenta) and magnetic field strength (low=blue to high=red) nine hours after the initiation of a CME.

Bad weather -- it came on fast and without warning. Its power was devastating. In its wake, more than six million people were left without power and in darkness.

It was a spring evening in 1989 when the severe space storm smashed into the Earth. More than a billion tons of electrically charged gas, which was blown off the sun three days earlier, plowed into the Earth's protective magnetic blanket, sending a torrent of current running toward the poles. In the Canadian province of Quebec, the flood of energy surged into electric lines, causing a system-wide power failure. The time from the onset of the storm to total blackout was just 90 seconds. When the lights finally came back on, millions of dollars in damage had been attributed to the storm.

Magnetic field lines from a CME appear to envelop users of the Cave Automatic Virtual Environment virtual reality tool. The University of Michigan's Stuart Feldman and Aaron Jacobovits developed the visualization application, which allows researchers to fly through the simulation results and find features. larger image (4K)

Humanity is maturing into a space-faring race. As we move from exploring space to using it to benefit mankind, the danger of some aspects of space weather has slowly been recognized. Space weather is a new term in the lexicon of science. It refers to surges of matter and magnetic energy blown off the sun that periodically sweep across interplanetary space. Twenty years ago the only consequence of space weather was a good aurora. Now we live in the age of the global village, where beepers, cell phones and the Internet keep many of us connected 24 hours a day. It is an age in which satellites are vital tools in communication and commerce, an age when the risk posed by a space storm reaching the outer atmosphere has become all too real.

Wherever there is weather, there is also the need for prediction. While Earth-bound storms travel a few thousand miles at most, space storms must cover the 100 million kilometer distance from the sun to the Earth. There is a lot of room for error in 100 million kilometers, as well as a lot of unknown physics to tame before a space weather prediction system can be put in place. This is the challenge Tamas Gombosi and his University of Michigan team of High Performance Computing and Communications (HPCC) Earth and Space Science (ESS) researchers believe they can tackle. Using computational systems that push the boundaries of state-of-the-art techniques, Gombosi and his collaborators hope to chart the pathways of space weather from the sun all the way to the Earth. Their recent success with the first sun-to-Earth simulation of a single space storm brings that goal and NASA's hopes for a working space weather prediction system one step closer to reality.

Tamas Gombosi. larger image (38K)

On being the right size

"The scales involved are enormous and enormously different," says Gombosi, professor of space science and aerospace engineering at the University of Michigan and principal investigator on the ESS grant. He explains the key feature that has limited space weather prediction: the sun has a radius of a hundred thousand kilometers, but the radius of the Earth is 100 times smaller. The distance between the two bodies, however, is 200 times larger. "Given the disparities in the size scales," says Gombosi, "tracking space weather is a very challenging computational problem."

The most powerful form of space storm is what researchers call a coronal mass ejection (CME). In a CME, a piece of the sun's outer atmosphere is blown into space at more than 500 kilometers per second. The power unleashed in a CME is staggering. Each CME launches billions of tons of matter, with an energy equivalent to 100 million atomic bombs, into space. Tracking the details of a CME's evolution as it crosses interplanetary space is the first step in a space weather prediction system. The next requirement is following the details of an evolved CME's collision with the Earth and its subsequent effect on the near-Earth space environment. The Michigan project is ambitious, aiming to simulate details of both phases of space weather.

Simulation by D.L. De Zeeuw, T.I. Gombosi, C.P.T. Groth, K.G. Powell and Q.F. Stout

The University of Michigan's models slice simulated space into a grid of cells. Using adaptive-mesh refinement, cells automatically divide into smaller cells as finer-scale activity occurs in them. Cells are grouped into blocks for greater efficiency on parallel computers. larger image (65K)

The principal tool of the project is "BATS-R-US," the imaginatively named computer code designed by the Michigan group to rein in the problem of space weather and its disparate scales. BATS-R-US stands for Block-Adaptive-Tree Solar-wind Roe Upwind Scheme. Embodied in that mouthful of a title is a series of innovative features that combine to make the Michigan team's effort unique in the world of computational physics.

The adjective Adaptive in BATS-R-US is one of the most important. It represents a significant advance in computational fluid dynamics in the last decade. Adaptive mesh refinement (AMR) is the capacity of a numerical code to put its resources where the action is. Resolution, which is the ability to capture the necessary level of detail, is among the most critical aspects of a simulation and often the best measure of its accuracy. If you can imagine creating a mosaic of the Mona Lisa with just four tiles, you have some idea of why low-resolution simulations are not worth much in terms of science. There is, however, an unfortunate trade-off in computational science between resolution and computer processing time. High-resolution simulations are also highly expensive.

"AMR codes get around this problem by dynamically changing the resolution," explains Ken Powell, co-principal investigator on the Michigan HPCC grant. Powell emphasizes that the BATS-R-US code can keep the resolution of the simulation high just where it is needed as the CME drives across interplanetary space toward the Earth. "The smallest zone in the simulations is 1/16 the size of the sun," he says, "while the largest is more than eight times the size of the sun. The resolution at any particular location in the grid changes, depending on what is happening there." The capacity to intelligently increase, and then relax, resolution as the simulation proceeds saves the Michigan group tens of thousands of computer processing hours. Without it the simulations and space weather prediction would become impossible.

Darren De Zeeuw, a research scientist in the Michigan group, explains how the BATS-R-US code was written from scratch for optimal speed. larger image (64K)

A journey of a thousand steps

Before they could simulate a space storm, the Michigan group first had to get the interplanetary equivalent of a beautiful day correct. A CME's propagation from the sun to the Earth could never be modeled correctly unless the space through which it propagated had the right properties. "We started out by simulating the normal solar wind in a domain that stretched from one solar radius all the way out to 215 solar radii (just beyond the Earth)," says Gombosi. The solar wind that fills interplanetary space has a considerable structure, even when the sun is quiet. Including a model of heating within the solar atmosphere helped the team get the background right. "We had to tweak our parameters a bit," says Gombosi, "but in the end we came up with an excellent fit to the solar wind data for the quiet sun." With the smooth seas worked out, Gombosi and his collaborators were ready to brew up a storm.

Exactly how a CME forms on the sun remains a mystery, so to create a storm in their simulations, the Michigan group helped nature along a little. "At the beginning of the simulation, we injected a small region of high density and closed magnetic field loops into the corona," says Gombosi. "It disturbed the delicate balance there, and we ended up with a nice CME." As the virtual CME exploded into interplanetary space, the researchers watched carefully to see where it would cross the Earth's orbit. Then, exercising a bit of God-like control, they reran the simulation, starting the Earth in just the right position to ensure a collision between it and the CME.

Tamas Gombosi, University of Michigan professor of space science and aerospace engineering, leads this HPCC Grand Challenge investigation. larger image (32K)

To catch the physics of the Earth-CME collision, the Michigan team faced a very tiny problem. The smallest cell in the team's sun-to-Earth calculation was 440 times the size of our planet. This is the Mona Lisa mosaic problem all over again. "We needed to recreate the Earth's grid so we could see what happened as the CME passed," says Gombosi. The Michigan group solved its problem with two related simulations. First, team members performed the large-scale simulation that tracked the CME as it crossed the Earth's orbit. Then they fed the values of the magnetic field, velocity and density in the passing storm into a second BATS-R-US simulation that focused on the near-Earth environment. "The smallest cell in that calculation was only a quarter of the Earth's radius," explains Gombosi. With the two linked calculations, the Michigan group could claim success in fielding the first fully 3D calculations for the evolution of space weather from the sun all the way to the Earth.

One of the more promising aspects of the Michigan group's success is that the simulations were faster than the duration of a real space storm. "Our calculation was over in less time than a real storm would have taken to reach the Earth," explains Darren De Zeeuw, a research scientist in the Michigan group. "We were faster than a speeding CME."

Simulation by D.L. De Zeeuw, T.I. Gombosi, C.P.T. Groth, K.G. Powell and Q.F. Stout

The outer heliosphere's slightly bent field lines trace the interstellar magnetic field while the spiral field lines convect out with the solar wind. The inset shows a detailed view of the bow shock. larger image (73K)

The adaptive parallel universe

The extraordinary speed of the BATS-R-US code comes directly from the "Block" adjective in its name and represents one of the most innovative aspects of the Michigan team's effort. "We knew we would be using parallel computers when we began the project," says Powell, "so we tried to optimize the construction of the adaptive mesh refinement with the parallelization in mind." A parallel computer is programmed to take a single problem and distribute it across many processors at once. If the researchers do their homework right, they can get the code to speed up by a factor nearly equal to the number of processors. The traditional approach to solving problems with parallel machines is domain decomposition. Each processor gets a chunk, or slab, of the physical domain that needs to be simulated. "Domain decomposition is fine when the grid is fixed," explains Powell, "but it's not so straightforward with an adaptive mesh code because the domain is changing as the resolution in a region is increased and then relaxed."

Code performance in billion floating-point operations per second (gigaflops) on a variety of machines. Comparing the black dashed line to the colored line for a computer can show, for example, whether running on 32 processors will run 32 times faster than one processor. larger image (41K)

Facing the problem head-on, the Michigan team dreamed up a novel solution. The most important thing in a parallel calculation is that no processor be left idle. To ensure this does not occur in the BATS-R-US code, the computational space is divided into blocks of grid cells. Each block was a fixed size, say 10x10x10 grid cells. The block becomes the unit of the calculation. Each processor is then tuned to work on so many blocks at once. If the resolution inside a block needs to increase, it is simply divided into new 10x10x10 blocks. The new blocks are redistributed across the processors to ensure each CPU has an equivalent workload. "We never have an unbalanced load on more than one processor," explains De Zeeuw. "That means latency didn't hurt us at all." The Block strategy has proven enormously successful for the Michigan group. "We really have gotten AMR to work effectively on parallel machines," continues De Zeeuw. "We see the speed of the code increase linearly with the number of processors, all the way up to massively parallel machines." De Zeeuw explains that the group recently ran the code on a machine with 1,490 processors and achieved more than 99 percent of the scaling speed up.

The real world

The group's success marks only the beginning of its efforts to create a tool for real space weather prediction. "The model we used to describe the Earth is really too simple," says Powell. The physics of the near-Earth environment is more complicated than that of interplanetary space. There are variations in the degree of ionization (how many free electrons are running around) and in the chemistry in the upper layers of Earth's atmosphere. The coupling of these layers to the planet's magnetic blanket (the magnetosphere) is also quite complicated. To produce models that can accurately follow the Earth's response to a CME, the Michigan group has teamed up with a number of the nation's best researchers in space physics.

Ken Powell and the UM group optimized the performance of adaptive mesh refinement with parallel computers in mind. larger image (40K)

"We are working with NCAR (the National Center for Atmospheric Research) and Rice University," says Gombosi. He explains that these groups have developed sophisticated codes to simulate different parts of the Earth's upper reaches. The Rice team has a numerical model for the inner regions of the magnetosphere. The NCAR team has developed a numerical code called TIMEGCM or Thermosphere-Ionosphere-Mesosphere-Electro-Global-Circulation Model. As the name implies, it was built to handle a good portion of the outer regions of the Earth's atmosphere.

Linking the BATS-R-US codes to models built by other groups, but not optimized for parallel machines, poses a considerable challenge for the Michigan researchers. "We are basically taking their physics kernels and building new numerics around them," says Powell. "We will act as the glue that holds the different codes together." The pay-off promises to make the effort more than worthwhile because the final, coupled codes will track current flow and chemistry in detail as a CME pours energy into the near-Earth environment. This is a feature that a true predictive system must include.

Simulation by D.L. De Zeeuw, T.I. Gombosi, C.P.T. Groth, K.G. Powell and Q.F. Stout

Coronal mass ejections (CMEs) burst out from the sun at irregular intervals. Shown are magnetic field lines (white) and magnetic field strength (blue = low to red = high) 20 hours into a CME simulation. larger image (41K)

"When you start doing something as ambitious as this," says Gombosi, summarizing his team's approach to the problem, "you always have to make compromises in terms of the physics. Hopefully we are now moving to the point that we can make fewer of these compromises and get closer to the real systems." With society becoming ever-more-dependent on the environment hundreds to thousands of miles from the Earth, the Michigan team's progress holds the promise of a true sea change in our approach to space weather.

Credits for Insights Magazine go to the following people along with the writers and photographers who contribute to each issue and the researchers and specialists whose material is highlighted:

Program manager: Dr. Eugene L. Tu
Deputy program manager: Dr. William R. Van Dalsem
Editor, writer, photographer and designer: Judy Conlon
Web designer and programmer: Sue Cox

Insights was published by the HPCC Program office and produced by Raytheon contractor staff at NASA Ames Research Center.