This article is the fifth in a series on the nine NASA High Performance Computing and Communications (HPCC) Earth and Space Sciences Project Science Team II Grand Challenge Investigations. Grand Challenges are fundamental problems of widespread scientific impact whose solutions can be advanced by applying high-performance computing techniques.
Schoolchildren learn about the sun and planets in their earliest science classes. It turns out that these heavenly bodies "are only a tiny fraction of the solar system," said Tamas Gombosi, University of Michigan (UM) professor of space science and aerospace engineering. Over 99 percent of the volume dominated by the presence of the sun - the heliosphere - is filled by the star's outer atmosphere, or solar wind.
"The solar wind is the medium that carries information from the sun" to the rest of the solar system, Gombosi said. The solar wind flows outward as a gaseous mixture of protons (about 96 percent), helium atoms (about 4 percent) and minor ions (less than 0.1 percent). Ions are atoms with a high positive charge caused by being nearly stripped of electrons.
Expanding into the heliosphere, the solar wind brings life, destruction and mystery. Earth could not survive without the sun's light and heat. On the other hand, charged solar wind phenomena sometimes wreak havoc on satellites and power distribution systems. Comets were never thought to emit X-rays until some recently did just that when passing through the inner heliosphere. Scientists are now examining the fast-rotating Jupiter and Saturn and their moons to obtain more details about effects on them.
Spacecraft placed throughout the solar system observe these interactions but merely "give you point-wise data," said Timur Linde, UM aerospace engineering graduate student. "We need a large-scale model to explain the observations." Linde has joined Gombosi and other UM researchers to build a "single, global model that encompasses the heliosphere from the sun to the planets to the outer edge of the solar system," said Gombosi, the Grand Challenge's project director.
A smart grid in the driver's seat
The solar wind begins at the corona, or shortly above the visible surface of the sun. The natural, "quiet solar wind has two states, fast (700 to 800 kilometers per second) and slow (300 to 400 kilometers per second)," said UM's Clinton Groth, assistant research scientist in space physics. After approximately 16 billion kilometers, "the supersonic solar wind transitions through a shockwave, becomes subsonic and forms the heliospheric tail," Gombosi said. "The tail is well past Pluto, but we don't exactly know how far past." Because the sun's magnetic field permeates the entire solar wind, the computer model tracking the flow must describe how magnetized, ionized gases behave. The computer code does that by combining and solving equations written by Leonhard Euler (1707-1783) and James Maxwell (1831-1879). The output of these magnetohydrodynamic (MHD) equations is changes in mass, momentum, energy levels and magnetic field.
To do MHD as efficiently as possible, the UM group took the radical step of writing a code from scratch for the Grand Challenge effort. The 18-month-old code is playfully called BATS-R-US, for Block-Adaptive-Tree Solar-wind Roe Upwind Scheme.
BATS-R-US slices space into a grid of boxes or cells, like an accounting spreadsheet, except the MHD equations are solved in every cell. "Typically, when we get out to Earth in a heliosphere model, the planet fits in one cell," Groth said. With such huge differences in scales, a uniform grid with cells tiny enough to capture details in planets and smaller bodies is impossible on current supercomputers. So, "we make the grid smart" using adaptive mesh refinement (AMR), said UM associate professor of aerospace engineering Ken Powell.
Similar to a car's automatic transmission shifting gears, AMR intuitively divides certain cells again and again during a simulation - up to 15 times for the Earth-heliosphere coupling, Groth said. "There are small cells where a lot is going on, and big cells where little is going on," Powell explained. An oversight module of the program acts as the driver, marking cells for division and telling them when to stop dividing according to changes in physical criteria. Because the program works "independent of what criteria you choose," BATS-R-US can model the full variety of heliospheric phenomena, said Darren De Zeeuw, UM assistant research scientist in space physics.
"Before, people wrote a solar wind code, a comet code, and so on," Gombosi said. "There is a switch in our code to do such things as the outer heliosphere, comets or the Earth." Turning on the comet switch, the group simulated the interaction of Comet Hyakutake and the solar wind. Germany's ROSAT satellite observed X-ray emissions from the comet in 1996, the first detection of its kind. A BATS-R-US model run showed that a plausible X-ray source is solar-wind ions stealing electrons from non-charged atoms and molecules in the comet's atmosphere.
Building with blocks
While a critical component scientifically, AMR on a cell-by-cell basis is not very fast across multiple computer processors. Instead, BATS-R-US uses groups of cells called blocks. All blocks contain the same number of cells, regardless of whether cells are big or small. "You measure performance on a given computer to determine the best block size," or number of cells per block, explained Quentin Stout, UM professor of computer science. "You take one processor and time it using different block sizes." With the ideal size programmed into BATS-R-US, new blocks form within existing blocks when cells divide.
This efficient approach speeds the code up in several ways. Computing multiple cells per block requires less communication among processors. Also, "whether it is a little block or a big block, the work is exactly the same," said UM's Hal Marshall, associate research scientist in parallel computing. "You get one block fast, all the blocks are fast." Overall, blocks fit the UM team's strategy of doing "everything in parallel from the beginning," Marshall said, and the result is code that scales up very well with more computer processors.
Solutions show prediction potential
The most recent performance enhancement to BATS-R-US is an MHD solver specially tuned for heliospheric studies. A solver is a way of writing the MHD equations, including converting them into algebraic form so supercomputers can solve them. The team had been using the powerful Roe (for Phillip Roe, UM professor of aerospace engineering) solver, a staple in aerospace design software.
"The Roe solver is among the most accurate," Linde said, because it picks up on any type of effect in a gas. "However, we do not always need the biggest gun to shoot the prey. Sometimes this isn't as important as it seems," he continued. "We want an adequate solver. You use just as much power as you need and get the same accurate result." A subset of the Roe solver was folded into BATS-R-US in late 1997. It performs four times faster than its predecessor.
In various combinations, advances in BATS-R-US yield as much as 70 billion floating-point operations per second on NASA HPCC's 512-processor CRAY T3E at Goddard Space Flight Center, located at Greenbelt, Md. That is 3,000 times faster than the workstation code the group was using at the outset of their HPCC cooperative agreement in August 1996. Such performance enables simulations to "describe the time evolution of material to Earth faster than it actually occurs," Gombosi said. It has been a significant factor in the project's model to date - the march of a coronal mass ejection (CME) past Earth.
CMEs are gigantic bubbles of ionized hydrogen and electrons that burst away from the sun at irregular intervals. "A CME rides on top of the expanding flow of the solar wind," Groth said. "Magnetic and pressure disturbances propagate faster than ambient flow conditions," at speeds up to 2,000 kilometers per second. If a CME hits Earth, Gombosi explained, associated energetic particles can pass through and damage satellite instruments. A CME interacting with Earth's magnetic field "generates fairly turbulent magnetic fields and energetic electrons called killer electrons," Gombosi said. Resulting magnetic storms often knock out satellites and power grids.
The UM simulation followed a CME's path from its formation for 40 hours. "It is 17 times faster on the CRAY T3E, taking about 2 1/2 hours on 512 processors," De Zeeuw said. Modeling far faster than real-time demonstrates the model's potential for predicting if and when the bubble will hit the Earth and what the effects will be. "You run the code and get results in two to three hours instead of two to three days," Gombosi said.
A predictive capability could have prevented the failure of a $200 million AT&T satellite if as scientists believe, this failure was caused by a January 1997 CME. "Operators would then put spacecraft in safety modes and bring power grids down from peak," Gombosi said. However, "the code is far from being implemented for predictions, for which you need more than two days' warning."
Today, the Air Force predicts by experience. If they observe a combination of events, they use intuition and simple computer models," Gombosi said. "There is no model you can take, put in the initial conditions and then run a prediction. More than that, you need to create a form that is user-friendly and reliable. Non-specialists can break a code in ways you have never imagined."
Gombosi estimates that it will take three to five years to ready the code for regular, everyday use. "The work we are doing is the equivalent of a company's R&D department," Gombosi said. "Our role is to make sure the product can be mass-produced and easily used by people who are not highly trained physicists."
For more details on this investigation, see the project's World Wide Web site at: http://hpcc.engin.umich.edu/HPCC/.