Cooperative Research Agreement spurs industry change
Vijay Naik and his team at IBM's T.J. Watson Research Center in Yorktown Heights, NY, developed a dynamic resource and job management system for massively parallel processing machines to improve system efficiency, user interactivity, and job turnaround time. by Judy Conlon |
Issue 4, January 1998 Welcome to the on-line version of NASA's Insights Magazine.
Insights is published by the High Performance Computing and Communications (HPCC) Program Office. To receive a hard copy of Insights or request an address change contact Judy Conlon or write to: NASA HPCC Insights, Mail Stop 269-3, Moffett Field, California 94035-1000, USA
| |
|
The goal behind NASA's High Performance Computing and Communications (HPCC) Program's Cooperative Research Agreement (CRA) "is to promote U.S. leadership in high-performance computing," according to David Bailey, NASA computer scientist. This three-year CRA concentrates on using a highly parallel computing environment for solving critical problems.
With a freight-train of competitive pressures moving through industry on a daily basis, IBM knew the benefits of CRA participation. There is an ever increasing demand for more computing power and a growing use of parallel computing. So IBM went to bat against other vendors with scalable parallel computers and was awarded the $22 million cooperative research agreement. As part of the agreement, IBM led a consortium that included Boeing Computer Services, Rensselaer Polytechnic Institute, Lockheed Missile and Space Research, Centric Engineering Systems, Intelligent Aerodynamics, and Rice University. IBM also installed three Scalable POWERparallel Systems (SP2) as the hardware testbeds for the research - the first of them 160-node SP2 at the Ames Research Center, the second a 48-node SP2 at Langley Research Center and the third a 8-node SP2 at Rice University. Agreements like this one are about collaboration, according to Richie Jacobovits, IBM project executive for the CRA. "If parallel computing is to become an established architecture for the general computing marketplace, it requires collaboration between industry and government labs. IBM and NASA shared the goal of establishing a broad and self-sustaining commercial market for parallel computing. NASA benefits because growth in this market means the availability of increasingly more powerful and mature parallel computers at lower cost for industry and government. IBM benefits because the CRA enabled the porting of Computational Aerosciences (CAS) codes to parallel architectures, the development of new CAS applications and the development of parallel system skills in industry and academia. All of these elements increase the demand for parallel computers." The CRA facilitates collaboration by offering private firms advantageous rights to patents and other intellectual property from the joint research. Through this spirit of cooperation, the CRA, completed last June, catapults change in the computing industry. Market sales of Massively Parallel Processor (MPP) systems climb. More early adopters strive toward higher performance and tackle more complex problems. IBM and other companies change their research and development course. The future direction of computing takes a new hue. Change in market trend Since placing the SP2 at NASA, IBM's SP line has attracted customers flocking to play the supercomputing game. Although Vijay Naik, IBM research staff member, is not sure how many of IBM's customers made their decision to buy based on NASA's selection of the SP, he thinks "a large number of customers were probably influenced by NASA's decision." He also presents a flattering report of NASA's parallel benchmarks - a common suite of measurements used by industry to test these machines.
"Because NASA has been a leader in high-performance computing for four decades, we could not ask for a better validation of the IBM SP2. We meet NASA's stringent tests. It certainly is prestigious and a morale boost for IBM upon re-entering the parallel high-performance computing field," adds Naik. "What carried the day for IBM is that its parallel computer had the best memory subsystem," says Thomas Lasinski, technical manager and computer futurist at NASA's Numerical Aerodynamics Simulation facility. "NASA evaluated the SP against a set of performance benchmarks. The decision, made on the basis of those numbers, was that the SP2 offered us the best price/performance." Despite the skepticism of supercomputing users in the early 90's, "we knew participating in NASA's parallel supercomputing research would make IBM a force to be reckoned with in the parallel system market," states Jacobovits. "Before the CRA, there were only a handful of parallel supercomputing installations, but today IBM has delivered parallel processors to over 2500 installations."
IBM launched its parallel computing line, the SP1, the year before HPCC kicked off the CRA. "We wanted NASA to test our product and verify its high performance and reliability," explains Jacobovits. "Having a technically demanding customer like NASA participate in the testing and validating of a large testbed system is an important checkpoint before these systems are released for production use in mission-critical computing environments."
To illustrate a simulation run on the IBM SP2, Vijay Naik shows a representation of carbon nanotubes. Simulations like this one, which require large computational resources, are used to understand the molecular mechanics involved in carbon nanotubes.
"Ford Motor Company has used Centric's software in the design process of new vehicles." - Steve Refai, Centric In addition to having the fastest parallel computer, IBM was best able to assemble a consortium and was positioned to exploit parallel computing commercially. By IBM placing its system with multiple nodes in a large government lab, IBM has the advantage of trying it out while paving the way to the commercial market for applications ranging from the scientific to non-scientific. Better design tools for the market One such application for the SP that resulted from the CRA is Centric's Spectrum TM Multiphysics engineering analysis software product for parallel computer systems. "With the CRA, we had the critical mass that enabled the release of Spectrum TM. We had the machine, the funding and the impetus to make a real product," says Steve Rifai, Centric director. "Without the CRA, we wouldn't have the revenues from parallel processing that we do."
With almost 80 percent of its business devoted to parallel system software, Centric has amassed the experience and knowledge to keep automotive, aeronautic and many other companies in shape. For example, Ford Motor Company has used Centric's software in the design process of new vehicles. Spectrum's parallel technology provides Ford engineers with highly accurate and detailed thermal/ computational fluid dynamic calculations in hours instead of the weeks that were previously required to obtain predictive engineering answers. The HPCC program is creating its own parallel computing renaissance on both sides of the fence - for industry and for government. "That's been the goal of the HPCC project from the beginning - to stimulate both the computing platforms and the funding that support those individuals who develop applications on these highly parallel computing platforms," adds David Bailey, who is simulating vortex roll up phenomena. Converting code to run on the 160-node SP2 proves faster than running his application on the C90. "This is a node-to-node performance measurement: It's not a matter of having to run on five or eight processors of the SP to match one CPU performance on the Cray. That's unusual." The CRA proved that problems requiring more memory and CPU cycles can be tackled in less time on the SP2 than on more conventional, vector-type supercomputers. "The SP2 is a powerful computer system with more than twice the main memory and disc space of the C90. So that's the carrot," Bailey says. "Four or five years ago, we had trouble attracting users to distributed memory systems, but today for the first time, this CRA proves that IBM's SP demonstrates competitive performance."
For example, NASA gets a steady stream of interest from industry's most experienced aeronautic designers. Among recent early adopters is CRA member Boeing, a 20-year veteran in high-performance computing tool development. (See Propelling power of prediction in September 1997 issue of INSIGHTS.) |
Change in industries R&D course While industry is now gravitating toward massively parallel systems, there were lessons along the way for IBM and NASA in trying to field and operate a system with a large number of nodes. Bailey explains the first lesson: "System software that runs on 16 or 32 nodes does not run without crashing on 160 nodes. Second, CRA customers can help flesh out these problems and help solve them."
One of the major results of the CRA process is enabling IBM's system software on a large number of nodes. The CRA provided an excellent opportunity to study the pattern of SP usage according to Naik, who manages the Distributed Resource Management System (DRMS), a project funded by the CRA. This experience has helped him in better understanding the resource management issues on large systems in the presence of often unpredictable and varying demands on the resources made by the users. It's given him an opportunity to successfully develop and test the new DRMS programming model - a method that provides an easy-to-understand abstraction for users in developing reconfigurable applications. These applications reconfigure dynamically in response to internal and external changes, enabling the system to manage its resources at a much finer granularity than is possible with more conventional approaches. This translates into higher system performance, a natural result of users applying the DRMS model. Goal: user-friendly operating system The CRA has helped in improving the quality of the Massively Parallel Processor operating environment by moving the technology one step closer to more powerful and productive systems. The MPP operating systems are not yet as flexible or adaptive as the sequential processor, virtual memory environments where jobs move in and out of the system without the user being aware of it. "The question is," Naik says, "How can we make the operating system more functional and user friendly without the user doing all the work?"
For a while developing a single application to make use of 160 powerful processors seemed about as possible as a camel passing through the eye of a needle. Equally as challenging is enabling resources on a massively parallel machine for many users who have different applications with varying demands on the system resources.
For most single applications, CRA participants found converting code to run on the 160-node SP2 proves faster than running the same application on the Cray C90. "Certain weaknesses in the system software required the development of NASA's Portable Batch System (PBS), which allows users to submit a job for either batch or interactive use," says Bailey. "IBM's software LoadLeveler manages a variety of different machines within the same location. PBS, in addition to providing new features beyond LoadLeveler, takes us one step further by managing jobs in a geographically distributed computing resource, such as the NASA metacenter," adds Jacobovits. HPCC provided this metacenter for research, which was made up of two systems, the 160-node IBM SP2 at NASA Ames Research Center and the 30- node SP2 at NASA Langley Research Center. To act on the excellent features in PBS, Naik says, "we took an application-oriented approach and extended its job management concepts by taking advantage of the applications based on the DRMS programming model. This method enables dynamic job scheduling and allows applications to reconfigure and adapt to the internal changes in resource requirements or to the external changes in the demands on the system resources. Under DRMS, the resource management system dynamically makes changes to the resource allocation while running rather than when a job terminates." For Bob Walkup who runs IBM's Center for Technical Computing in Yorktown Heights, New York, the Distributed Resource Management System has given his center more functionality than was previously possible on the SP system: "Previous system software tailored for batched processing lacked the flexible allocation of resources that DRMS allows. For users who want to run their code on all available CPUs, DRMS enables applications to dynamically change their size over time depending on the load on the system."
In addition to making parallel applications reconfigurable, the DRMS programming model provides other benefits. Among those are computational steering, multiple application coordination and parallel checkpointing where a user only adds small changes to existing parallel programs. Computational steering and multiple application coordination are key features in solving complex, multi-disciplinary design problems. For parallel checkpointing, the DRMS programming model makes it easier for the run-time system to capture the state of the application and store that on persistent storage such as a sequential or parallel file system. Using the captured state, an application can be restored on the same or on a different number of processors. For example, "if a job is running on 64 processors and two processors go down, using the checkpoint state, the application can be reconfigured and restarted right away to run on the remaining 62 processors," states Naik. "You do not have to wait until the other two processors are repaired and brought back. This increases the system robustness and is extremely desirable in production environments with long running jobs." The CRA also has given industry a jump start in achieving more compute power by focusing its research on the scalability of these systems.
Paradigm shift Another challenge the CRA addressed is the code-conversion process when users move from a shared memory system to a distributed memory system. Under management pressure, Bailey adds, "many users won't escape deadlines to learn a highly parallel system and convert codes." Since 95 percent of NASA scientists still use a shared memory architecture for their work, using the SP2 requires porting their code to a distributed memory architecture. With better programming abstractions, the effort involved in code conversions will be minimized. At the same time, programmers will benefit from the price/performance of scalable systems. The lessons learned from the CRA have resulted in an ongoing dichotomy, according to Lasinski. "For most single applications, we were able to parallelyze the code, so it runs faster on the 160 SP-2 nodes than it could run on the 16 C-90 nodes." The SP2 skips past the C90 by a factor of three or four in price performance. However, Lasinski takes the opposite viewpoint of performance when measuring sustained throughput, which is more effective on a shared memory machine with a mixture of applications. "It's true that if you take our current workload codes, which have obviously been written and tuned for Crays, then those tend not to run as competitively on the SP2. So the challenge is how we can tune those codes without totally rewriting them so that they will run well on the SP2," adds Bailey. Accessing data in remote memories - the heart of supercomputing - is expensive. Lasinski believes that most applications can be reorganized to work but maintains that there are some fickle applications that can not tolerate the kind of latency in a distributed memory system. In many cases, programmers reorganize their code to gain better data locality, which translates into computing more locally within a node before fetching data from remote memory. Minimizing the amount of communication is key to distributed memory programming. According to Naik, "Another reason to move from a shared memory system to a scalable parallel system is the need for a million million floating-point operations per second (teraflop)-level machine. Reaching that level using vector-type machines is much more expensive than with the scalable supercomputer technology." A future look Beneath it all lurks a whirl of excitement over what changes are next. Parallel computing and distributed computing are merging according to Naik. "You'll discover that very large-scale systems will be built with today's scalable systems as well as many other emerging types of architectures. The software that manages these systems will make this computing environment successful because the operating complexities will be totally transparent to the end users who will only interact with their desktop." "One success of this CRA is that it has given us a platform from which we can build systems that will manage resources over a geographically distributed area - a computational powergrid - ," Jacobovits has promised. "If we succeed in this future effort, the seed was sown in the work we did the last three years on the CRA." To advance computing - HPCC's mission - you go beyond what's possible today by looking at new ways to solve problems. These are the kind of problems that go undetected in a development environment. The brightest light in Bailey's constellation is a new approach to converting old codes, writing new codes and achieving better performance: "I hope 10 years from now that it's not business as usual." The difficulty in achieving well-working system software on a 160-node machine gives NASA and CRA members a sweeping view of challenges to come when systems house hundreds of thousands of processors. Bailey is already contemplating 160,000 nodes on a massively parallel machine right next door to a network of 10,000 nodes with each node acting like a 16 CPU system.
|
[Return to Top] [Return to Cover Page]