NASA Logo, National Aeronautics and Space Administration
Computational Technologies Project

FINAL REPORT FOR THE COMPUTATIONAL TECHNOLOGIES PROJECT AT JPL

NASA Contract NAS7-03001
Task Order NMO710812

Submitted by: Robert Ferraro
Manager, Computational Technologies
Jet Propulsion Laboratory
California Institute of Technology

Introduction

The completion of this task order closes out 14 years of technology development and support for High-End Computing in collaboration with the NASA Goddard Space Flight Center (GSFC). This work has spanned three NASA Prime Contracts at the Jet Propulsion Laboratory, beginning in 1992 as the Earth and Space Sciences (ESS) Project at the start of the NASA High Performance Computing and Communications (HPCC) Program and ending with the Computational Technologies (CT) Project in the Earth Science Technology Office (ESTO).

During this time, JPL and GSFC have worked in partnership to develop science applications and parallel computing software technology, acquire and test emerging parallel computing hardware platforms, experiment with emerging high-speed networking technology, conduct basic research in parallel computing technologies, and support graduate students and post-doctoral students in areas of relevance to the application of parallel computers to NASA science questions.

Over the course of this project, JPL conducted research and development activities in accordance with the ESS Project Plan prepared by the ESS Project Manager at GSFC. (The ESS Project became the CT Project under ESTO when the HPCC Program terminated.) Areas of specific JPL responsibility included:

  • Develop new parallel scalable computational techniques, visualization software and tools, application-specific and generic computational libraries and library design formalisms, and made the software developed publicly available as warranted via the World Wide Web in accordance with JPL Procedure No. 46512 entitled “Disseminating JPL-Developed Software, Rev. 1, dated April 27, 2002.“
  • Assist in selection and evaluation of parallel computing testbeds and their comparison to other NASA testbeds, and project performance scaling of the CT Investigator Teams codes on full-up teraflops systems.
  • Perform software R&D in three major thrust areas:
    • Facilities which allow the rapid porting and tuning of evolving applications to high end computing architectures
    • Visualization tools which are capable of handling the data volumes associated with teraflops systems, and are accessible from remote locations
    • Parallel algorithms for NASA applications
  • Publish R&D results at conferences and in technical journals as warranted.
  • Prepare technical accomplishment reports as required by the CT Project manager.
  • Maintain a public website that publishes the annual reports of the R&D tasks undertaken at JPL with project funding.

When the ESS Project began, parallel computers were in their infancy – attaining hundreds of megaflops on NASA applications. Today, parallel computers achieve teraflops on NASA applications and have supplanted the vector supercomputers that were the norm a decade ago. Through this project, JPL has contributed to the adoption of parallel computing technology, within the Laboratory and throughout NASA, in the execution of Earth and space science research.

Computational Technologies Under ESTO

With the transition from the ESS Project under the HPCC Program to the CT Project under ESTO, JPL focused its activities on software technologies that enabled the use of parallel and distributed computers in the pursuit of science and engineering research and development. At the outset, each task produced an ESTO entry quad chart defining objective, approach, entry TRL (Technology Readiness Level) and a set of milestones that described the expected path of the research over the lifetime of the task. Upon conclusion of the task, a final report chart set was produced that included the entry quad chart, a final summary quad chart describing the task major accomplishments, and additional descriptive material as appropriate. These tasks (linked to their final report chart sets) were:

Major Accomplishments - The CT Project (Oct 2001–Sept 2005)

This section highlights some of the accomplishments at JPL under the CT Project.

NASA Space Act Awards

“A Robust and Scalable Library for Parallel Adaptive Mesh Refinement” Charles Norton, Sept 2004 – for the PYRAMID adaptive mesh refinement software

“OnEarth, a Remote Imagery Web Map Server Complex.” Lucean Plesea, Nov 2004 – for the Format and Location Agnostic Image Data Access Subsystem

Publicly Released Software

ParVox is a parallel three-dimensional parallel volume rendering system capable of visualizing large time-varying earth science datasets in structured and unstructured grids. In addition, ParVox is a distributed visualization system that allows uses to control the parallel renderer on a remote supercomputer using the user's desktop workstation and viewing the results interactively.

Scientists use complex three-dimensional models to study complex earth phenomena, such as the tectonic process causing earthquakes, the heat transport in the ocean circulation, etc. Such models tend to run for tens or hundreds of years and thus produce hundreds of gigabytes of data. Visualization of such large volume of datasets is a big challenge by itself. The computing power, memory capacity and network bandwidth of a desktop computer are not sufficient to handle a dataset of this size. We solve the problem using parallel software rendering on supercomputers. The unique features of ParVox3.0 are:

  • Scalable to large number of processors and large volume of data
  • Multiple renderers in one system to support both structured and unstructured grid data.
  • Distributed visualization support with image compression

Several movies created with the ParVox software are available.

PYRAMID is a software library used for the automatic adaptive refinement of large unstructured meshes on parallel computers and clusters. This software library can be easily used in a variety of unstructured parallel computational applications, including parallel finite element, parallel finite volume, parallel visualization applications using triangular or tetrahedral meshes, and quality controlled parallel mesh generation from initial coarse element models. The library contains a suite of well designed, and efficiently implemented, modules that perform operations in a typical parallel AMR process. Among these are:

  • Mesh quality control during successive parallel adaptive refinement (typically guided by a local-error estimator)
  • Parallel adaptive refinement
  • Parallel mesh migration and load balancing
  • Parallel mesh partitioning using ParMetis
  • Visualization (Tecplot format)

RIVA (Remote Interactive Visualization and Analysis System) is a parallel terrain rendering system. RIVA can be used as an interactive system to explore and visualize large terrain datasets in 3-D perspective views. It can also be used as an animation tool to generate fly-by movies using high-resolution images and digital elevation.

RIVA has been used JPL to generate many movies. During the process of making movies, new features were added to the baseline system continuously whenever they are needed. As a consequence, RIVA have been evolved, improved, and augmented. It has many nice features that other terrain rendering systems don't have.

The PVM_Wrapper library can be used in place of PVM on a parallel computer with MPI‑1. It will wrap the PVM calls with MPI calls, and permit applications that have PVM's initSend, pack, send paradigm to be used with MPI but without having to rewrite the application.

Multi Surface Light Table (MSLT) is an interactive software tool for viewing earthquake faults under terrain. MSLT was developed in support of the QuakeSim project, which has created an earthquake-fault database and a set of earthquake-simulation software tools. MSLT visualizes the three-dimensional geometries of faults embedded below the terrain and animates time-varying simulations of stress and slip. The fault segments, represented as rectangular surfaces at dip angles, are organized into collections. An interface built into MSLT queries and retrieves fault definitions from the QuakeSim fault database. MSLT also reads time-varying output from one of the QuakeSim simulation tools, called "Virtual California." Stress intensity is represented by variations in color. Slips are represented by directional indicators on the fault segments. The magnitudes of the slips are represented by the duration of the directional indicators in time.

Technology Infusions

The Image Data Access Subsystem is the underlying technology in the OnEarth portal that is the home site of the WMS Global Mosaic, a high resolution global image mosaic of the earth, produced from more than 8200 individual Landsat 7 scenes.

PYRAMID has been incorporated into the JPL GeoFEST finite element modeling code for modeling earthquake fault slips. It is also in use at 4 major research institutions around the world.

PARVOX and RIVA are used at JPL to visualize the output from large scale science models. It has been used through the QuakeSim portal to visualize results from Earthquake models.

The Multi-surface Light Table is available as an analysis tool for visualizing earthquake fault slips through the QuakeSim portal.

The Distributed Mission Simulation software has been integrated into the JPL Team I Instrument System design. It implements a V&V system that verifies and validates observation scenario, data acquisition, signal fidelity, and end-to-end data flow. It is also used by the JPL Project Design Center supporting Team X (Spacecraft System design) as a V&V system that verifies the performance of subsystems with respect to science-return.

The PVM_Wrapper library was incorporated into the UCLA global circulation model and used to port the software to several parallel computing platforms.

The Cluster Computing Technology has been adopted into the Repeat Orbit Interferometry data processing (ROIPAC) and Low Thrust Trajectory Analysis software at JPL.

WIGLAF has been incorporated into the processing systems at JPL for ROIPAC, the low thrust trajectory optimization software QLAW, MMPAT & GA optimization of spacecraft power systems, and the NEMO multi-million atom simulations software.

Major Accomplishments - the HPCC ESS Project (1992—2001)

This section highlights some of the accomplishments at JPL under the ESS Project.

Software Technology and Applications

Wavelet Transform-based data compression - This software was used on the Deep Space 1 Mission to support CCD and IR data compression throughout the operation phase of the mission (1999-2001). It was a spin-off of a parallel data compression algorithm developed for interactive distributed visualization software.

Optical Systems Modeling - parallelization and optimization of tools that were used for Next Generation Space Telescope (now call the James Webb Space Telescope) design.

MODTool - Parallel Millimeter-wave Optics Design Tool for quick study and design of Millimeter-wave instruments. This software was inserted into the JPL Team I (Instrument Design Lab). A NASA Space Award for this software was made in 2003.

PYRAMID - Parallel Unstructured Adaptive Mesh Refinement Library supporting 2-D (triangular) and 3-D (tetrahedral) large-scale finite-element meshing. It has been used for active device modeling at JPL and for CFD simulations at NASA MSFC. The latest version of this package is publicly available.

Scientific Software Modernization - Development of object-oriented and modernization programming techniques, based on advances in Fortran 90/95, to address design of complex large scale scientific computations. The techniques were applied to wavefront sensing and control optics software which is used by many missions including NGST, SIRTF, and SIM. The techniques were also used in the PYRAMID unstructured adaptive meshing software development.

SceneGen - Distributed Instrument POV data generator - used for the MICAS (Miniature Imaging Camera and Spectrometer) instrument by the Deep Space 1 Mission to model system response for optical navigation algorithm verification and science scenario sequence validation.

Distributed Mosaic Server - used by Mars Odyssey to plan the Mars surface observation campaign during the science phase of the mission. Data served was from the Viking image database.

ParNaRV - Parallel algorithms for Near-Realtime Visualization of Mars Images. Parallelization of mosaic generation and stereo image correlation using beowulf cluster computers. Delivered to JPL Multimission Image Processing Lab.

NEMO-1D - Nanoelectronic modeling tool that enables the simulation of carrier transport through a 1-D heterostructure on a fully quantum mechanical basis. Parallelized original code to study anisotropic hole transport. Delivered to JPL Center for Integrated Space Microsystems.

NEMO-3D - Nanoelectronic Modeling tool, that enables the electronic structure simulation of systems consisting of several million atoms. Delivered to JPL Center for Integrated Space Microsystems.

GENES - Genetically engineered nanoelectronic devices and classical circuits. Genetic algorithm based optimization and design exploration tools that has NEMO 1-D and circuit simulator SPICE embedded into it. Delivered to JPL Center for Integrated Space Microsystems.

Distributed In-Situ Simulation System - a software environment applied in the Mars Rover Technology Program to integrate a Rover simulation (ROAM) and instrument operation simulation (PanCam) for in-situ mission scenario exploration.

Matpar - Parallel extensions to MATLAB. Delivered to JPL Machine Learning Group.

BEOLIN - architecture extension for PVM, used by PVM users to support Beowulf architecture. Incorporated into PVM 3.4.3.

Beowulf Cluster Software Development - parallel algorithms for various applications on beowulf clusters. Basis for the NEMO-3D, GENES, ParNaRV, and NEMO-1D work at JPL.

For the MISR Instrument Team - Model-based Data-flow and process-flow analysis for potential massively parallel ground data system

Technical Support to ESS Grand Challenge Teams

An Earth System Model Atmosphere/Ocean Dynamics and Tracers Chemistry, PI: R. Mechoso/UCLA

Visualization Tools: RIVA was used to generate Atlantic Ocean Surface temperature animations from three datasets (ocean surface temperature, ocean surface height, and the global earth image mosaic). ParVox was used to visualize the 3-D variables produced by the ocean circulation model (current velocity, ocean temperature, and salinity). The movies and snapshots have been used in publications, technical presentations, and video tapes.

Code design and optimization support for the Atmospheric General Circulation Model, including detailed algorithmic and code-timing analysis, parallel scalability, and single-node performance analysis massively parallel systems.

Simulations of the Earth's Core and Mantle Dynamics, PI: P.Olson/JHU

Code design and optimization support to the Earth's Interior Modeling team in achieving code performance of core and mantle simulation codes at 630 and 180 gigaflops, respectively.

Parallel Computing Hardware and Networking

Concurrent Supercomputing Consortium (1993) - With other consortium members, and in partnership with Intel, acquired the first 1024 processor Intel Paragon supercomputer. This asset was capable of 38.5 gigaflops, a new record for its day. The ESS project used it for Earth and Space science applications development, and for software technology research.

Cray T3D Testbed at JPL (1994) - With HPCC Program sponsorship, and in partnership with Cray Research, Inc., acquired one of the first Cray T3D massively parallel processors. Consisting of 256 processing elements and 16 gigabytes of memory, the machine was capable of 38 gigaflops peak performance. The ESS project used it for Earth and Space science applications development, and for software technology research.

Workshop on Enabling Technologies for Peta(FL)OPS Computing (1994) - convened the first workshop (with sponsorship from NASA, DOE, NSF, NSA BMDO ARPA) to examine the technology path to achieving sustained petaflops supercomputing. More than 60 invited contributors from industry, academia, and government participated. The report was published as book by MIT Press.

ACTS Keck/GCM Experiment (1996) - This pair of experiments demonstrated the feasibility of using long path delay satellite links to establish meta-computing and control/data acquisition networks for remote collaboration, observation, and control of science experiments in hostile environments.

HP X-CLASS EXEMPLAR at Caltech/JPL (1998) - In partnership with NSF, NASA, and HP, acquired a 16 Hypernode supercomputer. Each hypernode contained 16 processors sharing 4 Gbytes of memory. This was one of the first of a new generation of coherent share memory supercomputers. The ESS project used it for Earth and space science applications development, and for software technology research.

Beowulf Class Cluster Computing (1998–2001) - Evaluation of cluster computing issues ranging from system monitoring software, application performance compared to traditional supercomputing systems, and gigabit networking. The experience was transitioned to various vendors including PSSC labs and Myricom Corp.

SGI Origin2000 (1999) - In partnership with JPL Ocean Science and the JPL Supercomputing Project, acquired an SGI Origin 2000. This architecture was a first in cache coherent global shared memory design. The ESS project used it for Earth and Space science applications development, and for software technology research.

Basic Research

During the HPCC Program years, JPL led research into new parallel computing architectures and technologies with the goal of reducing the risk to achieving petaflops computing capability in the earliest possible time-frame. With co-sponsorship from NSF and DARPA, JPL led design studies for a hybrid technology multi-threaded (HTMT) architecture and the incorporation of processor in memory technology into massively parallel computing architectures.

JPL also sponsored a number of Graduate Student Research Program fellowships and National Research Council Resident Research Associates with ESS Project funds. These students and post-docs conducted research activities in parallel computing software and tools as well as parallel algorithms and applications.

Conclusion

JPL's participation in the HPCC ESS Project, and its transition into the CT Project, has resulted in the maturing of parallel computing technology for use by NASA in addressing challenging problems in Earth and space science research, and in improving the engineering capabilities applied to NASA missions. Much of the technology developed over the course of this project was transitioned to operational use with JPL and within science research activities in NASA and other institutions.

USA.gov NASA Logo - nasa.gov