P.M. Lyster, J.W. Larson, W. Sawyer, C.H.Q. Ding1, L.-P. Chang2, R. Ménard3, R. Rood4, S. E. Cohn4
University of Maryland Earth Systems Science Interdisciplinary Center*
Email to lysterp@mail.nih.gov
Additional Affiliations
1. Lawrence Berkeley National Laboratory, Berkeley, CA 94720
2. General Sciences Corporation, a subsidiary of Science Applications
International Corporation
3. University of Maryland at Baltimore County, Joint Center for
Earth Systems Technology
4. NASA/GSFC Data Assimilation Office
* Changed name from: Joint Center for Earth Systems Science
This is the third annual report to the testbed that has
arisen out of the Cooperative Agreement (NCCS5-150):
Grand Challenge Applications and
Enabling Scalable Computing Testbed(s) in Support
of High Performance Computing under the NASA High Performance
Computing and Communications Earth and Space Sciences Project
(HPCC ESS).
This covers the
period October 1, 1997 to September 30, 1998, and should be read
as a follow on to the annual reports for
1996 and
1997.
This is a collaborative project that has involved scientists from
the University of Maryland Earth Systems Science Interdisciplinary
Center (ESSIC), the NASA/Goddard Space Flight Center Data Assimilation
Office (DAO), and Lawrence Berkeley National
Laboratory.
The primary objectives of this Grand Challenge HPCC project are: to investigate the use of high performance computing to advance the science of data assimilation; to facilitate the migration of NASA's atmospheric Four Dimensional Data Assimilation (4DDA) software to scalable parallel systems; and to serve as discerning developers and testers of modern parallel computer technology. The present project concentrates on (i) the Core computing algorithm of the operational Goddard Earth Observing System Data Assimilation System (GEOS DAS) and (ii) the parallel Kalman filter. The Core software is comprised of the model which is a three-dimensional General Circulation Model (GCM) and the analysis which is the GEOS Interface code and the Physical-space Statistical Analysis System (PSAS). Before discussing the detailed work of the current project it is useful to broaden the context and now describe the complete environment in which data assimilation is being carried out at the NASA Data Assimilation Office (DAO). Note, some of this material is reproduced from previous reports for the purpose of completeness. A recent article by Jarrett Cohen on the Project appeared in the NASA's Insights Magazine, and an article appeared in Supercomputing97 by Lyster et al. 1997e. Also, the reader is encouraged to view the document: "The Computational Complexity of Atmospheric Data Assimilation" [Lyster, 1998b] for a discussion on both the computational complexity of GEOS DAS and comments on scalability and performance of the parallel applications.
Atmospheric data assimilation produces accurate gridded datasets that are used by meteorologists for weather analysis and forecasts; it is also being used as a tool for climate research. The DAO conducts reanalysis of archived earth-science datasets as well as real-time mission support analysis and forecasts for the climate research community. There are various definitions of data assimilation; the present project broadly describes 4DDA as the combination of a range of observations with physically consistent model forecasts to produce a best estimate of the state of the atmosphere. The DAO's Goddard Earth Observing System (GEOS) Data Assimilation System (DAS) is described in the Algorithm Theoretical Basis Document [DAO, 1996]. The DAO is preparing to move GEOS DAS to distributed-memory parallel computing platforms. The new system will be designated GEOS-3. This will be used for DAO's normal activities, and it will be an important for of NASA's Earth Science Enterprise in the coming years. It is also an important environment for studying new parallel computing technology under HPCC.
The end-to-end GEOS DAS is shown schematically in Figure 1. The present DAS, whose Core system (described in the Appendix A ) runs on multitasking Cray C90, J90, and SGI Origin 2000 computers, is fed from the Global Telecommunication System (GTS). Meteorological data, including satellite-retrieved temperature profiles, are given to the Goddard Distributed Active Archive Center (DAAC). These data proceed through a data reduction and preprocessing stage before being ingested by the Core system. In the future, new datatypes will be provided via the EOS Data and Information System (EOSDIS) Core System (ECS). Postprocessing stages include: Quality Assurance of Data Sets (QuADS) inspects the gridded data sets produced by the Core system for I/O error and general physical consistency; Adaptive Tuning of Error Statistics System (ATESS) produces observation and forecast error statistics for the GEOS DAS; and DAO On-Line Monitoring System (DOLMS) monitors observation and gridded data streams and makes them available for graphical presentation. There are a number of different modes of operation for the DAS. Briefly, mission support involves real-time, or First-Look, data assimilation and sometimes the production of up to 10-day model forecasts. For current data sets made available from the Goddard DAAC, this mode of operation ingests about 50 megabytes of data per day into the Core system (about 500,000 observations per day). In the next two years this number will increase to 150 megabytes per day (1.5 million observations per day). The output analysis (gridded) datasets are about 1 gigabyte per day in real-time mode, while the production of model-forecast fields can increase this quantity by over an order of magnitude. Periodically, the DAO conducts reanalysis projects that involve multi-year analysis whose data sets are then studied and distributed to the climate research community. In this mode of operation, the DAO plans for a production rate of 30 days of assimilation per wall-clock day.
The end-to-end GEOS DAS is a complex operation that is being updated using modern Software Engineering methods [Pressman, 1993]. There are over 100 people directly involved in DAO research and operations. The need for Software Engineering has become apparent in a complex environment where issues such as Formal Software Testing, Configuration Management (CM), version control and maintenance, multiple hardware platforms, and the coordination of distributed personnel are important. The DAO enthusiastically embraced Software Engineering and CM after it became apparent that the organization had reached critical mass in terms of personnel and technology. The GEOS-3 System Development Plan is available on the DAO intranet, and the GEOS-3 Software Plan (section 4 of the SDP) is referenced here. DAO is also subject to a yearly review by its external computing panel [Farrell et al., 1996, 1997].
As part of the DAO's preparation to provide support to NASA's Earth Science Enterprise in 1999, and as part of a Grand Challenge project that is being funded by the NASA High Performance Computing and Communications (HPCC) Earth and Space Sciences (ESS) program, a Core system that is based on the Message Passing Interface (MPI) parallel library is being developed using a Fortran 90 modular approach -- the Goddard Earth Modeling System (GEMS) [DAO, 1997]. Before 1997, key components of Core GEOS DAS, including the GCM (in collaboration with Max Suarez at GSFC) and an early version PSAS, have been parallelized [Ding and Ferraro, 1995] [Ding and Ferraro, 1996] (in this report, we shall refer to this parallel version of the early PSAS as the prototype parallel PSAS). These form the basis for the design of their respective parts of the parallel system. A discussion on the status of this prototyping effort (as of 1997) can be found in the article that appeared in Supercomputing97 by Lyster et al. (1997e). During 1997 and 1998 much of the work has involved transferring the knowledge gained in the prototyping of parallel code into the actual scientific code. The goal is to produce a scientifically valid and optimized parallel Core system. Most of the technological achievements of the current report deal with this effort.
Data assimilation refers to the integration of observational data into numerical models. It is a process that involves comparing observations with model results as well as estimating the observation and model error statistics, in order to extract the maximum amount of information from observations and improve the quantitative abilities of the model. To simplify the computation in a number of present-day operational data assimilation algorithms, the error statistics are often prescribed (e.g., modeled as homogeneous and isotropic), but not in this research with the Kalman filter where the dynamical model is used to evolve the error statistics. Early work in the present HPCC project developed the first Kalman filter for the assimilation of actual atmospheric observations -- in this case the filter is used to assimilate trace chemical constituents in the stratosphere where the dynamics may be assumed two-dimensional (horizontal isentropic surfaces) over timescales of days to weeks [Lyster et al. 1997d]. The observations are chemical constituents from instruments on board NASA's Upper Atmosphere Research Satellite (UARS).
The Kalman filter has three unknown error covariances; the initial error covariance, the observation error covariance, and a covariance of transport modeling errors. In this past year we have developed strategies for modeling the unknown covariances based on the principle of minimizing data shocks, and developed a chi-square validation tool used to estimate free covariance parameters. We found, for instance, that the measurement error accounts for only a small fraction of the observation error used in the assimilation system. The main source of error in the observations, called the representativeness error, is due to sub-grid scale variability arising from the mismatch between grid point model averaging and spatial observational averaging. We also investigated the effect of variance and correlation dynamics in assimilation experiments and found that flow-dependent error correlations accounts to a large extent for the optimality of the Kalman filter system. This work is discussed in the publications of [Ménard et al. 1998a and 1998b]
The Kalman filter was research that was enabled by HPCC (the present algorithm was developed under the Round One of the HPCC ESS Project). However the operational system (GEOS DAS) has been in use for a number of years, and the main thrust of the present project is to increase throughput and thus enable faster scientific discovery and development of more sophisticated algorithms. The DAO has been engaged in a wide range of scientific research using GEOS DAS and the reader is encouraged to further view the DAO Home Page.
During the year the work on the Core GEOS DAS system went through a fundamental shift. It was realized that the software would not reach the 50 and 100 gigaflop/s milestones and that the best interests of the DAO and HPCC would be served if the project be more directed toward producing parallel software that is subject to the full range of Software Engineering standards for operational software, viz, scientifically valid, portable, maintainable, and capable of meeting throughput requirements. A discussion of the computational complexity of atmospheric data assimilation software can be found at [Lyster 1998b] including some comments on why it is difficult to build scalable GEOS DAS software. Appendix C shows the GEOS DAS System Performance and Throughput Requirements. The following shows the historical context of this work, then summarizes the work that is being performed on the GCM, PSAS, and System Integration.
3.1 Historical Context
The parallelization of the GEOS GCM started in early 1997 as a common effort with the DAO's Modeling Group to attain a production code which could be integrated with the Analysis (GEOS Interfaces and PSAS) to form a complete data assimilation system for the AM-1 launch foreseen for June 1998 (note: the launch date is now for the Spring of 1999). The MPI parallelization was initiated in July 1997 when a Fortran 90 version of the GEOS GCM v5.9 system was made available by the DAO modeling group.
In order to ensure support for the AM-1 mission, the DAO had to shift in the short term from the planned MPI F90 version to a multitasking F77 version in August 1997. This policy shift diverted modeling personnel to the realization of the multitasked F77 GEOS GCM, and the F90 GCM code (i.e., the mainline MPI development GCM) became isolated from the production baseline. In spite of this, Will Sawyer proceeded with the work on the MPI GCM from July -- November, 1997 with the goal of meeting the December 1997 HPCC milestones. The resulting MPI GEOS GCM implementation was completed in late 1997 and extensively benchmarked in December 1997 and January 1998, but its performance was insufficient to meet the December 1997 milestones.
Although this code is not used for DAO production at this time, it has been used to acquire performance statistics and to analyze the software for potential bottlenecks. In this respect it is a "prototype" which has been of value in obtaining MPI programming strategies and expertise, and in expediting the MPI GCM production system which the DAO will move to in 1999.
Following the renegotiation, from January to April 1998, a detailed plan for the migration of the production GEOS GCM (which went through revisions 6.0 - 6.5 in that period) to MPI was proposed, resulting in the document: "GEOS-2.x (multitasked) to GEOS-3.x (MPI) GCM Migration Plan." From February to June, 1998, utilities to support the MPI GCM were improved, the GCM's parallel I/O concept (developed by Rob Lucchesi) was integrated into the prototype, three papers containing prototype results were submitted to (and accepted by) conferences, and prototype versions of the GCM grid transformations were implemented, validated and benchmarked. Many of these results were presented in May 1998 at the DAO Scientific Advisory Panel meeting.
Some of the prerequisites of the migration plan were satisfied by the production GEOS GCM v6.6 (June 1998) system, and the plan was put into action in July, 1998, with the successful re-parallelization and optimization of the v6.6 dynamical core. Further prerequisites, e.g., the reinstitution of a Fortran 90 framework, were satisfied by the GCM v6.8 (September 1998). The implementation of still others, e.g., unit tests for individual components, are still in the planning stage, indicating that the completion date suggested in the plan might not be met.
The v6.6 dynamical core parallelization was completed in mid-August. The code was validated by stringent comparison against the sequential version, and it was benchmarked on the T3E. Dynamical core optimization work on the SGI Origin took place at NASA Ames in late August, 1998. The result of these efforts (see Web References below) was a scalability of the dynamical core to the full extent of the Cray T3E (512 processors) and the Origin (64 processors), with an absolute performance of 16 GFlop/s on 512 T3E (300 MHz.) processors.
Currently the re-parallelization of the GCM v6.8 grid transformations -- which now support stretched grids as opposed to only uniform grids previously -- is underway. With the continued support of the DAO's modeling group to fulfill the plan's prerequisites and deliver key software components, the planned GEOS3-GCM target completion date of December 1998 still seems possible.
3.3 Work on the Physical-space Statistical Analysis System (PSAS)
PSAS is a parallel code that updates a
gridded forecast field (the forecast is usually obtained from
a GCM) with data to obtain an optimal,
or ``analysis'' field. At the beginning of the CAN, the PI team had
a parallelized PSAS that was developed out of circa 1993 PSAS algorithm.
The parallelization was performed by Chris Ding and coworkers at JPL
(prototype parallel PSAS). Overall performance of this algorithm
for 80,000 observations is indicated in Table 1 (note that the
Intel Paragon results are 4-byte precision while all others are 8-byte
precision).
| Machine | PSAS Solver | Whole PSAS | Milestone |
| 512 pes | gigaflop/s | gigaflop/s | |
| Intel Paragon | 18.3 | - | HPCC-1 (pre CAN) |
| T3D | 12.1 | 10.4 | CAN 10 gigaflop/s |
| T3E-600 | 32.0 | 29.0 | |
| T3E-900 | 33.8 | 37.5 | ... |
The scaling plot for the results on the T3D and T3E-600 are
shown in Figure 2, indicating for the first time that scaling
is beginning to degrade at high performance due to communication
overhead, and possible load imbalance.
In summary, with some work on load-balancing,
scaling the number of observations, using shmem, optimizing
FORTRAN 77 code, and going to single precision we may be
able to reach 50 gigaflop/s on the T3E-1200. The current code
has 27% parallel overhead (15% corresponds to load-imbalance
and 10% is parallel communication cost) for 80,000 observations on
512 pes of the T3E-900 so that may cause problem in scaling to
100 gigaflop/s. The important thing to note is that prototype
parallel PSAS has indicated that a scalable scientific GEOS-3 PSAS
can be developed which approaches the GEOS DAS System
Performance and Throughput Requirements (Appendix C).
Under the terms of the February Renegotiation the DAO is currently working toward developing a scientifically valid, scalable PSAS. This involves the transfer of parallel technology from the prototype parallel PSAS to the scientifically valid GEOS-2 PSAS. The software is currently being extensively documented [Guo et al. 1998, Larson et al. 1998], and in October 1998 a workshop will be held at the Data Assimilation Office to teach the community about the development strategy and the PSAS algorithm itself.
3.4 Work on the Integrated Core GEOS DAS
Much of the current work is focused on the development of the parallel GCM and PSAS. The document "GEOS-3.X Core System (MPI) Integration Strategy" [Lyster, 1998] is on the DAO Intranet (available on request). This document indicates how the full MPI GEOS DAS will be assembled in 1999 from the GCM and PSAS modules. It also shows a team structure that includes help from HPCC in-house personnel and contractors.
3.5 Comment on Reproducibility
There have been suggestions that the DAO's has overly stringent
requirements for reproducibility. It is important to note that the DAO
recognizes that so-called 'non-zero difference' results (i.e.,
results that differ to machine precision) occur naturally as a consequence
of a number of activities, for example, single processor optimization,
changing compilers, or small changes to the algorithm that may change
the order (or 'commute') arithmetic. Currently the DAO specifies that
for the same executable with the same input data on the same number of
processors the same result (to the precision specified by the arithmetic)
must be obtained. Round-off differences due to the kinds of modifications
mentioned above are allowed but should all be investigated. When significant
modifications are made to the DAS algorithm scientific validation must
be carried out.
More specific details may be found in the
DAO Fortran 90 Software Standards
[DAO 1998] (available on request).
DAO members are active in the international community. This is evidenced by the number of presentations given by team members.
Members Jay Larson, Will Sawyer, Chris Ding, and Peter Lyster attended the Science Team Meeting in San Jose at Supercomputing97; there the team presented a contributed talk that was well attended.
Frequent HPCC lunches have been held at GSFC with Clark Mobarry and Spencer Swift sometimes in attendance.
Jay Larson and Will Sawyer contributed their reviews at an allocations meeting for the distribution of GSFC T3E resources to the wider community.
Peter Lyster regularly reviews proposals for NASA's Atmospheric Chemistry Modeling and Analysis Program, NASA Headquarters.
The Lyster PI team is currently focused on developing scientifically valid MPI GCM and PSAS. This process is expected to continue through the Spring of 1999 after which a full GEOS-3 DAS will be assembled. According to the document "GEOS-3.X Core System (MPI) Integration Strategy" the GEOS Interfaces (including on-line Quality Control) will not be parallelized for this first instantiation. Subsequent phases of development will allow for: concurrent scientific development of GEOS-3; parallelization of GEOS Interfaces; and optimization of the MPI software for scalability and performance. The exact nature of the new (renegotiated) HPCC payment milestones for this PI team will be determined after the PSAS Workshop to be held at Goddard Space Flight Center in late October 1998. The two-dimensional trajectory code of Kevin Olsen has been incorporated into a full Lagrangian filter, and will be subject to scientific tests using the same UARS data (methane and ozone retrieved products) that was used for the Kalman filter [Ménard et al. 1998a, and 1998b]. New milestones for this algorithm will be ascertained after the October PSAS workshop.
Peter Lyster
Email to lys@dao.gsfc.nasa.gov
Mail: MCC1 Suite No. 200, 7501 Forbes Blvd, Seabrook, MD 20706
Figure 3 This is from a study using the Kalman filter of a tropical atmospheric wave-breaking event in the stratosphere. The visualization uses isosurfaces of constant mixing ratio of methane to depict the three-dimensional structure and evolution of the stratosphere. The Vis5D tool was used to produce the images from datasets that were produced by the Kalman filter. Data is from Upper Atmosphere Research Satellite (UARS) Cryogenic Limb Array Etalon Spectrometer (CLAES) measurements of September 6-14, 1992. The study also uses wind data from NASA/Goddard Space Flight Center's Data Assimilation Office for driving fields. The experiments have 18 vertical levels in the atmosphere. Horizontally, the resolution is 5 degrees longitude by 4 degrees latitude. There are three different zones: polar vortex (value of methane mixing ratio is low), mid-latitudes (surf zone), and tropics (mixing ratio is high). In the visualization, an isosurface is used to render surfaces of constant value of mixing ratio. The time taken for the 8 day runs was 20 hours of wall-clock time on 128 processors of the GSFC Cray T3D. This figure is an image from the award-winning video: Images of Earth and Space: SC97 Edition, NASA Scientific Visualization Studio, Goddard Space Flight Center.
Peer Reviewed Papers
C. H. Q. Ding, P. M. Lyster, J. W. Larson, J. Guo, A. da Silva: Atmospheric Data Assimilation on Distributed-Memory Parallel Computers. International Conference and Exhibition on High-Performance Computing and Networking (HPCN Europe '98), Springer-Verlag, Lecture Notes in Computer Science, (1998).
P. M. Lyster, J. W. Larson, J. Guo, W. Sawyer, A. da Silva, and I. Stajner: Progress in the Parallel Implementation of the Physical-space Statistical Analysis System (PSAS). Making its Mark: Proc. Seventh ECMWF Workshop on the Use of Parallel Processors in Meteorology, Eds. G-R. Hoffmann and N. Kreitz, 382-393 (World Scientific, 504pp, 1998). See also: http://www.wspc.com/books/compsci/3679.html
P. M. Lyster, S. E. Cohn, R. Ménard, L.-P. Chang, S.-J. Lin, and R. Olsen: Parallel Implementation of a Kalman Filter for Constituent Data Assimilation. Mon. Wea. Rev., 125, 1674-1686 (1997).
P. M. Lyster, J. W. Larson, W. Sawyer, C. H. Q. Ding, J. Guo, A. M. da Silva, L. L. Takacs: Parallel Computing at NASA Data Assimilation Office (DAO). Proc. Supercomputing97, San Jose (November 1997). Available at http://www.supercomp.org/sc97/proceedings/TECH/LYSTER/INDEX.HTM
W. Sawyer, L.L. Takacs, A. da Silva, and P.M. Lyster: Parallel Grid Manipulations in Earth Science Calculations. Proceedings of the 3rd International Meeting on Vector and Parallel Processing (VECPAR'98). Springer-Verlag, Lecture Notes in Computer Science (1998).
J. Syktus, J. Chappell, R. Oglesby, J. Larson, S. Marshall, B. Saltzman: Latitudinal dependence of signal-to-noise patterns from two general circulation models with CO2 forcing, Climate Dynamics, 13(5):293-302 (1997).
Manuscripts in Preparation
P. M. Lyster: The Computational Complexity of Atmospheric Data Assimilation. To be submitted to Int. J. Appl. Sci. Comp.
R. Ménard, L.-P. Chang, P. M. Lyster, and S. E. Cohn: Stratospheric Assimilation of Chemical Tracer Observations Using a Kalman filter. Part I: Formulation. To be submitted to Quart. J. Roy. Meteor. Soc.
R. Ménard, L.-P. Chang, P. M. Lyster, and S. E. Cohn: Stratospheric Assimilation of Chemical Tracer Observations Using a Kalman filter. Part II: Results. To be submitted to Quart. J. Roy. Meteor. Soc.
P. M. Lyster, S. E. Cohn, R. Ménard, and L.-P. Chang: A Domain Decomposition for Error Covariance Matrices Based on a Latitude-Longitude Grid. to be submitted to Computer Physics Communications.
Non-Peer Reviewed Papers
J. Guo, J. W. Larson, P. M. Lyster, and G. Gaspari: Documentation of the Physical-space Statistical Analysis System (PSAS) Part II: The Factored-Operator Error Covariance Model Formulation, DAO Office Note 98, NASA Data Assimilation Office, Greenbelt, MD (1998).
J. W. Larson, J. Guo, P. M. Lyster: Documentation of the Physical-space Statistical Analysis System (PSAS) Part III: Software Implementation, DAO Office Note 98, NASA Data Assimilation Office, Greenbelt, MD (1998).
P. M. Lyster, W. Sawyer, and L. L Takacs: Design of the Goddard Earth Observing System (GEOS) Parallel General Circulation Model (GCM). DAO Office Note No. 97-13, NASA Goddard Space Flight Center, Greenbelt, Maryland (1997).
P. M. Lyster, J. W. Larson, C. H. Q. Ding, J. Guo, W. Sawyer, A. M. da Silva, and I. Stajner: Design of the Goddard Earth Observing System (GEOS) Parallel Physical-space Statistical Analysis System (PSAS). DAO Office Note No. 97-05, NASA Goddard Space Flight Center, Greenbelt, Maryland (1997).
W. Sawyer, R. Lucchesi, P.M. Lyster, L.L Takacs, A. Molod, J. Larson, S. Nebuda, C. Pabon-Ortiz: Parallelization of DAO Atmospheric General Circulation Model. Proceedings of the Fourth International Workshop on Applied Parallel Computing (PARA98). Springer-Verlag, Lecture Notes in Computer Science, 1998.
W. Sawyer, R. Lucchesi, P.M. Lyster, L.L Takacs, A. Molod, J. Larson, S. Nebuda, C. Pabon-Ortiz: Parallelization Aspects of an Atmospheric General Circulation Model for Data Assimilation. Proceedings of 1998 Advanced Simulation Technologies Conference, High Performance Computing Symposium. Adrian Tentner, Ed. Society for Computer Simulation International, 1998. ISBN: 1-56555-145-1.
J. W. Larson, P. M. Lyster, W. Sawyer, C. H. Q. Ding, J. Guo, A. M. da Silva, L. L. Takacs: Progress in the Design and Optimization of the Parallel Goddard Data Assimilation System (DAS). High Performance Computing 1997 Grand Challenges in Computer Simulation, Ed. A. Tentner, p. 52., Society for Computer Simulation International (1997).
W. Sawyer, R. Lucchesi, P. M. Lyster, L. L. Takacs, A. Molod, J. Larson, S. Nebuda, C. Pabon-Ortiz: Parallelization Aspects of an Atmospheric General Circulation Model for Data Assimilation. Proc. 1998 Advanced Simulation Technologies Conference, High Performance Computing Symposium, Ed. A. Tentner, Society for Computer Simulation International, ISBN 1 56555 145 1 (1998).
W. Sawyer, R. Lucchesi, P. M. Lyster, L. L. Takacs, A. Molod, J. Larson, S. Nebuda, C. Pabon-Ortiz: Parallelization of DAO Atmospheric General Circulation Model. Proc. Fourth International Workshop on Applied Parallel Computing (PARA98), Springer-Verlag, Lecture Notes in Computer Science, (1998).
Invited Talks
J. Larson: Parallelization of Elements of the Goddard Earth Observing System Data Assimilation System (GEOS DAS), NASA Computational Aerosciences (CAS) Workshop '98, NASA Ames Research Center, (August 25-27, 1998).
P. M. Lyster: Four Dimensional Data Assimilation. 16th International Conference on the Numerical Simulation of Plasmas, Santa Barbara, CA (February 10-12, 1998).
P. M. Lyster: Techniques for Large-Scale Computing in Atmospheric Data Assimilation. IMACS'98: International Conference on Scientific Computing and Mathematical Modeling Alicante, Spain (June 25-27, 1998).
P. M. Lyster: Atmospheric Data Assimilation at the NASA Data Assimilation Office (DAO). CAS98: Computing in Atmospheric Sciences Workshop 1998, L'Imperial Palace, Annecy, France, (June 30, July 1-2, 1998).
P. M. Lyster: The Kalman Filter and High Performance Computing at NASA's Data Assimilation Office (DAO). NASA Computational Aerosciences (CAS) Workshop '98, NASA Ames Research Center, (August 25-27, 1998).
W. Sawyer: Parallel Grid Manipulations in Earth Science Calculations. High Performance Computing and Communications Program Computational Aerosciences Workshop (HPCCP/CAS98). NASA Ames Research Center (August 25-27, 1998).
Conference Presentations
D. Dee, L. Rukhovets, A. da Silva, J. Larson: An adaptive buddy check for on-line quality control of observations, EGS General Assembly Meeting, Nice, France, April, 1998.
J. Larson: Incorporating Parallel Computing into the Goddard Earth Observing System Data Assimilation System (GEOS DAS), Second International Workshop on Software Engineering and Code Design in Parallel Meteorological and Oceanographic Applications, Scottsdale, Arizona, June, 1998 .
W. Sawyer: Parallelization Aspects of an Atmospheric General Circulation Model for Data Assimilation. 1998 Advanced Simulation Technologies Conference, High Performance Computing Symposium, Boston, MA, USA, April 7, 1998.
W. Sawyer: Parallelization of the GEOS GCM Code. Workshop on Variable Resolution Regional Climate Modeling and Data Assimilation. University of Quebec at Montreal. Montreal, Quebec, Canada, May 14, 1998.
W. Sawyer: Parallelization of the DAO Atmospheric General Circulation Model. Fourth International Workshop on Applied Parallel Computing (PARA98). High Performance Computing Center North (HPC2N). Umea, Sweden, June 17, 1998.
W. Sawyer: Parallel Grid Manipulations in Earth Science Calculations. Third International Meeting on Vector and Parallel Processing (VECPAR'98). Faculdade de Engenharia da Universidade do Porto, Portugal, June 22, 1998.
The PI Project contributed a video showing the dynamical processes associated with the assimilation of Methane that has been retrieved from the UARS satellite. This was part of the video: Images of Earth and Space: SC97 Edition, NASA Scientific Visualization Studio, Goddard Space Flight Center. This was shown at the High Performance Computing and Communications booth at Supercomputing97, San Jose, CA (November, 1997). This video won an award of ``Excellence'' in the Society for Technical Communications 1997-98 International Technical Video Competition. It will be displayed at the Society's annual conference in May 1998.
A recent article on the Project appeared in the Insights magazine and can be viewed at http://www.hq.nasa.gov/hpcc/insights/vol6/index.htm
Peter Lyster was the NASA advisor for Sonetra Howard, who graduated from North Carolina State University in September 1997. Her thesis was on the hardware implementation of a Kalman filter using custom microcircuits.
Peter Lyster supervised Heather Kilcoyne in the Meteorology Department at the University of Maryland. She completed her M.Sc. in November 1997 with a scholarly paper titled ``Past and Current Methods of Data Assimilation at Leading Weather and Analysis Centers'' which can be viewed at http://www.datafront.com/paper
Peter Lyster is now supervising Tijana Janjic who is a graduate student at the University of Maryland.
PI Team members have been instrumental in bringing Software Technology and Practises to the DAO.
The following is taken from the publication by Lyster et al. (1997e) which appeared in Supercomputing97.
GEOS DAS uses a grid-point based atmospheric general circulation model GCM [Takacs et al., 1994]. Other physical processes are computed using parameterized models for turbulence, short- and long-wave radiation, moisture, and land surface processes. Analysis algorithms are used to combine the inherently unstructured data, with (hopefully) known errors, into the structured models whose errors are also quantified. This can be regarded as a problem in the field of stochastic modeling and estimation. One of the difficulties of atmospheric data assimilation arises from the large size of the state space. Current GCMs have more than 107 gridpoints and there are approximately 105 meteorological observations (including satellite-retrieved temperature profiles) reported world wide in a typical six-hourly (or synoptic) assimilation period. The current analysis at the DAO is an on-line Quality Control (QC) algorithm and the Physical-space Statistical Analysis System (PSAS) [Cohn et al.,1997]. A key part of PSAS performs an iterative conjugate-gradient solve of a moderately dense (26% full) 105 x 105 matrix. The heritage code for the Core system is mostly FORTRAN 77.
Note: DAO Intranet documents are available by request
P. Lyster: Request for Renegotiation, Grand Challenge Applications and Enabling Scalable Computing Testbed in Support of High Performance Computing, PI Project: Atmospheric Four Dimensional Data Assimilation: Applications in High Performance Computing (February 17, 1998; revised Feb 26, 1998).
P.M. Lyster, J.W. Larson, W. Sawyer, C.H.Q. Ding, L.-P. Chang, R. Menard, R. Rood, S.E. Cohn: Grand Challenge Applications and Enabling Scalable Computing Testbed in Support of High Performance Computing: Four Dimensional Data Assimilation Financial Year 1996 Annual Report (1996).
P.M. Lyster, J.W. Larson, W. Sawyer, C.H.Q. Ding, L.-P. Chang, R. Menard, R. Rood, S.E. Cohn: Grand Challenge Applications and Enabling Scalable Computing Testbed in Support of High Performance Computing: Four Dimensional Data Assimilation Financial Year 1997 Annual Report (1997).
P.M. Lyster. Four Dimensional Data Assimilation: HPCC 10 Gigaflop/s Milestone Submission (1997).
P.M. Lyster. GEOS-3.X Core System (MPI) Integration Strategy. DAO Intranet, available on request (1998).
Go to the GEOS-3 Software Plan
Download a paper [Lyster et al., 1997a] that describes the design of the
Parallel
GEOS General Circulation Model (GCM)
Download a paper [Lyster et al., 1997b] that describes the design of the
Parallel
GEOS Physical-space Statistical Analysis System (PSAS)
View a paper [Lyster et al., 1998b] that describes the Computational Complexity of Atmospheric Data Assimilation.
The DAO Benchmarking page is available only on the DAO Intranet, but may be made available to individuals on request.
W. Sawyer, and A. da Silva. ProTeX: A Sample Fortran 90 Source Code Documentation System. DAO Office Note No. 97-11, NASA Goddard Space Flight Center, Greenbelt, Maryland.
DAO Staff. DAO Fortran 90 Software Standards (1997). /intranet/pages/GEOS3/Software/Tools/standards/software_standards. Available on request.
W. Sawyer, L.L. Takacs, A. da Silva, and P.M. Lyster. The Design of PILGRIM: A Parallel Integrated Library for Grid Manipulation used for Earth Science Calculations (1997). /intranet/pages/GEOS3/Software/Core/Hermes/PILGRIM/pilgrim.
W. Sawyer, L.L. Takacs, R. Lucchesi and R. Todling. GEOS-2.x (multitasked) to GEOS-3.x (MPI) GCM Migration Plan: Requirements and Architectural Design. /intranet/pages/GEOS3/Software/Core/GCM/Design/migration.
Summaries of work by W. Sawyer, S. Cheung, and B. Nelson on optimizing the GCM parallel dynamical core for the SGI Origin machines can be obtained by requesting access to DAO Intranet documents. Also view Samson Cheung's page at Ames.
Download a paper [Lyster et al., 1997d] on the
Parallel Kalman Filter,
---------------------------------------------------------------------
Group 1: 2 x 2.5 x 70 GCM resolution
400,000 obs/day
5 days/wallclock day nominal performance
This corresponds to the current operational (multitasking) GEOS DAS
Main Memory (GB) 2.2 (per image)
"Disk" (GB) 8.0
Mass Storage (GB) 2270.2 (this is o/p per year)
Volume of Data (GB) 6.2 (produced per day per image)
Gflop/s sustained 0.3 (per image)
Duration of Run 5 days/WCday (continuous operation, single image)
---------------------------------------------------------------------
Group 2: 2 x 2.5 x 70 GCM resolution
800,000 obs/day
30 days/wallclock day nominal performance
This corresponds to the end of year 1999
Main Memory (GB) 2.3 (per image)
"Disk" (GB) 47.2
Mass Storage (GB) 13264.8 (this is o/p per year)
Volume of Data (GB) 36.3 (produced per day per image)
Gflop/s sustained 5.6 (per image)
Duration of Run 30 days/WCday (continuous operation, single image)
---------------------------------------------------------------------
Group 3: 1 x 1 x 70 GCM resolution
800,000 obs/day
30 days/wallclock day nominal performance
This corresponds to year 2000
Main Memory (GB) 6.0 (per image)
"Disk" (GB) 225.0
Mass Storage (GB) 74217.0 (this is o/p per year)
Volume of Data (GB) 203.0 (produced per day per image)
Gflop/s sustained 44.0 (per image)
Duration of Run 30 days/WCday (continuous operation, single image)
---------------------------------------------------------------------------