PADRE

Dan Quinlan
Lawrence Livermore National Laboratory
Livermore, CA 94550
PADRE Web Page: http://www.c3.lanl.gov/dquinlan/PADRE.html

February 19, 1999

Objective

PADRE is an acronym for Parallel Asynchronous Distribution and Routing Environment. The purpose of the PADRE library is to support the requirements of interoperability within the POOMA, Overture and other parallel libraries (A++/P++ are serial/parallel array classes within Overture). To support interoperability of independently developed parallel libraries PADRE presents a set of common data distribution mechanisms which define reasonably arbitrary distributions of data and most importantly the communication support to work with these distributions. By providing this level of abstraction PADRE provides an important part of the functionality, portability, and interoperability required in many modern parallel libraries.

The increasing complexity of parallel scientific (numerical) software, just as in all other areas of software development, demands increasing modularity and reusability of software components. For parallel scientific applications, the management of data distribution and interprocess- and interprocessor communication is an issue of some complexity that is not typically a consideration in more mainstream areas of computing.

A natural evolution of solutions to the complexity of data distribution may be observed. Early parallel machines provided `native' communication libraries, e.g. CMMD on the Thinking Machines CM-5. Next came attempts at standardization, e.g. PVM, MPI, and the currently emerging MPI-2, often built on top of native libraries. While providing facilities for communication, libraries such as these do not provide support for the management or organization of distributed data.

The lack of any general-purpose distributed-data management facilities motivated the development of libraries such as Multiblock PARTI, PGSLib, and KeLP. At this level of abstraction (unlike at the PVM/MPI level) it appears unreasonable to hope for a single satisfactory solution because of widely differing distribution and communication requirements, sometimes even within a single application or parallel library.

The sophistication of distributed-data management libraries (or distribution libraries) is such that it behooves the parallel application or library designer to use existing libraries rather than build them from scratch. Unfortunately, their interfaces are of sufficient complexity that the use of more than one such library, or offering the user the choice of distribution library to be used, will almost certainly greatly complicate the parallel library code, particularly in the former case wherein the same data may be alternately or even simultaneously be `managed' by more than one distribution library.

The objective of PADRE is to provide access to a wide number of different distributions while supporting their interoperability. In so doing, PADRE provdes interoperability between different libraries using any of the distribution mechanisms that PADRE uses internally.

Approach

PADRE internally leverages a number of publicly available distribution libraries. These libraries are used as sub-libraries which PADRE. PADRE then adds both higher level interfaces and interoperability between the different sub-libraries. The power and usefulness of PADRE derives as much from the abstract framework that it provides as the manifest functionality of the code itself. These abstractions represent the distillation of considerable previous work spanning numerous iterations and refinement of embedded codes (that is, not distinct libraries such as PADRE) to solve much the same problems.

Recent FY98 accomplishments

  1. Initial design, implementation, and initial software release January 1998. The initial design of PADRE took place with Steve Karmesin this was done to better permit it's use within POOMA and to provide interoperability between POOMA, Overture and other frameworks. We were unable to obtain the same distribution mechanism as is used within POOMA and so have used KELP from UCSD which can support largely the same distributions, thus we can be interoperable with POOMA applications, under a few minimal constraints. The interoperability with POOMA is the subject of continuing work.
  2. Incorporation into P++ and OVERTURE Framework (with test codes, etc.) PADRE replaced the distribution mechanisms more tightly embedded in P++ greatly improving the design of P++ by isolating the significant data distribution abstraction layer in a separate library. This provided P++ with a better distribution mechanism and more powerful communication mechanisms, plus the support for more complex optimizations (consistent with recent papers published by Bassetti, Davis, and Quinlan; on serial and parallel performance optimizations for object-oriented frameworks).
  3. Parallel Indirect Addressing: Initial parallel indirect addressing support developed PADRE work on indirect addressing support provided a most of the indirect addressing support for use in complex geometry applications using overlapping grids (but not all of what was required).
  4. KeLP into PADRE: The introduction of KeLP (from Scott Baden's work at UCSD) provides a greatly simplified development of the parallel indirect addressing and has been an important part of providing the more complete implementation of the PADRE indirect addressing support. The use of KeLP also provides the principle table-based distribution mechanisms and compliments the algorithm based distribution mechanisms that were obtained through the use of Multi-Block PARTI (from University of Maryland).
  5. Communication optimization support: Communication support within PADRE permits the use of an independent model (different from that found within the distribution libraries within PADRE). This alternative model provides much higher level multidimensional abstractions, the point is to provide more sophisticated handles (similar to MPI) for all the messages required for an update of a distributed object's ghost boundaries. This abstraction permits the hidden use of higher performance mechanisms and optimizations for message latency hiding (consistent with recent papers published by Bassetti, Davis, and Quinlan; on serial and parallel performance optimizations for object-oriented frameworks).

Plans for FY99

Future work will evolve to the use of other aspects of the DOE 2000 work and there use within Overture. Although we are committed to the distribution of PADRE as a simplifying mechanism to provide portability in parallel libraries in general, we would like to additionally work on the use of PETSC within Overture as a means to solve the elliptic equations represented by the CFD (pressure for incompressible N-S flows) and electric field computations within our CFD and PIC work respectively. The CFD combustion work is being done in conjunction with Caterpillar Inc., and the PIC and EMF computations are being done within cosmology simulations at LANL along with the Grand Challenge in Accelerator Physics at LANL (in particular as part of work to handle the complex accelerator geometries and provide adaptive mesh refinement support).

  1. Work with Jarek Nieplocha at PNL: We are working with Jarek to put the Global Array (GA) parallel library distributions and communication mechanisms (one-sided communications) into PADRE. This serves two important purposes: a) it provides P++ as a library for C++ users of GA who have bothered Jarek about this for sometime (greatly simplifying the means by which he can satisfy his customer base using C++ with GA. b) It provides the one-sided message passing mechanisms to P++ and the Overture framework so that we have one-sided messaging available to us. This one-sided messaging forms an important part of the infrastructure required for the parallel grid generation work within Overture and should greatly simplify that part of the parallel design. We will also evaluate its possible use as an alternative infrastructure for the indirect addressing support within P++ and PADRE as well. c) the use of GA distributions within PADRE and Overture will permit interoperability between numerous GA applications and the Overture framework (allowing them to address more complex geometry within their applications).
  2. Specific to the Communication mechanisms within PADRE, we have not yet added the SMP support, this is part of the near-term work and compliments the other optimization research for message latency hiding support in PADRE. The work here is to make communication much faster and more powerful in PADRE than that which most programmers can afford the invest time in supporting. Much of this has been done, the SMP support is what is left to be added.
  3. The use of PETSC within Overture and SAMRAI for the elliptic solves provides: a) robust mechanisms to complement the possiblely more efficient but less stable Multigrid based methods currently under development. b) A baseline for the performance of the of the newer research on elliptic methods for overlapping grids, adaptive mesh grids, unstructured grids, block structured grids, cartisan based adaptive refinement grids and embedded boundary grid methods. c) adaptive mesh refinement for elliptic solvers is proposed using more traditional CG methods, we will use PETSC to define these and use these as a baseline for the evaluation of different adaptive mesh refinement elliptic solution methods given different distributions of the initial data for steady-state and time-dependent applications.

Tool Availability

Software, including documentation, is available at http://www.c3.lanl/PADRE. Users require a C++ compiler and MPI, nothing more is required.

Who are the participants and affiliations

About this document ...

PADRE Home Page

This document was generated using the LaTeX2HTML translator Version 96.1-g (June 11, 1996) Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.

The command line arguments were:
latex2html -split 0 ProjectPage.tex.

The translation was initiated by Dan Quinlan on Fri Feb 19 16:21:16 PST 1999

Dan Quinlan
Fri Feb 19 16:21:16 PST 1999