Inter-operability of Global Arrays with PETSc

PETSc (the Portable, Extensible Toolkit for Scientific Computation) is developed by the Argonne National Laboratory. It is part of the ACTS Toolkit in the DOE 2000 Initiative. PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations. It employs the MPI standard for all message-passing communication, and is written in a data-structure-neutral manner to enable easy reuse and flexibility.

The following summarizes the inter-operability status of Global Arrays and PETSc:

Inter-operability

Global Arrays is inter-operable with PETSc. In an application using Global Arrays, the PETSc solvers can be called to solve PDEs that require solving large-scale, sparse nonlinear systems of equations. The only issue related to inter-operablity is how to convert the data structures of Global Arrays to those of PETSc before calling the PETSc solvers, and how to convert the data structures of PETSc back to Global Arrays after calling the PETSc solvers. Fortunately, PETSc provides enough mechanisms for this purpose. For vector operations, there are VecCreateMPI(), VecSetValues(), VecGetArray(), VecRestoreArray(), etc. The same functions exist for matrix operation.

The packages used in the testing are:

Global Arrays Version 3.0
PETSc Version 2.0.24

Instructions for using PETSc in a Global Arrays application

PETSc online documentation is a well-maintained site for PETSc resources. Examples can be accessed both online or from the package itself.

A typical scenario to use PETSc in a Global Arrays application is that there is a global array x that represents the approximate solution initialized with some initial values. It needs to call one of the PETSc solvers to solve the problem, and restore the results back to x.

Here are the instructions for implementing an example Ax = b, where A is the matrix defining the linear system, b is the right-hand side, and x is the approximate solution and an global array.

Initialize PETSc (PetscInitialize())
Convert the global array x to the PETSc format
- Create a PETSc Vector pets_x (VecCreateMPI())
- Get the range of pets_x which resides in the local process(or) (VecGetOwnershipRange())
- Get access to the local portion of pets_x (VecGetArray())
- Get the corresponding data block (the range of pets_x in local process(or)) in the global array x (ga_get())
- Put the data block to pets_x (VecRestoreArray())
- Create the linear solver and set various options
- Solve the linear system
- Write the solution back to Global Array
  - Get access to the local portion of pets_x (VecGetArray())
  - Put the local portion of solution back to global array x (ga_put())
  - Close the access to the local portion of pets_x (VecRestoreArray())

There are detailed instructions for setting up environment variables on different platforms with the PETSc package. Users on Cray T3E at NERSC only need to the load the petsc module: insert

module load petsc

into the .login file.

Discussion

Data conversion between the Global Arrays and PETSc is the key issue for inter-operability. PETSc provides several ways to create Vectors and Matrices and to set values to them. We found that the most efficient way to connect the Global Arrays and PETSc is to use the GetArray and RestoreArray mechanism. GetArray and RestoreArray are not intended to set values though, they open a window to access and update the local Vector/Matrix of PETSc. Global Arrays provide the one-sided operations, get and put, which are a perfect match for PETSc's GetArray and RestoreArray mechanism. The array segment of Global Arrays can be sent to or received from PETSc in block fashion, instead of updating element by element.

Here is how it works:

From Global Arrays to PETSc
1. Access the local portion of PETSc Vector/Matrix.
2. Use ga_get() to get the corresponding section of Global Array.
3. Close the access to (also update) the local portion of PETSc Vector/Matrix.
From PETSc to Global Arrays
1. Access the local portion of PETSc Vector/Matrix.
2. Use ga_put() to put the PETSc data in the corresponding section of Global Array.
3. Close the access to the local portion of PETSc Vector/Matrix.

Inter-operability of Global Arrays with PETSc