Disk Resident Arrays Operations

Disk Resident Arrays

Disk Resident Arrays (DRA) extend the Global Arrays (GA) programming model to disk. The library encapsulates the details of data layout, addressing and I/O transfer in disk arrays objects. Disk resident arrays resemble global arrays except that they reside on the disk instead of the main memory. The main features of this model are:

Data can be transfered (copied) between disk and global memory.
I/O operations have nonblocking interface to allow overlapping of I/O with computations.
All I/O operations are collective.
Either whole or sections of global arrays can be transferred between GA memory and the disk.
Reshaping and transpose operation are allowed during the transfer.
Disk resident arrays might be temporary or persistent (saved after program termination and available in next runs).
Persistent disk arrays can be accessed by any program executing on arbitrary number of processors.
Distribution and internal data layout on the disk is optimized for large data transfer.
Hints provided by the user are utilized to guide optimization of the library performance for the specific I/O patterns.

DRA can take advantage of a shared file system of a collection of independent filesystems accessible from individual computing nodes.

Note that DRAs make use of an internally defined data type dra_size_t to specify the dimensions and indices of DRAs in C. This data type is designed to prevent integer overflows from potentially large 1-dimensional DRAs.

dra_init

   int DRA_Init(int max_arrays, double max_array_size,
             double total_disk_space, double max_memory)
         max_arrays                             [input]
         max_array_size                         [input]
         total_disk_space                       [input]
         max_memory                             [input]

Initializes Disk Resident Array I/O subsystem.

max_array_size , total_disk_space and max_memory are given in bytes.

max_memory specifies how much local memory per processor the application is willing to provide to the DRA I/O subsystem for buffering.

The value of -1 for any of input arguments means: "don't care ", "don't know" , or "use defaults ".

dra_terminate

   int DRA_Terminate()

Close all open disk resident arrays and shut down DRA I/O subsystem.

ndra_create

     int NDRA_Create(int type, int ndim, dra_size_t dims[], char *name,
                     char *filename, int mode, dra_size_t reqdims[], int *d_a)
             type                       [input]  ! MA type identifier
             ndim                       [input]  ! Dimension of DRA
             dims[ndim]                 [input]  ! Dimensions of DRA
             name                       [input]  ! Name of DRA
             filename                   [input]  ! Name of file containing DRA
             mode                       [input]  ! READ; WRITE; READ/WRITE
             reqdims[ndim]              [input]  ! Typical request size
             d_a                        [output] ! DRA handle

Creates an N-dimensional DRA with specified dimensions and data type (represented by MA type handel). The dimension of the DRA is specified by the variable ndim and the physical dimensions of the DRA are specified in the array dims. The variable name is an internal name that can be used to identify the DRA and the variable filename represents the name of an abstract meta-file that will be used to store the data on disk. The variable mode can be used to restrict the behavior of the DRA and can be set using the predefined values DRA_R (read), DRA_W (write), and DRA_RW (read/write). The array reqdims contains the dimensions of request to the DRA; if any of the entries are set to -1, the ndra_create routine will attempt to use default values to come up with a guess for the size of this value. The variable d_a is an integer handle that is assigned to the DRA when it is created and can be used to access the DRA later in the program.

he data in disk resident array is implicitly initialized to 0 .

Access permissions (read, write, read& write) are set in mode . These are set using defined in the header files dra.fh (Fortran) and dra.h (C) preprocessor constants: DRA_R, DRA_W, DRA_RW.

The pair [rdim1, rdim2] specifies dimensions of a typical request. The value of -1 for either of them means "unspecified". The layout of the data on the disk(s) is determined based on the values of these arguments. Performance of the DRA operations depends on the dimensions (section shape) of the requests. If data layout is optimized for column-like sections, performance of DRA operations for row-like sections might be seriously degraded. This is analogous to the effect of wrong loop ordering yielding frequent cache misses in the following example .

              double precision a(1000, 1000)
              do i = 1, 1000
                 do j = 1, 1000
                    a(i,j) = dfloat(i+j)
                 enddo
              enddo

instead of

              do j = 1, 1000
                 do i = 1, 1000
                    a(i,j) = dfloat(i+j) 
                 enddo
              enddo

dra_set_default_config

     int DRA_Set_default_config(int numfiles, numioprocs)
              numfiles                  [input]
              numioprocs                [input]

This subroutine allows users to control the number of files that a DRA is broken up into and to control the number of processors doing IO, provided the DRA is being created on an open filesystem. If the DRA is
being created on local disk, then this subroutine has no effect. The original settings for these two variables is that both the number of files and the number of IO procs equals the number of SMP nodes being used by the calculation. Other settings can be chosen, however, to create DRAs composed of larger or smaller numbers of files and IO processors. These settings may provide better IO bandwidth on some platforms. The dra_set_default_config subroutine can be called multiple times throughout the program. Each DRA is created with whatever default configuration is applicable at the time of creation.

dra_open

     int DRA_Open(char *filename, int mode, int *d_a)
              filename                  [input]
              mode                      [input]  ! READ; WRITE; READ/WRITE
              integer d_a               [output]  ! DRA handle

Open and assign DRA handle to disk resident array stored in DRA meta-file filename.Disk resident arrays that are created with DRA_Create or NDRA_Create and saved by calling DRA_Close can be later opened and accessed by the same or different application.

Attributes of the disk resident array can be found by calling DRA_Inquire .

dra_write

     int DRA_Write(int g_a, int d_a, int *request)
              g_a                       [input]  ! GA handle
              d_a                       [input]  ! DRA handle
              request                   [output] ! request id

Write asynchronously specified global array to specified disk resident array.

The dimensions and type of arrays represented by handles g_a and d_a must match. If dimensions don't match, DRA_Write_section should be used instead.

The operation is by definition asynchronous but it might be implemented as synchronous i.e., it would return only when the I/O is completed.

request can be used by DRA_Probe or DRA_Wait for completion of the associated operation.

ndra_write

     int NDRA_Write(int g_a, int d_a, int *request)
             g_a                        [input] ! GA handle
             d_a                        [input] ! DRA handle
             request                    [output] ! request id

N-dimensional asynchronous write from specified global array to specified disk resident array.

The dimension, physical dimensions, and type of arrays represented by handles g_a and d_a must match. If the physical dimensions don't match, NDRA_Write_section should be used instead.

The operation is by definition asynchronous but it might be implemented as synchronous i.e., it would return only when the I/O is completed.

request can be used by DRA_Probe or DRA_Wait for completion of the associated operation.

ndra_write_section

     int NDRA_Write_section(int transp, int g_a, int glo[], int ghi[],
                            int d_b, dra_size_t dlo[], dra_size_t dhi[],
                            int *req)
              transp                    [input]  ! transpose operator flag
              g_a                       [input]  ! GA handle
              glo[ndim]                 [input]  ! array of lower indices on GA
              ghi[ndim]                 [input]  ! array of upper indices on GA
              d_b                       [input]  ! DRA handle
              dlo[ndim]                 [input]  ! array of lower indices on DRA
              dhi[ndim]                 [input]  ! array of upper indices on DRA
              req                       [output] ! request id

Asynchronously write specified global array section to specified disk resident array section.

              OP(g_a[glo:ghi]) --> d_a[dlo:dhi]

The transpose operator OP reverses the ordering of the the dimensions. The function returns an error if the two sections sizes are mismatched. See NDRA_Write specs for a discussion of request.

ndra_read

     int NDRA_Read(int g_a, int d_a, int *request)
              g_a                       [input]  ! GA handle
              d_a                       [input]  ! DRA handle
              request                   [output] ! request id

N-dimensional asynchronous read to the specified global array from the specified disk resident array.

The dimension, physical dimensions, and type of arrays referred to by handles g_a and d_a must match. If dimensions don't match, NDRA_Write_section could be used instead.

See NDRA_Write specs for discussion of request .

ndra_read_section

     int NDRA_Read_section(int transp, int g_a, int glo[], int ghi[],
                           int d_a, dra_size_t dlo[], dra_size_t dhi[],
                           int *request)
              logical transp                    [input]  ! transpose operator flag
              integer g_a                       [input]  ! GA handle
              integer glo(ndim)                 [input]  ! array of lower indices on GA
              integer ghi(ndim)                 [input]  ! array of upper indices on GA
              integer d_b                       [input]  ! DRA handle
              integer dlo(ndim)                 [input]  ! array of lower indices on DRA
              integer dhi(ndim)                 [input]  ! array of upper indices on DRA
              integer request                   [output] ! request id

N-dimensional asynchronous read to specified global array section from specified disk resident array section:

                OP(d_a[dlo:dhi]) -->  g_a[glo:ghi]

The transpose operator reverses the ordering of the dimensions. The function returns an error if the two sections sizes are mismatched. See NDRA_Write specs for discussion of request .

dra_probe

     int DRA_Probe(int request, int *compl_status)
              request                   [input]  ! request id
              compl_status              [output] ! completion status

Tests for completion of NDRA_Write / Read or NDRA_Write / Read_section operation which set the value passed in request argument.

compl_status equals 0 means the operation has been completed.

compl_status not equal 0 means "not done yet ".

dra_wait

     int DRA_Wait(int request)
              request                   [input]  ! request id

Blocks until completion of NDRA_Write / Read or NDRA_Write / Read_section operation which set the value passed in request argument.

ndra_inquire

     int NDRA_Inquire(int d_a, int *type, int *ndim, dra_size_t dims[],
                      char *name, char *filename)
              d_a                       [input]  ! DRA handle
              type                      [output] ! DRA data type
              ndim                      [output] ! Dimension of DRA
              dims[ndim]                [output] ! Array of dimensions of DRA
              name                      [output] ! DRA name
              filename                  [output] ! DRA filename

Return type, dimension, dimensions, name of disk resident array, and filename of DRA meta-file associated with d_a handle.

dra_delete

     int DRA_Delete(int d_a)
              d_a                       [input]  ! DRA handle

Delete a disk resident array associated with d_a handle. Invalidate handle. The corresponding DRA meta-file is destroyed.

dra_close

     int DRA_Close(int d_a)
              d_a                       [input]  ! DRA handle

Close DRA meta-file associated with d_a handle and deallocate data structures corresponding to this disk array. Invalidate d_a handle. The array on the disk is persistent.

dra_flick

     void DRA_Flick()

Returns control to DRA for a VERY short time to improve progress of pending asynchronous operations.

dra_print_internals

     void DRA_Print_internals(d_a)
                integer d_a                      [input]  ! DRA handle

A call to this subroutine causes the program to dump all the internal information about the disk resident array to standard output. Only the information on processor 0 is written out.