Chapter 5: Distributed and shared memory parallelization

5.1 Distributed memory

1. How to make your MPI code work with Agrif ?

In your Makefile add the keyword -DAGRIF_MPI

CPPFLAGS += -DAGRIF_MPI

MPI initilization

use Agrif_Mpp
call MPI_INIT(status)
call Agrif_MPI_Init()

2. convert local indices

You have to tell Agrif how to convert local indices (on a given proc) into global workspace:

Example: Agrif_Invloc

subroutine Agrif_Invloc ( indloc, procnum, dir, indglob )

        integer, intent(in)  :: indloc   ! local index  (input)

        integer, intent(in)  :: procnum  ! current MPI proc id

        integer, intent(in)  :: dir      ! direction (1,2,3)

        integer, intent(out) :: indglob  ! global index (output)

        select case( dir )

                case(1)   ;   indglob = indloc + ishiftmpi(procnum)

                case(2)   ;   indglob = indloc + jshiftmpi(procnum)

                case(3)   ;   indglob = indloc

        end select

end subroutine Agrif_Invloc

Currently, all grids are integrated sequentially:

sequence	\(i_1\)		\(i_2\)		\(i_3\)
grid	\(G_0\)	\(\rightarrow\)	\(G_1\)	\(\rightarrow\)	\(G_2\)	\(\rightarrow\) ...

For each grid, all processors are used.

Processors utilisation for each grid when grids are integrated sequently

\(i_1\)

\(i_2\)

\(i_3\)

Sister grids (same parent) could be integrated concurently:

sequence	\(i_1\)		\(i_2\)
grid	\(G_0\)	\(\rightarrow\)	\(G_1\) // \(G_2\)	\(\rightarrow\) ...

For each sequence, processors are distributed on the grids of same level.

Distribution of processors on the grids of same level

\(i_1\)

\(i_2\)

5.2 Shared Memory

Potentially less efficient (currently all the interpolation/restriction work is done by one thread), potential use of nested parallelism is being studied\[2mm]

Example:

!$OMP MASTER
        Call Agrif_Bc_Variable(u_id,procname=Interp_MyTraceur)
!$OMP END MASTER