Persistent Communication

Background

Unpredictability of Non-blocking Point-to-point Communication

  • The Rendezvous protocol is employed for large message size in most MPI implementations.
  • Communication delay of this protocol is unpredictable in two or more simultaneously communications.
  • Two timing charts of the rendezvous protocol, best and worst cases, are shown in the right-hand figures.
  • In the worst case, the ready message corresponding the request message cannot be transmitted because the network interface is busy to transfer data.

Persistent Communication

  • Initializations of send/receive operations are issued prior to starting the actual transmission and request objects are obtained.
  • Messages are transmitted using request objects.
  • Request objects are reused without initialization.
  • Since many MPI implementations employ nonblocking communication, the behavior of persistent communication is the same as non-blocking communication.

Design

Design Criteria of Persistent Remote DMA

  • Unpredictable behavior is mitigated.
  • Since all communication patterns have been known at the MPI_Start/MPI_Startall primitives, network interfaces can be utilized.
  • K computer and Fujitsu FX10, commercialize version of K have 4 DMA engines for interconnect.

Persistent Remote DMA Protocol

  • If message size is small, the eager protocol is used and the same behavior of existing MPI implementation in the eager protocol.
  • Otherwise the following protocol is employed

MPI_Isend and MPI_Irecv are implemented in the same way

Evaluation

  • Prototype system for K and FX10 computers has been implemented using PMPI facility without modification of original library implementation.
  • Benchmark: 2-dimensional 5-point stencil code using persistent communication
  • Machine: Fujitsu FX10 at University of Tokyo