Enhanced RDMA setup: Enhanced Remote Direct Memory Access (RDMA) Connection Establishment
A better handshake for setting up Remote Direct Memory Access, so that neither side has to be the client.
Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S. Wise, "Enhanced Remote Direct Memory Access (RDMA) Connection Establishment", RFC 6581, DOI 10.17487/RFC6581, April 2012, <https://www.rfc-editor.org/info/rfc6581>.
@misc{rfc6581,
series = {Request for Comments},
number = 6581,
howpublished = {RFC 6581},
publisher = {RFC Editor},
doi = {10.17487/RFC6581},
url = {https://www.rfc-editor.org/info/rfc6581},
author = {A. Kanevsky and C. Bestler and R. Sharp and S. Wise},
title = {{Enhanced Remote Direct Memory Access (RDMA) Connection Establishment}},
pagetotal = 25,
year = 2012,
month = apr,
} RFC text
Contents
Internet Engineering Task Force (IETF) A. Kanevsky, Ed.
Request for Comments: 6581 Dell Inc.
Updates: 5043, 5044 C. Bestler, Ed.
Category: Standards Track Nexenta Systems
ISSN: 2070-1721 R. Sharp
Intel
S. Wise
Open Grid Computing
April 2012
Enhanced Remote Direct Memory Access (RDMA)
Connection Establishment
Abstract
This document updates RFC 5043 and RFC 5044 by extending Marker
Protocol Data Unit (PDU) Aligned Framing (MPA) negotiation for Remote
Direct Memory Access (RDMA) connection establishment. The first
enhancement extends RFC 5044, enabling peer-to-peer connection
establishment over MPA / Transmission Control Protocol (TCP). The
second enhancement extends both RFC 5043 and RFC 5044, by providing
an option for standardized exchange of RDMA-layer connection
configuration.
Status of This Memo
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by
the Internet Engineering Steering Group (IESG). Further
information on Internet Standards is available in Section 2 of
RFC 5741.
Information about the current status of this document, any
errata, and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc6581.
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction ....................................................3
1.1. Summary of Changes Affecting RFC 5044 ......................4
1.2. Summary of Changes Affecting RFC 5043 ......................4
2. Requirements Language ...........................................4
3. Definitions .....................................................4
4. Motivations .....................................................7
4.1. Standardization of RDMA Read Parameter Configuration .......7
4.2. Enabling MPA Mode ..........................................9
4.3. Lack of Explicit RTR in MPA Request/Reply Exchange ........10
4.4. Limitations on ULP Workaround .............................11
4.4.1. Transport Neutral APIs .............................11
4.4.2. Work/Completion Queue Accounting ...................11
4.4.3. Host-based Implementation of MPA Fencing ...........12
5. Enhanced MPA Connection Establishment ..........................13
6. Enhanced MPA Request/Reply Frames ..............................14
7. Enhanced SCTP Session Control Chunks ...........................15
8. MPA Error Reporting ............................................16
9. Enhanced RDMA Connection Establishment Data ....................17
9.1. IRD and ORD Negotiation ...................................18
9.2. Peer-to-Peer Connection Negotiation .......................20
9.3. Enhanced Connection Negotiation Flow ......................21
10. Interoperability ..............................................21
11. IANA Considerations ...........................................22
12. Security Considerations .......................................23
13. Acknowledgements ..............................................23
14. References ....................................................23
14.1. Normative References .....................................23
14.2. Informative References ...................................24
1. Introduction
When used over the Transmission Control Protocol (TCP), the current
Remote Direct Data Placement (RDDP) [RFC5041] suite of protocols
relies on the MPA [RFC5044] protocol for both connection
establishment and for markers for TCP layering.
A typical model for establishing an RDMA connection has the following
steps:
o The passive side (responder) Upper Layer Protocol (ULP) listens
for connection requests.
o The active side (initiator) ULP submits a connection request using
an RDMA endpoint, the desired destination, and the parameters to
be used for the connection. Those parameters include both RDMA-
layer characteristics, such as the number of simultaneous RDMA
Read Requests to be allowed, and application-specific data.
o The passive side ULP receives a connection request that includes
the identity of the active side and the requested connection
characteristics. The passive side ULP uses this information to
decide whether to accept the connection, and if it is to be
accepted, how to create and/or configure the local RDMA endpoint.
o If accepting, the responder submits its acceptance of the
connection request, which in turn generates the accept message to
the initiator. This responder accept operation includes the RDMA
endpoint to be used and the connection characteristics (both the
RDMA configuration and any application-specific Private Data to be
transferred to the initiator).
o The active side receives confirmation that the connection has been
accepted, what the configured connection characteristics are, and
any application-supplied Private Data.
Currently, MPA only supports a client-server model for connection
establishment, forcing peer-to-peer applications to interact as
though they had a client-server relationship. In addition,
negotiation of some parameters specific to the Remote Direct Memory
Access Protocol (RDMAP) [RFC5040] are left to ULP negotiation.
Providing an optional ULP-independent format for exchanging these
parameters would be of benefit to transport neutral RDMA
applications.
1.1. Summary of Changes Affecting RFC 5044
This document enhances the MPA connection setup protocol [RFC5044].
First, it adds exchange and negotiation of the parameters necessary
to support RDMA Read Requests. Second, it adds a message that serves
as a Ready to Receive (RTR) indication from the initiator to the
responder as the last message of connection establishment and adds
negotiation of which type of message to use for carrying the RTR
indication into MPA Request/Reply Frames.
RTR indications are optional and are carried by existing RDMA message
types, specifically a zero-length FULPDU Send message, a zero-length
RDMA Read message, or a zero-length RDMA write message. The presence
vs. absence of the RTR indication and the type of RDMA message to use
are negotiated by control flags in Enhanced RDMA connection
establishment data specified by this document (see Section 9). RDMA
implementations are often tightly integrated with application
libraries and hardware, hence the flexibility to use more than one
type of RDMA message enables implementations to choose message types
that are less disruptive to the implementation structure. When an
RTR indication is used, and MPA connection setup negotiation
indicates support for multiple RDMA message types as RTR indications
by both the initiator and responder, the initiator selects one of the
supported RDMA message types as the RTR indication at the initiator's
sole discretion.
1.2. Summary of Changes Affecting RFC 5043
This document enhances [RFC5043] by adding new Enhanced Session
Control Chunks that extend the currently defined Chunks with the
addition of Inbound RDMA Read Queue Depth (IRD) and Outbound RDMA
Read Queue Depth (ORD) negotiation.
2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
3. Definitions
Active Side: See Initiator.
Consumer: The ULPs or applications that lie above MPA and Direct
Data Placement (DDP). The Consumer is responsible for making TCP
or Stream Control Transmission Protocol (SCTP) connections,
starting MPA and DDP connections, and generally controlling
operations. See [RFC5044] and [RFC5043].
CRC: Cyclic Redundancy Check
Completion Queue (CQ): A Consumer-accessible queue where the RDMA
device reports completions of Work Requests. A Consumer is able
to reap completions from a CQ without requiring per-transaction
support from the kernel or other privileged entity. See [RDMAC].
Completion Queue Entry (CQE): Transport- and device-specific
representation of a Work Completion. A CQ holds CQEs. See
[RDMAC].
FULPDU: Framed Upper Layer Protocol PDU. See FPDU of [RFC5044].
Inbound RDMA Read Request Queue (IRRQ): A queue that is associated
with an RDMA connection that tracks active incoming simultaneous
RDMA Read Request Messages. See [RDMAC].
Inbound RDMA Read Queue Depth (IRD): The maximum number of incoming
simultaneous RDMA Read Request Messages an RDMA connection can
handle. See [RDMAC].
Initiator: The endpoint of a connection that sends the MPA Request
Frame. The initiator is the active side of the connection
establishment. See [RFC5044].
IRD: See Inbound RDMA Read Queue Depth.
MPA Fencing: MPA responder connection establishment logic that
ensures that no ULP messages will be transferred until the
initiator's first message has been received.
MPA Request Frame: Data sent from the MPA initiator to the MPA
responder during the Startup Phase. See [RFC5044].
MPA Reply Frame: Data sent from the MPA responder to the MPA
initiator during the Startup Phase. See [RFC5044].
ORD: See Outbound RDMA Read Queue Depth.
Outbound RDMA Read Queue Depth (ORD): The maximum number of
simultaneous RDMA Read Requests that can be issued for the RDMA
connection. This should be less than or equal to the peer's IRD.
See [RDMAC].
Passive Side: See Responder.
Private Data: A block of data exchanged between MPA endpoints during
initial connection setup. See [RFC5044].
Queue Pair (QP): A Queue Pair is the set of Work Queues associated
exclusively with a single Endpoint (first defined in [VIA]). The
Send Queue (SQ), Receive Queue (RQ), and Inbound RDMA Read Queue
(IRQ) are considered to be part of the Queue Pair. The
potentially shared Completion Queue (CQ) and Shared Receive Queue
(SRQ) are not. See [RDMAC].
Remote Peer: The MPA protocol implementation on the opposite end of
the connection. Used to refer to the remote entity when
describing protocol exchanges or other interactions between two
nodes. See [RFC5044].
Responder: The connection endpoint that responds to an incoming MPA
connection request (the MPA Request Frame). The responder is the
passive side of the connection establishment. See [RFC5044].
Ready to Receive (RTR): RTR is an indication provided by the last
connection establishment message sent from the initiator to the
responder. An RTR indicates that the initiator is ready to
receive messages and that connection establishment is completed.
Startup Phase: The initial exchanges of an MPA connection that
serves to more fully identify MPA endpoints to each other and pass
connection-specific setup information to each other. See
[RFC5044].
Shared Receive Queue (SRQ): A shared pool of Receive Work Requests
posted by the Consumer that can be allocated by multiple RDMA
endpoints (QP). See [RDMAC].
Tagged (DDP) Message: A DDP Message that targets a Tagged Buffer
that is explicitly advertised to the Remote Peer through exchange
of an STag (memory handle), offset in the memory region identified
by STag, and length [RFC5040].
Untagged (DDP) Message: A DDP Message that targets an Untagged
Buffer associated with a queue specified the by Queue Number (QN).
[RFC5040].
Work Queue: An element of a QP that allows user-space applications
to submit Work Requests directly to network hardware (first
defined in [VIA]). Specific Work Queues include the Send Queue
(SQ) for transmit requests, Receive Queue (RQ) for receive
requests specific to a single endpoint, and Shared Receive Queues
(SRQs) for receive requests that can be allocated by one or more
endpoints. See [RDMAC].
Work Queue Element (WQE): Transport- and device-specific
representation of a Work Request. See [RDMAC].
Work Request: An elementary object used by Consumers to enqueue a
requested operation (WQEs) onto a Work Queue. See [RDMAC].
4. Motivations
The goal of this document is two-fold. The first is to extend
support from the current client-server model for RDMA connection
setup to a peer-to-peer model. The second is to add negotiation of
the RDMA Read Queue size for both sides of an RDMA connection.
4.1. Standardization of RDMA Read Parameter Configuration
Most RDMA applications are developed using a transport-neutral
Application Programming Interface (API) to access RDMA services based
on a "Queue Pair" paradigm as originally defined by the Virtual
Interface Architecture [VIA], refined by the Direct Access
Programming Library [DAPL], and most commonly deployed with the
OpenFabrics API [OFA].
These transport-neutral APIs seek to provide a common set of RDMA
services whether the underlying transport is, for example, RDDP over
MPA, RDDP over SCTP, or InfiniBand.
The common model for establishing an RDMA connection has the
following steps:
o The passive side ULP listens for connection requests.
o The active side ULP submits a connection request using an RDMA
endpoint ("Queue Pair"), the desired destination, and the
parameters to be used for the connection. Those parameters
include both RDMA-layer characteristics, such as the number of
simultaneous RDMA Read Requests to be allowed, and application-
specific data (typically referred to as "Private Data").
o The passive side ULP receives a connection request, which includes
the identity of the active side and the requested connection
characteristics. The passive side ULP uses this information to
decide whether to accept the connection, and if it is to be
accepted, how to create and/or configure the RDMA endpoint.
o If accepting, the passive side ULP submits its acceptance of the
connection request. This local accept operation includes the RDMA
endpoint to be used and the connection characteristics (both the
RDMA configuration and any application-specific Private Data to be
returned).
o The active side receives confirmation that the connection has been
accepted, what the configured connection characteristics are, and
any application-supplied Private Data.
As currently defined, DDP connection establishment requires the ULP
to encode the RDMA configuration in the application-specific Private
Data. This results in undesirable duplication of logic to cover RDMA
characteristics of both InfiniBand and RDDP for each ULP, and to
specify for InfiniBand and RDDP the extraction of the RDMA
characteristics for each ULP.
Both RDDP and InfiniBand support an initial Private Data exchange;
therefore, a standard definition of the RDMA characteristics within
the Private Data section would enable common connection establishment
APIs to format the RDMA characteristics based on the same API
information used when establishing either protocol to form the
connection. The application would then only have to indicate that it
was using this standard format to enable common connection
establishment procedures to apply common code to properly parse these
fields and configure the RDMA endpoints accordingly. Exchange of
parameters necessary to perform RDMA Read operations is a common
usage of the initial Private Data exchange.
One of the RDMA operations that is defined in [RDMAC] is an RDMA
Read. RDMA Read operations are performed using an untagged message
sent from a Queue Pair (QP) on the local endpoint to a QP on the
remote endpoint targeting the Inbound RDMA Read Request Queue (QN=1
or Inbound RDMA Read Request Queue (IRRQ)) associated with the
connection. RDMA Read responses transfer data associated with each
RDMA Read Request from the remote endpoint to the local endpoint
using tagged messages. An inbound RDMA Read Request remains on the
IRRQ from the time that it is received until the time that the last
tagged message associated with the RDMA request is acknowledged. The
IRRQ is associated with a QP but is not a Work Queue. Instead, the
IRRQ is a stand-alone queue that is used to manage RDMA Read Requests
associated with a QP. See [RDMAC], Section 6 for more information
regarding QPs and IRRQ. One of the characteristics that must be
configured for a QP is the size of the IRRQ. This parameter is
called the Inbound RDMA Read Queue Depth (IRD). Another
characteristic of a QP that must be configured is a local limit on
the number of simultaneous outbound RDMA Read Requests based on the
size of the remote endpoint QP's IRRQ. This parameter is call the
Outbound RDMA Read Queue Depth (ORD). ORD is used to limit the
number of simultaneous RDMA Read Requests such that the local
endpoint does not overrun the remote endpoint's IRRQ depth or IRD.
Note that outbound RDMA Reads are submitted to a QP's Send Queue at
the local peer, not to a separate outbound RDMA Read Request queue on
the local peer. The local endpoint uses ORD to strictly limit
simultaneous Read Requests so that IRRQ overruns do not occur at the
remote endpoint.
Determination of the values of the ORD and IRD are left to the ULP by
the current RDDP suite of protocols and also by [RDMAC]. Since this
negotiation of ORD and IRD is typical, it is desirable to provide a
common mechanism as described in this document.
4.2. Enabling MPA Mode
MPA defines encoding of DDP Segments in Framed Upper Layer Protocol
PDUs (FULPDUs). Generation of FULPDUs requires the ability to
periodically insert MPA Markers and to generate the MPA CRC-32c for
each frame. Reception may require parsing/removing the markers after
using them to identify MPA Frame boundaries and validation of the
MPA-CRC32c.
A major design objective for MPA was to ensure that the resulting TCP
stream would be fully compliant for any and all TCP-aware
middleboxes. The challenge is that while only some TCP payload
streams are a valid stream of MPA FULPDUs, any sequence of bytes is a
valid TCP payload stream. The determination that a given stream is
in a specific MPA mode cannot be made at the MPA or TCP layer.
Therefore, enabling of MPA mode is handled by the ULP.
The MPA protocol can be viewed as having two parts:
o a specification of generation and reception of MPA FULPDUs. This
is unchanged by enhanced RDMA connection establishment.
o a pre-MPA exchange of messages to enable a specific MPA mode for
the TCP connection. Enhanced RDMA connection establishment
extends this protocol with two new features.
In typical implementations, generation and reception of MPA FULPDUs
is handled by hardware. The exchange of the MPA Request and Reply
Frames is then handled by host software. As will be explained, this
implementation split impedes applications that are not compatible
with the client-server assumptions in the current MPA Request/Reply
exchange.
4.3. Lack of Explicit RTR in MPA Request/Reply Exchange
The exchange of MPA Request and Reply messages to place a TCP
connection in MPA mode is specified in [RFC5044]. This protocol
provides many benefits to the design of MPA FULPDU hardware:
o The ULP is responsible for specifying the exact MPA Mode (Markers
enabled or disabled, CRC-32c enabled or suppressed) and the point
in the TCP streams (inbound and outbound) where MPA Frames will
begin.
o Before the first MPA Frame is transmitted, all pre-MPA mode TCP
payloads will have been acknowledged by the peer. Therefore, it
is never necessary to generate a retransmission that mixes pre-MPA
and MPA payload.
o Before MPA reception is enabled, all incoming pre-MPA mode TCP
payloads will have been acknowledged. Therefore, the host will
never receive a TCP segment that mixes pre-MPA and MPA payload.
The limitation of the current MPA Request/Reply exchange is that it
does not define a Ready to Receive (RTR) indication that the active
side would send, so that the passive side can know that the last non-
MPA payload (the MPA Reply) had been received.
Instead, the role of an RTR indication is piggybacked on the first
MPA FULPDU sent by the active side. This is actually a valuable
optimization for all applications that fit the classic client-server
model. The client only initiates the connection when it has a
request to send to the server, and the server has nothing to send
until it has received and processed the client request.
Even applications where the server sends some configuration data
immediately can easily send the same information as application
Private Data in the MPA Reply. So the currently defined exchange
works for almost all applications.
Many peer-to-peer applications, especially those involving cluster
calculations (frequently using Message Passing Interface (MPI)
[UsingMPI] or [RDS]), have no natural client or server roles ([PPMPI]
[OpenMP]). Typically, one member of the cluster is arbitrarily
selected to initiate the connection when the distributed task is
launched, while the other accepts it. At startup time, however,
there is no way to predict which node will have the first message to
actually send. Immediately establishing the connections is valuable
because it reduces latency once results are ready to transmit and it
validates connectivity throughout the cluster.
The lack of an explicit RTR indication in the MPA Request/Reply
exchange forces all applications to have a first message from the
connection initiator, whether or not this matches the application
communication model.
4.4. Limitations on ULP Workaround
The requirement that the RDMA connection initiator sends the first
message does not appear to be onerous on first examination. The
natural question is why the application layer would not simply
generate a dummy message when there is no other message to submit.
There are three factors that make this workaround unsuitable for many
peer-to-peer applications:
o Transport-Neutral APIs.
o Work/Completion Queue Accounting.
o Host-based implementation of MPA Fencing.
4.4.1. Transport-Neutral APIs
Many of these applications access RDMA services using a transport-
neutral API such as [DAPL] or [OFA]. Only RDDP over TCP [RFC5044]
has a first message requirement. Other RDMA transports, including
RDDP over SCTP (see [RFC5043]) and InfiniBand (see [IBTA]), do not.
Application or middleware communications can be expressed as
transport-neutral RDMA operations, allowing lower software layers to
translate to transport and device specifics. Having a distinct extra
message that is required only for one transport undermines the
application's goal of being transport neutral.
4.4.2. Work/Completion Queue Accounting
RDMA local APIs conventionally use Work Queues to submit requests
(Work Queue elements or WQEs) and to asynchronously receive
completions (in Completion Queues or CQs).
Each Work Request can generate a Completion Queue Entry (CQE).
Completions for successful transmit Work Requests are frequently
suppressed, but the CQ capacity must account for the possibility that
each will complete in error. A CQ can receive completions from
multiple Work Queues.
CQs are defined to allow hardware RDMA implementations to generate
CQEs directly to a user-space-mapped buffer. This enables a user-
space RDMA Consumer to reap completions without requiring kernel
intervention.
A hardware RDMA implementation cannot reasonably wait for an
available slot in the CQ. The queue must be sized such that an
overflow will not occur. When an overflow does occur, it is
considered a catastrophic error and will typically require tearing
down all RDMA connections using that CQ.
This style of interface is very efficient, but places a burden on the
application to properly size each CQ to match the Work Queues that
feed it.
While the format of both WQEs and CQEs is transport and device
dependent, a transport-neutral API can deal with WQEs and CQEs as
abstract transport- and device-neutral objects. Therefore, the
number of WQEs and CQEs required for an application can be transport
and device neutral.
The capacity of the Work Queues and CQs can be calculated in an
abstract transport- and device-neutral fashion. If a dummy operation
approach is used, it would require lower layers to know the usage
model, and would disrupt the calculations by inserting a dummy
"operation" Work Request and filtering out the matching completion.
The lower layer does not know the usage model on which the queue
sizes are built, nor does it know how frequently an insertion will be
required.
4.4.3. Host-based Implementation of MPA Fencing
Many hardware implementations of RDDP using MPA/TCP do not handle the
MPA Request/Reply exchange in hardware, rather they are handled by
the host processor in software. With such designs, it is common for
the MPA Fencing to be implemented in the user-space, device-specific
library (commonly referred to as a 'User Verbs' library or module).
When the generation and reception of MPA FULPDUs are already
dedicated to hardware, a Work Completion can only be generated by an
untagged message, since arrival of a message for a tagged buffer does
not necessarily generate a completion and is done without any
interaction with ULP [RFC5040].
5. Enhanced MPA Connection Establishment
Below we provide an overview of Enhanced Connection Setup. The goal
is to allow standard negotiation of the ORD/IRD setting on both sides
of the RDMA connection and/or to negotiate the initial data transfer
operation by the initiator when the existing 'client sends first'
rule does not match application requirements.
The RDMA connection initiator sends an MPA Request, as specified in
[RFC5044]; the new format defined here allows for:
o Standardized negotiation of ORD and IRD.
o Negotiation of RTR functionality and the RDMA message type to use
as the RTR indication.
The RDMA connection responder processes the MPA Request and generates
an MPA Reply, as specified in [RFC5044]; the new format completes the
negotiation.
The local interface needs to provide a way for a ULP to request the
use of explicit RTR indication on a per-application or per-connection
basis when an explicit RTR indication will be required. Piggybacking
the RTR on a Client's first message is a valuable optimization for
most connections.
The RDMA connection initiator MUST NOT allow any later FULPDUs to be
transmitted before the RTR indication. One method to achieve this is
to delay notifying the ULP that the RDMA connection has been
established until after any required RTR indication has been
transmitted.
All MPA exchanges are performed via TCP prior to RDMA establishment,
and are therefore signaled via TCP and not via RDMA completion.
6. Enhanced MPA Request/Reply Frames
Enhanced RDMA connection establishment uses an alternate format for
MPA Requests and Replies as follows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | |
+ Key (16 bytes containing "MPA ID Req Frame") +
4 | (4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65) |
+ Or (16 bytes containing "MPA ID Rep Frame") +
8 | (4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65) |
+ +
12 | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 |M|C|R|S| Res | Rev | PD_Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ ~
~ Private Data ~
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Key: Unchanged from [RFC5044].
M: Unchanged from [RFC5044].
C: Unchanged from [RFC5044].
R: Unchanged from [RFC5044].
S: One, if the Private Data begins with the enhanced RDMA connection
establishment data; 0 otherwise.
Res: One bit smaller than in [RFC5044]; otherwise unchanged. In
[RFC5044], the 'Res' field, in which the newly defined 'S' bit
resides, is reserved for future use. [RFC5044] specifies that
'Res' MUST be set to zero when sending and MUST NOT be checked on
reception, making use of 'S' bit backwards compatibility with the
original MPA Frame format. When the 'S' bit is set to zero, no
additional Private Data is used for enhanced RDMA connection
establishment; therefore, the resulting MPA Request and Reply
Frames are identical to the unenhanced protocol.
Rev: This field contains the revision of MPA. To use any enhanced
connection establishment feature, this MUST be set to two or
higher. If no enhanced connection establishment features are
desired, it MAY be set to one. A host accepting MPA connections
MUST continue to accept MPA Requests with version one, even if it
supports version two.
PD_Length: Unchanged from [RFC5044]. This is the total length of
the Private Data field, including the enhanced RDMA connection
establishment data, if present.
Private Data: Unchanged from [RFC5044]. However, if the 'S' flag is
set, Private Data MUST begin with enhanced RDMA connection
establishment data (see Section 9).
7. Enhanced SCTP Session Control Chunks
Enhanced RDMA connection establishment uses the first 32 bits of the
Private Data field for IRD and ORD negotiation in the "DDP Stream
Session Initiate" and "DDP Stream Session Accept" SCTP Session
Control Chunks.
The type of the SCTP Session Control Chunk is defined by a Function
Code (see [RFC4960]). [RFC5043] already defines codes for 'DDP
Stream Session Initiate' and 'DDP Stream Session Accept', which are
equivalent to an MPA Request Frame and an accepting MPA Reply Frame.
Enhanced RDMA connection establishment requires three additional
function codes listed below:
Enhanced DDP Stream Session Initiate: 0x005
Enhanced DDP Stream Session Accept: 0x006
Enhanced DDP Stream Session Reject: 0x007
The Enhanced Reject function code MUST be used to indicate rejection
of enhanced DDP stream session for a configuration that would have
been accepted for unenhanced DDP stream session negotiation.
The enhanced DDP stream session establishment follows the same rules
as the standard DDP stream session establishment as defined in
[RFC5043]. ULP-supplied Private Data MUST be included for Enhanced
DDP Stream Session Initiate, Enhanced DDP Stream Session Accept, and
Enhanced DDP Stream Session Reject messages, and MUST follow the
enhanced RDMA connection establishment data in the DDP Stream Session
Initiate and the Enhanced DDP Stream Session Accept messages.
Private Data length MUST NOT exceed 512 bytes in any message,
including enhanced RDMA connection establishment data.
Private Data MUST NOT be included in the DDP Stream Session TERM
message.
Received Extended DDP Stream Session Control messages SHOULD be
reported to the ULP. If reported, any supplied Private Data MUST be
available for the ULP to examine. For example, a received Extended
DDP Stream Session Control message is not reported to ULP if none of
the requested RTR indication types are supported by the receiver. In
this case, the Provider MAY generate a reject reply message
indicating which RTR indication types it supports.
The enhanced DDP stream management MUST use the DDP stream session
termination function code to terminate a stream established using
enhanced DDP stream session function codes.
[RFC5043] already supports either side sending the first DDP Message
since the Payload Protocol Identifier (PPID) already distinguishes
between Session Establishment and DDP Segments. The enhanced RDMA
connection establishment provides the ULP a transport-independent way
to support the peer-to-peer model.
The following additional Legal Sequences of DDP Stream Session
messages are defined:
o Enhanced Active/Passive Session Accepted: as with Section 6.2 of
[RFC5043], but with the extended opcodes as defined in this
document.
o Enhanced Active/Passive Session Rejected: as with Section 6.3 of
[RFC5043], but with the extended opcodes as defined in this
document.
o Enhanced Active/Passive Session Non-ULP Rejected: as with Section
6.4 of [RFC5043], but with the extended opcodes as defined in this
document.
8. MPA Error Reporting
The RDMA connection establishment protocol is layered upon the
protocols defined in [RFC5040] and [RFC5041]. Any enhanced RDMA
connection establishment error generates an MPA termination message
to a peer. [RFC5040] defines a triplet of protocol layers, error
types, and error codes for error specification. MPA negotiation for
RDMA connection establishment uses the following layer and error type
for MPA error reporting:
Layer: 0x2 - LLP Error Type: 0x0 - MPA
While [RFC5044] defines four error codes, [RFC5043] does not define
any. Enhanced RDMA connection establishment extends the error codes
defined in [RFC5044] by adding three new error codes. Thus, enhanced
RDMA connection establishment is backward compatible with both
[RFC5043] and [RFC5044].
The following error codes are defined for enhanced RDMA connection
establishment negotiation:
Error Code Description
--------------------------------------------------------
0x05 Local catastrophic
0x06 Insufficient IRD resources
0x07 No matching RTR option
9. Enhanced RDMA Connection Establishment Data
Enhanced RDMA connection establishment places the following 32 bits
at the beginning of the Private Data field of the MPA Request and
Reply Frames or the "DDP Stream Session Initiate" and "DDP Stream
Session Accept" SCTP Session Control Chunks. ULP-specified Private
Data follows this field. The maximum amount of ULP-specified Private
Data is therefore reduced by 4 bytes. Note that this field MUST be
sent in network byte order, with the IRD and ORD encoded as 14-bit
unsigned integers.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 |A|B| IRD |C|D| ORD |
4 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
IRD: Inbound RDMA Read Queue Depth.
ORD: Outbound RDMA Read Queue Depth.
A: Control Flag for connection model.
B: Control Flag for use of a zero-length FULPDU (Send) RTR
indication.
C: Control Flag for use of a zero-length RDMA Write RTR indication.
D: Control Flag for use of a zero-length RDMA Read RTR indication.
9.1. IRD and ORD Negotiation
The IRD and ORD are used for negotiation of Inbound RDMA Read Request
Queue depths for both endpoints of the RDMA connection. The IRD is
used to configure the depth of the Inbound RDMA Read Request Queue
(IRRQ) on each endpoint. ORD is used to limit the number of
simultaneous outbound RDMA Read Requests allowed at any given point
in time in order to avoid IRRQ overruns at the remote endpoint. In
order to describe the negotiation of both local endpoint and remote
endpoint ORD and IRD values, four terms are defined:
Initiator IRD: The IRD value sent in the MPA Request or "DDP Stream
Session Initiate" SCTP Session Control Chunk. This is the value
of the initiator's IRD at the time of the MPA Request generation.
The responder sets its local ORD value to this value or less. The
initiator IRD is the maximum number of simultaneous inbound RDMA
Read Requests that the initiator can support for the requested
connection.
Initiator ORD: The ORD value in the MPA Request or "DDP Stream
Session Initiate" SCTP Session Control Chunk. This is the initial
value of the initiator's ORD at the time of the MPA Request
generation and also a request to the responder to support a
responder IRD of at least this value. The initiator ORD is the
maximum number of simultaneous outbound RDMA Read operations that
the initiator desires the responder to support for the requested
connection.
Responder IRD: The IRD value returned in the MPA Reply or "DDP
Stream Session Accept" SCTP Session Control Chunk. This is the
actual value that the responder sets for its local IRD. This
value is greater than or equal to the initiator ORD for successful
negotiations. The responder IRD is the maximum number of
simultaneous inbound RDMA Read Requests that the responder
actually can support for the requested connection.
Responder ORD: The ORD value returned in the MPA Reply or "DDP
Stream Session Accept" SCTP Session Control Chunk. This is the
actual value that the responder used for ORD and is less than or
equal to the initiator IRD for successful negotiations. The
responder ORD is the maximum number of simultaneous outbound RDMA
Read operations that the responder will allow for the requested
connection.
The relationships between these parameters after a successful
negotiation is complete are the following:
initiator ORD <= responder IRD
responder ORD <= initiator IRD
The responder and initiator MUST pass the peer's provided IRD and ORD
values to the ULP, in addition to using the values as calculated by
the preceding rules.
The responder ORD SHOULD be set to a value less than or equal to the
initiator IRD. If the initiator ORD is insufficient to support the
selected connection model, the responder IRD MAY be increased; for
example, if the initiator ORD is 0 (RDMA Reads will not be used by
the ULP) and the responder supports use of a zero-length RDMA Read
RTR indication, then the responder IRD can be set to 1. The
responder MUST set its ORD at most to the initiator IRD. The
responder MAY reject the connection request if the initiator IRD is
not sufficient for the ULP-required ORD and specify the required ORD
in the MPA Reject Frame responder ORD. Thus, the TERM message MUST
contain Layer 2, Error Type 0, Error Code 6.
Upon receiving the MPA Accept Frame from the responder, the initiator
MUST set its IRD at least to the responder ORD and its ORD at most to
the responder IRD. If the initiator does not have sufficient
resources for the required IRD, it MUST send a TERM message to the
responder indicating insufficient resources and terminate the
connection due to insufficient resources. Thus, the TERM message
MUST contain Layer 2, Error Type 0, Error Code 6.
The initiator MUST pass the responder provided IRD and ORD to the ULP
for both MPA Accept and Reject messages. The initiator ULP can
decide its course of action. For example, the initiator ULP may
terminate the established connection and renegotiate the responder
ORD.
An all ones value (0x3FFF) indicates that automatic negotiation of
the IRD or ORD is not desired, and that the ULP will be responsible
for it. The responder MUST respond to an initiator ORD value of
0x3FFF by leaving its local endpoint IRD value unchanged and setting
the IRD to 0x3FFF in its reply message. The initiator MUST leave its
local endpoint ORD value unchanged upon receiving a responder IRD
value of 0x3FFF. The responder MUST respond to an initiator IRD
value of 0x3FFF by leaving its local endpoint ORD value unchanged,
and setting ORD to 0x3FFF in its reply message. The initiator MUST
leave its local endpoint IRD value unchanged upon receiving a
responder ORD value of 0x3FFF.
9.2. Peer-to-Peer Connection Negotiation
Control Flag A value 1 indicates that a peer-to-peer connection model
is being performed, and value 0 indicates a client-server model.
Control Flag B value 1 indicates that a zero-length FULPDU (Send) RTR
indication is requested for the initiator and supported by the
responder, respectively, 0 otherwise. Control Flag C value 1
indicates that a zero-length RDMA Write RTR indication is requested
for the initiator and supported by the responder, respectively, 0
otherwise. Control Flag D value 1 indicates that a zero-length RDMA
Read RTR indication is requested for the initiator and supported by
the responder, respectively, 0 otherwise. The initiator MUST set
Control Flag A to 1 for the peer-to-peer model. The initiator MUST
set each Control Flag B, C, and D to 1 for each of the options it
supports, if Control Flag A is set to 1.
The responder MUST support at least one RTR indication option if it
supports Enhanced RDMA connection establishment. If Control Flag A
is 1 in the MPA Request message, then the responder MUST set Control
Flag A to 1 in the MPA reply message. For each initiator-supported
RTR indication option, the responder SHOULD set the corresponding
Control Flag if the responder can support that option in an MPA
reply. The responder is not required to specify all RTR indication
options it supports. The responder MUST set at least one RTR
indication option if it supports more than one initiator-specified
RTR indication option. The responder MAY include additional RTR
indication options it supports, even if not requested by any
initiator specified RTR indication options. If the responder does
not support any of the initiator-specified RTR indication options,
then the responder MUST set at least one RTR indication type option
it supports.
Upon receiving the MPA Accept Frame with Control Flag A set to 1, the
initiator MUST generate one of the negotiated RTR indications. If
the initiator is not able to generate any of the responder-supported
RTR indications, then it MUST send a TERM message to the responder
indicating failure to negotiate a mutually compatible connection
model or RTR option, and terminate the connection. Thus, the TERM
message MUST contain Layer 2, Error Type 0, Error Code 7. The ULP
can negotiate a ULP-level RTR indication when a Provider-level RTR
indication cannot be negotiated.
The initiator MUST set Control Flag A to 0 for the client-server
model. The responder MUST set Control Flag A to 0 if Control Flag A
is 0 in the request. If Control Flag A is set to 0, then Control
Flags B, C, and D MUST also be set to 0. On reception, if Control
Flag A is set to 0, then Control Flags B, C, and D MUST be ignored.
9.3. Enhanced Connection Negotiation Flow
The RTR indication type and ORD/IRD negotiation follows the following
order:
initiator (MPA Request) --> The initiator sets Control Flag A to 1
to indicate the peer-to-peer connection model and sets its initial
IRD/ORD on the local endpoint of the connection. The initiator
also sets Control Flags B, C, and D to 1 for each initiator-
supported option of RTR indication.
responder (MPA Reply) <-- The responder matches the initiator's
Control Flag A value and sets ORD/IRD to its local endpoint values
based upon the initiator's initial ORD/IRD values and the number
of simultaneous RDMA Read Requests required by the ULP. The
responder sets Control Flags B, C, and D to 1 for each responder-
supported option of RTR indication options for the peer-to-peer
connection model. The responder also sets its IRD/ORD to actual
values.
initiator (First RDMA Message) --> After the initiator modifies its
ORD/IRD to match the responder's values as stated above, the
initiator sends the first message of the negotiated RTR indication
option. If no matching RTR indication option exists, then the
initiator sends a TERM message.
The initiator or responder MUST generate the TERM message that
contains Layer 2, Error Type 0, Error Code 5 when it encounters
any error locally for which the special Error Code is not defined
in Section 8 before resetting the connection.
10. Interoperability
The initiator requests enhanced RDMA connection establishment by
sending an enhanced RDMA establishment request; an enhanced responder
is REQUIRED to respond with an enhanced RDMA connection establishment
response, whereas an unenhanced responder treats the enhanced request
as incorrectly formatted and closes the TCP connection. All
responders are REQUIRED to issue unenhanced RDMA connection
establishment responses in response to unenhanced RDMA connection
establishment requests.
The initiator MUST NOT use the enhanced RDMA connection establishment
formats or function codes when no enhanced functionality is desired.
The responder MUST continue to accept unenhanced connection requests.
There are three initiator/responder cases that involve enhanced MPA:
both the initiator and responder, only the responder, and only the
initiator. The enhanced MPA Frame is defined by field 'S' set to 1.
Enhanced MPA initiator and responder: If the responder receives an
enhanced MPA message, it MUST respond with an enhanced MPA
message.
Enhanced MPA responder only: If the responder receives an unenhanced
MPA message ('S' is set to 0), it MUST respond with an unenhanced
MPA message.
Enhanced MPA initiator only: If the responder receives an enhanced
MPA message and it does not support enhanced RDMA connection
establishment, it MUST close the TCP connection and exit MPA.
From a standard RDMA connection establishment point of view, the
enhanced MPA Frame is improperly formatted as stated in [RFC5044].
Thus, both the initiator and responder report TCP connection
termination to an application locally. In this case, the
initiator MAY attempt to establish an RDMA connection using the
unenhanced MPA protocol as defined in [RFC5044] if this protocol
is compatible with the application, and let the ULP deal with ORD
and IRD and peer-to-peer negotiations.
A note for potential future enhancements for connection establishment
negotiation: It is possible to further extend formatting of Private
Data of the MPA Request and Reply Frames and to use other bits from
the "Res" field to indicate additional Private Data formatting.
11. IANA Considerations
IANA has added the following entries to the "SCTP Function Codes for
DDP Session Control" registry created by Section 3.5 of [RFC6580]:
0x0005, Enhanced DDP Stream Session Initiate, [RFC6581]
0x0006, Enhanced DDP Stream Session Accept, [RFC6581]
0x0007, Enhanced DDP Stream Session Reject, [RFC6581]
IANA has added the following entries to the "MPA Errors" registry
created by Section 3.3 of [RFC6580]:
0x2/0x0/0x05, - MPA Error / Local catastrophic error, [RFC6581]
0x2/0x0/0x06 - MPA Error / Insufficient IRD resources, [RFC6581]
0x2/0x0/0x07 - MPA Error / No matching RTR option, [RFC6581]
12. Security Considerations
The security considerations from RFC 5044 and RFC 5043 apply and the
changes in this document do not introduce new security
considerations. However, it is recommended that implementations do
sanity checking for the input parameters, including ORD, IRD, and the
control flags used for RTR indication option negotiation.
13. Acknowledgements
The authors wish to thank Sean Hefty, Dave Minturn, Tom Talpey, David
Black, and David Harrington for their valuable contributions and
reviews of this document.
14. References
14.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC4960] Stewart, R., "Stream Control Transmission Protocol", RFC
4960, September 2007.
[RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
Garcia, "A Remote Direct Memory Access Protocol
Specification", RFC 5040, October 2007.
[RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
Data Placement over Reliable Transports", RFC 5041,
October 2007.
[RFC5043] Bestler, C. and R. Stewart, "Stream Control Transmission
Protocol (SCTP) Direct Data Placement (DDP) Adaptation",
RFC 5043, October 2007.
[RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
Carrier, "Marker PDU Aligned Framing for TCP
Specification", RFC 5044, October 2007.
[RFC6580] Ko, M. and D. Black, "IANA Registries for the Remote
Direct Data Placement (RDDP) Protocols", RFC 6580, April
2012.
14.2. Informative References
[DAPL] "Direct Access Programming Library",
<http://www.datcollaborative.org/uDAPL_doc_062102.pdf>.
[IBTA] "InfiniBand Architecture Specification Release 1.2.1",
<http://www.infinibandta.org>.
[OFA] "OFA verbs & APIs", <http://www.openfabrics.org/>.
[OpenMP] McGraw-Hill, "Parallel Programming in C with MPI and
OpenMP", 2003.
[PPMPI] Morgan Kaufmann Publishers Inc., "Parallel Programming
with MPI", 2008.
[RDMAC] "RDMA Protocol Verbs Specification (Version 1.0)",
<http://www.rdmaconsortium.org/home/
draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>.
[RDS] Open Fabrics Association, "Reliable Datagram Socket",
2008,
<http://www.openfabrics.org/archives/spring2008sonoma>.
[UsingMPI] MIT Press, "Using MPI-2: Advanced Features of the Message
Passing Interface", 1999.
[VIA] Cameron, Don and Greg Regnier, "Virtual Interface
Architecture", Intel, April 2002.
Authors' Addresses
Arkady Kanevsky (editor)
Dell Inc.
One Dell Way, MS PS2-47
Round Rock, TX 78682
USA
Phone: +1-512-728-0000
EMail: arkady.kanevsky@gmail.com
Caitlin Bestler (editor)
Nexenta Systems
555 E El Camino Real #104
Sunnyvale, CA 94087
USA
Phone: +1-949-528-3085
EMail: Caitlin.Bestler@nexenta.com
Robert Sharp
Intel
LAD High Performance Message Passing, Mailstop: AN1-WTR1
1501 South Mopac, Suite 400
Austin, TX 78746
USA
Phone: +1-512-493-3242
EMail: robert.o.sharp@intel.com
Steve Wise
Open Grid Computing
4030 Braker Lane STE 130
Austin, TX 78759
USA
Phone: +1-512-343-9196 x101
EMail: swise@opengridcomputing.com
ISO Plain Language
On this page
§1Introduction
When used over the Transmission Control Protocol, the Remote Direct Data Placement suite of protocols, defined in RFC 5041, relies on the Marker PDU Aligned Framing protocol of RFC 5044 for both connection establishment and for the markers used for TCP layering.
A typical model for establishing an RDMA connection runs like this. The passive side, the responder, has an upper layer protocol that listens for connection requests. The active side, the initiator, submits a connection request using an RDMA endpoint, the desired destination, and the parameters to be used for the connection, which include RDMA-layer characteristics such as the number of simultaneous RDMA Read requests to be allowed, and application-specific data. The passive side receives that request, which includes the identity of the active side and the requested connection characteristics, and uses it to decide whether to accept and how to create or configure its local RDMA endpoint. If accepting, the responder submits its acceptance, which includes the RDMA endpoint to be used and the connection characteristics, and that generates the accept message to the initiator. The active side then receives confirmation that the connection was accepted, the configured connection characteristics, and any application-supplied private data.
MPA only supports a client-server model for connection establishment, forcing peer-to-peer applications to interact as though they had a client-server relationship. Negotiation of some parameters specific to the Remote Direct Memory Access Protocol of RFC 5040 is also left to the upper layer. Providing an optional upper-layer-independent format for exchanging those parameters would benefit transport-neutral RDMA applications.
The handshake has always had a shape where one side listens and the other calls. This RFC changes that so either side can call, and adds two numbers to the handshake that say how many reads each side can have in flight at once.
§4Motivations, Read Queue Depths and the Readiness Indication
The goal of the document is two-fold. The first is to extend support from the current client-server model for RDMA connection setup to a peer-to-peer model. The second is to add negotiation of the RDMA Read queue size for both sides of an RDMA connection.
Most RDMA applications are developed using a transport-neutral application programming interface to access RDMA services based on a Queue Pair paradigm, as originally defined by the Virtual Interface Architecture, refined by the Direct Access Programming Library, and most commonly deployed with the OpenFabrics API. These APIs seek to provide a common set of RDMA services whether the underlying transport is, for example, RDDP over MPA, RDDP over SCTP, or InfiniBand.
As currently defined, DDP connection establishment requires the upper layer to encode the RDMA configuration in the application-specific private data. That results in undesirable duplication of logic to cover the RDMA characteristics of both InfiniBand and RDDP for each upper layer, and in a need to specify the extraction of those characteristics for each upper layer. Because both RDDP and InfiniBand support an initial private data exchange, a standard definition of the RDMA characteristics within the private data section would let common connection establishment APIs format those characteristics from the same information used for either protocol.
One of the RDMA operations is an RDMA Read. Read operations are performed using an untagged message sent from a Queue Pair on the local endpoint to a Queue Pair on the remote endpoint, targeting the Inbound RDMA Read Request Queue associated with the connection. Read responses transfer data from the remote endpoint to the local endpoint using tagged messages. An inbound Read Request remains on the queue from the time it is received until the time that the last tagged message associated with it is acknowledged.
One characteristic that must be configured for a Queue Pair is the size of the inbound read queue, called the Inbound RDMA Read Queue Depth. Another is a local limit on the number of simultaneous outbound Read Requests, based on the size of the remote endpoint's inbound queue, called the Outbound RDMA Read Queue Depth. The outbound depth is used to limit the number of simultaneous Read Requests so that the local endpoint does not overrun the remote endpoint's queue. Outbound Reads are submitted to the Send Queue at the local peer rather than to a separate outbound read request queue. Their values are currently left to the upper layer, and because this negotiation is typical, a common mechanism is desirable.
The exchange of MPA Request and Reply messages that places a TCP connection in MPA mode is specified in RFC 5044. It gives the hardware several benefits: the upper layer specifies the exact MPA mode, meaning whether markers and the CRC-32c are enabled, and the point in each TCP stream where MPA frames begin; before the first MPA frame is transmitted all pre-MPA payload has been acknowledged by the peer, so it is never necessary to generate a retransmission that mixes pre-MPA and MPA payload; and before MPA reception is enabled all incoming pre-MPA payload has been acknowledged, so the host never receives a TCP segment that mixes pre-MPA and MPA payload.
The limitation is that the exchange defines no Ready to Receive indication, so the passive side cannot know that the last non-MPA payload, the MPA Reply, has been received. Instead the role of that indication is piggybacked on the first MPA frame sent by the active side. For the classic client-server model this is a valuable optimization, since the client only connects when it has a request to send and the server has nothing to send until it has processed the request. Even a server that sends configuration data immediately can send the same information as application private data in the MPA Reply.
Many peer-to-peer applications, especially those involving cluster calculations, have no natural client or server roles. Typically one member of the cluster is arbitrarily selected to initiate the connection when the distributed task is launched, while the other accepts it. At startup time there is no way to predict which node will have the first message to actually send, and establishing the connections immediately is valuable because it reduces latency once results are ready and validates connectivity throughout the cluster. The lack of an explicit Ready to Receive indication forces all applications to have a first message from the connection initiator, whether or not that matches the application model.
The natural question is why the application layer would not simply generate a dummy message when there is nothing else to send. Three factors make that unsuitable for many peer-to-peer applications. First, transport-neutral APIs: only RDDP over TCP has the first message requirement, while RDDP over SCTP and InfiniBand do not, so an extra message required by only one transport undermines the goal of being transport neutral. Second, work and completion queue accounting: a completion queue must be sized for the possibility that every work request completes in error, and inserting a dummy work request and filtering out its completion would require the lower layer to know the application's usage model and how often an insertion would be required. Third, host-based implementation of MPA fencing: when the generation and reception of MPA frames are already dedicated to hardware, a work completion can only be generated by an untagged message, since the arrival of a message for a tagged buffer does not necessarily generate a completion.
The two numbers are the size of the queue that holds incoming read requests and the limit on how many reads may be outstanding. Today each application has to work them out for itself, once for one transport and again for another, and the RFC puts them into the handshake so the same code works everywhere.
The old handshake assumed the caller speaks first, and the responder only learns the caller's side is ready when the first real message arrives. For two peers that is wrong. Sending a dummy message to fill the gap would work, but it would break the promise that the same application code runs over any transport, it would upset the queue sizing the hardware needs, and it would produce a completion that the application did not ask for.
- Transport-neutral APIsOnly RDDP over TCP needs a first message; RDDP over SCTP and InfiniBand do not, so an extra message required by one transport breaks the promise of one application code.
- Queue accountingA dummy work request and the filtering of its completion would force the lower layer to know the application's usage model and how often a dummy would be needed.
- Host-based fencingWhen frame generation is in hardware, only an untagged message produces a work completion, so a dummy message creates a completion the application did not ask for.
§5Enhanced MPA Connection Establishment
The goal of the enhanced connection setup is to allow standard negotiation of the outbound and inbound read queue depth settings on both sides of the RDMA connection, and to negotiate the initial data transfer operation by the initiator when the existing rule that the client sends first does not match the application requirements.
The RDMA connection initiator sends an MPA Request as specified in RFC 5044, in the new format defined by this document, which allows for standardized negotiation of the two queue depths and for negotiation of the Ready to Receive functionality and the RDMA message type to use for that indication. The responder processes the request and generates an MPA Reply, and the new format completes the negotiation.
The local interface needs to provide a way for an upper layer to request the use of an explicit Ready to Receive indication on a per-application or per-connection basis when one will be required. Piggybacking the indication on the client's first message remains a valuable optimization for most connections.
The RDMA connection initiator must not allow any later frames to be transmitted before the Ready to Receive indication. One method of achieving this is to delay notifying the upper layer that the RDMA connection has been established until after any required indication has been transmitted. All MPA exchanges are performed via TCP prior to RDMA establishment, and are therefore signalled via TCP and not via RDMA completion.
The handshake gains a flag saying which side is which and a flag saying which kind of empty message will be used to signal readiness. Once the reply comes back, the initiator sends that empty message before any real data, and the responder now knows the setup is complete.
- MPA RequestThe initiator sets control flag A to 1 for a peer-to-peer model, sets its initial queue depths, and sets flags B, C and D for each readiness indication it supports.
- MPA ReplyThe responder matches the flag A value, sets its own queue depths from the initiator's values and the reads the application needs, and sets the flags for the indications it supports.
- First RDMA messageThe initiator adjusts its queue depths to the responder's values and sends the negotiated readiness indication. If no indication matches, it sends a termination message instead.
§6Enhanced MPA Request and Reply Frames
Enhanced RDMA connection establishment uses an alternate format for MPA Requests and Replies. The 16-byte Key is unchanged, holding the text MPA ID Req Frame for a request and MPA ID Rep Frame for a reply. The M, C and R flags are unchanged. The S flag is new: it is one if the private data begins with the enhanced RDMA connection establishment data, and zero otherwise.
The Res field is one bit smaller than in RFC 5044; otherwise it is unchanged. In RFC 5044 the Res field, in which the newly defined S bit resides, is reserved for future use, and RFC 5044 specifies that it must be set to zero when sending and must not be checked on reception. That is what makes use of the S bit backwards compatible with the original MPA frame format. When the S bit is zero, no additional private data is used for enhanced connection establishment, and the resulting request and reply frames are identical to the unenhanced protocol.
The Rev field contains the revision of MPA. To use any enhanced connection establishment feature it must be set to two or higher. If no enhanced features are desired it may be set to one. A host accepting MPA connections must continue to accept requests with version one, even if it supports version two.
The PD_Length field is unchanged from RFC 5044. It is the total length of the private data field, including the enhanced connection establishment data if present. The private data itself is unchanged, except that if the S flag is set it must begin with the enhanced connection establishment data.
The interoperability rules follow from the S flag. If the responder receives an enhanced MPA message it must respond with an enhanced MPA message. If it receives an unenhanced message, with S set to zero, it must respond with an unenhanced message. If it receives an enhanced message and does not support enhanced connection establishment, it must close the TCP connection and exit MPA; from a standard connection establishment point of view the enhanced frame is improperly formatted, so both sides report TCP connection termination locally. In that last case the initiator may attempt to establish an RDMA connection using the unenhanced protocol if that is compatible with the application, and let the upper layer deal with the queue depths and the peer-to-peer negotiation.
The new format is the old one with a single extra bit. If that bit is zero the frame is byte for byte the old frame, so a peer that knows nothing about this RFC sees a normal handshake and answers normally.
§9The Enhanced Connection Establishment Data
Enhanced RDMA connection establishment places 32 bits at the beginning of the private data field of the MPA Request and Reply frames, or of the DDP Stream Session Initiate and DDP Stream Session Accept SCTP Session Control chunks. Upper-layer private data follows this field, so the maximum amount of upper-layer private data is reduced by 4 bytes. The field must be sent in network byte order, with the two queue depths encoded as 14-bit unsigned integers.
The layout is a control flag A for the connection model, a control flag B for the use of a zero-length frame RTR indication, then 14 bits of inbound read queue depth, then a control flag C for the use of a zero-length RDMA Write RTR indication, a control flag D for the use of a zero-length RDMA Read RTR indication, and then 14 bits of outbound read queue depth.
Four bytes at the front of the private data: two numbers for the queue depths and four one-bit switches, one for the connection model and three for the kinds of empty message that could signal readiness.
| Field | Name | Size | Value | Description |
|---|---|---|---|---|
FLAG-A | A | 1 bits | 1 | Control flag for the connection model. 1 means peer-to-peer and 0 means client-server. |
FLAG-B | B | 1 bits | 1 | Control flag for the use of a zero-length frame RTR indication. |
IRD | IRD | 14 bits | Inbound RDMA Read Queue Depth, as a 14-bit unsigned integer in network byte order. | |
FLAG-C | C | 1 bits | 1 | Control flag for the use of a zero-length RDMA Write RTR indication. |
FLAG-D | D | 1 bits | 1 | Control flag for the use of a zero-length RDMA Read RTR indication. |
ORD | ORD | 14 bits | Outbound RDMA Read Queue Depth, as a 14-bit unsigned integer in network byte order. |
§9.1Negotiating the Queue Depths and the Connection Model
Four terms describe the negotiation. The Initiator IRD is the value sent in the request, the initiator's inbound read queue depth at the time the request was generated. The responder sets its local outbound depth to this value or less, and the initiator IRD is the maximum number of simultaneous inbound Read Requests the initiator can support for the requested connection.
The Initiator ORD is the value in the request. It is the initial value of the initiator's outbound depth and also a request that the responder support a responder IRD of at least this value. It is the maximum number of simultaneous outbound Read operations the initiator wants the responder to support.
The Responder IRD is the value returned in the reply. It is the actual value the responder sets for its local inbound queue depth, and it is greater than or equal to the initiator ORD for a successful negotiation. The Responder ORD is the value returned in the reply, the actual value the responder used for its outbound depth, and it is less than or equal to the initiator IRD for a successful negotiation. After a successful negotiation the responder ORD is at most the initiator IRD and the initiator ORD is at most the responder IRD.
The responder and initiator must pass the peer's provided values to the upper layer as well as using them. The responder outbound depth should be set to a value at most the initiator IRD. If the initiator ORD is insufficient to support the selected connection model, the responder IRD may be increased; for example, if the initiator ORD is 0, meaning the upper layer will not use RDMA Reads, and the responder supports a zero-length RDMA Read RTR indication, the responder IRD can be set to 1. The responder may reject the connection request if the initiator IRD is not sufficient for the required outbound depth, and specify the required value in the reject frame, in which case the termination message must carry layer 2, error type 0, error code 6.
On receiving the accept frame the initiator must set its IRD at least to the responder ORD and its ORD at most to the responder IRD. If the initiator does not have sufficient resources for the required IRD it must send a termination message indicating insufficient resources and terminate the connection, again with layer 2, error type 0, error code 6. The initiator must pass the responder values to the upper layer for both accept and reject messages, and the upper layer can then decide its course of action, for example terminating the connection and renegotiating.
An all-ones value, 0x3FFF, indicates that automatic negotiation is not desired and that the upper layer will be responsible for it. The responder must answer an initiator ORD of 0x3FFF by leaving its local IRD unchanged and setting the IRD to 0x3FFF in its reply, and the initiator must leave its local ORD unchanged on receiving a responder IRD of 0x3FFF. The responder must answer an initiator IRD of 0x3FFF by leaving its local ORD unchanged and setting ORD to 0x3FFF in its reply, and the initiator must leave its local IRD unchanged on receiving a responder ORD of 0x3FFF.
The connection model. Control flag A set to 1 indicates a peer-to-peer model, and 0 a client-server model. Flag B set to 1 indicates that a zero-length frame RTR indication is requested by the initiator and supported by the responder. Flag C does the same for a zero-length RDMA Write RTR indication, and flag D for a zero-length RDMA Read RTR indication. The initiator must set flag A to 1 for the peer-to-peer model, and must set each of flags B, C and D to 1 for each option it supports if flag A is 1. The responder must support at least one RTR indication option if it supports enhanced connection establishment. If flag A is 1 in the request, the responder must set flag A to 1 in the reply. For each initiator-supported option the responder should set the corresponding flag if it can support that option, and it must set at least one option if it supports more than one initiator-specified option. If the responder does not support any of the initiator-specified options it must still set at least one option that it does support.
On receiving the accept frame with flag A set to 1, the initiator must generate one of the negotiated RTR indications. If it cannot generate any of the responder-supported indications it must send a termination message indicating failure to negotiate a mutually compatible connection model or RTR option, with layer 2, error type 0, error code 7. The upper layer can negotiate an upper-layer-level indication when a provider-level one cannot be negotiated.
For the client-server model the initiator must set flag A to 0, the responder must set flag A to 0 if the request has it at 0, and flags B, C and D must also be zero. On reception, if flag A is 0 then flags B, C and D must be ignored.
Each side says how many incoming reads it can queue and how many outgoing reads it wants to make, and the reply settles it so that neither side exceeds the other's queue. A value of all ones in either field means the application wants to handle that itself. One control flag says whether this is a two-equal-peers connection or the old request-and-answer one, and three more say which kinds of empty message are acceptable as the readiness signal.
| Term | Meaning |
|---|---|
| Initiator IRD | The initiator's inbound queue depth sent in the request. The responder sets its local outbound depth to this value or less. |
| Initiator ORD | The initiator's outbound queue depth, and a request that the responder support an inbound depth of at least this value. |
| Responder IRD | The inbound queue depth the responder actually sets, at least the initiator ORD for a successful negotiation. |
| Responder ORD | The outbound queue depth the responder actually uses, at most the initiator IRD for a successful negotiation. |
§7SCTP Chunks and Error Reporting
Enhanced RDMA connection establishment uses the first 32 bits of the private data field for the queue depth negotiation in the DDP Stream Session Initiate and DDP Stream Session Accept SCTP Session Control chunks. The type of an SCTP Session Control chunk is defined by a function code. RFC 5043 already defines codes for DDP Stream Session Initiate and DDP Stream Session Accept, which are equivalent to an MPA Request frame and an accepting MPA Reply frame. Three additional function codes are defined here: Enhanced DDP Stream Session Initiate, 0x005; Enhanced DDP Stream Session Accept, 0x006; and Enhanced DDP Stream Session Reject, 0x007. The enhanced reject code must be used to indicate rejection of an enhanced DDP stream session for a configuration that would have been accepted for unenhanced negotiation.
The enhanced DDP stream session establishment follows the same rules as the standard establishment in RFC 5043. Upper-layer private data must be included for the enhanced initiate, accept and reject messages, and must follow the enhanced connection establishment data in the initiate and accept messages. The private data length must not exceed 512 bytes in any message, including the enhanced data, and private data must not be included in the DDP Stream Session TERM message.
Received enhanced DDP stream session control messages should be reported to the upper layer, and any supplied private data must be available for it to examine. A message is not reported to the upper layer if none of the requested RTR indication types are supported by the receiver, in which case the provider may generate a reject reply message indicating which indication types it does support. Enhanced DDP stream management must use the DDP stream session termination function code to terminate a stream established using the enhanced function codes.
On errors, the protocol is layered on RFC 5040 and RFC 5041, and any enhanced connection establishment error generates an MPA termination message. MPA negotiation uses layer 0x2, the LLP error type 0x0 for MPA. RFC 5044 defines four error codes and RFC 5043 defines none, and this document adds three: 0x05 for a local catastrophic error, 0x06 for insufficient IRD resources, and 0x07 for no matching RTR option. That keeps enhanced connection establishment backward compatible with both RFC 5043 and RFC 5044.
The same negotiation exists for the SCTP version of RDMA, using three new chunk types. When something goes wrong the peers exchange a termination message carrying a code that says which of three things went wrong: something local broke, there were not enough queue resources, or no readiness signal was agreed on.
- 0x05Local catastrophic — A local error was encountered for which no specific error code is defined.
- 0x06Insufficient IRD resources — The peer's inbound queue depth was not sufficient for the outbound depth the application needs.
- 0x07No matching RTR option — No mutually compatible connection model or readiness indication could be negotiated.
§12Security Considerations
The security considerations from RFC 5044 and RFC 5043 apply, and the changes in this document introduce no new security considerations. It is, however, recommended that implementations sanity check the input parameters, including the two queue depths and the control flags used for the RTR indication option negotiation.
No new security rules. The RFC does ask implementations to check the numbers and flags they receive rather than trusting them, since a bad value could cause trouble later.