RFC 632 UNKNOWN PERFORMANCE · FLOW CONTROL
Throughput degradations for single packet messages
Why out-of-order single-packet messages could reduce ARPANET throughput.
RFC EDITOR
Opderbeck, H., "Throughput degradations for single packet messages", RFC 632, DOI 10.17487/RFC0632, May 1974, <https://www.rfc-editor.org/info/rfc632>.
BIBTEX
@misc{rfc632,
series = {Request for Comments},
number = 632,
howpublished = {RFC 632},
publisher = {RFC Editor},
doi = {10.17487/RFC0632},
url = {https://www.rfc-editor.org/info/rfc632},
author = {H. Opderbeck},
title = {{Throughput degradations for single packet messages}},
pagetotal = 6,
year = 1974,
month = may,
} RFC text
Contents
Network Working Group H. Opderbeck
Request for Comments: #632 UCLA-NMC
NIC # 30239 20 May 1974
Throughput Degradations for Single Packet Messages
The transmission of digitized speech over the ARPANET represents a new
dimension in the use of packet switching systems. The throughput and
delay requirements for this newly emerging application area are quite
different from the throughput and delay requirements for interactive use
or file transfers. In particular, we need to achieve a high throughput
for small messages since long messages result in long source delays to
fill the large buffers. Therefore we are currently studying the
throughput limits for single-packet messages. We realize that up to now
little attempt was made to optimize throughput for low delay traffic.
It was nevertheless surprising for us to find out that the observed
throughput for single-packet messages is in many cases only about one
fourth of what one would expect. In what follows we are going to
explain why this happens and what could be done to correct this
situation.
On April 1, 1974, we sent, using the IMP message generator, single-
packet messages at the highest possible rate ("RFNM-driven") from the
MOFFET-IMP to the SRI-IMP. There are two three-hop paths from MOFFET to
SRI, one of them involving two 230.4 kbs circuits. Since there was
hardly any interfering traffic we expected an average round-trip delay
of not more than 100 msec. Assuming that there are, on an average, 3
messages in transmission between MOFFET and SRI and assuming a message
length of about 1000 bits this should result in a throughout of more
than 30 kbs. The observed through was, however, less than 8 kbs. A
repetition of the experiment showed the same result. A more detailed
analysis of the collected data revealed that an average number of 3.5
messages were simultaneously in transmission between MOFFET and SRI.
The throughput degradation could therefore not have been due to
interfering traffic between these two sites. Also the channel
utilization for all channels that were involved in the transmission was
less than 40 percent. The observed mean round-trip times between MOFFET
and SRI, however, were about 500 msec. Since these large round-trip
times were obviously not due to physical limitations, we studied the
flow control mechanism for single-packet messages and were able to come
up with an explanation for this undesirable behavior.
When a single-packet message arrives at the destination IMP out of order
(i.e., the logically preceding message has not yet arrived there) it is
not accepted by the destination IMP. It is rather treated as a request
for the allocation of one reassembly buffer. The corresponding ALLOCATE
is then sent back to the source IMP only after the RFNM for the previous
message has been processed. We therefore may have the following
sequence of events:
1 MSG(i) sent from SOURCE-IMP (message i is sent from the source
IMP to the destination IMP).
2 MSG(i+1) sent from SOURCE-IMP.
3 MSG(i+1) arrives at DEST-IMP (due to an alternate path or a line
error, message (i+1) arrives at the destination IMP out of
order; it is treated as a request for one reassembly buffer
allocation and then discarded).
4 MSG(i) arrives at DEST-IMP (message i arrives at the destination
IMP; it is put on the proper HOST output queue).
5 RFNM(i) sent from DEST-IMP (after message i has been accepted by
the destination HOST the RFNM is sent to the source IMP).
6 ALL(i+1) sent from DEST-IMP (only after the RFNM for message i
has been processed can the ALLOCATE for message i + 1 be sent).
7 RFNM(i) arrives at SOURCE-IMP.
8 ALL(i+1) arrives at SOURCE-IMP.
9 MSG(i+1) arrives at DEST-IMP (now message i+1 is put on the
proper HOST output queue.)
10 RFNM(i+1) sent form DEST-IMP.
11 RFNM(i+1) arrives at SOURCE-IMP.
Note that the round-trip time for message i+1 is the time interval
between event 2 and event 11. Therefore the round-trip time for message
i+1 is more than twice as large as it would have been if it had arrived
in order, other conditions being unchanged. Therefore a line error will
in many cases not only delay the message in error but also the next
single-packet message if this message follows the preceding message
within 125 msec, the error retransmission timeout interval. Also, a
faster, alternate path to the destination IMP can actually slow down the
transmission since it causes messages to arrive there out of order.
This situation becomes even worse when we consider RFNM-driven single-
packet message traffic. Table 1 shows a possible sequence of events.
We again assume that message i+1 reaches the destination IMP before
message i.
SOURCE IMP DESTINATION IMP
MSG(i) sent
MSG(i+1) sent
MSG(i+2) sent MSG(i+1) arr
MSG(i+3) sent MSG(i) arr
RFNM(i) sent
ALL(i+1) sent
MSG(i+2) arr
MSG(i+3) arr
RFNM(i) arr
MSG(i+4) sent
ALL(i+1) arr MSG(i+4) arr
MSG(i+1) sent MSG(i+1) arr
RFNM(i+1) sent
RFNM(i+1) arr ALL(i+2) sent
MSG(i+5) sent
ALL(i+2) arr MSG(i+5) arr
MSG(i+2) sent MSG(i+2) arr
RFNM(i+2) sent
RFNM(i+2) arr ALL(i+3) sent
MSG(i+6) sent
ALL(i+3) arr MSG(i+6) arr
MSG(i+3) sent MSG(i+3) arr
RFNM(i+3) sent
RFNM(i+3) arr ALL(i+4) ent
MSG(i+7) sent
ALL(i+4) arr MSG(i+7) arr
MSG(i+4) sent MSG(i+4) arr
RFNM(i+4) sent
RFNM(i+4) arr ALL(i+5)
MSG(i+8) sent
ALL(i+5) arr
MSG(i+5) sent
Table 1.
Retransmission Pattern for the Current Flow Control Scheme
(Since the traffic is RFNM-driven, the arrival of RFNM i, i+1, ... is
followed by the sending of message i+4, i+5, ...)
The most interesting fact about this sequence of events is that the
arrival of message i+1 before message i at the destination IMP causes
not only messages i+1 but all future messages to be retransmitted--
though we do not assume that any of the future messages arrive out of
order. The table also shows that the round-trip times for message i+4
and all future messages is more than four times as large as it would be
without these undesirable retransmissions. It is also noteworthy that,
once this retransmission pattern has established itself, there is almost
no way the system can recover from this condition other than
interrupting the input stream at the source IMP. A single arrival out
of order of any of the later user or control messages, for instance,
will not change this retransmission pattern. The normal flow of
single-packet messages will only reestablish itself if, for example,
message i+4, i+5, and i+6 are simultaneously delayed for several hundred
milliseconds such that messages i+1, i+2, and i+3 can be retransmitted
in the meantime. The probability of occurrence of such an event is,
however, extremely small. Therefore one can consider the system as
being trapped in this undesirable retransmission condition. The
"normal" flow of messages, on the other hand, represents only the
transient behavior of the system since there is always a finite
probability that two message arrive out of order due to transmission
errors.
As mentioned before, the system can only recover from this throughput
(and delay) degradation if the input stream of single-packet messages is
interrupted. In case of speech transmission, however, this might not
occur for a long time. Therefore speech transmission systems would in
many cases have to work with only one fourth of the expected single-
packet bandwidth. Since this is clearly an unacceptable condition we
looked for a modification of the current flow control scheme which
corrects this situation. In what follows we describe two methods that
could be used to avoid the undesirable retransmission of messages.
Recall that a single-packet message is rejected at destination IMP and
later retransmitted if the RFNM for the preceding message has not yet
been sent to the source IMP. This is mainly done to prevent the
occurrence of reassembly lockup conditions [1]. Therefore the problem
cannot be solved by simply accepting all single-packet messages without
additional measures to prevent deadlocks. This could lead to a
reassembly lockup if a large number of single-packet messages from
several source IMPs arrives at their common destination IMP out of
order. In this case the destination IMP might not be able to accept
those messages that are in order because of the lack of reassembly
buffers. As a result the system is deadlocked. Any solution of the
throughput degradation problem must guarantee that all messages that
arrive in order can be accepted by the destination IMP.
One way to achieve this goal is to reject single-packet messages that
arrive out of order only if the buffer requirement(s) of the preceding
messages(s) is not known. In the previous examples we have seen that
the destination IMP continuously rejected messages although it knew the
buffer requirements for the messages that had to be delivered first. As
the buffer requirements become known, the necessary number of buffers
can be set aside and future single-packet messages can be accepted
without the danger of deadlock. Therefore the undesirable retransmission
pattern cannot establish itself. Table 2 shows the sequence of events
for this policy if message i+1 arrives before message i at the
destination IMP.
SOURCE IMP DESTINATION IMP
MSG (i) sent
MSG(i+1) sent
MSG(i+2) sent MSG(i+1) arr. (rejected)
MSG(i+3) sent MSG(i) arr. (HOST output)
RFNM(i) sent
ALL (i+1) sent
MSG(i+2) arr (stored)
MSG(i+3) arr (stored)
RFNM(i) arr
MSG(i+4) sent
ALL(i+1) arr MSG(i+4) arr (stored)
MSG(i+1) sent MSG(i+1) arr (HOST output)
RFNM(i+1) sent
RFNM(i+1) arr RFNM(i+2) sent
MSG(i+5) sent RFNM(i+3) sent
RFNM(i+4) sent
Table 2.
Sequence of Events for Modified Flow Control Scheme
Note that in this modified scheme only one message, message i+1, is
retransmitted. In view of the fact that the IMPs have plenty of
reassembly buffer space it is, however, desirable to avoid this one
retransmission, too. This is particularly important for the
transmission of speech which depends on a steady flow of data and will
be disrupted by a sudden large delay. Therefore we suggest a second
method to solve the throughput degradation problem which, in most cases,
will not require any retransmissions.
Suppose all single-packet messages are initially accepted (or stored).
Currently single-packet messages that arrive out of order are rejected
because of the possibility of a deadlock. But let us take a closer look
at the situation where all single-packet messages are accepted (or
stored) such that there is no reassembly buffer available for messages
that have to be delivered to their HOST(s) next. This is not really a
lockup condition because the source IMPs keep a copy of all single-
packet messages for which a RFNM has not yet been received. Therefore
any single-packet message, which arrived out of order but was accepted
by the destination IMP nevertheless, can be deleted later without the
message being lost. The destination IMP only has to send an ALLOCATE
for each deleted single-packet message to the corresponding source IMP
when reassembly buffer space is available. This can also be considered
as deferred rejection. But now a retransmission is only necessary if
the destination IMP is really running out of reassembly buffers. In
this case, the physical limitations of the system are reached and we
cannot hope to gain large throughput increases by means of protocol
changes.
It is our intention to pursue this issue with the IMP development group
at BBN in the future. They agree that the two solutions we suggest
would improve the situation. However, they can think of alternative
solutions.
I acknowledge the help of Bill Naylor and Joe Katz in performing the
experiments which led to the discovery of the throughput degradation.
References:
[1] Kleinrock, L. and H. Opderbeck. "On a Possible Lockup condition
in the IMP Subnet Due to Message Sequencing", RFC #626.
[ This RFC was put into machine readable form for entry ]
[ into the online RFC archives by Alex McKenzie with ]
[ support from GTE, formerly BBN Corp. 11/99 ]
ISO Plain Language
On this page
PROBLEMA test of high-rate single-packet traffic found much lower throughput than expected, a concern for applications such as digitized speech that need fast delivery of small messages.
MECHANISMWhen a destination IMP receives a single-packet message out of order, it treats it as a buffer request and waits for the preceding message's RFNM; in RFNM-driven traffic, this can trigger repeated retransmissions.
THE PROPOSED CHANGESThe RFC describes two possible flow-control changes, but says the authors intended to pursue the issue with the IMP development group.
§1The Single-Packet Throughput Test
The authors were studying small-message throughput for applications such as digitized speech. In a test sending single-packet messages from MOFFET to SRI, they expected more than 30 Kbps but observed less than 8 Kbps. They measured about 3.5 messages in transmission at once and channel utilization below 40%, while mean round-trip time was about 500 milliseconds.
The RFC says the unexpectedly low throughput was not explained by interfering traffic or channel utilization, so the authors examined the flow-control mechanism.
The test moved fewer small messages per second than expected even though the channels were not heavily used. The authors traced the problem to how the IMPs handled messages arriving out of order.
§2How One Out-of-Order Message Delays Others
If a single-packet message arrives before its logical predecessor, the destination IMP does not accept it immediately. It treats it as a request for a reassembly buffer and sends an allocation only after processing the earlier message's RFNM. With RFNM-driven traffic, later messages may already have reached the destination IMP and be waiting for allocations.
The RFC's example shows that one out-of-order arrival can cause later messages to be retransmitted too, even if they arrived in order. Once this pattern starts, the authors say it may persist until the source stream is interrupted.
A message that arrives too early waits for its predecessor. In a fast stream, that delay can lead to repeated retransmissions of messages that follow it.
§3Two Possible Flow-Control Changes
The first proposed change would reject an out-of-order single-packet message only while the buffer needs of earlier messages are unknown. Once those needs are known and buffers are set aside, later messages could be accepted safely; in the example, only one message needs retransmission.
The second proposal would accept or store all single-packet messages at first, then delete one if the destination IMP needs its buffer for an earlier message. Because the source IMP retains unacknowledged single-packet messages, the destination could later send an allocation for a deleted message. The authors say they planned to discuss these options with the IMP development group.
One option accepts out-of-order messages once earlier buffer needs are clear. Another stores them when possible and asks for retransmission only if a buffer is truly needed for an earlier message.