Trailer encapsulation: Trailer encapsulations
A link-level trick for putting the protocol headers at the end of the packet.
Leffler, S. and M.J. Karels, "Trailer encapsulations", RFC 893, DOI 10.17487/RFC0893, April 1984, <https://www.rfc-editor.org/info/rfc893>.
@misc{rfc893,
series = {Request for Comments},
number = 893,
howpublished = {RFC 893},
publisher = {RFC Editor},
doi = {10.17487/RFC0893},
url = {https://www.rfc-editor.org/info/rfc893},
author = {S. Leffler and M.J. Karels},
title = {{Trailer encapsulations}},
pagetotal = 6,
year = 1984,
month = apr,
} RFC text
Contents
Network Working Group Samuel J. Leffler
Request for Comments: 893 Michael J. Karels
University of California at Berkeley
April 1984
Trailer Encapsulations
Status of this Memo
This RFC discusses the motivation for use of "trailer encapsulations"
on local-area networks and describes the implementation of such an
encapsulation on various media. This document is for information
only. This is NOT an official protocol for the ARPA Internet
community.
Introduction
A trailer encapsulation is a link level packet format employed by
4.2BSD UNIX (among others). A trailer encapsulation, or "trailer",
may be generated by a system under certain conditions in an effort to
minimize the number and size of memory-to-memory copy operations
performed by a receiving host when processing a data packet.
Trailers are strictly a link level packet format and are not visible
(when properly implemented) in any higher level protocol processing.
This note cites the motivation behind the trailer encapsulation and
describes the trailer encapsulation packet formats currently in use
on 3 Mb/s Experimental Ethernet, 10 Mb/s Ethernet, and 10 Mb/s V2LNI
ring networks [1].
The use of a trailer encapsulation was suggested by Greg Chesson, and
the encapsulation described here was designed by Bill Joy.
Motivation
Trailers are motivated by the overhead which may be incurred during
protocol processing when one or more memory to memory copies must be
performed. Copying can be required at many levels of processing,
from moving data between the network medium and the host's memory, to
passing data between the operating system and user address spaces.
An optimal network implementation would expect to incur zero copy
operations between delivery of a data packet into host memory and
presentation of the appropriate data to the receiving process. While
many packets may not be processed without some copying operations,
when the host computer provides suitable memory management support it
may often be possible to avoid copying simply by manipulating the
appropriate virtual memory hardware.
In a page mapped virtual memory environment, two prerequisites are
usually required to achieve the goal of zero copy operations during
packet processing. Data destined for a receiving agent must be
aligned on a page boundary and must have a size which is a multiple
of the hardware page size (or filled to a page boundary). The latter
restriction assumes virtual memory protection is maintained at the
page level; different architectures may alter these prerequisites.
Data to be transmitted across a network may easily be segmented in
the appropriate size, but unless the encapsulating protocol header
information is fixed in size, alignment to a page boundary is
virtually impossible. Protocol header information may vary in size
due to the use of multiple protocols (each with a different header),
or it may vary in size by agreement (for example, when optional
information is included in the header). To insure page alignment the
header information which prefixes data destined for the receiver must
be reduced to a fixed size; this is normally the case at the link
level of a network. By taking all (possibly) variable length header
information and moving it after the data segment a sending host may
"do its best" in allowing the receiving host the opportunity to
receive data on a page aligned boundary. This rearrangement of data
at the link level to force variable length header information to
"trail" the data is the substance of the trailer encapsulation.
There are several implicit assumptions in the above argument.
1. The receiving host must be willing to accept trailers. As this
is a link level encapsulation, unless a host to host negotiation
is performed (preferably at the link level to avoid violating
layering principles), only certain hosts will be able to converse,
or their communication may be significantly impaired if trailer
packets are mixed with non-trailer packets.
2. The cost of receiving data on a page aligned boundary should be
comparable to receiving data on a non-page aligned boundary. If
the overhead of insuring proper alignment is too high, the savings
in avoiding copy operations may not be cost effective.
3. The size of the variable length header information should be
significantly less than that of the data segment being
transmitted. It is possible to move trailing information without
physically copying it, but often implementation constraints and
the characteristics of the underlying network hardware preclude
merely remapping the header(s).
4. The memory to memory copying overhead which is expected to be
performed by the receiver must be significant enough to warrant
the added complexity in the both the sending and receiving host
software.
The first point is well known and the motivation for this note.
Thought has been given to negotiating the user of trailers on a per
host basis using a variant of the Address Resolution Protocol [2]
(actually augmenting the protocol), but at present all systems using
trailers require hosts sharing a network medium to uniformly accept
trailers or never transmit them. (The latter is easily carried out
at boot time in 4.2BSD without modifying the operating system source
code.)
The second point is (to our knowledge) insignificant. While a host
may not be able to take advantage of the alignment and size
properties of a trailer packet, it should nonetheless never hamper
it.
Regarding the third point, let us assume the trailing header
information is copied and not remapped, and consider the header
overhead in the TCP/IP protocols as a representative example [3]. If
we assume both the TCP and IP protocol headers are part of the
variable length header information, then the smallest trailer packet
(generated by a VAX) would have 512 bytes of data and 40+ bytes of
header information (plus the trailer header described later). While
the trailing header could have IP and/or TCP options included this
would normally be rare (one would expect most TCP options, for
example, to be included in the initial connection setup exchange) and
certainly much smaller than 512 bytes. If the data segment is
larger, the ratio decreases and the expected gain due to fewer copies
on the receiving end increases. Given the relative overheads of a
memory to memory copy operation and that of a page map manipulation
(including translation buffer invalidation), the advantage is
obvious.
The fourth issue, we believe, is actually a non-issue. In our
implementation the additional code required to support the trailer
encapsulation amounts to about a dozen lines of code in each link
level "network interface driver". The resulting performance
improvement more than warrants this minor investment in software.
It should be recognized that modifying the network (and normal link)
level format of a packet in the manner described forces the receiving
host to buffer the entire packet before processing. Clever
implementations may parse protocol headers as the packet arrives to
find out the actual size (or network level packet type) of an
incoming message. This allows these implementations to avoid
preallocating maximum sized buffers to incoming packets which it can
recognize as unacceptable. Implementations which parses the network
level format on the fly are violating layering principles which have
been extolled in design for some time (but often violated in
implementation). The problem of postponing link level type
recognition is a valid criticism. In the case of network hardware
which supports DMA, however, the entire packet is always received
before processing begins.
Trailer Encapsulation Packet Formats
In this section we describe the link level packet formats used on the
3 Mb/s Experimental Ethernet, and 10 Mb/s Ethernet networks as well
as the 10 Mb/s V2LNI ring network. The formats used in each case
differ only in the format and type field values used in each of the
local area network headers.
The format of a trailer packet is shown in the following diagram.
+----+-------------------------------------------------+----+
| LH | data | TH |
+----+-------------------------------------------------+----+
^ ( ^ ) ^
LH:
The fixed-size local network header. For 10 a Mb/s Ethernet,
the 16-byte Ethernet header. The type field in the header
indicates that both the packet type (trailer) and the length of
the data segment.
For the 10 Mb/s Ethernet, the types are between 1001 and 1010
hexadecimal (4096 and 4112 decimal). The type is calculated as
1000 (hex) plus the number of 512-byte pages of data. A
maximum of 16 pages of data may be transmitted in a single
trailer packet (8192 bytes).
data:
The "data" portion of the packet. This is normally only data
to be delivered to the receiving processes (i.e. it contains no
TCP or IP header information). Data size is always a multiple
of 512 bytes.
TH:
The "trailer". This is actually a composition of the original
protocol headers and a fixed size trailer prefix which defines
the type and size
of the trailing data. The format of a trailer is shown below.
The carats (^) indicate the page boundaries on which the receiving
host would place its input buffer for optimal alignment when
receiving a trailer packet. The link level receiving routine is able
to locate the trailer using the size indicated in the link level
header's type field. The receiving routine is expected to discard
the link level header and trailer prefix, and remap the trailing data
segment to the front of the packet to regenerate the original network
level packet format.
Trailer Format
+----------------+----------------+------~...~----------+
| TYPE | HEADER LENGTH | ORIGINAL HEADER(S) |
+----------------+----------------+------~...~----------+
Type: 16 bits
The type field encodes the original link level type of the
transmitted packet. This is the value which would normally be
placed in the link level header if a trailer were not generated.
Header length: 16 bits
The header length field of the trailer data segment. This
specifies the length in bytes of the following header data.
Original headers: <variable length>
The header information which logically belongs before the data
segment. This is normally the network and transport level
protocol headers.
Summary
A link level encapsulation which promotes alignment properties
necessary for the efficient use of virtual memory hardware facilities
has been described. This encapsulation format is in use on many
systems and is a standard facility in 4.2BSD UNIX. The encapsulation
provides an efficient mechanism by which cooperating hosts on a local
network may obtain significant performance improvements. The use of
this encapsulation technique currently requires uniform cooperation
from all hosts on a network; hopefully a per host negotiation
mechanism may be added to allow consenting hosts to utilize the
encapsulation in a non-uniform environment.
References
[1] "The Ethernet - A Local Area Network", Version 1.0, Digital
Equipment Corporation, Intel Corporation, Xerox Corporation,
September 1980.
[2] Plummer, David C., "An Ethernet Address Resolution Protocol",
RFC-826, Symbolics Cambridge Research Center, November 1982.
[3] Postel, J., "Internet Protocol", RFC-791, USC/Information
Sciences Institute, September 1981.
ISO Plain Language
On this page
§1What a Trailer Encapsulation Is
A trailer encapsulation is a link level packet format employed by 4.2BSD UNIX and others. A trailer, or trailer encapsulation, may be generated by a system under certain conditions in an effort to minimise the number and size of memory-to-memory copy operations performed by a receiving host when processing a data packet. Trailers are strictly a link level packet format and are not visible, when properly implemented, in any higher level protocol processing.
The note cites the motivation behind the trailer encapsulation and describes the packet formats used on 3 Mb/s Experimental Ethernet, 10 Mb/s Ethernet and 10 Mb/s V2LNI ring networks. The formats differ only in the format and type field values used in each of the local area network headers. The use of a trailer encapsulation was suggested by Greg Chesson, and the encapsulation described here was designed by Bill Joy.
The status of the document is worth stating plainly, because it is unusual: this RFC discusses the motivation for using trailer encapsulations and describes the implementation of such an encapsulation on various media. This document is for information only, and is explicitly not an official protocol for the ARPA Internet community.
A trailer encapsulation is a way of writing a packet so that the receiving machine can hand the data to the program that wants it without copying it first. The headers go at the end of the packet instead of the beginning. This is a local network technique, and the RFC says plainly that it is not an Internet standard.
§2Why Trailers Exist
Trailers are motivated by the overhead that may be incurred during protocol processing when one or more memory to memory copies must be performed. Copying can be required at many levels, from moving data between the network medium and the host's memory to passing data between the operating system and user address spaces. An optimal network implementation would expect to incur zero copy operations between delivery of a data packet into host memory and presentation of the appropriate data to the receiving process. While many packets may not be processed without some copying, when the host computer provides suitable memory management support it may often be possible to avoid copying simply by manipulating the appropriate virtual memory hardware.
In a page mapped virtual memory environment, two prerequisites are usually required to achieve zero copy operations. Data destined for a receiving agent must be aligned on a page boundary and must have a size that is a multiple of the hardware page size, or be filled to a page boundary. Data to be transmitted may easily be segmented into the appropriate size, but unless the encapsulating protocol header information is of fixed size, alignment to a page boundary is virtually impossible. Header information may vary in size because of the use of multiple protocols, each with a different header, or by agreement, for example when optional information is included. To ensure page alignment, the header information that prefixes the data must be reduced to a fixed size, which is normally the case at the link level. Taking all the possibly variable length header information and moving it after the data segment lets the sending host give the receiving host the opportunity to receive data on a page aligned boundary. That rearrangement is the substance of the trailer encapsulation.
The argument rests on several implicit assumptions. The receiving host must be willing to accept trailers, and since this is a link level encapsulation, unless a host to host negotiation is performed, only certain hosts will be able to converse, or their communication may be significantly impaired if trailer packets are mixed with non-trailer packets. The cost of receiving data on a page aligned boundary should be comparable to receiving data on a non-page aligned boundary. The size of the variable length header information should be significantly less than that of the data segment, because although it is possible to move trailing information without physically copying it, implementation constraints and the characteristics of the network hardware often preclude merely remapping the headers. And the copying overhead expected at the receiver must be significant enough to warrant the added complexity in both the sending and receiving host software.
The note examines each assumption. The first is well known and is the motivation for the document. The second is, to the authors' knowledge, insignificant. On the third, taking TCP and IP headers as the variable length information, the smallest trailer packet generated by a VAX would have 512 bytes of data and 40 or more bytes of header information, plus the trailer header; the larger the data segment, the better the ratio. The fourth they believe to be a non-issue: the additional code required to support the encapsulation amounts to about a dozen lines in each link level network interface driver.
One cost is acknowledged. Modifying the network and normal link level format of a packet in this manner forces the receiving host to buffer the entire packet before processing it. Clever implementations may parse protocol headers as the packet arrives to find the actual size or network level packet type, avoiding preallocating maximum sized buffers to packets they can recognise as unacceptable, but implementations that parse the network level format on the fly are violating layering principles. The problem of postponing link level type recognition is a valid criticism, though for network hardware that supports DMA the entire packet is always received before processing begins anyway.
The trick only pays off if the data can land in memory exactly where the receiving program will read it, which in practice means starting on a page boundary. A header of unpredictable length in front of the data makes that impossible, so the header moves to the back and the data is padded to a round size. The RFC lists the conditions under which this is worth doing, and the cost, which is that the receiver must hold the whole packet before it can start on it.
- Every host must agreeUnless hosts negotiate, only some will understand trailers, and mixing trailer and non-trailer packets impairs communication.
- Alignment must be cheapReceiving data on a page boundary should cost about the same as receiving it anywhere else.
- Headers must be smallThe trailing header should be much smaller than the data segment, since headers often cannot simply be remapped.
- Copying must be worth avoidingThe copying overhead at the receiver has to be significant enough to justify the extra code on both sides.
§3The Trailer Packet Format
A trailer packet has three parts: a fixed-size local network header, the data, and the trailer. For 10 Mb/s Ethernet the local network header is the 16-byte Ethernet header, and the type field in that header indicates both that the packet is a trailer and the length of the data segment. For 10 Mb/s Ethernet the types are between 1001 and 1010 hexadecimal, that is 4096 and 4112 decimal, and the type is calculated as 1000 in hexadecimal plus the number of 512-byte pages of data. A maximum of 16 pages of data may be transmitted in a single trailer packet, which is 8192 bytes.
The data portion of the packet is normally only data to be delivered to the receiving processes, so it contains no TCP or IP header information, and its size is always a multiple of 512 bytes. The trailer is a composition of the original protocol headers and a fixed size trailer prefix that defines the type and size of the trailing data.
The receiving side works from the length already in the header. The link level receiving routine is able to locate the trailer using the size indicated in the link level header's type field. It is expected to discard the link level header and the trailer prefix, and to remap the trailing data segment to the front of the packet, which regenerates the original network level packet format.
The packet has a normal link-level header, then the data, then the trailer. The link header's type field says how much data there is, so the receiver knows where the trailer starts, and it can then move the data back to the front of the packet to recover the original layout.
+----+-------------------------------------------------+----+
| LH | data | TH |
+----+-------------------------------------------------+----+
^ ( ^ ) ^§4The Trailer Header
The trailer itself has three fields. The type field is 16 bits and encodes the original link level type of the transmitted packet: the value that would normally be placed in the link level header if a trailer were not generated. The header length field is 16 bits and specifies the length in bytes of the header data that follows it. The original headers are of variable length and hold the header information that logically belongs before the data segment, normally the network and transport level protocol headers.
The trailer carries three things: what the packet type would have been, how many bytes of headers follow, and those headers themselves.
- Type16 bits. The link level type the packet would have carried if a trailer had not been generated.
- Header length16 bits. The length in bytes of the header data that follows.
- Original headersVariable length. The network and transport level headers that logically belong in front of the data.
§5Summary and Status
The note summarises a link level encapsulation which promotes the alignment properties necessary for the efficient use of virtual memory hardware facilities. The encapsulation format is in use on many systems and is a standard facility in 4.2BSD UNIX. It provides an efficient mechanism by which cooperating hosts on a local network may obtain significant performance improvements.
The use of the encapsulation technique currently requires uniform cooperation from all hosts on a network; the authors hope that a per host negotiation mechanism may be added, allowing consenting hosts to use the encapsulation in a non-uniform environment. Thought had been given to negotiating the use of trailers on a per host basis using a variant of the Address Resolution Protocol, but at present all systems using trailers require hosts sharing a network medium to uniformly accept trailers or never transmit them, and the latter is easily arranged at boot time in 4.2BSD without modifying the operating system source code.
All the machines on the segment have to agree: either they all understand trailers or none of them send them. The RFC notes that a per-host negotiation would be better, and that the designers did think about it, but it was not built.