TRILL Directory: Directory Assistance Problem and High-Level Design Proposal
Using a directory of where servers are placed to stop a TRILL network flooding frames it does not need to.
Dunbar, L., Eastlake 3rd, D., Perlman, R., and I. Gashinsky, "Directory Assistance Problem and High-Level Design Proposal", RFC 7067, DOI 10.17487/RFC7067, November 2013, <https://www.rfc-editor.org/info/rfc7067>.
@misc{rfc7067,
series = {Request for Comments},
number = 7067,
howpublished = {RFC 7067},
publisher = {RFC Editor},
doi = {10.17487/RFC7067},
url = {https://www.rfc-editor.org/info/rfc7067},
author = {L. Dunbar and D. Eastlake 3rd and R. Perlman and I. Gashinsky},
title = {{Directory Assistance Problem and High-Level Design Proposal}},
pagetotal = 15,
year = 2013,
month = nov,
} RFC text
Contents
Internet Engineering Task Force (IETF) L. Dunbar
Request for Comments: 7067 D. Eastlake
Category: Informational Huawei
ISSN: 2070-1721 R. Perlman
Intel
I. Gashinsky
Yahoo
November 2013
Directory Assistance Problem and High-Level Design Proposal
Abstract
Edge TRILL (Transparent Interconnection of Lots of Links) switches
currently learn the mapping between MAC (Media Access Control)
addresses and their egress TRILL switch by observing the data packets
they ingress or egress or by the TRILL ESADI (End-Station Address
Distribution Information) protocol. When an ingress TRILL switch
receives a data frame for a destination address (MAC&Label) that the
switch does not know, the data frame is flooded within the frame's
Data Label across the TRILL campus.
This document describes the framework for using directory services to
assist edge TRILL switches in reducing multi-destination frames,
particularly unknown unicast frames flooding, and ARP/ND (Address
Resolution Protocol / Neighbor Discovery), thus improving TRILL
network scalability and security.
Status of This Memo
This document is not an Internet Standards Track specification; it is
published for informational purposes.
This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Not all documents
approved by the IESG are a candidate for any level of Internet
Standard; see Section 2 of RFC 5741.
Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc7067.
Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction ....................................................3
2. Terminology .....................................................4
3. Impact of Massive Number of End Stations ........................5
3.1. Issues of Flooding-Based Learning in Data Centers ..........5
3.2. Two Examples ...............................................6
4. Benefits of Directory-Assisted TRILL Edge .......................7
5. Generic Operation of Directory Assistance .......................8
5.1. Information in Directory for Edge RBridges .................8
5.2. Push Model and Requirements ................................9
5.3. Pull Model and Requirements ...............................11
6. Recommendation .................................................12
7. Security Considerations ........................................12
8. Acknowledgements ...............................................13
9. Informative References .........................................14
1. Introduction
Edge TRILL (Transparent Interconnection of Lots of Links) switches
(devices implementing [RFC6325], also known as RBridges) currently
learn the mapping between destination MAC addresses and their egress
TRILL switch by observing data packets or by the ESADI (End-Station
Address Distribution Information) protocol. When an ingress RBridge
(Routing Bridge) receives a data frame for a destination address
(MAC&Label) that RBridge does not know, the data frame is flooded
within that Data Label across the TRILL campus. (Data Labels are
VLANs or FGLs (Fine-Grained Labels [FGL]).
This document describes a framework for using directory services in
environments where such services are available, such as typical data
centers, to assist edge TRILL switches. This assistance can reduce
multi-destination frames, particularly ARP [RFC826], ND [RFC4861],
and unknown unicast, thus improving TRILL network scalability. In
addition, the information provided by a directory can be more secure
than that learned from the data plane (see Section 7).
Data centers, especially Internet and/or multi-tenant data centers,
tend to have a large number of end stations with a wide variety of
applications. Their networks differ from enterprise campus networks
in several ways that make them attractive for the use of directory
assistance, in particular:
1. Data center topology is often based on racks and rows.
Furthermore, a Server/VM (virtual machine) Management System
orchestrates the assignment of guest operating systems to servers,
racks, and rows; it is not done at random. So, the information
necessary for a directory is normally available from that
Management System.
2. Rapid workload shifting in data centers can accelerate the
frequency of the physical servers being reloaded with different
applications. Sometimes, applications loaded into one physical
server at different times can belong to different subnets. When a
VM is moved to a new location or when a server is loaded with a
new application with different IP/MAC addresses, it is more likely
that the destination address of data packets sent out from those
VMs are unknown to their attached edge RBridges.
3. With server virtualization, there is an increasing trend to
dynamically create or delete VMs when the demand for resources
changes, to move VMs from overloaded servers to less loaded
servers, or to aggregate VMs onto fewer servers when demand is
light. This results in the more frequent occurrence of multiple
subnets on the same port at the same time and a higher change rate
for VMs than for physical servers.
Both items 2 and 3 above can lead to applications in one subnet being
placed in different locations (racks or rows) or one rack having
applications belonging to different subnets.
2. Terminology
The terms "VLAN" and "Data Label" are used interchangeably with
"Subnet" in this document, because it is common to map one subnet to
one VLAN or FGL.
Bridge: Device compliant with IEEE Std 802.1Q-2011 [802.1Q].
Data Label: VLAN or FGL
EoR: End-of-Row switches in a data center. Also known as
aggregation switches.
End Station: Guest OS running on a physical server or on a virtual
machine. An end station in this document has at least one
IP address, at least one MAC address, and is connected to a
network.
FGL: Fine-Grained Label [FGL]
IS-IS: Intermediate System to Intermediate System routing protocol.
TRILL uses IS-IS [IS-IS] [RFC6326].
RBridge: "Routing Bridge", an alternate name for a TRILL switch.
ToR: Top-of-Rack switches in a data center. Also known as access
switches in some data centers.
TRILL: Transparent Interconnection of Lots of Links [RFC6325]
TRILL Switch: A device implementing the TRILL protocol [RFC6325].
VM: Virtual Machine
3. Impact of Massive Number of End Stations
This section discusses the impact of a massive number of end stations
in a TRILL campus using Data Centers as an example.
3.1. Issues of Flooding-Based Learning in Data Centers
It is common for Data Center networks to have multiple tiers of
switches, for example, one or two Access Switches for each server
rack (ToR), aggregation switches for some rows (or EoR switches), and
some core switches to interconnect the aggregation switches. Many
aggregation switches deployed in data centers have high port density.
It is not uncommon to see aggregation switches interconnecting
hundreds of ToR switches.
+-------+ +------+
+/------+ | +/-----+ |
| Aggr11| + ----- |AggrN1| + EoR switches
+---+---+/ +------+/
/ \ / \
/ \ / \
+---+ +---+ +---+ +---+
|T11|... |T1x| |T21| .. |T2y| ToR switches
+---+ +---+ +---+ +---+
| | | |
+-|-+ +-|-+ +-|-+ +-|-+
| |... | | | | .. | |
+---+ +---+ +---+ +---+ Server racks
| |... | | | | .. | |
+---+ +---+ +---+ +---+
| |... | | | | .. | |
+---+ +---+ +---+ +---+
Figure 1: Typical Data Center Network Design
The following problems could occur when TRILL is deployed in a data
center with a large number of end stations and when the end stations
in one subnet/Label are placed under multiple edge RBridges:
- Unnecessary filling of slots in the MAC address learning table
of edge RBridges, e.g., RBridge T11, due to T11 receiving
broadcast/multicast traffic (e.g., ARP/ND, cluster multicast,
etc.) from end stations under other edge RBridges that are not
actually communicating with any end stations attached to T11.
- Packets being flooded across a TRILL campus when their
destination MAC addresses are not in the ingress RBridge's MAC
address to the egress RBridge cache.
3.2. Two Examples
Consider a data center with 1,600 server racks. Each server rack has
at least one ToR switch. The ToR switches are further divided into 8
groups, with each group being connected by a set of aggregation
switches. There could be 4 to 8 aggregation switches in each set to
achieve load sharing for traffic to/from server racks. Let's
consider the following two scenarios for the TRILL campus boundary if
TRILL is deployed in this data center environment:
- Scenario #1: TRILL campus boundary starts at the ToR switches:
If each server rack has one ToR, there are 1,600 edge RBridges.
If each rack has two ToR switches, then there will be 3,200
edge RBridges.
In this scenario, the TRILL campus will have more than 1,600
(or 3,200) + 8*4 (or 8*8) nodes, which is a large IS-IS area.
Even though a mesh IS-IS area can scale up to thousands of
nodes, it is challenging for aggregation switches to handle
IS-IS link state advertisement among hundreds of parallel
ports.
If each ToR has 40 downstream ports facing servers and each
server has 10 VMs, there could be 40*10 = 400 end stations
attached. If those end stations belong to 8 Labels, then the
total number of MAC&Label entries learned by each edge RBridge
in the worst case might be 400*8 = 3,200, which is not a large
number.
- Scenario #2: TRILL campus boundary starts at the aggregation
switches:
With the same assumptions as before, the number of nodes in the
TRILL campus will be less than 100, and aggregation switches
don't have to handle IS-IS link state advertisements among
hundreds of parallel ports.
However, the number of MAC&Label <-> Egress RBridge mapping
entries to be learned and managed by the RBridge edge node can
be very large. In the example above, each edge RBridge has 200
edge ports facing the ToR switches. If each ToR has 40
downstream ports facing servers and each server has 10 VMs,
there could be 200*40*10 = 80,000 end stations attached. If
all those end stations belong to 1,600 Labels (50 per Data
Label) and each Data Label has 200 end stations, then under the
worst-case scenario, the total number of MAC&Label entries to
be learned by each edge RBridge can be 1,600*200=320,000, which
is very large.
4. Benefits of Directory-Assisted TRILL Edge
In some environments, particularly data centers, the assignment of
applications to servers, including rack and row selection, is
orchestrated by Server (or VM) Management System(s). That is, there
is a database or multiple databases that have the knowledge of where
each application is placed. If the application location information
can be fed to RBridge edge nodes through some form of directory
service, then there is much less chance of RBridge edge nodes
receiving unknown MAC destination addresses, therefore less chance of
flooding.
Avoiding unknown unicast address flooding to the TRILL campus is
especially valuable in the data center environment, because there is
a higher chance of an edge RBridge receiving packets with an unknown
unicast destination address and broadcast/multicast messages due to
VM migration and servers being loaded with different applications.
When a VM is moved to a new location or a server is loaded with a new
application with a different IP/MAC addresses, it is more likely that
the destination address of data packets sent out from those VMs is
unknown to their attached edge RBridges. In addition, gratuitous ARP
(IPv4 [RFC826]) or Unsolicited Neighbor Advertisement (IPv6
[RFC4861]) sent out from those newly migrated or activated VMs have
to be flooded to other edge RBridges that have VMs in the same
subnets.
The benefits of using directory assistance include:
- Avoids flooding an unknown unicast destination address across
the TRILL campus. The directory-enforced MAC&Label <-> Egress
RBridge mapping table can determine if a data packet needs to
be forwarded across the TRILL campus.
When multiple RBridge edge ports are connected to end stations
(servers/VMs), possibly via bridged LANs, a directory-assisted
edge RBridge won't need to flood unknown unicast destination
data frames to all ports of the edge RBridges in the frame's
Data Label when it ingresses a frame. It can depend on the
directory to locate the destination. When the directory
doesn't have the needed information, the frames can be dropped
or flooded depending on the policy configured.
- Reduces flooding of decapsulated Ethernet frames with an
unknown MAC destination address to a bridged LAN connected to
RBridge edge ports.
When an RBridge receives a unicast TRILL data packet whose
destination Nickname matches with its own, the normal procedure
is for the RBridge to decapsulate it and forward the
decapsulated Ethernet frame to the directly attached bridged
LAN. If the destination MAC is unknown, the RBridge floods the
decapsulated Ethernet frame out all ports in the frame's Data
Label. With directory assistance, the egress RBridge can
determine if the MAC destination address in a frame matches any
end stations attached via the bridged LAN. Frames can be
discarded if their destination addresses do not match.
- Reduces the amount of MAC&Label <-> Egress RBridge mapping
maintained by edge RBridges. There is no need for an edge
RBridge to keep MAC entries of remote end stations that don't
communicate with the end stations locally attached.
- Eliminates ARP/ND being broadcast or multicast through the
TRILL core.
- Provides some protection against spoofing of source addresses
(see Section 7).
5. Generic Operation of Directory Assistance
There are two different models for directory assistance to edge
RBridges: Push Model and Pull Model. The directory information is
described in Section 5.1 below, while Section 5.2 discusses Push
Model requirements, and Section 5.3 Pull Model requirements.
5.1. Information in Directory for Edge RBridges
To achieve the benefits of directory assistance for TRILL, the
corresponding Directory Server entries will need, at a minimum, the
following logical data structure:
[IP, MAC, Data Label, {list of attached RBridge nicknames}, {list of
interested RBridges}]
The {list of attached RBridges} are the edge RBridges to which the
host (or VM) is attached as specified by the [IP, MAC, Data Label] in
the entry. The {list of interested RBridges} are the remote RBridges
that might have attached hosts that communicate with the host in this
entry.
When a host has multiple IP addresses, there will be multiple
entries.
The {list of interested RBridges} could get populated when an RBridge
queries for information, or information is pushed from a Directory
Server. The list is used to notify those RBridges when the host
(specified by the [IP, MAC, Data Label]) in the entry changes its
RBridge attachment. An explicit list in the directory is not needed
as long as the interested RBridges can be determined.
5.2. Push Model and Requirements
Under this model, Directory Server(s) push the MAC&Label <-> Egress
RBridge mapping for all the end stations that might communicate with
end stations attached to an RBridge edge node. If the packet's
destination address can't be found in the MAC&Label <-> Egress
RBridge table, the Ingress RBridge could be configured to:
simply drop a data packet,
flood it to the TRILL campus, or
start the pull process to get information from the Pull Directory
Server(s).
It may not be necessary for every edge RBridge to get the entire
mapping table for all the end stations in a campus. There are many
ways to narrow the full set down to a smaller set of remote end
stations that communicate with end stations attached to an edge
RBridge. A simple approach is to only push the mapping for the Data
Labels that have active end stations under an edge RBridge. This
approach can reduce the number of mapping entries being pushed.
However, the Push Model will usually push more entries of MAC&Label
<-> Egress RBridge mapping to an edge RBridges than needed. Under
the normal process of edge RBridge cache aging and unknown
destination address flooding, rarely used mapping entries would have
been removed. But it can be difficult for Directory Servers to
predict the communication patterns among applications within one Data
Label. Therefore, it is likely that the Directory Servers will push
down all the MAC&Label entries if there are end stations in the Data
Label attached to the edge RBridge. This is a disadvantage of the
Push Model compared with the Pull Model described below.
In the Push Model, it is necessary to have a way for an RBridge node
to request Directory Server(s) to push the mapping entries. This
method should at least include the Data Labels enabled on the
RBridge, so that the Directory Server doesn't need to push down the
entire set of mapping entries for all the end stations in the campus.
An RBridge must be able to get mapping entries when it is initialized
or restarted.
The Push Model's detailed method and any handshake mechanism between
an RBridge and Directory Server(s) is beyond the scope of this
framework document.
When a Directory Server needs to push a large number of entries to
edge RBridges, efficient data organization should be considered, for
example, with one edge RBridge nickname being associated with all the
attached end stations' MAC addresses and Data Labels. As shown in
Table 1 below, to make the data more compact, a representation can be
used where a nickname need only occur once for a set of Labels, each
of which occurs only once and each of which is associated with a set
of multiple IP and MAC address pairs. It would be much more bulky to
have each IP and MAC address pair separately accompanied by its Label
and by the nickname of the RBridge by which it is reachable.
+------------+---------+--------------------------------+
| Nickname1 |Label-1 | IP/MAC1, IP/MAC2, ,, IP/MACn |
| |-------- +--------------------------------+
| |Label-2 | IP/MAC1, IP/MAC2, ,, IP/MACn |
| |-------- +--------------------------------+
| | ...... | IP/MAC1, IP/MAC2, ,, IP/MACn |
+------------+-------- +--------------------------------+
| Nickname2 |Label-1 | IP/MAC1, IP/MAC2, ,, IP/MACn |
| |-------- +--------------------------------+
| |Label-2 | IP/MAC1, IP/MAC2, ,,IP/MACn |
| |-------- +--------------------------------+
| | | IP/MAC1, IP/MAC2, ,, IP/MACn |
+------------+-------- +--------------------------------+
| ------- |-------- +--------------------------------+
| | | IP/MAC1, IP/MAC2, ,, IP/MACn |
+------------+-------- +--------------------------------+
Table 1: Summarized Table Pushed Down from Directory
Whenever there is any change in MAC&Label <-> Egress RBridge mapping
that can be triggered by end stations being added, moved, or
decommissioned, an incremental update can be sent to the edge
RBridges that are impacted by the change. Therefore, something like
a sequence number has to be maintained by Directory Servers and
RBridges. Detailed mechanisms will be specified in a separate
document.
5.3. Pull Model and Requirements
Under this model, an RBridge pulls the MAC&Label <-> Egress RBridge
mapping entry from the Directory Server when its cache doesn't have
the entry. There are a couple of possibilities for triggering the
pulling process:
- The RBridge edge node can send a pull request whenever it
receives an unknown MAC destination, or
- The RBridge edge node can intercept all ARP/ND requests and
forward them or appropriate requests to the Directory Server(s)
that has the information on where the target end stations are
located.
The Pull Directory response could indicate that the address being
queried is unknown or that the requestor is administratively
prohibited from getting an informative response.
By using a Pull Directory, a frame with an unknown MAC destination
address doesn't have to be flooded across the TRILL campus and the
ARP/ND requests don't have to be broadcast or multicast across the
TRILL campus.
The ingress RBridge can cache the response pulled from the directory.
The timer for such a cache should be short in an environment where
VMs move frequently. The cache timer could be configured by the
Management System or sent along with the Pulled reply by the
Directory Server(s). It is important that the cached information be
kept consistent with the actual placement of addresses in the campus;
therefore, there needs to be some mechanism by which RBridges that
have pulled information that has not expired can be informed when
that information changes or becomes invalid for other reasons.
One advantage of the Pull Model is that edge RBridges can age out
MAC&Label entries if they haven't been used for a certain configured
period of time or a period of time provided by the directory.
Therefore, each edge RBridge will only keep the entries that are
frequently used, so its mapping table size will be smaller. Edge
RBridges would query the Directory Server(s) for unknown MAC
destination addresses in data frames or ARP/ND and cache the
response. When end stations attached to remote edge RBridges rarely
communicate with the locally attached end stations, the corresponding
MAC&VLAN entries would be aged out from the RBridge's cache.
An RBridge waiting for a response from Directory Servers upon
receiving a data frame with an unknown destination address is similar
to an Layer-3/Layer-2 boundary router waiting for an ARP or ND
response upon receiving an IP data packet whose destination IP is not
in the router's IP/MAC cache table. Most deployed routers today do
hold the packet and send ARP/ND requests to the target upon receiving
a packet with a destination IP not in its IP-to-MAC cache. When
ARP/ND replies are received, the router will send the data packet to
the target. This practice minimizes flooding when targets don't
exist in the subnet.
When the target doesn't exist in the subnet, routers generally resend
an ARP/ND request a few more times before dropping the packets. So,
if the target doesn't exist in the subnet, the router's holding time
to wait for an ARP/ND response can be longer than the time taken by
the Pull Model to get IP-to-MAC mapping from a Directory Server.
RBridges with mapping entries being pushed from a Directory Server
can be configured to use the Pull Model for targets that don't exist
in the mapping data being pushed.
A separate document will specify the detailed messages and mechanism
for RBridges to pull information from Directory Server(s).
6. Recommendation
TRILL should provide a directory-assisted approach. This document
describes a basic framework for directory assistance to RBridge edge
nodes. More detailed mechanisms will be described in a separate
document or documents.
7. Security Considerations
For general TRILL security considerations, see Section 6 of
[RFC6325].
Accurate mapping of IP addresses into MAC addresses and of MAC
addresses to the RBridges from which they are reachable is important
to the correct delivery of information. The security of specific
directory-assisted mechanisms for delivering such information will be
discussed in the document or documents specifying those mechanisms.
A directory-assisted TRILL edge can be used to substantially improve
the security of a TRILL campus over TRILL's default MAC address
learning from the data plane. Assume S is an end station attached to
RB1 trying to spoof a target end station T and that T is attached to
RB2. Perhaps S wants to steal traffic intended for T or forge
traffic as if it was from T.
With that default TRILL data-plane learning as described in
[RFC6325], S can impersonate T or any other end station in the same
Data Label (VLAN or FGL [FGL]) as S and possibly other Data Labels,
depending on how tightly VLAN admission and Appointed Forwarders
[RFC6439] are configured at the port by which S is connected to RB1.
S can just send native frames with the forged source MAC addresses of
T, perhaps broadcast frames for maximum effectiveness. With this
technique, S will frequently receive traffic intended for T and S can
easily forge traffic as being from T.
Such spoofing can be prevented to the extent that the network
RBridges (1) use trusted directory services as described above in
this document, (2) discard native frames received from a local end
station when the directory says that end stations should be remote,
and, (3) when appropriate, intercept ARP and ND messages and respond
locally. Under these circumstances, S would be limited to spoofing
targets on the same RBridge as the ingress RBridge for S (that is,
RB1 = RB2). RB1 would still need to learn which local end stations
were attached to which port, and S could confuse RB1 by sending
frames with the forged source MAC address of other end stations on
RB1. Although it would also still be restricted to frames in a VLAN
that would both be admitted by S's port of attachment and for which
that port is an Appointed Forwarder.
Security against spoofing could be even further strengthened by
adding port of attachment information to the directory and discarding
native frames that are received on the wrong port. This would limit
S to spoofing targets that were on the same link as S and in a VLAN
admitted by the port of that link's attachment to RB1 and for which
that port is an Appointed Forwarder (or, if the link is multiply
connected, in the same way at all of the ports by which the link is
attached to an RBridge).
Even without directory services, secure ND [RFC3971] or use of secure
ESADI (as described in [ESADI]) may also be helpful to security.
8. Acknowledgements
Thanks for comments and review from the following:
Sam Aldrin, David Black, Charlie Kaufman, Yizhou Li, and Erik
Nordmark
9. Informative References
[802.1Q] IEEE Std 802.1Q-2011, "IEEE Standard for Local and
metropolitan area networks - Virtual Bridged Local Area
Networks", May 2011.
[IS-IS] ISO/IEC, "Intermediate System to Intermediate System
intra-domain routeing information exchange protocol for
use in conjunction with the protocol for providing the
connectionless-mode network service (ISO 8473)", ISO/IEC
10589:2002.
[RFC826] Plummer, D., "Ethernet Address Resolution Protocol: Or
Converting Network Protocol Addresses to 48.bit Ethernet
Address for Transmission on Ethernet Hardware", STD 37,
RFC 826, November 1982.
[RFC3971] Arkko, J., Ed., Kempf, J., Zill, B., and P. Nikander,
"SEcure Neighbor Discovery (SEND)", RFC 3971, March 2005.
[RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
"Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
September 2007.
[RFC6325] Perlman, R., Eastlake 3rd, D., Dutt, D., Gai, S., and A.
Ghanwani, "Routing Bridges (RBridges): Base Protocol
Specification", RFC 6325, July 2011.
[RFC6326] Eastlake, D., Banerjee, A., Dutt, D., Perlman, R., and A.
Ghanwani, "Transparent Interconnection of Lots of Links
(TRILL) Use of IS-IS", RFC 6326, July 2011.
[RFC6439] Perlman, R., Eastlake, D., Li, Y., Banerjee, A., and F.
Hu, "Routing Bridges (RBridges): Appointed Forwarders",
RFC 6439, November 2011.
[ESADI] Zhai, H., Hu, F., Perlman, R., Eastlake 3rd, D., and O.
Stokes, "TRILL (Transparent Interconnection of Lots of
Links): ESADI (End Station Address Distribution
Information) Protocol", Work in Progress, July 2013.
[FGL] Eastlake 3rd, D., Zhang, M., Agarwal, P., Perlman, R., and
D. Dutt, "TRILL (Transparent Interconnection of Lots of
Links): Fine-Grained Labeling", Work in Progress, May
2013.
Authors' Addresses
Linda Dunbar
Huawei Technologies
5430 Legacy Drive, Suite #175
Plano, TX 75024, USA
Phone: +1-469-277-5840
EMail: ldunbar@huawei.com
Donald Eastlake
Huawei Technologies
155 Beaver Street
Milford, MA 01757 USA
Phone: +1-508-333-2270
EMail: d3e3e3@gmail.com
Radia Perlman
Intel Labs
2200 Mission College Blvd.
Santa Clara, CA 95054-1549 USA
Phone: +1-408-765-8080
EMail: Radia@alum.mit.edu
Igor Gashinsky
Yahoo
45 West 18th Street 6th floor
New York, NY 10011 USA
EMail: igor@yahoo-inc.com
ISO Plain Language
On this page
§1Why a Directory Would Help
Edge TRILL switches, also known as RBridges, currently learn the mapping between destination MAC addresses and their egress TRILL switch by observing data packets or by using the ESADI protocol. When an ingress RBridge receives a data frame for a destination address, a MAC and label pair, that it does not know, the frame is flooded within that data label across the TRILL campus. The data labels are VLANs or fine-grained labels.
The RFC describes a framework for using directory services, in environments where such services are available, to assist edge TRILL switches. The assistance can reduce multi-destination frames, particularly address resolution, neighbour discovery and unknown unicast traffic, and so improve the scalability of the TRILL network. The information a directory provides can also be more secure than what is learned from the data plane.
Data centres are the natural place for this, and the RFC gives three reasons. Their topology is built from racks and rows, and a management system for servers and virtual machines orchestrates the assignment of guest operating systems to servers, racks and rows, so the information a directory needs is normally already available from that management system. Workloads move quickly: applications loaded into one physical server at different times can belong to different subnets, and when a virtual machine is moved or a server is reloaded with a different application, the destination addresses of the packets it sends are more likely to be unknown to the edge switch it is attached to. And with server virtualization there is a growing trend to create and delete virtual machines as demand changes, to move them off overloaded servers, and to pack them onto fewer servers when demand is light, which results in more subnets on the same port at the same time and a faster rate of change than physical servers see.
Data centres already have a system that knows which server each virtual machine runs on, and where that server is. That is exactly the knowledge a TRILL switch needs to avoid sending a frame to every switch in the building.
§2The Words Used
The document uses VLAN and data label interchangeably with subnet, because it is common to map one subnet onto one VLAN or fine-grained label. A field device is a physical device placed in the network's operating environment, such as a plant, a city or a home. Field devices include sensors and actuators as well as routers and border routers, and they are usually, though not always, constrained in processor, memory, storage, bandwidth and sometimes power.
A sensor measures a physical quantity and converts it into a signal that a program or a person can read, such as current, pressure, humidity, oxygen or noise. An actuator controls a set of equipment, for example modulating the flow of a gas or liquid, controlling electricity distribution or performing a mechanical operation. A controller is a field device that can take sensor input and automatically change the environment by manipulating actuators. A data sink is a device that collects data from nodes in the network.
The border router of a low-power network connects it to another routing domain, such as a local network, a wide area network or the internet, where a different routing protocol may be in use; it acts as a routing device and may also host other functions such as collecting or aggregating data. A RBridge is a routing bridge, the alternative name for a TRILL switch. The other terms are the building blocks of a data centre: top-of-rack switches, also called access switches, end-of-row switches, also called aggregation switches, and the bridges themselves, which comply with IEEE 802.1Q.
The vocabulary is about the parts of the network: little battery-powered sensors and actuators, the routers that pass their messages along, and the border router that connects the whole thing to the rest of the internet.
§3How Bad the Flooding Gets
Data centre networks usually have several tiers of switches: one or two access switches for each server rack, aggregation switches for some rows, and core switches to interconnect the aggregation switches. Many aggregation switches have a high port density, and it is not uncommon to see one interconnecting hundreds of top-of-rack switches. Two problems follow when TRILL is deployed with a large number of end stations and the end stations in one subnet are spread under several edge switches. Slots fill up unnecessarily in the MAC address learning table of an edge switch, because it receives broadcast and multicast traffic such as address resolution and cluster multicast from end stations under other edge switches that are not communicating with anything attached to it. And packets are flooded across the campus whenever their destination address is not in the ingress switch's cache.
The RFC works through two ways of drawing the TRILL campus boundary. In the first, the boundary starts at the top-of-rack switches, so a data centre with 1,600 server racks has 1,600 edge switches, or 3,200 if each rack has two. The campus then has more than 1,600, or 3,200, plus the aggregation switches, which is a large IS-IS area. Even a meshed IS-IS area can scale to thousands of nodes, but it is challenging for aggregation switches to handle link state advertisements among hundreds of parallel ports. Each top-of-rack switch with 40 downstream ports and 10 virtual machines per server could have 400 end stations attached; if they belong to 8 labels, the worst case number of MAC and label entries each edge switch learns is 3,200.
In the second scenario the boundary starts at the aggregation switches, so the campus has fewer than 100 nodes and the aggregation switches no longer have to handle link state advertisements among hundreds of parallel ports. The cost moves to the edge switch's table. Each edge switch has 200 ports facing the top-of-rack switches, each of those has 40 downstream ports, and each server has 10 virtual machines, so 80,000 end stations are attached. If those belong to 1,600 labels, with 50 per data label and 200 end stations in each, the worst case number of MAC and label entries an edge switch must learn is 320,000.
There is a trade-off. Put the boundary low and the routing area is huge; put it high and each switch has to remember a quarter of a million addresses. Both are hard to live with, and both get worse as virtual machines move around.
| Aspect | Boundary at the top-of-rack switches | Boundary at the aggregation switches |
|---|---|---|
| Nodes in the campus | More than 1,600 or 3,200, plus the aggregation switches | Fewer than 100 |
| End stations attached to an edge switch | 400 | 80,000 |
| MAC and label entries per edge switch | Up to 3,200 | Up to 320,000 |
| Routing load | Aggregation switches handle link state advertisements among hundreds of parallel ports | Aggregation switches do not have to |
§4What a Directory Buys You
In some environments, particularly data centres, the assignment of applications to servers, including rack and row selection, is orchestrated by a server or virtual machine management system. There is a database, or several, that knows where each application is placed. If that location information can be fed to the edge switches through some form of directory service, the switches are much less likely to receive unknown destination addresses, and so much less likely to flood.
The benefits are five. Unknown unicast flooding across the campus is avoided, because the directory-enforced mapping table can say whether a packet needs to travel across the campus at all, and where it does, the switch can depend on the directory to locate the destination rather than sending the frame everywhere. When the directory does not have the information, the frames can be dropped or flooded according to the policy configured. Flooding of decapsulated Ethernet frames with an unknown destination address to a bridged local network is reduced, because the egress switch can check the destination against the hosts actually attached to that local network and discard frames that match nothing. The amount of mapping an edge switch must maintain falls, because it no longer needs MAC entries for remote end stations that never talk to the local ones. Address resolution and neighbour discovery traffic no longer has to be broadcast or multicast through the TRILL core. And a directory provides some protection against spoofing of source addresses.
The point of the directory is that a switch can tell the difference between "I do not know where this address is" and "this address is not on my network". The first needs a flood; the second needs nothing at all.
- No unknown unicast floodingThe directory says whether the packet has to cross the campus at all.
- No flooding to the local LANThe egress switch can drop a frame whose destination is not attached to it.
- A smaller mapping tableNo entries are kept for remote hosts that never talk to the local ones.
- No address lookups in the coreAddress resolution and neighbour discovery need not be broadcast through the core.
- Some protection from spoofingThe directory can contradict a forged source address.
§5Two Ways to Feed the Directory In
Whatever the model, the directory entry needs, at a minimum, the logical structure of an IP address, a MAC address, a data label, the list of edge switch nicknames the host is attached to, and the list of interested switches. The attached list is the edge switches to which the host is attached, as specified by the address and label in the entry. The interested list is the remote switches that might have attached hosts that talk to this one. A host with several IP addresses has several entries. The interested list could be filled in when a switch queries for information or when a server pushes it, and it is used to notify those switches when the host changes which switch it is attached to; an explicit list in the directory is not needed as long as the interested switches can be worked out.
In the push model, the directory server pushes the mapping for all the end stations that might communicate with the end stations attached to an edge node. If a packet's destination is not in the table, the ingress switch could drop it, flood it to the campus, or start the pull process. It is not necessary for every edge switch to receive the whole mapping; a simple approach is to push only the labels that have active end stations under the edge switch. The weakness of the model is that it usually pushes more entries than are needed, because directory servers have difficulty predicting the communication patterns among applications within one label, so they are likely to push all the entries for any label that has end stations attached to the edge switch. A switch also needs a way to ask the directory to push, and that request should at least name the labels enabled on the switch, and the switch must be able to get its entries when it starts or restarts. When the mapping changes, an incremental update can be sent to the affected switches, so something like a sequence number has to be kept by both the servers and the switches.
In the pull model, the edge switch asks the directory for the mapping entry when its cache does not have it. There are two ways to trigger that: the switch can send a pull request whenever it receives an unknown destination address, or it can intercept all address resolution and neighbour discovery requests and forward them, or the appropriate ones, to the directory server that knows where the targets are. The reply can say the address is unknown, or that the requester is administratively prohibited from getting an informative answer. The ingress switch caches the reply, and the cache timer should be short where virtual machines move often; it could be set by the management system or sent with the reply. There must also be a way to tell switches that have pulled information when it changes or becomes invalid. The advantage of the model is that the edge switch can age out entries it has not used for a while, so it only keeps the ones that are frequently used and its table stays smaller. A switch waiting for a directory reply is much like a router waiting for an address resolution reply, and where the target does not exist at all, routers generally retry a few times before dropping the packets, which takes longer than a pull does. A switch that already has pushed entries can also be configured to use the pull model for targets that are not in the pushed data.
There are two ways to use the directory. The directory can push the whole map down to every switch, which is simple but wasteful. Or a switch can ask for one entry when it needs one, which wastes less but needs a round trip.
§7Security and Source Address Spoofing
For general TRILL security considerations the RFC points to Section 6 of RFC 6325. It notes that accurate mapping of IP addresses to MAC addresses, and of MAC addresses to the switches they are reachable from, matters for correct delivery.
A directory-assisted edge can substantially improve the security of a campus over TRILL's default learning from the data plane. Take an end station S attached to RB1 that wants to impersonate a target T attached to RB2, perhaps to steal traffic meant for T or to forge traffic as if it came from T. With the default data-plane learning, S can impersonate T or any other end station in the same data label, and possibly other labels, depending on how tightly VLAN admission and appointed forwarders are configured at S's port. S can simply send native frames with the forged source MAC address of T, and broadcast frames for maximum effect, and will frequently receive traffic intended for T.
Such spoofing can be prevented to the extent that the network switches use trusted directory services, discard native frames received from a local end station when the directory says that end station should be remote, and, where appropriate, intercept address resolution and neighbour discovery messages and answer them locally. Under those circumstances S is limited to spoofing targets on the same switch it is attached to. Adding the port of attachment to the directory and discarding native frames received on the wrong port would tighten this further. Even without directory services, secure neighbour discovery or the use of secure ESADI may help.
A device can pretend to be someone else simply by sending frames with the wrong source address. A directory that says where everything really is lets the switch catch that, because the frame arrives on the wrong port.