Leaf-Spine Deployment and Best Practices Guide Deploying leaf-spine networks in the data center with Dell EMC Networking and Cisco Nexus multilayer switches Dell EMC Networking Solutions Engineering July 2017
Revisions Date Rev. Description Authors July 2017 1.0 Initial release Jim Slaughter, Shree Rathinasamy THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Copyright © 2017 Dell Inc. All rights reserved. Dell and the Dell EMC logo are trademarks of Dell Inc. in the United States and/or other jurisdictions.
Table of contents Revisions............................................................................................................................................................................. 2 1 Introduction ................................................................................................................................................................... 6 1.1 2 3 Typographical Conventions ..........................................................................................
7.1.2 S4048-ON OSPF configuration ........................................................................................................................26 7.2 Z9100-ON spine switch configuration...............................................................................................................27 7.2.1 Z9100-ON BGP configuration...........................................................................................................................30 7.2.2 Z9100-ON OSPF configuration ........
9.3.4 show uplink-state-group ...................................................................................................................................55 9.3.5 show spanning-tree rstp brief ...........................................................................................................................55 10 Example 4: Layer 2 with Dell EMC leaf and Cisco Nexus spine switches .................................................................57 10.1 Z9100-ON leaf switch configuration ......
1 Introduction Data center networks have traditionally been built in a three-layer hierarchical tree consisting of access, aggregation and core layers as shown in Figure 1. Core Spine 1 Aggregation Access Traditional network architecture Due to increasing east-west traffic within the data center (server-server, server-storage, etc.), an alternative to the traditional access-aggregation-core network model is becoming more widely used.
1.1 Typographical Conventions The command line examples in this document use the following conventions: 7 Monospace Text CLI examples Underlined Monospace Text CLI examples that wrap the page. This text is entered as a single command. Italic Monospace Text Variables in CLI examples Bold Monospace Text Used to distinguish CLI examples from surrounding text. Leaf-Spine Deployment and Best Practices Guide | Version 1.
2 Hardware overview This section briefly describes the hardware used to validate the examples in this guide. A complete listing of hardware and components used is provided in Appendix B. 2.1 Dell EMC Networking S3048-ON The Dell EMC Networking S3048-ON is a 1-Rack Unit (RU) switch with forty-eight 1GbE Base-T ports and four 10GbE SFP+ ports. In this guide, one S3048-ON switch supports out-of-band (OOB) management traffic in each rack. Dell EMC Networking S3048-ON 2.
2.4 Dell EMC Networking Z9100-ON The Dell EMC Networking Z9100-ON is a 1-RU, multilayer switch with thirty-two ports supporting 10/25/40/50/100GbE plus two 10GbE ports. The Z9100-ON is used both as a leaf and as a spine in examples in this guide. As a leaf, the Z9100-ON is able to support servers with a wide range of network adapter speeds such as 25GbE. As a spine, the Z9100-ON provides top performance with 100GbE support. Dell EMC Networking Z9100-ON 2.
3 Leaf-spine overview The connections between leaf and spine switches can be layer 2 (switched) or layer 3 (routed). The terms “layer 3 topology” and “layer 2 topology” in this guide refer to these connections. In both topologies, downstream connections to servers, storage and other endpoint devices within the racks are layer 2 and connections to external networks are layer 3.
3.2 Layer 2 leaf-spine topology In a layer 2 leaf-spine network, traffic between leafs and spines is switched (except for a pair of edge leafs) as shown in Figure 8. VLT is used for multipathing and load balancing traffic across the layer 2 leaf-spine fabric. Connections from hosts to leaf switches are also layer 2. For connections to external networks, layer 3 links are added between the spines and a pair of edge leafs.
4 Management network The OOB management network is isolated from the leaf-spine production network. It is the same for the layer 2 and layer 3 leaf-spine topologies. An S3048-ON installed in each rack provides 1GbE connectivity to the management network. The RJ-45 OOB management ports on each spine and leaf switch are connected to the S3048-ON switches as shown in Figure 9. PowerEdge server iDRACs and Chassis Management Controllers (CMCs) are also connected for server administration.
5 Protocols used in the leaf-spine examples This section provides an overview of the protocols used in constructing the leaf-spine network examples in this guide. The first three protocols are used in all layer 2 and layer 3 topology examples: • • • VLT, Section 5.1 Uplink Failure Detection (UFD), Section 5.2 RSTP, Section 5.3 The remaining protocols are only used in the layer 3 topology examples: • • • 5.1 Routing protocols, Section 5.
Note: Downstream connections from leaf switches configured for VLT do not necessarily have to be configured as LAGs if other fault tolerant methods are preferred (e.g. multipath IO). In this guide, examples 1 and 2 use LAGs to downstream servers while examples 3 and 4 do not. 5.2 UFD If a leaf switch loses all connectivity to the spine layer, by default the attached hosts continue to send traffic to that leaf without a direct path to the destination.
Dell EMC recommends enabling fast fall-over with BGP. Fast fall-over terminates external BGP sessions of any directly adjacent peer if the link to reach the peer goes down without waiting for the hold-down timer to expire. Note: BGP fast fall-over is enabled manually on Dell EMC switches and is enabled by default on Cisco Nexus 5600 series switches. Examples using EBGP (BGPv4) are provided in the layer 3 topology examples in this guide. 5.4.
5.6 ECMP The nature of a leaf-spine topology is that leaf switches are no more than one hop away from each other. As shown in Figure 10, Leaf 1 has two equal cost paths to Leaf 4, one through each spine. The same is true for all leafs. Spine 1 Spine 2 Path 1 Path 2 ECMP Leaf 1 VLTi Leaf 2 Leaf 3 Rack 1 VLTi Leaf 4 Rack 2 Use of ECMP in a layer 3 topology ECMP is a routing technique used in a layer 3 leaf-spine topology for load balancing packets along these multiple equal cost paths.
6 Layer 3 configuration planning 6.1 BGP ASN configuration When EBGP is used, an autonomous system number (ASN) is assigned to each switch. Valid private, 2-byte ASNs range from 64512 through 65534. Figure 11 shows the ASN assignments used for leaf and spine switches in the BGP examples in this guide.
10.0.1.1/32 10.0.2.1/32 VLTi 10.0.2.2/32 10.0.1.2/32 10.0.2.3/32 Rack 1 VLTi 10.0.2.4/32 Rack 2 Loopback addressing All loopback addresses used are part of the 10.0.0.0/8 address space with each address using a 32-bit mask. In this example, the third octet represents the layer, “1” for spine and “2” for leaf. The fourth octet is the counter for the appropriate layer. For example, 10.0.1.1/32 is the first spine switch in the topology while 10.0.2.4/32 is the fourth leaf switch. 6.2.
The point-to-point IP addresses used in this guide are shown in Figure 13: Spine 1 A Leaf 1 VLTi B C Leaf 2 Rack 1 Spine 2 D E F G Leaf 3 H VLTi Leaf 4 Rack 2 Point-to-point IP addresses Note: The example point-to-point addresses use a 31-bit mask to save address space. This is optional and covered in RFC 3021. Below is an example when setting an IP address with a 31-bit mask on a Dell EMC S4048-ON. The warning message can be safely ignored on point-to-point interfaces.
7 Example 1: Layer 3 with Dell EMC leaf and spine switches This section provides BGP and OSPF configuration examples to build the layer 3 leaf-spine topology shown in Figure 14. Dell EMC Networking S4048-ON switches are used at the leaf layer and Dell EMC Networking Z9100-ON switches are used at the spine layer. Note: The BGP ASNs and IP addresses defined in Section 6 are used here.
Note: All switch configuration files for the topology in Figure 14 are contained in the attachment named Example1_config_files.pdf. The files may be edited as needed in a plain text editor and commands pasted directly into switch consoles. Dell EMC Networking switches start at their factory default settings per Appendix A. 7.1 S4048-ON leaf switch configuration The following configuration details are for S4048-Leaf1 and S4048-Leaf2 in Figure 14.
Configure the VLT interconnect between S4048-Leaf1 and S4048-Leaf2. In this configuration, add interfaces fortyGigE 1/53-54 to static port channel 127 for the VLT interconnect. The backup destination is the management IP address of the VLT peer switch. Enable peer routing. Note: Dell EMC recommends that the VLTi is configured as a static LAG, without LACP, per the commands shown below.
Create a VLAN interface containing the server-facing port channel(s). Use the same VLAN ID on both leafs. Create a switched virtual interface (SVI) by assigning an IP address to the VLAN interface. The address must be unique but on the same network on both leaf switches. Note: In this example, Server 1’s NIC is configured as an LACP NIC team. It is assigned the IP address 172.16.1.7/24. The SVI on either leaf, 172.16.1.1 or 172.16.1.2, is specified as Server 1’s default gateway.
Configure a route map and IP prefix-list to redistribute all loopback addresses and leaf networks via BGP or OSPF. The command seq 10 permit 10.0.0.0/8 ge 24 includes all addresses in the 10.0.0.0/8 address range with a mask greater than or equal to 24. This includes all loopback addresses used as router IDs as well as the 10.60.1.0/24 network used on Leafs 3 and 4 as shown in Figure 14. The command seq 20 permit 172.16.0.0/16 ge 24 includes the 172.16.1.
7.1.1 S4048-ON BGP configuration Use these commands to configure BGP. If OSPF is used, skip to section 7.1.2. First, enable BGP with the router bgp ASN command. The ASN is from Figure 11. The bgp bestpath as-path multipath-relax enables ECMP. The maximum-paths ebgp 2 command specifies the maximum number of parallel paths to a destination to add to the routing table. This number should be equal to or greater than the number of spines, up to 64. BGP neighbors are configured and fast fall-over is enabled.
7.1.2 S4048-ON OSPF configuration Use these commands to configure OSPF. Skip this section if BGP is used. First, enable OSPF is enabled with the router ospf process-id command (valid range is 1-65535). Add the connected networks to OSPF area 0. The maximum-paths 2 command enables ECMP and specifies the maximum number of parallel paths to a destination to add to the routing table. This number should be equal to or greater than the number of spines, up to 64.
7.2 Z9100-ON spine switch configuration The following configuration details are for Z9100-Spine1 and Z9100-Spine2 in Figure 14. Note: On Z9100-ON switches, Telnet is enabled and SSH is disabled by default. Both services require the creation of a non-root user account to login. If needed, it is a best practice to use SSH instead of Telnet for security. SSH can optionally be enabled with the command: (conf)#ip ssh server enable.
Configure the four point-to-point interfaces connected to leaf switches. Assign IP addresses per Table 1. Configure a loopback interface to be used as the router ID. This is used with BGP or OSPF. Note: If multiple loopback interfaces exist on a system, the interface with the highest numbered IP address is used as the router ID. This configuration only uses one loopback interface. 28 Z9100-Spine1 interface fortyGigE 1/1/1 description Leaf 1 fo1/49 ip address 192.168.1.
Configure a route map and IP prefix-list to redistribute all loopback addresses and leaf networks via BGP or OSPF. The command seq 10 permit 10.0.0.0/8 ge 24 includes all addresses in the 10.0.0.0/8 address range with a mask greater than or equal to 24. This includes all loopback addresses used as router IDs as well as the 10.60.1.0/24 network used on Leafs 3 and 4 as shown in Figure 14. The command seq 20 permit 172.16.0.0/16 ge 24 includes the 172.16.1.
7.2.1 Z9100-ON BGP configuration Use these commands to configure BGP. If OSPF is used, skip to section 7.2.2. First, enabled BGP with the router bgp ASN command. The ASN is from Figure 11. The bgp bestpath as-path multipath-relax enables ECMP. The maximum-paths ebgp 2 command specifies the maximum number of parallel paths to a destination to add to the routing table. In this topology, there are two equal cost best paths from a spine to a host, one to each leaf that the host is connected.
7.2.2 Z9100-ON OSPF configuration Use these commands to configure OSPF. Skip this section if BGP is used. First, enable OSPF with the router ospf process-id command (valid range is 1-65535). Add the connected networks to OSPF area 0. The maximum-paths 2 command enables ECMP and specifies the maximum number of parallel paths to a destination to add to the routing table. In this topology, there are two equal cost best paths from a spine to a host, one to each leaf that the host is connected.
7.3.1 show ip bgp summary When BGP is configured, this command shows the status of all BGP connections. Each spine has four neighbors (the four leafs) and each leaf has two neighbors (the two spines). This command also confirms BFD is enabled on the 6th line of output. Z9100-Spine-1#show ip bgp summary BGP router identifier 10.0.1.
Neighbor ID 10.0.2.1 10.0.2.2 10.0.2.3 10.0.2.4 Pri 1 1 1 1 State FULL/DR FULL/DR FULL/DR FULL/DR Dead Time 00:00:32 00:00:34 00:00:35 00:00:35 Address 192.168.1.1 192.168.1.3 192.168.1.5 192.168.1.7 Interface Fo 1/1/1 Fo 1/2/1 Fo 1/3/1 Fo 1/4/1 Area 0 0 0 0 Dead Time Address 00:00:38 192.168.1.0 00:00:39 192.168.2.0 Interface Fo 1/49 Fo 1/50 Area 0 0 S4048-Leaf-1#sh ip ospf neighbor Neighbor ID 10.0.1.1 10.0.1.2 7.3.
B EX 10.0.2.4/32 B EX 10.60.1.0/24 via via via via via 192.168.2.0 192.168.1.0 192.168.2.0 192.168.1.0 192.168.2.0 20/0 00:03:56 20/0 00:03:56 Note: The command show ip route can also be used to verify the information above as well as static routes and direct connections. 7.3.4 show ip route ospf On switches with OSPF configured, this command is used to verify the OSPF entries in the Routing Information Base (RIB). Entries with multiple paths shown are used with ECMP.
O E2 10.0.2.4/32 O E2 10.60.1.0/24 O O O O O O 192.168.1.2/31 192.168.1.4/31 192.168.1.6/31 192.168.2.2/31 192.168.2.4/31 192.168.2.6/31 via via via via via via via via via via via 192.168.2.0, 192.168.1.0, 192.168.2.0, 192.168.1.0, 192.168.2.0, 192.168.1.0, 192.168.1.0, 192.168.1.0, 192.168.2.0, 192.168.2.0, 192.168.2.
7.3.6 show vlt brief This command is used to validate VLT configuration status on leaf switches in this topology. The Inter-chassis link (ICL) Link Status, Heart Beat Status and VLT Peer Status must all be up. The role for one switch in the VLT pair is primary and its peer switch (not shown) is assigned the secondary role. Ensure Peer-Routing is shown as enabled.
7.3.9 show uplink-state-group This command is used to validate the UFD status on leaf switches in this topology. Status: Enabled, Up indicates UFD is enabled and no interfaces are currently disabled by UFD.
8 Example 2: Layer 3 with Dell EMC leaf and Cisco Nexus spine switches In this section, the Dell EMC Networking Z9100-ON spines used in the previous example are replaced with Cisco Nexus 5600 series spines as shown in Figure 15. BGP and OSPF configuration examples are included. S4048-ON leaf switch configuration is identical to that covered in Section 7.1 and is not repeated in this section. Note: The BGP ASNs and IP addresses defined in Section 6 are used here.
8.1 Nexus 5600 series spine switch configuration The following configuration details are for Nexus5600-Spine1 and Nexus5600-Spine2 in Figure 15. First, set the hostname, enable LLDP and disable switchport as the default port type. Configure the management interface and default management route.
8.1.1 Nexus5600-Spine1 interface loopback 0 description Router ID ip address 10.0.1.1/32 no shutdown Nexus5600-Spine2 interface loopback 0 description Router ID ip address 10.0.1.2/32 no shutdown end copy running-config startup-config end copy running-config startup-config Nexus 5600 series BGP configuration Use these commands to configure BGP and BFD. If OSPF is used, skip to Section 8.1.2. First, enable the BGP and BFD features.
Nexus5600-Spine1 8.1.2 Nexus5600-Spine2 neighbor 192.168.1.1 remote-as 64701 address-family ipv4 unicast bfd neighbor 192.168.2.1 remote-as 64701 address-family ipv4 unicast bfd neighbor 192.168.1.3 remote-as 64702 address-family ipv4 unicast bfd neighbor 192.168.2.3 remote-as 64702 address-family ipv4 unicast bfd neighbor 192.168.1.5 remote-as 64703 address-family ipv4 unicast bfd neighbor 192.168.2.5 remote-as 64703 address-family ipv4 unicast bfd neighbor 192.168.1.
8.
11 BGP AS-PATH entrie(s) using 110 bytes of memory 2 neighbor(s) using 16384 bytes of memory Neighbor State/Pfx 192.168.1.0 192.168.2.0 8.2.2 AS MsgRcvd MsgSent TblVer InQ 54 59 58 66 0 0 0 0 64601 64602 OutQ Up/Down 0 00:47:30 5 0 00:33:00 4 show ip ospf neighbor When OSPF is configured, this command shows the state of all connected OSPF neighbors. In this configuration, each spine has four neighbors (the four leafs) and each leaf has two neighbors (the two spines).
10.0.2.1/32, ubest/mbest: 1/0 *via 192.168.1.1, [20/0], 00:51:59, 10.0.2.2/32, ubest/mbest: 1/0 *via 192.168.1.3, [20/0], 00:56:12, 10.0.2.3/32, ubest/mbest: 1/0 *via 192.168.1.5, [20/0], 00:52:07, 10.0.2.4/32, ubest/mbest: 1/0 *via 192.168.1.7, [20/0], 00:56:08, 10.60.1.0/24, ubest/mbest: 2/0 *via 192.168.1.5, [20/0], 00:52:07, *via 192.168.1.7, [20/0], 00:52:23, 172.16.1.0/24, ubest/mbest: 2/0 *via 192.168.1.1, [20/0], 00:51:59, *via 192.168.1.
10.0.2.1/32, ubest/mbest: 1/0 *via 192.168.1.1, Eth2/1, [110/20], 00:32:09, ospf-1, type-2 10.0.2.2/32, ubest/mbest: 1/0 *via 192.168.1.3, Eth2/2, [110/20], 00:31:49, ospf-1, type-2 10.0.2.3/32, ubest/mbest: 1/0 *via 192.168.1.5, Eth2/3, [110/20], 00:31:42, ospf-1, type-2 10.0.2.4/32, ubest/mbest: 1/0 *via 192.168.1.7, Eth2/4, [110/20], 00:31:30, ospf-1, type-2 10.60.1.0/24, ubest/mbest: 2/0 *via 192.168.1.5, Eth2/3, [110/20], 00:31:30, ospf-1, type-2 *via 192.168.1.
8.2.5 show bfd neighbors This command may be used to verify BFD is properly configured and sessions are established as indicated by Up in the RH (Remote Heard) and State columns on the Nexus spine and Up in the State column on the Dell EMC leaf. Note: The output for S4048-Leaf1 shown is for BGP configurations as indicated by a B in the Clients column. On OSPF configurations, the output is identical except there is an O in the Clients column. Nexus spine output is the same for either protocol.
9 Example 3: Layer 2 with Dell EMC leaf and spine switches This section provides configuration information to build the layer 2 leaf-spine topology shown in Figure 16. Dell EMC Networking S4048-ON switches are used at the leaf layer and Dell EMC Networking S6010-ON switches are used at the spine layer. VLAN 10, VLAN 20 Spine 1-6010 All spines Fo 1/25-1/28 VLTi Fo 1/31-1/32 Spine 2-S6010 Fo 1/25-1/28 Po 1 Fo 1/51 VLAN 10, VLAN 20 All leafs Po 2 Fo 1/53 Fo 1/51 Leaf 1-S4048 Te 1/48 IP address : 192.
First, configure the serial console enable password and disable Telnet. L2-Leaf1-S4048 L2-Leaf2-S4048 enable configure enable configure enable sha256-password enable_password no ip telnet server enable enable sha256-password enable_password no ip telnet server enable Set the hostname, configure the OOB management interface and default gateway. Enable LLDP. Enable RSTP as a precaution. Note: In this layer 2 topology, the RSTP root bridge is configured at the spine level.
L2-Leaf1-S4048 vlt domain 127 peer-link port-channel 127 back-up destination 100.67.194.2 unit-id 0 L2-Leaf2-S4048 vlt domain 127 peer-link port-channel 127 back-up destination 100.67.194.1 unit-id 1 Interface Te 1/48 connects downstream to Server 1 and is configured as an RSTP edge port. Interfaces Fo 1/51 and Fo 1/53 connect to the spines upstream and are configured in LACP port channel 1. The port channel is configured for VLT.
L2-Leaf1-S4048 L2-Leaf2-S4048 shutdown shutdown interface Vlan 20 no ip address tagged TenGigabitEthernet 1/48 tagged Port-channel 1 shutdown interface Vlan 20 no ip address tagged TenGigabitEthernet 1/48 tagged Port-channel 1 shutdown Configure UFD. This shuts the downstream interfaces if all uplinks fail. The hosts attached to the switch use the remaining LACP port member to continue sending traffic across the fabric.
Enable RSTP as a precaution. L2-Spine1-S6010 is configured as the primary RSTP root bridge using the bridge-priority 0 command. L2-Spine2-S6010 is configured as the secondary RSTP root bridge using the bridge-priority 4096 command. L2-Spine1-S6010 hostname L2-Spine1-S6010 L2-Spine2-S6010 hostname L2-Spine2-S6010 interface ManagementEthernet 1/1 ip address 100.67.194.15/24 no shutdown management route 0.0.0.0/0 100.67.194.254 interface ManagementEthernet 1/1 ip address 100.67.194.
Interfaces Fo 1/25-28 connect to the leaf switches downstream via LACP port channels. Port channel 1 has members Fo 1/25 and Fo 1/26 and port channel 2 has members Fo 1/27 and Fo 1/28. The port channels are configured for VLT.
VLANs 10 and 20 are configured on each switch. Port-channels 1 and 2 are tagged in both VLANs. Note: The shutdown/no shutdown commands on a VLAN have no effect unless the VLAN is assigned an IP address (configured as an SVI). Finally, exit configuration mode and save the configuration with the end and write commands. 9.
Delay-Restore timer: Delay-Restore Abort Threshold: Peer-Routing : Peer-Routing-Timeout timer: Multicast peer-routing timeout: L2-Leaf1-S4048#show vlt brief VLT Domain Brief -----------------Domain ID: Role: Role Priority: ICL Link Status: HeartBeat Status: VLT Peer Status: Local Unit Id: Version: Local System MAC address: Remote System MAC address: Remote system version: Delay-Restore timer: Delay-Restore Abort Threshold: Peer-Routing : Peer-Routing-Timeout timer: Multicast peer-routing timeout: 9.3.
L2-Spine1-S6010#show vlt mismatch L2-Spine1-S6010# L2-Leaf1-S4048#show vlt mismatch L2-Leaf1-S4048# 9.3.4 show uplink-state-group This command is used to validate the UFD status on leaf switches in this topology. Status: Enabled, Up indicates UFD is enabled and no interfaces are currently disabled by UFD.
Po 127 128.128 Interface Name ---------Po 1 Po 2 Po 127 Role -----Desg Desg Desg 128 600 PortID -------128.2 128.3 128.128 Prio ---128 128 128 FWD(vltI) Cost ------188 188 600 0 0 Sts ----------FWD FWD FWD f48e.382b.0869 128.128 Cost ------0 0 0 Link-type --------(vlt) P2P (vlt) P2P (vltI)P2P Edge ---No No No L2-Leaf1-S4048#show spanning-tree rstp brief Executing IEEE compatible Spanning Tree Protocol Root ID Priority 0, Address f48e.382b.
10 Example 4: Layer 2 with Dell EMC leaf and Cisco Nexus spine switches This section provides configuration information to build the layer 2 leaf-spine topology shown in Figure 17. Dell EMC Networking Z9100-ON switches are used at the leaf layer and Cisco Nexus 7000 series switches are used at the spine layer.
10.1 Z9100-ON leaf switch configuration The following section outlines the configuration commands issued to the Z9100-ON leaf switches to build the topology in Figure 17. The commands detailed below are for L2-Leaf1-Z9100 and L2-Leaf2-Z9100. The configuration commands for L2-Leaf3-Z9100 and L2-Leaf4-Z9100 are similar and are provided in the attachments. Note: On Z9100-ON switches, Telnet is enabled and SSH is disabled by default. Both services require the creation of a non-root user account to login.
Convert interfaces connected to the Nexus 5600 spines from their default speed of 100GbE to 40GbE. L2-Leaf1-Z9100 stack-unit 1 port 1 portmode single speed 40G no-confirm stack-unit 1 port 3 portmode single speed 40G no-confirm L2-Leaf2-Z9100 stack-unit 1 port 1 portmode single speed 40G no-confirm stack-unit 1 port 3 portmode single speed 40G no-confirm Configure the VLT interconnect between Leaf1 and Leaf2.
no ip address port-channel-protocol LACP port-channel 1 mode active no shutdown no ip address port-channel-protocol LACP port-channel 1 mode active no shutdown interface Port-channel 1 description To Spines no ip address portmode hybrid switchport vlt-peer-lag port-channel 1 no shutdown interface Port-channel 1 description To Spines no ip address portmode hybrid switchport vlt-peer-lag port-channel 1 no shutdown VLANs 10 and 20 are configured on each switch. Port-channel 1 is tagged in both VLANs.
10.2 Nexus 7000 series spine switch configuration The following sections outline the configuration commands issued to the Nexus 7000 series switches to build the topology in Figure 17. First, enable the LACP and virtual port channel (vPC) features. Configure the hostname, management IP address and default management route. Note: Cisco enables Rapid Per VLAN Spanning Tree Plus (RPVST+), its implementation of RSTP, on Nexus 7000 series switches by default.
Create a vPC domain and vPC peer link between the two spine switches. On spine 1, assign a role priority of 1 to make it the vPC primary. Specify the management IP address of the vPC peer as the vPC peer-keepalive destination. In this example, interfaces Ethernet 3/5 and 3/6 are used to create the vPC peer link. Interfaces are configured as trunk ports and allow applicable VLANs. 62 L2-Spine1-Nexus7K vpc domain 1 role priority 1 peer-keepalive destination 100.67.184.28 source 100.67.184.
Configure port channels and member ports for downstream connectivity to the leaf switches. Finally, exit configuration mode and save the configuration with the end and copy running-config startup-config commands.
10.3 Example 4 validation In addition to sending traffic between hosts, the configuration shown in Figure 17 can be validated with the commands shown in this section. For more information on commands and output, see the Command Line Reference Guide for the applicable switch (links to documentation are provided in Appendix C). 10.3.1 show vpc This command displays the vPC status on the Nexus spine switches. Peer status and vPC keep-alive status must be as shown.
10.3.2 show vpc consistency-parameters This command pinpoints inconsistencies between vPC peers on the Nexus spine switches. Depending on the severity of the misconfiguration, vPC may either warn the user (Type-2 misconfiguration) or suspend the port channel (Type-1 misconfiguration). In the specific case of a VLAN mismatch, only the VLAN that differs between the vPC member ports is suspended on the port channels.
10.3.3 show spanning-tree This command validates spanning tree is enabled on all VLANs on the Nexus spine switches and all interfaces are forwarding (Sts column shows FWD). One of the spine switches (L2-Spine1-Nexus7K in this example) is the root bridge on all VLANs. L2-Spine1-Nexus7K# show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 1 Address 8478.ac11.
Address 8478.ac11.e341 This bridge is the root Hello Time 2 sec Max Age 20 sec Bridge ID Priority Address Hello Time Interface ---------------Po1 Po2 Po20 10.3.4 Role ---Desg Desg Desg Forward Delay 15 sec 20 (priority 0 sys-id-ext 20) 8478.ac11.e341 2 sec Max Age 20 sec Forward Delay 15 sec Sts --FWD FWD FWD Cost --------1 1 1 Prio.Nbr -------128.4096 128.4097 128.
A Dell EMC Networking switch factory default settings All Dell EMC Networking switches in this guide can be reset to factory defaults as follows: Dell#restore factory-defaults stack-unit unit# clear-all Proceed with factory settings? Confirm [yes/no]:yes Factory settings are restored and the switch reloads. After reload, enter A at the [A/C/L/S] prompt as shown below to exit Bare Metal Provisioning mode. This device is in Bare Metal Provisioning (BMP) mode.
B Validated hardware and operating systems The following table includes the hardware and operating systems used to validate the examples in this guide. Switches and operating systems used in this guide 69 Switch OS / Version Dell EMC Networking S3048-ON DNOS 9.11.2.0 P0 Dell EMC Networking S4048-ON DNOS 9.11.2.0 P0 Dell EMC Networking S6010-ON DNOS 9.11.2.0 P0 Dell EMC Networking Z9100-ON DNOS 9.11.2.0 P0 Cisco Nexus 7000 series (validated with Nexus 7004) NX-OS 6.
C Technical support and resources Dell EMC TechCenter is an online technical community where IT professionals have access to numerous resources for Dell EMC software, hardware and services.
D Support and Feedback Contacting Technical Support Support Contact Information Web: http://support.dell.com/ Telephone: USA: 1-800-945-3355 Feedback for this document We encourage readers to provide feedback on the quality and usefulness of this publication by sending an email to Dell_Networking_Solutions@Dell.com. 71 Leaf-Spine Deployment and Best Practices Guide | Version 1.
