Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment A guide with basic examples for deploying the Dell Force10 MXL modular switch into the access layer of a Cisco Nexus network Authors Network Enabled Solutions Team Jason Pearce, Manjesh Siddamurthy Networking Enterprise Technologist Kevin Horton
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment This document is for informational purposes only and may contain typographical errors and technical inaccuracies. The content is provided as is, without express or implied warranties of any kind. © 2012 Dell Inc. All rights reserved. Dell and its affiliates cannot be responsible for errors or omissions in typography or photography. Dell, the Dell logo, and PowerEdge are trademarks of Dell Inc.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Contents Introduction .............................................................................................................. 5 Content Overview ..................................................................................................... 6 Document Conventions ............................................................................................... 7 Device Introduction ..............................................
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Figures Figure 1. Dell Force10 MXL Switch .................................................................................. 5 Figure 2. Example Network Topology .............................................................................. 7 Figure 3. MXL External Interfaces ................................................................................... 8 Figure 4. FlexIO 40GbE QSFP+ Module....................................
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Introduction This document is an easy-to-use guide of recommended basic deployment practices for the Dell Force10 MXL in the access layer of a Cisco Nexus network environment. The Dell Force 10 MXL—an I/O module for the Dell PowerEdge M1000e chassis—is a 10 and 40 gigabit Ethernet multilayer switch targeted at deployment in data center networks.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Content Overview This document is broken up topically to allow the steps and concepts of MXL switch deployment to be addressed separately in a simple manner as detailed in Document Conventions. An introduction to the Force10 MXL hardware and its connectivity and management options is presented in Device Introduction.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Document Conventions This document is intended as a quick reference guide for use during basic deployment of a Force10 MXL switch. As such, many details, configuration options, and specific features of the MXL are left out of this document.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Device Introduction The Dell Force10 MXL is an I/O Module for the M1000e modular server chassis that is a line-speed, multilayer, 10 and 40Gb Ethernet switch that provides 32 internal 10GbE links—enabling full connectivity to M420 quarter-height server blades—and flexible 10 and 40Gb Ethernet options for external connectivity. With as much as 240 Gigabits of full-duplex external connectivity, it offers a favorable 1.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Figure 4. FlexIO 40GbE QSFP+ Module The FlexIO 40GbE QSFP+ module provides two 40Gb Ethernet ports that can be connected using Dell Force10 QSFP+ transceivers or Dell Force10 twinax cabling. Additionally Dell offers breakout fiber and twinax cables that enable each 40GbE port to be split into four 10GbE links that will connect with compatible 10GbE fiber transceivers or SFP+ ports .
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Due to the availability of FlexIO modules with varying port and link count, the external interface numbering on the MXL can be difficult to understand. Table 1 details the external interface numbering in various modular configurations of the MXL switch. This numbering follows the potential10GbE links skipping numbers where available link count is less than the potential.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Management Connectivity The MXL switch provides a number of methods for connectivity to its management command-line interface (CLI). Out-of-band console connectivity is available via serial on a physical USB type-A port on the face of the switch when used with the provided cable that is shipped with each MXL (see the previous section— External Interfaces—for help identifying the port).
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Dell Force10 MXL Deployment Initial Configuration This section provides a rapid introduction to some common MXL initial deployment tasks. For more detailed information on deploying the MXL see the Force10 MXL User Guide. The examples that are shown here start in privileged execution mode. On the serial console, privileged execution mode can be reached by using the “enable” command.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment MXL1(conf)#enable password NewEnablePassword MXL1(conf)#exit MXL1# Configuring a username allows an individual to authenticate and is required for remote management. The root user (with password calvin) is automatically configured on the MXL during its first boot to enable rapid remote management during deployment. It is highly recommended to remove the default root user once local user accounts are configured.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment MXL1(conf)#exit MXL1# For additional information regarding SSH please refer to the MXL User Guide. After configuring SSH and having confirmed the configuration is correct by logging in using SSH, it is recommended to disable Telnet for security reasons. Disable the Telnet Service on the MXL Telnet provides simple remote connectivity to the MXL’s CLI and is enabled by default.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Split an MXL 40Gb Ethernet Interface into Four 10Gb Ethernet Links The external QSFP+ 40Gb Ethernet ports can be configured as four separate 10Gb Ethernet links. Physical connectivity is enabled by an optical split fiber cable or a split twinax cable. When a 40GbE port is run in quad mode, it provides four 10Gb Ethernet interfaces that number sequentially starting with the port number of the 40GbE interface.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Access Port Downlink Configuration Server’s network interfaces’ default configuration works with a single subnet available on the link and no VLAN tagging. To provide connectivity to this type of server interface, the matching switch interface that it connects to should be configured as an access port in the server’s designated VLAN.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment MXL1(conf-if-te-1)#exit MXL1(conf)#exit MXL1# For administrative convenience the above steps can be applied to a range of switch interfaces during configuration. In the following example, the downlinks of MXL1 in the example network are all configured as access ports with interfaces 1-8 on VLAN 11, interfaces 9-24 on VLAN 12, and interfaces 25-32 on VLAN 13.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Trunk Port Downlink Configuration Trunk ports can participate in multiple VLANs over one Ethernet interface and are often used for connection to virtualization hosts and other VLAN aware applications. To keep the traffic of the different VLANs from mixing, a numbered tag is inserted in each Ethernet frame (with the optional exception of the interface’s “native” VLAN). To deploy a trunk port on an MXL running PVST follow these four steps.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment And the “portmode hybrid” command allows a trunk port to also carry a single untagged (or native) VLAN. Configure a Native VLAN for a Trunk Port Interface MXL2#configure MXL2(conf)#interface vlan 13 MXL2(conf-if-vl-13)#untagged tengigabitethernet 0/1 MXL2(conf-if-vl-13)#no shutdown MXL2(conf-if-vl-13)#exit MXL2(conf)#exit MXL2# Now that the expected VLANs are configured for the trunk port, it is ready to be enabled.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Link Aggregation Group Configuration As network switches interconnect more servers and other end nodes, a single active link between devices—especially between switches—often does not provide enough throughput or enough link resiliency for the applications running on the network. A common solution for this is to use link aggregation.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Trunk Port Uplink Configuration with MLAG at Top of Rack The preferred deployment topology for an MXL in the access layer of a Cisco Nexus network is to use the Nexus vPC feature between two top-of-rack peer switches to provide a multi-chassis LAG connection to the MXL.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment N5K1 (conf-vpc-10)#peer-keepalive destination 172.25.188.61 source 172.25.188.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment N5K1 (conf)#interface Ethernet 1/1-2 N5K1 (conf-if-range)#switchport mode trunk N5K1 (conf-if-range)#channel-group 20 mode active N5K1 (conf-if-range)#exit N5K1 (conf)#exit N5K1# This same command set is now run on N5K2 to configure its half of the multi-chassis LAG.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Enable Per-VLAN Spanning Tree on the MXL Most Cisco Nexus networks run per-VLAN spanning tree and the MXL is capable of natively participating in the version of the spanning tree protocol. Here are the commands required to enable it.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Enable the MXL Uplink Port-Channels Now that the expected spanning-tree, port-channel, switchport, and VLAN settings are in place; the following commands to enable the link members of the uplink port-channel can be run.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Po 20 128.21 [...snipped...] 26 128 1400 FWD 1400 24589 0023.04ee.be0a 144.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Trunk Port Uplink Configuration with Per-VLAN Spanning Tree The availability of Per-VLAN spanning tree on the MXL allows it to natively integrate into the spanningtree environment of a Cisco Nexus network. In the following example the MXL2 switch is configured for the PVST environment of the example network. As shown in the below diagram, it will have two LAGs that uplink it to two top-of-rack Nexus switches.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment N5K2 (conf)#interface port-channel 40 N5K2 (conf-if-po-40)#switchport mode trunk N5K2 (conf-if-po-40)#exit N5K2 (conf)#interface Ethernet 1/17 - 18 N5K2 (conf-if-range)#switchport mode trunk N5K2 (conf-if-range)#channel-group 40 mode active N5K2 (conf-if-range)#exit N5K2 (conf)#exit N5K2# Prepare the Nexus Per-VLAN Spanning Tree Instances In the following CLI commands N5K1 is configured as primary or secondary root of the three VLANs on
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment MXL2#configure MXL2(conf)#interface port-channel 30 MXL2(conf-if-po-30)#switchport MXL2(conf-if-po-30)#exit MXL2(conf)#exit MXL2# And now the commands that configure MXL2’s LAG connection to N5K2.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment MXL2#configure MXL2(conf)#interface range tengigabitethernet 0/35 - 36 MXL2(conf-if-range-te-0/35-36)#port-channel-protocol lacp MXL2(conf-if-range-te-0/35-36-lacp)#port-channel 40 mode active MXL2(conf-if-range-te-0/35-36-lacp)#exit MXL2(conf-if-range-te-0/33-36)#no shutdown MXL2(conf-if-range-te-0/35-36)#exit MXL2(conf)#exit MXL2# Verify the MXL2’s Spanning-tree State Below is the output from MXL2 showing the general spanning-tree sta
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Spanning Tree Protocol Failover and Failback Observations The primary role of any Spanning Tree Protocol is to prevent loops from occurring in a network topology. The secondary role is to re-converge after a topology change (link failure) as quickly as possible. The Force10 MXL blade switch supports Spanning Tree Protocol (IEEE 802.1d), Rapid Spanning Tree Protocol (802.1w), Multiple Spanning Tree Protocol (IEEE 802.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Force10 MXL blade switch was the link with the lowest path cost and should, from the Force10 MXL’s perspective, have always been in the Forwarding Status and Root Role when it was available. In this lab 6 different tests were performed for the MSTP to MSTP testing, and then again for the Rapid PVST+ to PVST testing. Each test was performed while a series of pings were in progress.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment As mentioned previously the Nexus switches can run in one of two Spanning Tree modes, MSTP or Rapid PVST+ Here is the command for configuring the Nexus Switches to run MSTP: N5K2#configure N5K2 (conf)#spanning-tree mode mst N5K2 (conf)#exit N5K2# The process for configuring the MXL switch to run MSTP. First, check to see if STP is already configured. If it is, then disable and remove it from the configuration, before configuring MSTP.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24576, Address 547f.ee7a.7301 Root Bridge hello time 2, max age 20, forward delay 15, max hops 19 Bridge ID Priority 32768, Address 001e.c9f1.0153 Configured hello time 2, max age 20, forward delay 15, max hops 20 Bpdu filter disabled globally CIST regional root ID Priority 24576, Address 547f.ee7a.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Te 1/2 Dis 128.187 128 200000 DIS 2000 P2P No No No Te 1/30 Dis 128.215 128 200000 DIS 2000 P2P No No No Te 1/31 Dis 128.216 128 200000 DIS 2000 P2P No No No Te 1/32 Dis 128.217 128 200000 DIS 2000 P2P No No No Te 1/43 Altr 128.228 128 2000 2000 P2P No No No Te 1/45 Root 128.230 128 2000 ... ...
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment 64 bytes from 192.168.0.42: icmp_seq=15 ttl=254 time=3.035 ms Verify that Spanning Tree is running as expected after the Local Interface Failure Simulation What is shown below as a result of the show spanning-tree command is: 1) Interface 1/43 has a status of Forwarding and is in the Role of Root 2) Interface 1/45 has a status of Disabled and is in the Role of Discarding.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Te 1/31 128.216 128 200000 DIS 4000 32768 001e.c9f1.0153 128.216 Te 1/32 128.217 128 200000 DIS 4000 32768 001e.c9f1.0153 128.217 Te 1/43 128.228 128 2000 FWD 4000 28672 547f.ee6c.21c1 128.131 Te 1/45 128.230 128 2000 DIS 4000 32768 001e.c9f1.0153 128.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Root Bridge. This scenario was observed with only a single ping being lost during the failback, indicating that the re-convergence took about 1 second. 3) Failover: Backbone Failure Simulation (Backbone Fast). In this test a shutdown was issued on N5K1’s interface 1/25. This caused MXL2 to place its interface 1/43 interface into the Role of Designated.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment MXL2(conf-rstp)#disable MXL2(conf-rstp)#exit MXL2(conf)#no protocol spanning-tree rstp MXL2(conf)#exit MXL2# Enable PVST Spanning Tree on the MXL Here are the commands to configure PVST: MXL2#configure MXL2(conf)#protocol spanning-tree pvst MXL2(conf-pvst)#no disable MXL2(conf-pvst)#exit MXL2(conf)#exit MXL2# Verify that Spanning Tree is Running as Expected What is show below as a result of the show spanning-tree command is: 1) The MXL’
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Te 1/43 128.228 128 2000 BLK Te 1/45 128.230 128 2000 FWD 2000 28682 547f.ee6c.21c1 128.131 2000 24586 547f.ee7a.7301 128.133 Interface Name Role PortID Prio Cost Sts Cost Link-type Edge BpduFilter ---------- ------ -------- ---- ------- ----------- ------- --------- ---- ---------Te 1/43 Altr 128.228 128 2000 Te 1/45 Root 128.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Request 10 timed out 64 bytes from 192.168.0.42: icmp_seq=14 ttl=254 time=3.032 ms 64 bytes from 192.168.0.42: icmp_seq=15 ttl=254 time=3.035 ms ...
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Te 1/43 Root 128.228 128 2000 Te 1/45 Dis 128.230 128 2000 FWD DIS 2002 2002 P2P P2P No No No No Nexus Rapid PVST+ and MXL PVST – Remainder of Test Observations After the Local Port Failover and Failback scenarios were observed, the following 4 scenarios were also observed 1) Failover: Uplink Failure Simulation (Uplink Fast). In this test a shutdown was issued on N5K1’s interface 1/5.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Uplink Failback (Uplink Fast) 1s 1s Upstream Failover (Backbone Fast) 1s 1s Upstream Failback (Backbone Fast) 1s 1s Appendices Appendix A: Referenced Network Topology and Device Configurations Figure 4.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment N5K2 is the primary root of VLANs 11 and 13’s spanning tree instances and the secondary root of VLAN 12’s. Its port-channel 10 uses Ethernet interfaces 1/17-20, port-channel 20 uses Ethernet interfaces 1/1-2, and its port-channel 40 uses Ethernet interfaces 1/25-26.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment MXL1 Running Configuration MXL2 Running Configuration ! Version 8.3.16.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment interface TenGigabitEthernet 0/9 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/10 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/11 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/12 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/13
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment interface TenGigabitEthernet 0/19 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/20 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/21 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/22 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/23
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment interface TenGigabitEthernet 0/29 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/30 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/31 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/32 no ip address switchport spanning-tree pvst edge-port no shutdown ! interface TenGigabitEthernet 0/33
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment interface fortyGigE 0/49 no ip address shutdown ! interface fortyGigE 0/53 no ip address shutdown ! interface ManagementEthernet 0/0 ip address 172.25.188.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment N5K1 Running Configuration version 5.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment interface Ethernet1/8 interface Ethernet1/9 interface Ethernet1/10 interface Ethernet1/11 interface Ethernet1/12 interface Ethernet1/13 interface Ethernet1/14 interface Ethernet1/15 interface Ethernet1/16 interface Ethernet1/17 switchport mode trunk channel-group 10 mode active interface Ethernet1/18 switchport mode trunk channel-group 10 mode active interface Ethernet1/19 switchport mode trunk channel-group 10 mode active interface Ethe
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment N5K2 Running Configuration version 5.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment interface Ethernet1/9 interface Ethernet1/10 interface Ethernet1/11 interface Ethernet1/12 interface Ethernet1/13 interface Ethernet1/14 interface Ethernet1/15 interface Ethernet1/16 interface Ethernet1/17 switchport mode trunk channel-group 10 mode active interface Ethernet1/18 switchport mode trunk channel-group 10 mode active interface Ethernet1/19 switchport mode trunk channel-group 10 mode active interface Ethernet1/20 switchport mo
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Appendix B: Basic Terminology Bridging Bridging—commonly called switching—is frame-by-frame layer-2 forwarding of Ethernet traffic with forwarding decisions generally based on each frame’s source and destination MAC address. A simple bridge has three available actions to perform on received traffic—filter, forward, or flood.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment LAG Link Aggregation Group (LAG) is a configured bundle of Ethernet links that are treated as the same logical Ethernet link. There are multiple terms that apply to LAGs including channel group, port channel, trunk, and even some server Ethernet interface teaming involves a collection of links that would be considered a LAG. However while channel group and port channel always apply to LAG use, trunk and teaming do not.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment Out-of-Band An out-of-band interface provides management connectivity to a device without participating in or relying on a device’s in-band (normal-use) data interfaces. On a switch this means that an out-of-band interface does not send or receive traffic from the switched links—neither bridged nor routed.
Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment STP Spanning-Tree Protocol (STP)—see Spanning Tree. Switching Switching, in an Ethernet context, is a specific technology but the term has largely been generalized to mean layer-2 Ethernet bridging. Ethernet switching is in fact the employment of ASIC technologies to implement Ethernet traffic forwarding and filtering in specialized circuits and memory structures designed for high throughput, low latency, and low cost performance.
