Dell EMC SmartFabric OS10 Switch Configuration Guide for VxRail 4.7 Deploying S5200F-ON series switches for a VxRail 4.7 cluster Abstract This document provides Dell EMC SmartFabric OS10 switch configuration examples and topology options for a VxRail 4.7 cluster deployment using S5200F-ON series switches.
Revisions Date Description August 2019 Initial release The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any software described in this publication requires an applicable software license. © 2019 Dell Inc. or its subsidiaries. All Rights Reserved.
Table of contents Revisions.............................................................................................................................................................................2 1 2 Introduction ...................................................................................................................................................................5 1.1 Supported switches and operating systems ................................................................................
7.1.3 Configure interfaces .........................................................................................................................................23 7.1.4 Configure VLT...................................................................................................................................................25 7.2 Dual switch without VLT ...................................................................................................................................26 7.2.
1 Introduction Our vision at Dell EMC is to be the essential infrastructure company from the edge, to the core, and to the cloud. Dell EMC Networking ensures modernization for today’s applications and for the emerging cloud-native world. Dell EMC is committed to disrupting the fundamental economics of the market with an open strategy that gives you the freedom of choice for networking operating systems and top-tier merchant silicon.
1.1 Supported switches and operating systems The examples provided in this deployment guide use VxRail 4.7 nodes connected to S5248F-ON switches running Dell EMC SmartFabric OS10. Dell EMC Networking supports the following switches and operating systems for VxRail 4.7 and later: Supported Dell EMC PowerSwitches and operating systems 1.2 Typographical conventions The CLI and GUI examples in this document use the following conventions: 1.
2 Hardware overview This section briefly describes the hardware used to validate this deployment. Appendix A contains a complete listing of hardware and software validated for this guide. 2.1 Dell EMC PowerSwitch S5248F-ON The S5248F-ON is a 1-Rack Unit (RU), multilayer switch with 48x25GbE, 4x100GbE, and 2x200GbE ports. This guide uses two S5248F-ONs as leaf switches. Dell EMC PowerSwitch S5248F-ON 2.
Note: The deployment examples in this guide build a VxRail cluster consisting of four P570 VxRail nodes. 2.3.1 Dell EMC VxRail P Series node VxRail P Series nodes are ideal for CPU-intensive workloads such as databases. P Series nodes support up to 44 CPU cores, 1536GB memory, and 24TB hybrid or 46TB all-flash storage. 2.3.2 Dell EMC VxRail V Series node VxRail V Series nodes are graphics-ready for uses such as high-end 2D/3D visualization.
VxRail optimizes network traffic by splitting it across rNDC uplinks and by using Network I/O control (NIOC) shares for different traffic types. See the Dell EMC VxRail Network Planning Guide for more information. P, V, S, and E Series VxRail nodes also include a 1GbE BASE-T integrated Dell Remote Access Card (iDRAC) for OOB management. Note: P, V, and S Series nodes support additional network adapters in PCIe slots. See the Dell EMC VxRail Network Planning Guide for more information.
3 Topology options VxRail may be deployed using a single or dual switch topology. Using a single switch provides a lower initial cost but creates a single point of failure. A dual switch configuration helps ensure high availability by eliminating this single point of failure. A dual switch configuration may be used with or without VLT. Dell EMC recommends a dual switch configuration with VLT. The sections that follow explain the different options. 3.
• • • • • Guarantees fast convergence if either a link or a device fails Provides link-level resiliency Assures high availability Allows a single device to use a LAG across two upstream switches Provides Layer 2 multipathing Note: While VxRail nodes use active and standby network adapters instead of LAGs, other servers in the rack can connect to the VLT switch pair with an LACP LAG for active/active Layer 2 multipathing.
3.2 Single switch In a single switch topology, all VxRail nodes connect to a single switch. This topology is not recommended as the switch becomes a single point of failure. S5248F-ON 1 2 VxRail node 1 1 2 25GbE connections VxRail node 2 1 2 VxRail node n Rack 1 Single switch topology 12 Dell EMC SmartFabric OS10 Switch Configuration Guide for VxRail 4.
4 VxRail in the data center 4.1 Leaf-spine network Dell EMC recommends using a leaf-spine network in the data center with leaf switches configured as VLT peers. The switches and VxRail nodes covered in this guide are shown in Rack 1 in Figure 10 and are incorporated into a data center’s leaf-spine network. Spine1 Spine2 Leaf-2A Leaf-2B L3 L2 L3 S5248F-Leaf1B S5248F-Leaf1A VLTi Leaf-nA Leaf-nB VLTi VLTi Additional VxRail clusters, servers, storage, etc.
4.2 OOB Management network The Out-of-band (OOB) Management network is an isolated network for remote management of servers, switches, and storage devices using their dedicated hardware management ports. It is also commonly used to carry heartbeat messages sent between switches configured as VLT peers.
5 Configuration planning 5.1 VLANs and IP addresses VLANs and IP addresses used for VxRail node traffic must be planned before switch configuration and VxRail deployment can begin. VxRail node traffic is divided into five or more VLANs, as shown in Table 1.
5.2 VxRail network configuration table Information provided in the VxRail network configuration table is used during VxRail deployment. The values used for this deployment example are shown in the right column. The VLANs and IP addresses used are based on the information from Table 2. Note: For additional information on the VxRail network configuration table, see the Dell EMC VxRail Network Planning Guide.
Row 34 35 36 37 38 39 40 41 42 43 Category vSAN VM Networks … (unlimited number) Solutions Logging 44 45 Description Values used Ending address for IP pool Subnet mask VLAN ID Starting address for IP pool Ending address for IP pool Subnet mask VLAN ID VM Network name and VLAN ID 172.16.12.40 255.255.255.0 1612 172.16.13.1 172.16.13.40 255.255.255.
Hostname IP Address vxvcenter.dell.local 172.16.11.100 vxpsc.dell.local 172.16.11.101 vxman.dell.local 172.16.11.102 vxinsight.dell.local 172.16.11.103 An NTP server is not required but is recommended. If an NTP server is not provided, VxRail uses the time that is set on VxRail node 1. Note: For this deployment guide, the PowerEdge server used as the jump box also provides DNS and NTP services for the VxRail cluster.
6 Switch configuration prerequisites 6.1 Check switch OS version S5248F-ON switches must be running SmartFabric OS10.5.0.0 or later. Run the show version command to check the OS version. Dell EMC recommends upgrading to the latest release available on Dell Digital Locker (account required). OS10# show version S5248F-ON# show version Dell EMC Networking OS10-Enterprise Copyright (c) 1999-2019 by Dell Inc. All Rights Reserved. OS Version: 10.5.0.0 Build Version: 10.5.0.0.
6.3 Factory default configuration The configuration commands in the sections that follow begin with S5248F-ON switches at their factory default settings. Dell EMC PowerSwitches running SmartFabric OS10 can be reset to their default configuration using the serial console port as follows: OS10# delete startup-configuration Proceed to delete startup-configuration [confirm yes/no(default)]:y OS10# reload System configuration has been modified.
7 Configure switches This section covers switch configuration for S5248F-ON switches in the different topologies. Commands for all examples are provided as attachments to this guide. 7.1 Dual switch with VLT This example uses a four-node VxRail cluster connected to a pair of switches configured with VLT, as shown in Figure 12. Dell EMC recommends using this topology.
7.1.1 General settings Configure the hostname, OOB management IP address, and default gateway. Enable IPv6 MLD snooping to allow discovery of the VxRail nodes. Specify an NTP server accessible by the switch. The example shows an NTP server on the OOB management network with a different IP address than the one used by the VxRail nodes on the External Management network. Note: Default spanning tree settings are used in this deployment.
7.1.
Note: Flow control is enabled on all network interfaces in ESXi by default. For more information, see VMware vSAN Network Design.
7.1.4 Configure VLT Use 100GbE interfaces for the VLTi between the two leaf switches. This example uses interfaces 1/1/53 and 1/1/54. Remove each interface from Layer 2 mode with the no switchport command. Create the VLT domain. The backup destination is the OOB management IP address of the VLT peer switch. Configure the interfaces used as the VLTi with the discovery-interface command. Enable peer routing. Note: For more information on VLT, see the Dell EMC SmartFabric OS10 User Guide Release 10.5.0.
7.2 Dual switch without VLT This example uses a four-node VxRail cluster connected to a switch pair without VLT, as shown in Figure 13. S5248F-Leaf1A 1/1/20 19 18 S5248F-Leaf1B 17 1/1/17 18 19 20 LACP 25GbE connections 100GbE LACP connections, ports 1/1/53-54 VxRail node 1 VxRail node 2 VxRail node 3 Note: The jump box/DNS server, not shown, is connected to port 1/1/9 on each switch.
7.2.1 General settings Configure the hostname, OOB management IP address, and default gateway. Enable IPv6 MLD snooping to allow discovery of the VxRail nodes. Specify an NTP server accessible by the switch. In this example, this is an NTP server on the OOB management network, with a different IP address than the one used by the VxRail nodes on the External Management network. Note: Default spanning tree settings are used in this deployment.
7.2.
Note: Flow control is enabled on all network interfaces in ESXi by default. For more information, see VMware vSAN Network Design.
7.2.4 Configure the switch interconnect Create a port channel to use between the two leaf switches, 127 in this example. Use the switchport mode trunk command to enable the port channel to carry traffic for multiple VLANs, and allow all VLANs on the port channel. Use 100GbE interfaces for the port channel, 1/1/53 and 1/1/54 in this example. Add each interface to the port channel as LACP active members with the channel-group 127 mode active command.
7.3 Single switch This example uses a four-node VxRail cluster connected to a single switch, as shown in Figure 14. S5248F-ON 1/1/23 21 19 17 18 20 22 24 25GbE connections VxRail node 1 VxRail node 2 Note: The jump box/DNS server, not shown, is connected to ports 1/1/9 and 1/1/10. VxRail node 3 VxRail node 4 Rack 1 Four-node cluster connected to a single switch The commands in the following sections are run to complete the switch configuration.
General settings – single switch S5248F-ON configure terminal hostname S5248F-ON interface mgmt1/1/1 no ip address ip address 100.67.172.38/24 no shutdown management route 0.0.0.0/0 100.67.172.254 ipv6 mld snooping enable ntp server 100.67.10.20 7.3.2 Configure VLANs In this section, VLANs are configured per Table 2. Create the External Management VLAN and assign an IP address. Create the vMotion, vSAN, VM Network, and Internal Management VLANs.
7.3.3 Configure interfaces Configure the interfaces for the jump box or laptop computer connections to be used during VxRail deployment. This example uses 10GbE interfaces 1/1/9:1 and 1/1/10:1. Note: See Appendix B to change the native speed of S5248F-ON port groups from 25GbE to 10GbE for jump box or laptop computer connections. While only one connection to the jump box is required, two may be used for redundancy. Add these interfaces as access ports on VLAN 1611 (the External Management VLAN).
S5248F-ON spanning-tree port type edge flowcontrol receive on flowcontrol transmit off no shutdown interface ethernet1/1/18 description Node1_Port2 switchport mode trunk switchport access vlan 1611 switchport trunk allowed vlan 1612-1615,3939 spanning-tree port type edge flowcontrol receive on flowcontrol transmit off no shutdown interface ethernet1/1/19 description Node2_Port1 switchport mode trunk switchport access vlan 1611 switchport trunk allowed vlan 1612-1615,3939 spanning-tree port type edge flowcon
S5248F-ON interface ethernet1/1/23 description Node4_Port1 switchport mode trunk switchport access vlan 1611 switchport trunk allowed vlan 1612-1615,3939 spanning-tree port type edge flowcontrol receive on flowcontrol transmit off no shutdown interface ethernet1/1/24 description Node4_Port2 switchport mode trunk switchport access vlan 1611 switchport trunk allowed vlan 1612-1615,3939 spanning-tree port type edge flowcontrol receive on flowcontrol transmit off no shutdown end write memory 35 Dell EMC Smart
8 Switch validation After switches are configured and devices are connected, the switch CLI is used to validate the network configuration. This section provides a list of the most common commands and their output for the examples used in this guide. 8.1 General validation commands The commands and output shown in this section are for S5248F-Leaf1A in the dual switch with VLT topology. The output of its peer, S5248F-Leaf1B, is similar.
Note: The command output for the dual switch without VLT topology is shown below. In this example, Port channel 127 is up (U), and DYNAMIC indicates LACP is used. Ports 1/1/53 and 1/1/54 are port channel members, and (P) indicates each is up and active.
Note: For the dual switch without VLT topology, the VLAN portion of the command output is as follows: NUM Status Description 1 Active 1/1/11:1,1/1/12:1,1/1/13-1/1/16,1/1/21-1/1/52 1611 Active External_Mgmt 1612 Active vMotion 1613 Active vSAN 1614 Active VM_Network_A 1615 Active VM_Network_B 3939 Active Internal_Mgmt Q Ports A Eth1/1/1-1/1/8,1/1/10:1, A T A T T T T T T T T T T Po127 Po127 Eth1/1/9:1,1/1/17-1/1/20 Eth1/1/17-1/1/20 Po127 Eth1/1/17-1/1/20 Po127 Eth1/1/17-1/1/20 Po127 Eth1/1/
Note: The command output is the same for all topologies in this guide. 8.1.5 show lldp neighbors The show lldp neighbors command is useful for identifying connected equipment. Interface 1/1/9:1 is connected to the jump box. Interfaces 1/1/17-20 are connected to the Broadcom adapter in each of the VxRail nodes, and 1/1/53-54 are connected to the other leaf.
8.1.6 show vrrp brief In this deployment, VRRP is configured on the External Management VLAN, 1611, to provide gateway redundancy for management traffic. The output from the show vrrp brief command shows the master and virtual IP addresses and whether the switch is in the master or backup role. The switch configured with the largest priority value, shown in the Priority column, becomes the master.
8.2.2 show vlt domain_id backup-link This command is used to verify VLT peers are communicating on the backup link over the OOB management network. The Destination is the management IP address of the peer. The Peer Heartbeat status must be Up. S5248F-Leaf1A# show vlt 127 backup-link VLT Backup Link -----------------------Destination : 100.67.172.37 Peer Heartbeat status : Up Heartbeat interval : 30 Heartbeat timeout : 90 Destination VRF : default 8.2.
9 Deploy VxRail A laptop computer or jump box with a web browser for the VxRail user interface is required. It is either directly plugged into a leaf switch or able to logically reach the VxRail External Management VLAN from elsewhere on the network. By default, the initial VxRail Manager IP address is 192.168.10.200/24. After initial configuration, the address changes to its new address on the External Management network. The new VxRail Manager address used in this guide is 172.16.11.
In a web browser on the laptop computer or jump box, connect to https://192.168.10.200 and deploy VxRail using the planning data in Table 3. Note: Step-by-step VxRail deployment GUI screenshots are beyond the scope of this guide. 9.1 Management network VLAN ID During VxRail 4.7 deployment, the user is prompted for the Management Network VLAN ID in the GUI. This refers to the External Management VLAN, which is VLAN 1611 in this guide.
A Validated components for S5248F-ON using VxRail 4.7 The following tables include hardware, software, and firmware that was used to configure and validate the examples in this guide. A.1 Dell EMC PowerSwitches Switches and operating system versions A.2 Qty Item OS version 2 Dell EMC PowerSwitch S5248F-ON leaf switches 10.5.0.0 1 Dell EMC PowerSwitch S3048-ON management switch 10.5.0.0 DAC Cables DAC cables Qty Item Dell PN 8 DAC-SFP28-25G-3.
A.4 VxRail appliance software The examples in this guide were validated using VxRail 4.7 appliance software. The software consists of the component versions that are provided below: VxRail appliance software component versions 45 Item Version VxRail Manager 4.7.211 build 13893929 VMware ESXi 6.7 EP09 build 13644319 VMware vCenter Server Appliance 6.7 U2a 13643870 VMware vSAN 6.7 EP09 VMware vRealize Log Insight 4.6.
B S5248F-ON port groups In the example used in this guide, interface eth1/1/9 is connected to a 10GbE adapter on a PowerEdge server used as a jump box. On S5248F-ON, this interface must be changed from its native speed of 25GbE to 10GbE using the port-group command. Note: In place of a jump box, a laptop computer containing a 1GbE BASE-T network adapter may be directly connected to an S5248F-ON leaf switch for VxRail deployment as described in Chapter 9.
Verify the settings with the following command: S5248F-Leaf1A(conf-pg-1/1/3)# do show port-group Port-group port-group1/1/1 port-group1/1/2 port-group1/1/3 (Output truncated) Mode Eth 25g-4x Eth 25g-4x Eth 10g-4x Ports FEM 1 2 3 4 5 6 7 8 9 10 11 12 - The four interfaces in port-group1/1/3 are now set to 10GbE. Note: The mode command changes the native speed of all interfaces in the port group, eth 1/1/9-1/1/12. In this guide, only port 1/1/9 is used.
C Technical resources Dell EMC Networking Guides Dell EMC SmartFabric OS10 User Guide Release 10.5.0 Manuals and documentation for Dell EMC PowerSwitch S3048-ON Manuals and documentation for Dell EMC PowerSwitch S5248F-ON Dell EMC Networking Layer 3 Leaf-Spine Deployment and Best Practices with OS10 Dell EMC VxRail Network Planning Guide Dell EMC VxRail support and documentation (account required) Dell EMC VxRail Support Matrix (account required) Dell EMC VxRail Appliance 4.5.x and 4.7.
D Support and feedback Contacting Technical Support Support Contact Information Web: http://www.dell.com/support Telephone: USA: 1-800-945-3355 Feedback for this document Readers are encouraged to provide feedback on the quality and usefulness of this publication by sending an email to Dell_Networking_Solutions@Dell.com. 49 Dell EMC SmartFabric OS10 Switch Configuration Guide for VxRail 4.
