Dell EMC Solutions for Microsoft Azure Stack HCI Networking Guide Abstract This guide provides Microsoft Azure Stack HCI networking deployment examples using Dell EMC Storage Spaces Direct Ready Nodes and Dell EMC Networking switches.
Revisions Date Description May 2019 Revised Appendix B March 2019 Initial release The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any software described in this publication requires an applicable software license. © 2019 Dell Inc.
Table of contents Revisions.............................................................................................................................................................................2 1 2 3 4 Introduction ...................................................................................................................................................................6 1.1 Typographical conventions .........................................................................................
8 7.6 Configure VLT...................................................................................................................................................29 7.7 Configure DCB..................................................................................................................................................30 Switch validation .........................................................................................................................................................
10.5 Get-DnsClientServerAddress ...........................................................................................................................49 10.6 View VLAN IDs .................................................................................................................................................49 10.7 Get-NetAdapterRdma .......................................................................................................................................50 10.
1 Introduction This guide provides Dell EMC Solutions for Microsoft Azure Stack HCI greenfield network deployment examples using Dell EMC Storage Spaces Direct Ready Nodes and Dell EMC Networking switches. S2D is included with Microsoft Windows Server 2019 Datacenter and Windows Server 2016 Datacenter editions. S2D uses locally attached drives on industry-standard servers to create highly available softwaredefined storage. S2D has two deployment options: converged and hyper-converged.
2 Hardware Overview This section briefly describes the hardware used to validate the deployment examples in this document. Appendix A contains a complete listing of hardware and software validated for this guide. 2.1 Dell EMC Networking S5248F-ON The Dell EMC Networking S5248F-ON is a 1-Rack Unit (RU), multilayer switch with 48xSFP28 25GbE ports, 2xQSFP28-DD ports (2x100GbE per port), and 4xQSFP28 100GbE ports. This guide uses two S5248F-ONs as leaf switches for in-band traffic. Section 2.
2.4 Management system A management system is used to deploy and manage the S2D cluster. In this guide, a Dell EMC PowerEdge R640 server is used as the S2D management system. It also serves as the Active Directory (AD) domain controller and DNS server for the examples provided. Optionally, existing AD and DNS infrastructure may be used provided it has network connectivity to the S2D Ready Nodes via the in-band management network.
3 Topology S2D clusters are limited to 16 nodes. In this deployment, four S2D Ready Nodes are installed in a single rack along with two leaf switches and one OOB management switch. The devices are cabled as shown in Figure 5.
The two leaf switches are connected to spine switches upstream (not shown). Using two leaf switches provides fault tolerance and twice the bandwidth. Note: Spine switch configuration is beyond the scope of this guide and is covered in Dell EMC Networking Layer 3 Leaf-Spine Deployment and Best Practices with OS10EE. 3.1 Connections to leaf switches Each S5248F-ON switch has 48x25GbE ports available for connections to S2D Ready Nodes and other servers in the rack.
3.2 Connections to OOB management switch The OOB management network is an isolated network for remote management of servers, switches, and other devices. It is also used to carry heartbeat messages sent between leaf switches configured as VLT peers. Dell EMC recommends using at least one S3048-ON switch per 42-RU rack for OOB management network connections. Each S3048-ON has forty-eight 1GbE Base-T ports for connections to server iDRACs and leaf switch management ports as shown in Figure 7.
4 Configuration planning The network adapters installed in the Ready Nodes determine which Remote Direct Memory Access (RDMA) options are available and whether Data Center Bridging (DCB) is required or not. 4.1 RDMA overview S2D Ready Nodes use network adapters that are RDMA capable. RDMA significantly increases throughput and lowers latency by performing direct memory transfers between servers. There are two implementations of RDMA supported with S2D Ready Nodes: 1. RDMA over Converged Ethernet (RoCE).
4.4 Adapter/RDMA/DCB options This guide is applicable to the following network adapter, RDMA, and DCB combinations: • • 4.5 QLogic QL41262 – iWARP with or without DCB Mellanox ConnectX-4 Lx – RoCEv2 with or without DCB VLANs and IP addressing Table 2 shows the three VLANs used in this deployment. VLAN 1611 uses the 10GbE connections for inband management and VM traffic. VLANs 1613 and 1614 are for storage traffic on the 25GbE links.
5 Configure network adapters Note: Exact iDRAC steps in this section may vary depending on hardware, software and browser versions used. See the PowerEdge server documentation for steps to connect to the iDRAC. 5.1 Reset network adapters to factory defaults Note: These steps are only necessary for network adapters that have been modified from their factory default settings. 1. 2. 3. 4. Connect to the server's iDRAC in a web browser and launch the virtual console.
6. On the Main Configuration Page for the port, click the Default button followed by Yes to load the default settings. Click OK to acknowledge the Success message. 7. Click Finish. Notice if the message indicates a reboot is required for changes to take effect. 8. Click Yes to save changes. Click OK to acknowledge the Success message. 9. Repeat steps 5-8 for each network adapter port listed on the Device Settings page. 10.
6. On the Main Configuration Page, click Device Level Configuration. Device Level Configuration menu item 7. Verify Virtualization Mode is set to None. Virtualization Mode set to None 8. Click the Back button to return to the Main Configuration Page. 9. On the Main Configuration Page, click NIC Configuration.
10. On the NIC Configuration page, make the following settings: a. b. c. d. e. f. Link Speed – SmartAN NIC + RDMA Mode – Enabled RDMA Operational Mode – iWARP Boot Protocol – None Virtual LAN Mode – Disabled Virtual LAN ID – 1 QLogic NIC port settings configured 11. Click the Back button to return to the Main Configuration Page. 12. On the Main Configuration Page, click Data Center Bridging (DCB) Settings.
13. On the Data Center Bridging (DCB) Settings page, ensure the DCBX Protocol feature is set to Disabled. DCBX Protocol setting Note: DCBx is a protocol used to exchange PFC and ETS configuration details between devices, such as a switch and a network adapter. DCBx is not used in this deployment because Microsoft recommends disabling it for S2D deployments. ETS and PFC settings are configured manually on the switch and the network adapter later in this guide. 14.
5.3 Verify Mellanox ConnectX-4 Lx adapter settings This section verifies the Mellanox ConnectX-4 Lx adapters are configured properly. Note: The settings shown in this section are factory defaults. This section may be skipped if the Mellanox ConnectX-4 Lx adapters are known to be at their factory default settings. There are no RDMA or DCB settings in System Setup for this adapter. If the system is already in System Setup from Section 5.1, skip to step 4. 1. 2. 3. 4. 5.
7. On the NIC Configuration page, make sure the following items are configured: a. Legacy Boot Protocol - None b. Virtual LAN Mode - Disabled NIC Configuration 8. Click the Back button to return to the Main Configuration Page. 9. On the Main Configuration Page, click Device Level Configuration.
10. Verify Virtualization Mode is set to None. Virtualization Mode set to None 11. Click Back > Finish > Yes (to save the changes if applicable) > OK (to acknowledge the change success message if applicable). Repeat steps 5-11 above for the 2nd Mellanox adapter port. 12. When complete, exit System Setup and reboot the system.
6 Leaf switch configuration prerequisites 6.1 Check switch OS version Leaf switches must be running a supported version of OS10EE. Note: See Section 2.5 for minimum supported OS10EE versions by switch model. Run the show version command to check the OS version. Dell EMC recommends upgrading to the latest release available on Dell Digital Locker (account required). OS10# show version Dell EMC Networking OS10-Enterprise Copyright (c) 1999-2019 by Dell Inc. All Rights Reserved. OS Version: 10.4.2.
6.3 Factory default configuration The switch configuration commands in the sections that follow begin with the leaf switches at their factory default settings. Dell EMC Networking switches running OS10EE can be reset to their default configuration as follows: OS10# delete startup-configuration Proceed to delete startup-configuration [confirm yes/no(default)]:y OS10# reload System configuration has been modified.
7 Configure leaf switches This chapter details the configuration commands issued to the S5248F-ON leaf switches. The switches start at their factory default settings per Section 6.3. The commands in the sections that follow should be entered in the order shown. Note: S5248F-Leaf1A and S5248F-Leaf1B switch running-configuration files, with and without DCB, are provided as text file attachments to this .pdf. Section 1.2 describes how to access .pdf attachments.
7.1 General settings Enter configuration mode with the configure terminal command. Configure the hostname. If DHCP is not used, configure the OOB management IP address and default gateway. Specify an NTP server accessible by the switch. In this example, this is an NTP server on the OOB management network. 5. Enable RSTP as a precaution against loops.
7.3 Configure VLANs and VRRP In this section, VLANs are configured as shown in Table 2. VLAN 1611 is put in layer 3 mode by assigning an IP address. This allows routing of in-band management and/or VM traffic. VRRP is configured to provide gateway redundancy. 1. Create the in-band management and VM VLAN. 2. Assign a unique IP address on each switch to the VLAN interface. 3. Create a VRRP virtual router with the vrrp-group number command. Note: VRRP is an active/standby first hop redundancy protocol.
7.4 Configure storage interfaces Configure the interfaces for the storage connections. 25GbE interfaces 1/1/1 through 1/1/4 on both switches are used in this example. 1. 2. 3. 4. 5. Provide an interface description. Use the switchport mode trunk command to enable ports to carry traffic for multiple VLANs. Allow the two storage VLANs, 1613 and 1614, on the trunk ports. Set the MTU to its maximum size, 9216 bytes. This improves storage traffic performance.
7.5 S5248F-Leaf1A S5248F-Leaf1B switchport mode trunk switchport trunk allowed vlan 16131614 mtu 9216 flowcontrol receive on flowcontrol transmit off spanning-tree port type edge switchport mode trunk switchport trunk allowed vlan 16131614 mtu 9216 flowcontrol receive on flowcontrol transmit off spanning-tree port type edge Configure in-band management interfaces Configure the interfaces for connections to the in-band management and VM network (VLAN 1611).
7.6 Configure VLT The QSFP28-DD ports on the S5248F-ON are used for the VLTi between the two leaf switches as shown in Figure 5. The two physical QSFP28-DD ports provide four 100GbE interfaces, 1/1/49-1/1/52. Configure the VLTi interfaces as follows: 1. Provide an interface description. 2. Remove each interface from Layer 2 mode with the no switchport command. 3. Set the MTU to its maximum size, 9216 bytes. Configure the VLT domain as follows: 1.
7.7 S5248F-Leaf1A S5248F-Leaf1B discovery-interface ethernet1/1/491/1/52 peer-routing discovery-interface ethernet1/1/491/1/52 peer-routing end write memory end write memory Configure DCB Note: This section is only applicable if DCB used. If DCB is not used, skip to Chapter 8. See Chapter 4 for more information on DCB options. To configure DCB, complete the following steps: 1. Enter configuration mode with the configure terminal command. 2. Disable DCBx.
ETS configuration S5248F-Leaf1A S5248F-Leaf1B qos-map queue 0 queue 3 queue 5 qos-map queue 0 queue 3 queue 5 traffic-class queue-map qos-group 0-2,4,6-7 qos-group 3 qos-group 5 traffic-class queue-map qos-group 0-2,4,6-7 qos-group 3 qos-group 5 class-map type queuing q0 match queue 0 class-map type queuing q0 match queue 0 class-map type queuing q3 match queue 3 class-map type queuing q3 match queue 3 class-map type queuing q5 match queue 5 class-map type queuing q5 match queue 5 policy-map typ
Configure the interfaces participating in DCB. In this example, this includes the server-facing storage interfaces (Eth 1/1/1-1/1/4) and the VLTi interfaces (Eth 1/1/49-1/1/52). Note: The commands shown are in addition to the interface configuration commands issued in the preceding sections. 1. Disable transmit and receive link level flow control using the flowcontrol receive | transmit off commands. 2. Assign the PFC service policy and enable priority flow control on the interfaces. 3.
8 Switch validation Use the commands in this section to help verify the two leaf switches are configured correctly, and connections are properly established. The commands and output shown are for S5248F-Leaf1A. The output of its peer, S5248F-Leaf1B, is similar. Note: For additional commands and output related to the leaf-spine portion of the topology, such as BGP and Uplink Failure Detection (UFD), see Dell EMC Networking Layer 3 Leaf-Spine Deployment and Best Practices with OS10EE. 8.
A Eth1/1/21:1,1/1/22:1,1/1/23:1,1/1/24:1,1/1/31:1 1613 Active Storage_1 8.1.3 1614 Active 4094 Active Storage_2 T T T T T Eth1/1/1-1/1/4 Po1000 Eth1/1/1-1/1/4 Po1000 Po1000 show lldp neighbors The show lldp neighbors command is useful for identifying connected equipment by port. Note: Mellanox ConnectX-4 Lx adapters set to “Host in Charge” per Section 9.12 have LLDP disabled.
8.2 VLT validation commands 8.2.1 show vlt domain_id This command is used to validate the VLT configuration status. The Role for one switch in the VLT pair is primary, and its peer switch, not shown, is assigned the secondary role. The VLTi Link Status and VLT Peer Status must both be up.
8.2.3 show vlt domain_id mismatch This command highlights configuration issues between VLT peers. Mismatch examples include incompatible VLT configuration settings and VLAN differences. Any mismatches reported should be resolved before proceeding.
9 S2D Ready Node configuration and deployment These steps are applicable to Windows Server 2016 or 2019 Datacenter with Server Core or Server with Desktop Experience installations. The network adapter names shown in the command examples may vary depending on adapters installed in the system. For PowerShell commands that require a network adapter name, use the Get-NetAdapter cmdlet to retrieve the name for the associated port. Note: This chapter ends with Section 9.13 - Configure DCB on Ready Nodes.
9.2 Ensure DCB is disabled on Intel X710 adapters DCB, if used, is only configured on the QLogic or Mellanox adapters for this deployment. Dell EMC recommends disabling DCB on the Intel X710 adapters. On each of the Ready Nodes, ensure Data Center Bridging is disabled for both Intel X710 network adapter ports as follows: 1. In Windows Server, go to Device Manager > Network Adapters. 2. Double click on the first Intel Ethernet 10G X710 port to open the Properties dialog box. 3.
9.3 Install roles and features On each of the Ready Nodes, install the Hyper-V, Failover Clustering, and, if DCB is used, the Data Center Bridging roles and features using the Install-WindowsFeature PowerShell cmdlet. The command below also installs the management tools for Hyper-V and Failover Clustering and restarts the system.
The example above is for Ready Node 1. IP addresses used for the Ready Nodes are listed in Table 3. The default gateway is set to the leaf pair’s VRRP address, 172.16.11.254. 4. Configure the DNS server address on each Ready Node. PS> Set-DnsClientServerAddress -InterfaceAlias ‘vEthernet (Management)’ ServerAddresses 172.16.11.5 Note: Since the management system is not running Hyper-V, its two 10GbE NICs connected to the leaf switches are simply configured as a switch independent NIC team.
9.5.3 Enable RDMA RDMA is enabled on each storage network adapter port with the Enable-NetAdapterRdma cmdlet. The wildcard (*) may be used because the same command applies to both ports in slot 2. PS> Enable-NetAdapterRdma -Name 'SLOT 2*' RDMA is enabled for Live Migration traffic with the following PowerShell command: PS> Set-VMHost –VirtualMachineMigrationPerformanceOption SMB 9.5.4 Enable iWARP on QLogic adapters Note: Skip this step for Mellanox adapters because they do not use iWARP.
9.6 Join Ready Nodes to the Active Directory domain Before the cluster is created, the nodes must be a part of an Active Directory (AD) domain. This task can be performed with the Add-Computer cmdlet. Run the following command on each Ready Node that will be a part of the Storage Spaces Direct cluster. Note: Connecting to AD services using the host management network may require routing to the AD network. Ensure that this is in place before proceeding to domain join.
If there are existing partitions or other data on the drives, the drives must be cleaned before creating the host cluster. Warning: With the exception of the boot drives, the following script removes all data from all disks on the hosts specified in $ServerList. For more information, see https://docs.microsoft.com/en-us/windowsserver/storage/storage-spaces/deploy-storage-spaces-direct.
9.8 Run the Test-Cluster cmdlet Before creating the S2D cluster, ensure the Ready Nodes are properly configured and are ready for the cluster creation. This is done using the Test-Cluster cmdlet.
---Cluster-01 Review the cluster creation report for any warnings. The warning shown in the command output above is because a cluster witness has not been configured. A witness is configured in the following section. 9.10 Configure a cluster witness Microsoft recommends configuring a witness for a cluster with three or four nodes and requires a witness for a two-node cluster. Witness configuration is not required for clusters with five or more nodes. Note: For more information see https://docs.microsoft.
9.11 Enable S2D The Enable-ClusterS2D cmdlet puts the cluster in S2D mode. It creates the storage pool, configures the S2D cache drives, and creates capacity and performance tiers. 1. Run the following command on any one of the Ready Nodes to enable S2D on the cluster: PS> Enable-ClusterS2D –Verbose Note: The command only needs to be run on once as it applies to the entire cluster. The -Verbose option is useful as it shows the S2D configuration steps as they are executed on the screen.
9.13 Configure DCB on Ready Nodes This section is only necessary if DCB is used. See Chapter 4 for more information on DCB options. Run the following commands on each of the Ready Nodes in the cluster to configure DCB: 1. Create a new QoS policy with a match condition set to 445. The TCP port dedicated for Server Message Block (SMB) traffic is used to set the priority value to 3. PS> New-NetQosPolicy -Name 'SMB' –NetDirectPortMatchCondition 445 – PriorityValue8021Action 3 2.
10 PowerShell validation commands and output The following commands are run as needed on any of the S2D Ready Nodes to validate the configuration. Note: Some output in the examples shown has been truncated to fit the page. 10.1 Get-NetAdapter Use the Get-NetAdapter cmdlet to view physical and virtual network adapters and their status. The vEthernet (Mangement) adapter shown in the output below is created in Section 9.3.
10.3 Get-NetIPAddress Use the Get-NetIPAddress cmdlet to view IP addresses configured on the node. The output that follows is from Ready Node 1. PS> Get-NetIPAddress -AddressFamily IPv4 | Select InterfaceAlias, IPAddress InterfaceAlias -------------vEthernet (Management) SLOT 2 Port 1 SLOT 2 Port 2 Loopback Pseudo-Interface 1 10.4 IPAddress --------172.16.11.1 172.16.13.1 172.16.14.1 127.0.0.1 Get-NetRoute The Get-NetRoute cmdlet is used to verify the default gateway configured on the Ready Node.
10.7 Get-NetAdapterRdma Use the Get-NetAdapterRdma cmdlet to verify storage adapters have RDMA enabled. RDMA is enabled in Section 9.4.3. PS> Get-NetAdapterRdma Name ---vEthernet (Management) SLOT 2 Port 1 SLOT 2 Port 2 10.
10.10 Storage validation commands 10.10.1 Get-PhysicalDisk The Get-PhysicalDisk cmdlet is run to verify disk health, size, and media type on a Ready Node.
10.10.2 Get-StoragePool The Get-StoragePool cmdlet can be run on any node to determine available space in the shared storage pool. In the following example, the cluster has 58.45TB of capacity and 12.06TB used space. PS> Get-StoragePool -IsPrimordial $False FriendlyName Status Health ----------------- ------S2D on Cluster-01 OK Healthy Primordial ReadOnly Size ---------- -------- -------False False 58.45 TB AllocatedSize ------------12.06 TB 10.10.
10.11.2 Get-NetAdapterQos Use the Get-NetAdapterQos cmdlet to verify the NIC’s DCB operational parameters. Verify the OperationalTrafficClasses bandwidth and priorities match those configured on the switch, shown in Table 11, and OperationalFlowControl is enabled for priority 3 and 5 traffic as configured on the switch, shown in Table 12. Note: The following output is from the Mellanox adapter. The output for the QLogic adapter is similar.
10.11.3 Get-NetQosFlowControl Verify the priority flow control settings with the Get-NetQosFlowControl cmdlet. The output shown confirms PFC is enabled for priority 3 and 5 traffic. PS> Get-NetQosFlowControl Priority -------0 1 2 3 4 5 6 7 Enabled ------False False False True False True False False PolicySet --------Global Global Global Global Global Global Global Global IfIndex IfAlias ------- ------- 10.11.
11 Switch DCB validation commands If DCB is used, the commands in this section will help verify DCB is configured correctly. The commands and output shown are for S5248F-Leaf1A. The output of its peer, S5248F-Leaf1B, is similar. 11.1 Show queuing statistics Queuing statistics can be viewed to verify that traffic is applied to the correct queues and that storage and cluster heartbeat traffic (queues 3 and 5 respectively) is not dropped under heavy network loads.
A Validated components The following tables include the hardware, software, and firmware used to configure and validate the examples in this guide. A.1 Dell EMC Networking Switches Switches and OS versions Qty Item OS Version 2 Dell EMC Networking S5248F-ON leaf switch 10.4.2.2 1 Dell EMC Networking S3048-ON management switch 10.4.2.2 Note: See Table 1 for supported Dell EMC Networking switches and OS versions. A.
PowerEdge R740xd S2D Ready Node components with WS2019 A.3 Qty per Item node Firmware Driver 2 Intel Xeon Gold 6148 CPU @ 2.40GHz, 20 cores - - 12 16GB DDR4 DIMM (192GB total) - - 2 960GB SATA SSD - - 4 4TB SATA HDD - - 1 Dell HBA330 Storage Controller 16.17.00.03 2.51.24.80 1 Boot Optimized Storage Solution (BOSS) S1 Controller w/ 2x240GB SATA M.2 SSD 2.5.13.3016 1.2.0.1048 1 QLogic FastLinQ QL41262HxCU-DE 2x25GbE SFP28 14.10.07 8.37.35.
A.4 Dell EMC Networking Optics and Cables The optics and cables in the table below were used to validate the examples in this guide. Optics and cables A.5 Qty Item Part Number 16 SFP28-25G-SR or SFP28-25G-SR-NOF optical transceiver P7D7R or W4GPP 20 10G SFP+ SR optical transceiver WTRD1 18 LC/LC OM3 MMF patch cable - 2 DAC-Q28DD-200G-1.
B Port-groups In this deployment guide, S5248F-ON interfaces eth1/1/21-1/1/24 and 1/1/31 are connected to 10GbE Intel X710 adapters. These interfaces must be changed from their native speed of 25GbE to 10GbE. This is done on S5248F-ON switches using the port-group command. Note: The port-group command is used in Section 7.2 of this guide.
S5248F-ON port-group configuration options 60 Port group number Native physical interface Native speed Other supported speeds Non-native logical interface name 1/1/1 1/1/2 1/1/3 1/1/4 1/1/5 1/1/6 1/1/7 1/1/8 1/1/9 1/1/10 1/1/11 1/1/12 1/1/13 1/1/14 1/1/15 1/1/16 1/1/17 1/1/18 Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth Eth 25g-4x 25g-4x 25g-4x 25g-4x 25g-4x 25g-4x 25g-4x 25g-4x 25g-4x 25g-4x 25g-4x 25g-4x 100g-2x 100g-2x 100g-1x 100g-1x 100g-1x 100g-1x 10g-4x 10g-4x 10g-4x
C Technical resources Dell EMC Networking Guides OS10 Enterprise Edition User Guide Release 10.4.2.
D Fabric Design Center The Dell EMC Fabric Design Center (FDC) is a cloud-based application that automates the planning, design, and deployment of network fabrics that power Dell EMC compute, storage and hyper-converged infrastructure solutions. The FDC is ideal for turnkey solutions and automation based on validated deployment guides. FDC allows design customization and flexibility to go beyond validated deployment guides. For additional information, visit the Dell EMC Fabric Design Center.
E Support and feedback Contacting Technical Support Support Contact Information Web: http://www.dell.com/support Telephone: USA: 1-800-945-3355 Feedback for this document We encourage readers to provide feedback on the quality and usefulness of this publication by sending an email to Dell_Networking_Solutions@Dell.com.
