Dell EMC Networking – RDMA over Converged Ethernet (RoCE v1) Cheat Sheet RoCE v1 Configuration cheat sheet with Dell EMC OS10 Abstract Detailed RoCEv1 configuration with Dell EMC networking and OS10 August 2018 Dell EMC Networking – RDMA over Converged Ethernet (RoCE v1) Cheat Sheet
Revisions Revisions Date Description August 2018 Initial release Acknowledgements This paper was produced by the following members of the Dell EMC Networking Technical Marketing engineering team: Author: Mario Chow Other: The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Table of contents Revisions.............................................................................................................................................................................2 Acknowledgements .............................................................................................................................................................2 1 2 3 3 Introduction .............................................................................................................
1 Introduction Your typical next generation data center is a dynamic and scalable asset that every enterprise is learning to harness to achieve a competitive edge. It is a convergence point of multiple types of workloads and technologies that together deliver business applications, 1.1 Objective The objective of this document is to provide a set of valid Dell EMC networking configurations that apply to a converged environment consisting of RDMA, iSCSI, and LAN (Local Area Network) traffic. 1.
2 Setup Figure 1, describes the reference test environment used to deploy a converged data center. The test environment uses a distributed Spine-Leaf architecture, where the leaf layer acts as both Leaf and ToR (Top of Rack). The connected servers to the Leaf/ToR switches simulate a rack within a data center. Each server is simulating FCoE, and RDMA traffic, however, for the sake of this document, only RDMA traffic is highlighted.
Dell EMC Networking RoCE v1 Reference Environment. To demonstrate how Dell EMC networking ensures quality of service in a converged environment, congestion is created on the links between the leaf and spine. A port-channel sized at 20GbE is configured between the leaf and spine interlinks. Through the use of an IXIA traffic generator, LAN and iSCSI traffic are simulated. LAN traffic is running at 40Gbps, and iSCSCI is running at 10Gbps easily congesting the inter-links between the leaf and spine switches.
3 Configuration details The following section depicts in full detail the entire switch configuration for both the spines and leaf. For brevity, only one spine configuration is shown. The configuration is repeated for the other spine. 3.1 Spine – S4148Fs 3.2 S4148F (OS10) 1. Configure VLT a. Switch# conf b. Switch(conf)# vlt domain 1 c. Switch(conf-vlt-1)# discovery-interface ethernet1/1/1-1/1/2 d. Switch(conf-vlt-1)# end e. Switch# 2. Enable lldp a. Switch# conf b. Switch(conf)# lldp enable c.
f. Switch(config-qos-map)# end g. Switch# 6. Start configuring the class maps to be used with all respective policy maps – first one “queueing”. 4a. Queueing – This queues refer to the queues configured in step 3. a. Switch# conf t b. Switch(config)# class-map type queueing q0 c. Switch(config-cmap-queuing)# match queue 0 d. Switch(config-cmap-queuing)# exit e. Switch(config)# class-map type queuing q4 f. Switch(config-cmap-queuing)# match queue 4 g. Switch(config-cmap-queuing)# exit h.
c. d. e. f. g. h. i. j. k. l. Switch(config-pmap-queuing)# class q0 Switch(config-pmap-c-que)# bandwidth percent 10 Switch(config-pmap-c-que)# exit Switch(config-pmap-queuing)# class q4 Switch(config-pmap-c-que)# bandwidth percent 40 Switch(config-pmap-c-que)# exit Switch(config-pmap-queuing)# class q5 Switch(config-pmap-c-que)# bandwidth percent 50 Switch(config-pmap-c-que)# end Switch# 5b. PFC policy map – Policy map turning pfc “ON” on CoS 4 and 5 or iSCSI and RDMA flows respectively. a.
PFC is turned “ON” at the interface k. Switch(config-if-eth1/1/5)# end l. Switch# 3.3 Leaf – S4048-3, S4048-4, S4148U 3.4 S4048-ON-3 (OS9) 1. Enable lldp a. Switch# conf b. Switch(conf)# protocol lldp c. Switch(conf-lldp)# no dis d. Switch(conf-lldp)# exit e. Switch(conf)# 2. Enable dcb and assign 4 priority queues a. Switch# conf b. Switch(conf)# dcb enable pfc-queues 4 c. Switch(conf)# exit d. Switch# 3. Configure DCB, PFC, ETS profiles to be used for all three traffic patterns. Priority group a.
3.5 S4048-ON-4 & S4148U (OS10) To configure the S4048-ON-4 and S4148U repeat steps 2 – 8 from the S4148F section. Note: The configurations applied on interface eth1/1/5 should be repeated for all other interfaces. However, not all configurations apply. For example, interface eth1/1/53 is only receiving LAN traffic, therefore, only vlan 100 would be configured.
