Mellanox InfiniBand OFED Driver for VMware vSphere 5.x User Manual Rev 1.8.2 www.mellanox.
Rev 1.8.2 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
Rev 1.8.2 Table of Contents Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Document Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rev 1.8.2 List of Tables Table 1: Document Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Table 2: Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Table 3: Abbreviations and Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Table 4: Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rev 1.8.2 Document Revision History Table 1 - Document Revision History Revision Date Change Description 1.8.2 Sep 2013 No changes 1.8.1 Feb 2013 Added Section 4.1, “SCSI RDMA Protocol”, on page 14 and its subsections. 1.8.0 June 2012 Document restructuring and minor content updates 1.4.1-2.0.
Rev 1.8.2 About this Manual This document provides instructions for installing and using drivers for Mellanox Technologies ConnectX®-2 and ConnectX®-3 based network adapter cards in a VMware ESXi Server environment. Intended Audience This manual is intended for system administrators responsible for the installation, configuration, management and maintenance of the software and hardware of InfiniBand adapter cards. It is also intended for application developers.
Rev 1.8.2 Common Abbreviations and Acronyms Table 3 - Abbreviations and Acronyms Abbreviation / Acronym Whole Word / Description B (Capital) ‘B’ is used to indicate size in bytes or multiples of bytes (e.g., 1KB = 1024 bytes, and 1MB = 1048576 bytes) b (Small) ‘b’ is used to indicate size in bits or multiples of bits (e.g.
Rev 1.8.2 Glossary The following is a list of concepts and terms related to InfiniBand in general and to Subnet Managers in particular. It is included here for ease of reference, but the main reference remains the InfiniBand Architecture Specification. Table 4 - Glossary 8 Channel Adapter (CA), Host Channel Adapter (HCA) An IB device that terminates an IB link and executes transport functions. This may be an HCA (Host CA) or a TCA (Target CA).
Rev 1.8.2 Related Documentation Table 5 - Reference Documents Document Name Description MFT User’s Manual Mellanox Firmware Tools User’s Manual. See under docs/ folder of installed package. MFT Release Notes Release Notes for the Mellanox Firmware Tools. See under docs/ folder of installed package. Support and Updates Webpage Please visit http://www.mellanox.com > Products > Software > InfiniBand/VPI Drivers/VMware Drivers for downloads, FAQ, troubleshooting, future updates to this manual, etc.
Rev 1.8.2 Mellanox InfiniBand OFED Driver for VMware® vSphere Overview 1 Mellanox InfiniBand OFED Driver for VMware® vSphere Overview 1.1 Introduction to Mellanox InfiniBand OFED for VMware Mellanox OFED is a single Virtual Protocol Interconnect (VPI) software stack based on the OpenFabrics (OFED) Linux stack adapted for VMware, and operates across all Mellanox network adapter solutions supporting up to 56Gb/s InfiniBand (IB) and 2.5 or 5.0 GT/s PCI Express 2.0 and 3.0 uplinks to servers.
Rev 1.8.2 1.4 mlx4 InfiniBand Driver MLNX-OFED-ESXi package contains: • MLNX-OFED-ESXi-1.8.1.zip - Hypervisor bundle which contains the following kernel modules: • mlx4_core (ConnectX family low-level PCI driver) • mlx4_ib (ConnectX family InfiniBand driver) • ib_core • ib_sa • ib_mad • ib_umad • ib_ipoib • ib_cm • ib_srp 1.4.1 ULPs IPoIB The IP over IB (IPoIB) driver is a network interface implementation over InfiniBand.
Rev 1.8.2 2 Installing Mellanox InfiniBand OFED Driver for VMware vSphere Installing Mellanox InfiniBand OFED Driver for VMware vSphere This chapter describes how to install and test the Mellanox InfiniBand OFED Driver for VMware vSphere package on a single host machine with Mellanox InfiniBand hardware installed. The InfiniBand OFED driver installation on VMware ESXi Server 5.0 is done using VMware's VIB bundles. Please uninstall any previous Mellanox driver packages prior to installing the new version.
Rev 1.8.2 3 Uninstalling Mellanox InfiniBand OFED Driver To uninstall the driver: 1. Log into the ESXi5.0 server with root permissions. 2. List the existing InfiniBand OFED driver modules. #> esxcli software vib list | grep Mellanox net-ib-core 1.8.1 OEM.500.0.0.472560 Mellanox Partner Supported 2013-02-09 net-ib-ipoib 1.8.1 OEM.500.0.0.472560 Mellanox Partner Supported 2013-02-09 ... 3. Remove each module using the "esxcli software vib remove..." command.
Rev 1.8.2 Driver Features 4 Driver Features 4.1 SCSI RDMA Protocol 4.1.1 SRP Overview The InfiniBand package includes a storage module called SRP, which causes each InfiniBand port on the VMware ESX Server machine to be exposed as one or more physical storage adapters, also referred to as vmhbas. To verify that all supported InfiniBand ports on the host are recognized and up, perform the following steps: 1. Connect to the VMware ESX Server machine using the interface of VMware vSPhere Client. 2.
Rev 1.8.2 For example, to set the maximum number of SRP targets per SCSI host to four, run: cos# esxcfg-module ib_srp -s 'max_srp_targets=4' 6. To find out all SRP's parameters, run: cos# vmkload_mod -s ib_srp Default values are usually optimum per performance however, if you need to manually set the system to achieve better performance, tune the following parameters: • srp_sg_tablesize - Maximum number of scatter lists supported per I/O. • srp_cmd_per_num - Maximum number of commands can queue per lun.
Rev 1.8.2 4.2.2 Driver Features IPoIB Configuration 1. Install the Mellanox OFED driver for VMware 2. Verify the driver VIBs are installed correctly. Run: # esxcli software vib list | grep -i Mellanox (a.k.a “esxcfg-nics -l”) 3. Verify the uplinks state is "up". Run: # esxcli network nic list | grep -i Mellanox (a.k.a “esxcfg-nics -l”) See your VMware distribution documentation for additional information about configuring IP addresses.
Rev 1.8.2 5 Configuring the Mellanox InfiniBand OFED Driver for VMware vSphere 5.1 Configuring an Uplink To configure an Uplink: 1. Add the device as an uplink to an Existing vSwitch using the CLI. a. Log into the ESXi server with root permissions. b. Add an uplink to a vSwitch. #> esxcli network vSwitch standard uplink add -v 2. Verify the uplink is added successfully. #> esxcli network vSwitch standard list To remove the device locally: 1.
Rev 1.8.2 5.2.2 Configuring the Mellanox InfiniBand OFED Driver for VMware vSphere Networking The InfiniBand package includes a networking module called IPoIB, which causes each InfiniBand port on the VMware ESXi Server machine to be exposed as one or more physical network adapters, also referred to as uplinks or vmnics. To verify that all supported InfiniBand ports on the host are recognized and up, perform the following steps: 1.
Rev 1.8.2 For VGT mode, VLANs need to be configured in the installed guest operating system. This procedure may vary for different operating systems. See your guest operating system manual on VLAN configuration. In addition, for each new interface created within the virtual machine, at least one packet should be transmitted. For example: Create a new interface (e.g., ) with IP address .
Rev 1.8.2 Configuring the Mellanox InfiniBand OFED Driver for VMware vSphere /usr/local/ofed/etc/opensm/partitions.conf and include the line: key0=0x7fff,ipoib,mtu=5 : ALL=full; i. Restart OpenSM: /etc/init.d/opensmd restart To enable 4k mtu support: run “esxcli system module parameters set m=mlx4_core -p=mtu_4k=1”. Changes will take effect after the reboot. 5.2.5 High Availability High Availability is supported for both InfiniBand network and storage adapters.
