Mellanox InfiniBand OFED Driver for VMware vSphere 4.X User Manual Rev 1.4.1-2.0.000 Last Updated: June 28, 2011 www.mellanox.
NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
Rev 1.4.1- Table of Contents Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Preface . . . . . . . . .
List of Tables Table 1: Typographical Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Table 2: Abbreviations and Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Table 3: Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Table 4: Reference Documents . . . . . . . . . . . . . . . . . . . . . . . . .
Rev 1.4.1- Revision History Rev 4.
Rev 1.4.1- Preface This Preface provides general information concerning the scope and organization of this User’s Manual.
Rev 1.4.1- Documentation Conventions Typographical Conventions Table 1 - Typographical Conventions Description Convention File names file.
Rev 1.4.
Rev 1.4.1- Glossary The following is a list of concepts and terms related to InfiniBand in general and to Subnet Managers in particular. It is included here for ease of reference, but the main reference remains the InfiniBand Architecture Specification. Table 3 - Glossary Channel Adapter (CA), Host Channel Adapter (HCA) An IB device that terminates an IB link and executes transport functions. This may be an HCA (Host CA) or a TCA (Target CA).
Rev 1.4.1- Related Documentation Table 4 - Reference Documents Document Name Description MFT User’s Manual Mellanox Firmware Tools User’s Manual. See under docs/ folder of installed package. MFT Release Notes Release Notes for the Mellanox Firmware Tools. See under docs/ folder of installed package. Support and Updates Webpage Please visit http://www.mellanox.com > Products > Adapter IB/VPI SW/VMware Drivers for downloads, FAQ, troubleshooting, future updates to this manual, etc.
Rev 1.4.1- 1 Mellanox InfiniBand OFED Driver for VMware® vSphere Overview 1.1 Introduction to Mellanox InfiniBand OFED for VMware Mellanox OFED is a single Virtual Protocol Internconnect (VPI) software stack based on the OpenFabrics (OFED) Linux stack adapted for VMware, and operates across all Mellanox network adapter solutions supporting 10, 20 and 40Gb/s InfiniBand (IB); 10Gb/s Ethernet (10GigE); and 2.5 or 5.0 GT/s PCI Express 2.0 uplinks to servers.
Rev 1.4.1- Mellanox InfiniBand OFED Driver for VMware® vSphere Overview IPoIB, SRP Initiator y Utilities Diagnostic tools Performance tests y Firmware tools (MFT) y Documentation 1.4 mlx4 VPI Driver mlx4 is the low level driver implementation for the ConnectX® and ConnectX®-2 adapters designed by Mellanox Technologies. ConnectX/ConnectX-2 can operate as an InfiniBand adapter, as an Ethernet NIC. The Mellanox OFED driver for VMware supports InfiniBand and Ethernet NIC configurations.
Rev 1.4.1- instead, it controls a connection to an I/O controller—known as the SRP Target—to provide access to remote storage devices across an InfiniBand fabric. The SRP Target resides in an I/O unit and provides storage services. See Chapter 3.1, “SCSI RDMA Protocol”. 1.4.2 Mellanox Firmware Tools Mellanox Firmware Tools solution applies to VMware ESX Servers only. The Mellanox Firmware Tools (MFT) package is a set of firmware management tools for a single InfiniBand node.
Installation 2 Installation This chapter describes how to install and test the Mellanox InfiniBand OFED Driver for VMware vSphere package on a single host machine with Mellanox InfiniBand and/or Ethernet adapter hardware installed. The chapter includes the following sections: 2.1 y Section 2.1,“Hardware and Software Requirements,” on page 14 y Section 2.2,“InfiniBand OFED Driver Installation on VMware ESX/ESXi Server,” on page 15 y Section 2.2.
Rev 1.4.1- 2.2 InfiniBand OFED Driver Installation on VMware ESX/ESXi Server The InfiniBand OFED driver installation on VMware ESX Server 4.x is done using VMware's VIB bundles. Please uninstall any previous versions on ESX/ESXi before installing the new version. 2.2.1 Installing InfiniBand OFED Driver on a VMware ESXi Server ESXi package is available as a standalone or as part of the vSphere Management Assistant (vMA) virtual appliance from http://www.vmware.com.
Installation 2.2.2 Installing InfiniBand OFED Driver on a VMware ESX Server To install the driver package on a VMware ESX Server machine: 1. Set the machine to maintenance mode 2. Log into the service console as root and execute the following steps: Before installing, please verify if you have on your machine of the following ESX builds. Run : /usr/bin/vmware -v VMware ESX 4.1.0 build-348481' - ESX 4.1 Update 1 VMware ESX 4.1.0 build-260247' - ESX 4.1 VMware ESX 4.0.0 build-164009' - ESX 4.
Rev 1.4.1- 3 Driver Features 3.1 SCSI RDMA Protocol 3.1.1 SRP Overview The InfiniBand package includes a storage module called SRP, which causes each InfiniBand port on the VMware ESX Server machine to be exposed as one or more physical storage adapters, also referred to as vmhbas. To verify that all supported InfiniBand ports on the host are recognized and up, perform the following steps: 1. Connect to the VMware ESX Server machine using the interface of VMware VI Client. 2.
Rev 1.4.1- Driver Features cos# esxcfg-module ib_srp -s 5. To apply your changes, reboot the machine: cos# reboot For example, to set the maximum number of SRP targets per SCSI host to four, run: cos# esxcfg-module ib_srp -s 'max_srp_targets=4' 6.
Rev 1.4.1- y Unreliable Datagram IPoIB also supports the following software based enhancements: y Large Receive Offload y Ethtool support 3.2.2 IPoIB Configuration 1. Install the Mellanox OFED driver for VMware 2. Verify the drivers are installed correctly and linked up. Run: esxcfg-nics -l See your VMware distribution documentation for additional information about configuring IP addresses.
Working With VPI Rev 1.4.1-2.0.000 4 Working With VPI 4.1 VPI - Multi-Protocol Support This driver package supports Mellanox's multi-protocol VPI technology. VPI means the driver supports the coexistence of 10GigE NICs and IB HCAs on the same host (ESX server), and depending on the ConnectX device type also the coexistence of 10GigE and IB ports on the same HCA device. The following port configurations are supported in VPI: (IB,IB), (IB,ETH) and (ETH,ETH).
Rev 1.4.1-2.0.000 ESX cos> connectx_port_config -h for detailed info ESX cos> connectx_port_config -d -c< configuration> ESXi cos> connectx_port_config for detailed info ESXi cos> connectx_port_config < configuration> 4.1.2 Removing Corrupted Configuration Files In certain cases the old connectx_port_config script may leave behind some bogus port configuration. To delete them, follow the steps below: 1. Retrieve the current configuration string.
Working With VPI Rev 1.4.1-2.0.000 4.2.2 Networking The InfiniBand package includes a networking module called IPoIB, which causes each InfiniBand port on the VMware ESX Server machine to be exposed as one or more physical network adapters, also referred to as uplinks or vmnics. To verify that all supported InfiniBand ports on the host are recognized and up, perform the following steps: 1. Connect to the VMware ESX Server machine using the interface of VMware vSphere Client. 2.
Rev 1.4.1-2.0.000 Ethernet VLANs are implemented on InfiniBand using Partition Keys (See RFC 4392 for information). Thus, the InfiniBand network must be configured first. This can be done by configuring the Subnet Manager (SM) on your subnet. Note that this configuration is needed for both VLAN configuration modes, VGT and VST. For further information on the InfiniBand Partition Keys configuration for IPoIB, see the Subnet Manager manual installed in your subnet.
Working With VPI Rev 1.4.1-2.0.000 a. Open "Network Connections" b. Select the IPoIB adapter and right click on it c. Select "Properties" d. Press "Configure" and then go to the "Advanced" tab e. Select the payload MTU size and change it to 1500 f. Make sure that the firmware of the HCAs and the switches supports the MTU you wish to set. g. Configure your Subnet Manager (SM) to set the MTU value in the configuration file. The SM configuration for MTU value is per Partition Key (PKey).
Rev 1.4.1-2.0.000 4.2.7 High Availability High Availability is supported for both InfiniBand network and storage adapters. A failover port can be located on the same HCA card or on a different HCA card on the same system (for hardware redundancy). To define a failover policy for InfiniBand networking and/or storage, follow the instructions in the ESX Server Configuration Guide document.
