Mellanox WinOF VPI User Manual Rev 4.3 www.mellanox.
Rev 4.3 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
WinOF VPI for Windows User Manual Rev 4.3 Table of Contents Document Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Intended Audience . . . . . . . . . . . . . . . . . .
Rev 4.3 3.3.1 Verifying SMB Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.3.2 Verifying SMB Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.4 Verifying SMB Events that Confirm RDMA Connection . . . . . . . . . . . . . . . . . 30 Chapter 4 Driver Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.1 Configuring the InfiniBand Driver . . . . . . . . .
WinOF VPI for Windows User Manual 7.4.8 7.4.9 7.4.10 7.4.11 7.4.12 7.4.13 7.4.14 7.4.15 7.4.16 7.4.17 7.4.18 7.4.19 7.4.20 Rev 4.3 ibv_read_lat. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 ibv_send_bw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 ibv_send_lat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rev 4.3 List of Tables Table 1: Table 2: Table 3: Table 4: Table 5: Table 6: Table 7: Table 8: Table 9: Table 10: Table 11: Table 12: Table 13: Table 14: Table 15: Table 16: Table 17: Table 18: Table 19: Table 20: Table 21: Table 22: Table 23: Table 24: Table 25: Table 26: Table 27: Table 28: Table 29: Table 30: Table 31: Table 32: Table 33: Table 34: Table 35: Table 36: Table 37: 5 Document Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
WinOF VPI for Windows User Manual Rev 4.3 Document Revision History Table 1 - Document Revision History Document Revision Rev 4.3 Date January 6, 2013 Changes • • Changed the options description in Appendix 5.1.4.3,“Options,” on page 39 Added “perf_tuning”Appendix 5.1.4.2,“Synopsys,” on page 38 Added “-st” flag to Appendix 5.1.4.3,“Options,” on page 39 Added ‘ibdiagnet’ utility: Appendix 7.3.2.1,“ibdiagnet Options,” on page 50 Appendix 7.3.2.2,“ibdiagnet Output Files,” on page 50 Appendix 7.3.2.
Rev 4.3 Table 1 - Document Revision History Document Revision Rev 2.1.2 Date October 10, 2010 Changes • • • • 7 Removed section Debug Options. Updated Section 3, “Uninstalling Mellanox VPI Driver,” on page 11 Added Section 6, “InfiniBand Fabric,” on page 38 and its subsections Added Section 6.3, “InfiniBand Fabric Performance Utilities,” on page 71 and its subsections Rev 2.1.1.1 July 14, 2010 Removed all references of InfiniHost® adapter since it is not supported starting with WinOF VPI v2.1.
WinOF VPI for Windows User Manual Rev 4.3 About this Manual Scope The document describes WinOF Rev 4.3 features, performance, InfiniBand diagnostic, tools content and configuration. Additionally, this document provides information on various performance tools supplied with this version. Intended Audience This manual is intended for system administrators responsible for the installation, configuration, management and maintenance of the software and hardware of VPI (InfiniBand, Ethernet) adapter cards.
Rev 4.3 Documentation Conventions Table 2 - Documentation Conventions Description Convention Example File names file.
WinOF VPI for Windows User Manual Rev 4.
Rev 4.3 1 Introduction Introduction This User Manual addresses the Mellanox WinOF driver Rev 4.3 package distributed for: • Windows Server 2012 (64 bit only) • Windows Server 2008 (32 bit, and 64 bit) • Windows Server 2008 R2 (64 bit only) Mellanox WinOF is composed of several software modules that contain an InfiniBand and Ethernet driver. The Mellanox WinOF driver supports Infiniband and 40/10GB Ethernet ports. The port type is determined upon boot based on card’s capability and user setting. 1.
WinOF VPI for Windows User Manual 1.3.1 Rev 4.3 Downloading the Firmware Tools Package 1. Download Mellanox Firmware Tools Please download the current firmware tools package (MFT) from http://www.mellanox.com > Products > Software/Drivers > InfiniBand & VPI SW/Drivers > Firmware Tools. The tools package to download is "MFT Software for Windows_x64" for x64 architecture. 2. Install and Run WinMFT To install the WinMFT package, double click the MSI or run it from the command prompt.
Rev 4.3 2 Driver Features Driver Features The Mellanox VPI WinOF driver release introduces the following capabilities: • One or two ports • Up to 16 Rx queues per port • Rx steering mode (RSS) • Hardware Tx/Rx checksum calculation • Large Send Offload (i.e.
WinOF VPI for Windows User Manual • 2.1.3 Rev 4.3 Performance tests Reading Port Counters Statistics RoCE port statistics are not shown in the Windows network counters associated with Etherent interface. It is possible to read port statistics in the same way it is done for regular InfiniBand ports. The information is available by running vstat -c. 2.1.4 Setting RoCE » To set the RoCE, please perform the following: Step 1. Open Device Manager and expand System devices display pane. Step 2.
Rev 4.3 Driver Features Enter the following command to display the current status run of Mellanox WinOF VPI. vstat.exe hca_idx=0 uplink={BUS=PCI_E, SPEED=5.0 Gbps, WIDTH=x4, CAPS=2.5*x8} MSI-X={ENABLED=1, SUPPORTED=128, GRANTED=10, ALL_MASKED=N} vendor_id=0x02c9 vendor_part_id=4099 hw_ver=0x0 fw_ver=2.10.
WinOF VPI for Windows User Manual 2.2.2 Rev 4.3 Running an Example Test - ib_send_bandwidth Step 1.
Rev 4.3 Driver Features Step 2. Start the client: ndpingpong.exe c 15.4.12.128 2222 p1 Using 8 processors. Sender Frequency is 2597685 1 100000 3.80 98.56 262824.1 2 100000 3.80 98.68 526268.5 4 100000 3.81 102.53 1049904.8 8 100000 3.95 98.81 2023532.8 16 100000 3.97 98.29 4025846.5 32 100000 4.05 100.25 7896576.7 64 100000 4.18 101.02 15324826.5 128 100000 4.66 100.65 27483686.0 256 100000 6.46 99.14 39617564.3 512 100000 7.08 101.52 72318742.7 1024 100000 7.70 99.41 132958061.2 2048 100000 8.99 99.
WinOF VPI for Windows User Manual 2.5 Rev 4.3 Receive Side Scaling (RSS) Mellanox WinOF Rev 4.3 IPoIB and Ethernet drivers use NDIS 6.30 new RSS capabilities. The main changes are: • Supports unlimited number of processors (previously 64) • Individual network adapter RSS configuration usage To set the RSS capability for individual adapter instead of global setting, and to improve RSS on Windows 2012 server, set the registry keys listed in the table below: Table 4 - Registry Keys Setting Sub-key 2.
Rev 4.3 2.6.2 Driver Features Port Protocol Configuration Step 1. Display the Device Manager and expand “System devices”. Step 2. Right-click on the Mellanox ConnectX VPI network adapter and left-click Properties. Select the Port Protocol tab from the Properties sheet. The “Port Protocol” tab is displayed only if the NIC is a VPI (IB and ETH).
WinOF VPI for Windows User Manual Rev 4.3 The figure below is an example of the displayed Port Protocol sheet for a dual port VPI adapter card. Step 3. In this step, you can perform the following functions: • Choose HW Defaults option. If you choose the HW Defaults option, the port protocols will be determine according to the NIC’s hardware default values. • Choose the desired port protocol for the available port(s).
Rev 4.3 2.7 Driver Features Load Balancing, Fail-Over (LBFO) and VLAN Windows 2012 Microsoft supports load balancing as part of the operating system. plese refer to Microsoft document “Windows 2012 Link Teaming”. However, in previous operating systems (Windows 2008, Windows 2008 R2 and Windows 7) we have separate drivers, please see the below section: 2.7.
WinOF VPI for Windows User Manual 2.7.2 Rev 4.3 Creating a Load Balancing and Fail-Over (LBFO) Bundle LBFO is used to balance the workload of packet transfers by distributing the workload over a bundle of network instances and to set a secondary network instance to take over packet indications and information requests if the primary network instance fails. The following steps describe the process of creating an LBFO bundle. Step 1. Display the Device Manager. Step 2.
Rev 4.3 Driver Features To create a new bundle, perform the following: 1. Click the Create button. 2. Enter a (unique) bundle name. 3. Select a bundle type. 4. Select the adapters to be included in the bundle (that have not been associated with a VLAN). 5. [Optional] Select Primary Adapter. An active-passive scenario used for data transfer of link disconnecting. In such scenario, the system uses one of the other interfaces.
WinOF VPI for Windows User Manual Rev 4.3 To modify an existing bundle, perform the following: 1. Select the desired bundle and click Modify 2. Modify the bundle name, its type, and/or the participating adapters in the bundle 3. Click the Commit button To remove an existing bundle, select the desired bundle and click Remove. You will be prompted to approve this action. Notes on this step: a.
Rev 4.3 2.7.3 Driver Features Creating a Port VLAN You can create a Port VLAN either on a physical Mellanox ConnectX® EN adapter or a virtual bundle (team). The following steps describe how to create a port VLAN. Step 1. 25 Display the Device Manager.
WinOF VPI for Windows User Manual Step 2. Rev 4.3 Right-click a Mellanox network adapter (under “Network adapters” list) and left-click Properties. Select the VLAN tab from the Properties sheet. If a physical adapter has been added to a bundle (team), then the VLAN tab will not be displayed.
Rev 4.3 Driver Features Step 3. Click New to open a VLAN dialog window. Enter the desired VLAN Name and VLAN ID, and select the VLAN Priority. After installing the first virtual adapter (VLAN) on a specific port, the port becomes disabled. This means that it is not possible to bind to this port until all the virtual adapters associated with it are removed. When using a VLAN, the network address is configured using the VLAN ID. Therefore,the VLAN ID on both ends of the connection must be the same.
WinOF VPI for Windows User Manual 2.7.4 Rev 4.3 Removing a Port VLAN To remove a port VLAN, perform the following steps: Step 1. In the Device Manager window, right-click the network adapter from which the port VLAN was created. Step 2. Left-click Properties. Step 3. Select the VLAN tab from the Properties sheet. Step 4. Select the VLAN to be removed. Step 5. Click Remove and confirm the operation.
Rev 4.3 Deploying Windows Server 2012 with SMB Direct 3 Deploying Windows Server 2012 with SMB Direct 3.1 Overview The Server Message Block (SMB) Protocol is a network file sharing protocol implemented in Microsoft Windows. The set of message packets that defines a particular version of the protocol is called a dialect. The Microsoft SMB Protocol is a client-server implementation and consists of a set of data packets, each containing a request sent by the client or a response sent by the server.
WinOF VPI for Windows User Manual Rev 4.3 1. Start a long-running file copy to create a lasting session with the SMB Server. 2. Open a PowerShell window while the copy is ongoing, and run the following cmdlets to verify the connection is using the right SMB dialect and that SMB Direct is working: Get-SmbConnection Get-SmbMultichannelConnection netstat.
Rev 4.3 4 Driver Configuration Driver Configuration Once you have installed Mellanox WinOF VPI package, you can perform various modifications to your driver to make it suitable for your system’s needs Changes made to the Windows registry happen immediately, and no backup is automatically made. Do not edit the Windows registry unless you are confident regarding the changes. 4.1 Configuring the InfiniBand Driver 4.1.
WinOF VPI for Windows User Manual Rev 4.3 Step 1. Display the Device Manager. Step 2. Right-click a Mellanox ConnectX VPI adapter (under “System devices” list) and left-click Properties.
Rev 4.3 Driver Configuration Step 3. Select the Information tab from the Properties sheet. To save this information for debug purposes, click Save To File and provide the output file name.
WinOF VPI for Windows User Manual 4.2 Rev 4.3 Configuring the Ethernet Driver The following steps describe how to configure advanced features. Step 1. Display the Device Manager.
Rev 4.3 Driver Configuration Step 2. Right-click a Mellanox network adapter (under “Network adapters” list) and left-click Properties. Select the Advanced tab from the Properties sheet. Step 3. Modify configuration parameters to suit your system. Please note the following: a. For help on a specific parameter/option, check the help window at the bottom of the dialog. b.
WinOF VPI for Windows User Manual 5 Rev 4.3 Performance Tuning This document describes how to modify Windows registry parameters in order to improve performance. Please note that modifying the registry incorrectly might lead to serious problems, including the loss of data, system hang, and you may need to reinstall Windows. As such it is recommended to back up the registry on your system before implementing recommendations included in this document.
Rev 4.3 Performance Tuning b. Dual port traffic - Improves performance for running traffic on both ports simultaneously. c. Forwarding traffic - Improves performance for running scenarios that involve both ports (for example: via IXIA) d. Multicast traffic - Improves performance when the main traffic runs on multicast. 6. Click on “Run Tuning” button.
WinOF VPI for Windows User Manual Rev 4.3 6. Click on “Run Tuning” button. Clicking the "Run Tuning" button will activate the general tuning as explained above and change several driver registry entries for the current adapter and it's sibling device, if the sibling is an Ethernet device as well. It will also generate a log including the applied changes. Users can view this log to restore the previous values. The log path is: %HOMEDRIVE%\Windows\System32\LogFiles\PerformanceTunning.
Rev 4.3 Performance Tuning perf_tuning.exe -m -c1 -b -n perf_tuning -st -c1 [-c2 ] 5.1.4.3 Options Flag -s Description Single port traffic scenario. This option can be followed by one or two connection names. The tuning will restore the default settings on the second connection and performed on the first connection.
WinOF VPI for Windows User Manual Rev 4.3 Flag -f Description Forwarding traffic scenario. This option must be followed by two connection names. The tuning in this case is codependent.
Rev 4.3 Performance Tuning Flag -st Description Single stream traffic scenario. This option must be followed by one or two connection names for an Ethernet adapter. The tuning will restore the default settings on the second connection and performed on the first connection. This option automatically sets: • • • • • • SendCompletionMethod = 0 RecvCompletionMethod = 2 *ReceiveBuffers = 1024 UseRSSForRawIP = 1 UseRSSForUDP = 1 In Operating Systems support NDIS6.
WinOF VPI for Windows User Manual Rev 4.3 To improve performance, activate the performance tuning tool as follows: 1. Start the "Device Manager" (open a command line window and enter: devmgmt.msc). 2. Open "Network Adapters". 3. Right click the relevant Ethernet adapter and select Properties. 4. Select the "Advanced" tab and select Performance Options 5. Modify performance parameters (properties) as desired. 5.2.1.1 Performance Known Issues 5.2.
Rev 4.3 Performance Tuning The number of receive buffers (default 1024). • Send Buffers The number of sent buffers (default 2048). • Performance Options Configures parameters that can improve adapter performance. • Interrupt Moderation Moderates or delays the interrupts’ generation. Hence, optimizes network throughput and CPU utilization (default Enabled). • When the interrupt moderation is enabled, the system accumulates interrupts and sends a single interrupt rather than a series of interrupts.
WinOF VPI for Windows User Manual Rev 4.3 • Interrupt Moderation RX Packet Time Maximum elapsed time (in usec) between the receiving of a packet and the generation of an interrupt, even if the moderation count has not been reached (default 10). • Rx Interrupt Moderation Type Sets the rate at which the controller moderates or delays the generation of interrupts making it possible to optimize network throughput and CPU utilization.
Rev 4.3 Performance Tuning Sets the number of SA query retries once a query fails. The valid values are 1 - 64 (default 10). • SA Query Timeout Sets the waiting timeout (in millisecond) of an SA query completion. The valid values are 500 60000 (default 1000 ms). This document describes how to modify Windows registry parameters in order to improve performance.
WinOF VPI for Windows User Manual 6 Rev 4.3 OpenSM - Subnet Manager OpenSM v3.3.11 is an InfiniBand Subnet Manager. For Mellanox WinOF VPI to operate, OpenSM must be running on at least one host machine in the InfiniBand cluster. Please use the embedded OpenSM in the WinOF package for testing purpose and small cluster. Otherwise, we recommend using OpenSM from FabricIT EFM™ or UFM™. OpenSM can run as a Windows service which can be started manually from the following directory: \t
Rev 4.3 7 7.1 InfiniBand Fabric InfiniBand Fabric Network Direct Interface The Network Direct Interface (NDI) architecture provides application developers with a networking interface that enables zero-copy data transfers between applications, kernel-bypass I/O generation and completion processing, and one-sided data transfer operations. NDI is supported by Microsoft and is the recommended method to write InfiniBand application.
WinOF VPI for Windows User Manual 7.3.1 • Section 7.3.7,“ibping,” on page 64 • Section 7.3.8,“ibnetdiscover,” on page 65 • Section 7.3.9,“ibtracert,” on page 69 • Section 7.3.10,“sminfo,” on page 70 • Section 7.3.11,“ibclearerrors,” on page 72 • Section 7.3.12,“ibstat,” on page 72 • Section 7.3.13,“vstat,” on page 73 • Section 7.3.14,“osmtest,” on page 73 Rev 4.3 Utilities Usage This section first describes common configuration, interface, and addressing for all the tools in the package.
Rev 4.3 InfiniBand Fabric 2. Define the environment variable IBDIAG_DEV_IDX 7.3.1.3 Addressing This section applies to the ibdiagpath tool only. A tool command may require defining the destination device or port to which it applies. The following addressing modes can be used to define the IB ports: • Using a Directed Route to the destination: (Tool option ‘-d’) This option defines a directed route of output port numbers from the local port to the destination.
WinOF VPI for Windows User Manual 7.3.2 Rev 4.3 ibdiagnet ibdiagnet [-c ] [-v] [-r] [-o ] [-t ] [-s ] [-i ] [-p ] [-pm] [-pc] [-P <=>] [-lw <1x|4x|12x>] [-ls <2.5|5|10>] [-skip ] 7.3.2.
Rev 4.3 InfiniBand Fabric Table 5 - ibdiagnet (of ibutils) Output Files Output File Description ibdiagnet.mcfdbs A dump of the multicast forwarding tables of the fabric switches ibdiagnet.masks In case of duplicate port/node Guids, these file include the map between masked Guid and real Guids ibdiagnet.sm List of all the SM (state and priority) in the fabric ibdiagnet.pm A dump of the pm Counters values, of the fabric links ibdiagnet.
WinOF VPI for Windows User Manual Rev 4.3 7.3.2.3 ibdiagnet Error Codes 1 2 3 4 5 6 7.3.3 - Failed Failed Failed Failed Failed Failed to to to to to to fully discover the fabric parse command line options intract with IB fabric use local device or local port use Topology File load requierd Package ibportstate Enables querying the logical (link) and physical port states of an InfiniBand port. It also allows adjusting the link speed that is enabled on any InfiniBand port.
Rev 4.3 InfiniBand Fabric Table 6 - ibportstate Flags and Options (Continued) Flag Description -G/--Guid Use GUID address argument. In most cases, it is the Port GUID.
WinOF VPI for Windows User Manual Rev 4.3 2. Query the status of two channel adapters using directed paths. > ibportstate -C mlx4_0 -D 0 1 PortInfo: # Port info: DR path slid 65535; dlid 65535; 0 port 1 LinkState:.......................Initialize PhysLinkState:...................LinkUp LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.
Rev 4.3 InfiniBand Fabric # Show the new configuration > ibportstate -C mlx4_0 -D 0 1 PortInfo: # Port info: DR path slid 65535; dlid 65535; 0 port 1 LinkState:.......................Initialize PhysLinkState:...................LinkUp LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps LinkSpeedEnabled:................5.0 Gbps (IBA extension) LinkSpeedActive:.................5.
WinOF VPI for Windows User Manual Rev 4.3 Table 7 - ibroute Flags and Options Flag Description -D/--Direct Use directed path address arguments. The path is a comma separated list of out ports. Examples: ‘0’ – self port ‘0,1,2,1,4’ – out via port 1, then 2, ... -G/--Guid Use GUID address argument. In most cases, it is the Port GUID. Example: ‘0x08f1040023’ -M/--Multicast Show multicast forwarding tables. The parameters and specify the MLID range.
Rev 4.3 InfiniBand Fabric Unicast lids [0x3-0x7] of switch Lid 2 guid 0x0002c902fffff00a (MT47396 InfiniscaleIII Mellanox Technologies): Lid Out Destination Port Info 0x0003 021 : (Switch portguid 0x000b8cffff004016: 'MT47396 Infiniscale-III Mellanox Technologies') 0x0006 007 : (Channel Adapter portguid 0x0002c90300001039: 'sw137 HCA-1') 0x0007 021 : (Channel Adapter portguid 0x0002c9020025874a: 'sw157 HCA-1') 3 valid lids dumped 3.
WinOF VPI for Windows User Manual Rev 4.3 7.3.5.1 smpquery Applicable Hardware All InfiniBand devices. 7.3.5.2 smpquery Synopsys smpquery [-h] [-d] [-e] [-c] [-v] [-D] [-G] [-s ] [-L] [-u] [-V] [-C ] [-P ] [-t ] [--node-name-map ] [op params] 7.3.5.3 smpquery Options The table below lists the various flags of the command.
Rev 4.
WinOF VPI for Windows User Manual Rev 4.3 NeighborMTU:.....................2048 SMSL:............................0 VLCap:...........................VL0-7 InitType:........................0x00 VLHighLimit:.....................4 VLArbHighCap:....................8 VLArbLowCap:.....................8 InitReply:.......................0x00 MtuCap:..........................2048 VLStallCount:....................0 HoqLife:.........................31 OperVLs:.........................VL0-3 PartEnforceInb:............
Rev 4.3 InfiniBand Fabric # Node info: DR path slid 65535; dlid 65535; 0 BaseVers:........................1 ClassVers:.......................1 NodeType:........................Channel Adapter NumPorts:........................2 SystemGuid:......................0x0002c9030000103b Guid:............................0x0002c90300001038 PortGuid:........................0x0002c90300001039 PartCap:.........................128 DevId:...........................0x634a Revision:........................
WinOF VPI for Windows User Manual Rev 4.
Rev 4.3 InfiniBand Fabric Examples perfquery perfquery perfquery perfquery perfquery perfquery perfquery -r -e -R -e -R -R -R 32 1 # read performance counters and reset -r 32 1# read extended performance counters and reset 0x20 1 # reset performance counters of port 1 only -R 0x20 1# reset extended performance counters of port 1 only -a 32 # reset performance counters of all ports 32 2 0x0fff# reset only error counters of port 2 32 2 0xf000# reset only non-error counters of port 2 1.
WinOF VPI for Windows User Manual Rev 4.3 XmtPkts:.........................1803332 RcvPkts:.........................1799018 3. Read then reset performance counters from LID 2, port 1. > perfquery -r 2 1 # Port counters: Lid 2 port 1 PortSelect:......................1 CounterSelect:...................0x0100 SymbolErrors:....................0 LinkRecovers:....................0 LinkDowned:......................0 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:.........
Rev 4.3 InfiniBand Fabric Table 10 - ibping Flags and Options Flag 7.3.8 Description --errors, -e Shows send and receive errors (timeouts and others) --help, -h Shows the usage message --verbose, -v/-vvv/-v -v -v Increases the application verbosity level --version, -V Shows the version info --Lid, -L Use LID address argument --usage, -u Usage message --Guid, -G Uses GUID address argument. In most cases, it is the Port GUID.
WinOF VPI for Windows User Manual Rev 4.3 Table 11 - ibnetdiscover Flags and Options Flag Description -g, --grouping Show grouping. Grouping correlates IB nodes by different vendor specific schemes. It may also show the switch external ports correspondence. -H, --Hca_list List of connected CAs -S, --Switch_list List of connected switches -R, --Router_list List of connected routers -s, --show Show progress information during discovery --node-name-map Specify a node name map.
Rev 4.3 InfiniBand Fabric Table 11 - ibnetdiscover Flags and Options Flag Description -usage, -u Usage message --Ca, -C Use the specified ca_name --Port, -P Use the specified ca_port --timeout, -t Override the default timeout for the solicited mads --full, -f show full information (ports' speed and width) --show, -s show more information 7.3.8.3 Topology File Format The topology file format is largely intuitive.
WinOF VPI for Windows User Manual Rev 4.
Rev 4.3 InfiniBand Fabric Example # IB1 # Line cards 0x0008f104003f125c 0x0008f104003f125d 0x0008f104003f10d2 0x0008f104003f10d3 0x0008f104003f10bf # Spines 0x0008f10400400e2d 0x0008f10400400e2e 0x0008f10400400e2f 0x0008f10400400e31 0x0008f10400400e32 # GUID Node Name 0x0008f10400411a08 0x0008f10400411a28 0x0008f10400411a34 0x0008f104004119d0 7.3.
WinOF VPI for Windows User Manual Rev 4.3 Table 12 - ibtracert Flags and Options Flag Description --Lid, -L Use LID address argument --errors, -e Show send and receive errors --usage, -u Usage message --Guid, -G Use GUID address argument. In most cases, it is the Port GUID.
Rev 4.3 InfiniBand Fabric 7.3.10.1 sminfo Synopsys sminfo [-d(ebug)] [-e(rr_show)] [-s state] [-p prio] [-a activity] [-D(irect)] [-L(id)] [-u(sage)] [-G(uid)] [-C ca_name] [-P ca_port] [-t(imeout) timeout_ms] [V(ersion)] [-h(elp)] sm_lid | sm_dr_path [modifier] 7.3.10.2 sminfo Options The table below lists the various flags of the command. Most OpenIB diagnostics take the following common flags.
WinOF VPI for Windows User Manual Rev 4.3 Examples sminfo # local ports sminfo sminfo 32 # show sminfo of lid 32 sminfo -G 0x8f1040023 # same but using guid address 7.3.11 ibclearerrors ibclearerrors is a script which clears the PMA error counters in PortCounters by either waking the IB subnet topology or using an already saved topology file. 7.3.11.1 ibclearerrors Synopsys ibclearerrors [-h] [-N | -nocolor] [ | -C ca_name -P ca_port -t(imeout) timeout_ms] 7.3.11.
Rev 4.
WinOF VPI for Windows User Manual Rev 4.3 • It verifies the existing inventory, with all the object fields, and matches it to a presaved one. • A Multicast Compliancy test. • An Event Forwarding test. • A Service Record registration test. • An RMPP stress test. • A Small SA Queries stress test. It is recommended that after installing opensm, the user should run "osmtest -f c" to generate the inventory file, and immediately afterwards run "osmtest -f a" to test OpenSM.
Rev 4.3 InfiniBand Fabric Table 17 - osmtest Flags and Options Flag Description -m, --max_lid This option specifies the maximal LID number to be searched for during inventory file build (default to 100) -g, --guid This option specifies the local port GUID value with which OpenSM should bind. OpenSM may be bound to 1 port at a time. If GUID given is 0, OpenSM displays a list of possible port GUIDs and waits for user input.
WinOF VPI for Windows User Manual Rev 4.3 Table 17 - osmtest Flags and Options Flag 7.4 Description -vf This option sets the log verbosity level. A flags field must follow the -D option.
Rev 4.3 InfiniBand Fabric 7.4.1.1 ib_read_bw Synopsys ib_read_bw [-i(b_port) ib_port] [-m(tu) mtu_size] [-s(ize) message_size] [-n iteration_num] [-p(ort) PDT_port] [-b(idirectional)] [-o(uts) outstanding reads] [a(ll)] [-V(ersion)] 7.4.1.2 ib_read_bw Options The table below lists the various flags of the command. Table 18 - ib_read_bw Flags and Options Flag 7.4.
WinOF VPI for Windows User Manual Rev 4.3 7.4.2.1 ib_read_lat Synopsys ib_read_lat [-i(b_port) ib_port] [-m(tu) mtu_size] [-s(ize) message_size] [-t(xdepth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-o(uts) outstanding reads] [a(ll)] [-V(ersion)] [-C report cycles] [-H report histogram] [-U report unsorted] 7.4.2.2 ib_read_lat Options The table below lists the various flags of the command. Table 19 - ib_read_lat Flags and Options Flag 7.4.
Rev 4.3 InfiniBand Fabric 7.4.3.1 ib_send_bw Synopsys ib_send_bw [-i(b_port) ib_port] [-c(onnection_type) RC\UC\UD] [-m(tu) mtu_size] [s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-b(idirectional)] [-a(ll)] [-V(ersion)] 7.4.3.2 ib_send_bw Options The table below lists the various flags of the command. Table 20 - ib_send_bw Flags and Options Flag 7.4.
WinOF VPI for Windows User Manual Rev 4.3 7.4.4.1 ib_send_lat Synopsys ib_send_lat [-i(b_port) ib_port] [-c(onnection_type) RC\UC\UD] [-m(tu) mtu_size] [s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-a(ll)] [-V(ersion)] [-C report cycles] [-H report histogram] [-U report unsorted] 7.4.4.2 ib_send_lat Options The table below lists the various flags of the command. Table 21 - ib_send_lat Flags and Options Flag 7.4.
Rev 4.3 InfiniBand Fabric 7.4.5.1 ib_write_bw Synopsys ib_write_bw [-q num of qps] [-c(onnection_type) RC\UC\UD] [-i(b_port) ib_port] [m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [p(ort) PDT_port] [-b(idirectional)] [-a(ll)] [-V(ersion)] 7.4.5.2 ib_write_bw Options The table below lists the various flags of the command. Table 22 - ib_write_bw Flags and Options Flag 7.4.
WinOF VPI for Windows User Manual Rev 4.3 7.4.6.1 ib_write_lat Synopsys ib_write_lat [-i(b_port) ib_port] [-c(onnection_type) RC\UC\UD] [-m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-a(ll)] [-V(ersion)] [-C report cycles] [-H report histogram] [-U report unsorted] 7.4.6.2 ib_write_lat Options The table below lists the various flags of the command. Table 23 - ib_write_lat Flags and Options Flag 7.4.
Rev 4.3 InfiniBand Fabric 7.4.7.1 ibv_read_bw Synopsys ibv_read_bw [-i(b_port) ib_port] [-d ib device] [-o(uts) outstanding reads] [-m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-u qp timeout] [-S(l) sl type] [-x gid index] [-e(vents) use events] [-F CPU freq fail] [-b(idirectional)] [-a(ll)] [-V(ersion)] 7.4.7.2 ibv_read_bw Options The table below lists the various flags of the command.
WinOF VPI for Windows User Manual Rev 4.3 Table 24 - ibv_read_bw Flags and Options Flag 7.4.8 Description -Q, --cq-mod Generate Cqe only after <--cq-mod> completion -N, --no peak-bw Cancel peak-bw calculation (default with peak) ibv_read_lat This is a more advanced version of ib_read_lat ,and contains more flags and featurs than the older version and also improved algorithms. ibv_read_lat calculats the latency of RDMA read operation of message_sizeB between a pair of machines.
Rev 4.3 InfiniBand Fabric Table 25 - ibv_read_lat Flags and Options Flag 7.4.
WinOF VPI for Windows User Manual Rev 4.
Rev 4.3 InfiniBand Fabric perform a ping pong benchmark on which you send packet only after you receive one. Each of the sides samples the CPU clock each time they receive a send packet, in order to calculate the latency. 7.4.10.1 ibv_send_lat Synopsys ibv_send_lat [-i(b_port) ib_port] [-c(onnection_type) RC\UC\UD] [-d ib_device name] [-m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [I(nline_size) inline size] [-u qp timeout] [-S(L) sl type] [-x gid index] [-e(events) use events] [-n iterati
WinOF VPI for Windows User Manual Rev 4.3 Table 27 - ibv_send_lat Flags and Options Flag Description -g, --post= The number of posts for each qp in the chain (default tx_depth) -I, --inline_size= The maximum size of message to be sent in “inline mode” (default 0) -e, --events Inactive during CQ events (default poll) -g, --mcg= Sends messages to multicast group with qps attached to it.
Rev 4.3 InfiniBand Fabric Table 28 - ibv_write_bw Flags and Options Flag Description -c, --connection= Connection type RC/UC(default RC) -s, --size= The size of message to exchange (default 65536) -a, --all Runs sizes from 2 till 2^23 -t, --tx-depth= The size of tx queue (default 100) -n, --iters= The number of exchanges (at least 2, default 1000) -u, --qp-timeout= QP timeout.
WinOF VPI for Windows User Manual Rev 4.3 7.4.12.1 ibv_write_lat Synopsis ibv_write_lat [-i(b_port) ib_port] [-c(onnection_type) RC\UC\UD][-m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [-I(nline_size) inline size] [-u qp timeout] [-S(L) sl type] [-d ib_device name] [-x gid index] [-n iteration_num] [-p(ort) PDT_port] [-a(ll)] [-V(ersion)] [-C report cycles] [-H report histogram] [-U report unsorted] 7.4.12.2 ibv_write_lat Options The table below lists the various flags of the command.
Rev 4.3 InfiniBand Fabric 7.4.13 ibaddr Displays the lid (and range) as well as the GID address of the port specified (by DR path, lid, or GUID) or the local port by default. This utility can be used as simple address resolver.
WinOF VPI for Windows User Manual Rev 4.3 7.4.13.1 ibaddr Synopsis ibaddr [-d(ebug)] [-D(irect)] [-G(uid)] [-l(id_show)] [-g(id_show)] [-C ca_name] [-P ca_port] [-t(imeout) timeout_ms] [-V(ersion)] [-h(elp)] [] 7.4.13.
Rev 4.3 InfiniBand Fabric Table 30 - ibaddr Flags and Options Flags Description -t Overrides the default timeout for the solicited mads. 7.4.13.3 Multiple CA/Multiple Port Support When no IB device or port is specified, the port to use is selected by the following criteria: 1. the first port that is ACTIVE. 2. if not found, the first port that is UP (physical link up).
WinOF VPI for Windows User Manual Rev 4.3 7.4.14.2 ibcacheedit Options Table 31 - ibcacheedit Flags and Options Flags Description --switchguid BEFOREGUID:AFTERGUID Specifies a switchguid that should be changed. The before and after guid should be separated by a colon. On switches, port guids are identical to the switch guid, so port guids will be adjusted as well on switches. --caguid BEFOREGUID:AFTERGUID Specifies a caguid that should be changed.
Rev 4.3 InfiniBand Fabric 7.4.15.1 iblinkinfo Synopsis [-hcdl -C -P -p -S -G -D --load-cache ] 7.4.15.2 iblinkinfo Flags and Options Table 32 - iblinkinfo Flags and Options Flags 95 Description -S -G --port-guid start partial scan at the port specified by (hex format) -D start partial scan atthe port specified by the direct route path.
WinOF VPI for Windows User Manual Rev 4.3 Table 32 - iblinkinfo Flags and Options Flags --filterdownports Description Filter downports indicated in a ibnetdiscover cache. If a port was previously indicated as down in the specified cache, and is still down, donot output it in the resulting output. This option may be particularly useful for environments where switchesare not fully populated, thus much of the default iblinkinfo info is considered un-useful.
Rev 4.3 InfiniBand Fabric 7.4.16.2 ibqueryerrors Options Table 33 - ibqueryerrors Flags and Options Flags Description -s Suppresses the errors listed in the comma separated list provided. -c Suppresses some of the common "side effect" counters. These counters usually do not indicate an error condition and can be usually be safely ignored. -G -S --portguid Report results for the port specified.
WinOF VPI for Windows User Manual Rev 4.3 Table 33 - ibqueryerrors Flags and Options Flags Description --load-cache Loads and uses the cached ibnetdiscover data stored in the specified filename. May be useful for outputting and learning about other fabrics or a previous state of a fabric. Cannot be used if user specifies a direct route path. See ibnetdiscover for information on caching ibnetdiscover output. -R This option is obsolete (and has no effect). -d Raises the IB debugging level.
Rev 4.3 InfiniBand Fabric 7.4.17.
WinOF VPI for Windows User Manual Rev 4.3 7.4.17.2 ibsysstat Options Table 34 - ibsysstat Flags and Options Flags Description ping Verifies connectivity to server (default) host Obtains host information from server cpu Obtains cpu information from server -o, --oui Uses specified OUI number to multiplex vendor mads -S, --Server Starts in server mode (do not return) Debugging Flags Description NOTE: Most OpenIB diagnostics take the following common flags.
Rev 4.3 InfiniBand Fabric 2. If not found, the first port that is UP (physical link up). If a port and/or CA name is specified, the user request is attempted to be fulfilled, and will fail if it is not possible. 7.4.
WinOF VPI for Windows User Manual Rev 4.3 7.4.18.
Rev 4.3 InfiniBand Fabric Table 35 - perfquery Flags and Options Flags --vlxmitcounters Shows ticks waiting to transmit counters per VL. Thisis an optional counter. --swportvlcong Shows sw port VL congestion. This is an optional counter. --rcvcc Shows Rcv congestion control counters. This is an optional counter. --slrcvfecn Shows SL Rcv FECN counters. This is an optional counter. --slrcvbecn Shows SL Rcv BECN counters. This is an optional counter.
WinOF VPI for Windows User Manual Rev 4.3 Table 35 - perfquery Flags and Options Flags Description Addressing Flags Description -G Uses GUID address argument. In most cases, it is the Port GUID. Example: "0x08f1040023" -s Uses ’smlid’ as the target lid for SM/SA queries. -C Uses the specified ca_name. -P Uses the specified ca_port. -t override the default timeout for the solicited mads. 7.4.18.
Rev 4.3 InfiniBand Fabric 7.4.19.1 saquery Synopsis saquery [-h] [-d] [-p] [-N] [--list | -D] [-S] [-I] [-L] [-l] [-G] [-O] [-U] [-c] [-s] [-g] [-m] [-x] [-C ca_name] [-P ca_port][--smkey val] [-t(imeout) ] [--src-to-dst ] [--sgid-to-dgid ] [--node-name-map ] [ | | ] 7.4.19.2 saquery Options Table 36 - saquery Flags and Options Flags 105 Description -p Gets PathRecord info. -N Gets NodeRecord info.
WinOF VPI for Windows User Manual Rev 4.3 Table 36 - saquery Flags and Options Flags Description -C Uses the specified ca_name. -P Uses the specified ca_port. --smkey Uses SM_Key value for the query. Will be used only with "trusted" queries. If nonnumeric value (like’x’) is specified then saquery will prompt for a value. -t, -timeout Specifies SA query response timeout in milliseconds. Default is 100 milliseconds.
Rev 4.3 InfiniBand Fabric 7.4.20.
WinOF VPI for Windows User Manual Rev 4.3 7.4.20.2 smpdump Options Table 37 - smpdump Flags and Options Flags Description attr IBA attribute ID for SM attribute mod IBA modifier for SM attribute Debugging Flags Description NOTE: Most OpenIB diagnostics take the following common flags. The exact list of supported flags per utility can be found in the usage message and can be shown using the util_name -h syntax. -d Raises the IB debugging level. Can be used several times (-ddd or -d -d -d).
Rev 4.
WinOF VPI for Windows User Manual 8 Rev 4.3 Software Development Kit Software Development Kit (SDK) a set of development tools that allows the creation of InfiniBand applications for MLNX_VPI software package. The SDK package contains, header files, libraries, and code examples. To open the SDK package you must run the sdk.exe file and get the complete list of files. SDK package can be found under \IB\SDK Mellanox Technologies 110
Rev 4.3 Troubleshooting 9 Troubleshooting 9.1 InfiniBand Troubleshooting Issue 1. The IB interfaces is not up after the first reboot after the installation process is completed. Suggestion: To troubleshoot this issue, follow the steps bellow: 1. Check that the IB driver is running on all nodes by using 'vstat'. The vstat utility located at \tools, displays the status and capabilities of the network adaptor card(s). 2.
WinOF VPI for Windows User Manual Issue 5. Rev 4.3 The Ethernet driver fails to start. A yellow sign appears near the "Mellanox ConnectX 10Gb Ethernet Adapter" in the Device Manager display. Suggestion: This can happen due to a hardware error. Try to disable and re-enable "Mellanox ConnectX Adapter" from the Device Manager display. Issue 6. No connectivity to a Fault Tolerance bundle while using network capture tools (e.g., Wireshark).
Rev 4.3 113 Troubleshooting • Mellanox ConnectX EN 10Gbit Ethernet Adapter device detected that the link connected to port is up, and has initiated normal operation. • Mellanox ConnectX EN 10Gbit Ethernet Adapter device detected that the link connected to port is down. This can occur if the physical link is disconnected or damaged, or if the other end-port is down. • Mismatch in the configurations between the two ports may affect the performance.
WinOF VPI for Windows User Manual 10 Rev 4.3 Documentation • Under \Documentation: • License file • User Manual (this document) • MLNX_VPI_Installation Guide • MLNX_VPI_Release Notes Mellanox Technologies 114
Rev 4.
