Dell and the Dell logo are trademarks of Dell Inc. ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual 0J05YT, 0K6V3V Rev 1.0 www.mellanox.
NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 Table of Contents Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rev 1.0 List of Tables Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Table 2: Documents List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 Table 3: ConnectX-3 Mezzanine Card Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Table 4: Jumper Configuration . . . . . . . . . . . . . . . . . . . . . . . .
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 List of Figures Figure 1: 0J05YT Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 Figure 2: Board Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10 Figure 3: 0K6V3V Label . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rev 1.0 Revision History This document was first printed on February 14, 2012. Table 1 - Revision History Table 6 Date Rev February 2012 1.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 About this Manual This User Manual describes ConnectX®-3 Dual QSFP Port mezzanine card for the Dell Blade Servers. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the cards, installation instructions, and relevant documentation. Intended Audience This manual is intended for the installer and user of the mezzanine cards.
Rev 1.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 1 Overview This document is a User Manual for the Mellanox Technologies ConnectX®-3 40Gb/s and 56Gb/ s InfiniBand dual-port mezzanine cards for the Dell Blade Servers. The cards described in this manual have the following main features: • InfiniBand Architecture Specifications v1.2.1 compliant • PCI Express 3.0 (1.1 and 2.
Rev 1.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual 2 Card Interfaces 2.1 I/O Interfaces Rev 1.0 Each mezzanine card includes the following interfaces: • PCI Express (PCIe) • I2C-compatible connector (for debug) Figure 5: 0J05YT/0K6V3V Card 2.1.1 InfiniBand Interface The ConnectX®-3 mezzanine cards are compliant with the InfiniBand Architecture Specification, Release 1.2.1. It has two compliant InfiniBand ports with four Tx/Rx pairs of SerDes.
Rev 1.0 Card Interfaces 2.1.2 PCI Express Interface The ConnectX®-3 mezzanine cards support PCI Express 3.0 (1.1 and 2.0 compatible) . The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations. 2.1.3 I2C-compatible Interface A three-pin header on the mezzanine cards is provided as the I2C-compatible interface. Figure 6: I2C Connector 2.2 Power All adapter cards receive 12V and 3.3V power from the PCI Express connector.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 A jumper inserted into the drill on each adapter card indicates to the device whether an on-board Flash device is to be used or not. Table 4 provides information on this jumper. Table 4 - Jumper Configuration Description Card Default Configuration Option Flash present/ not present connection open – Flash present connection shorted – Flash not present connection open – Flash present 2.3.
Rev 1.0 Card Interfaces Table 5 - 0J05YT VPD Layout 14 Offset (Decimal) Item Value Format Description 59 SerialNumber OO0J05YTMMM MMYMDSSSS %STR_SPC “00..00XXXX..
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.
Rev 1.0 Card Interfaces Table 6 - 0K6V3V VPD Layout Offset (Decimal) 16 Item Value Format Description 53 Revision RV %STR PCB revision 56 VPD Keyword SN STR Serial Number 58 Length 0x14 59 SerialNumber OO0K6V3VMMM MMYMDSSSS %STR_SPC “00..00XXXX..
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.
Rev 1.0 Card Installation 3 Card Installation 3.1 Hardware RequirementsTBD Before installing the mezzanine card, make sure that the system meets the hardware and software requirements listed in Table 7 - Jumper Configuration Requirement Hardware Software Operating Systems/Distributions 3.2 Description Used with Dell Blade Servers Refer to the Dell Blade Servers Manuals Installation Kit Make sure all of the parts are in the kit before you start the installation.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 Use the documentation supplied with the Dell Blade Servers to remove and replace the blade from the chassis. The card shown below has grounding clips that must come in contact with the chassis after the card is installed. Figure 7: Grounding Clip Make sure that you are properly grounded. Make sure that the equipment, both while in the chassis and while removed from the chassis, is properly grounded to prevent ESD.
Rev 1.0 Card Installation DIMMS can be hot. Allow sufficient time for the blade components to cool before starting this procedure. 1. 2. 3. Remove the blade from the chassis. Remove the cover from the blade. Open the retaining clip. The card can be placed in either of the two available slots. 4. Expose the socket to be used for the new card. a. When replacing an existing card, remove the card from the socket.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 Figure 10: Remove the Cover Hold the mezzanine card by the edges only. Line up the mezzanine card so that the pins of the mezzanine card are over the sockets in the blade server. 5. Plug the card into the socket by placing your thumb over the UPC symbol and pressing down until the card reaches the bottom. Figure 11: Press on the Card 6. Close the retaining clip.
Rev 1.0 Card Installation Figure 12: Single Card in Slot C Figure 13: Two Cards Installed 22 7. Replace the blade server cover. 8.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual 3.3 Rev 1.0 Safety Warnings 1. Installation Instructions Read all installation instructions before connecting the equipment to the power source. 2. Over Temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: °C (°F). To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation openings. 3.
Rev 1.0 Driver Software and Firmware 4 Driver Software and Firmware 4.1 Driver Software 4.1.1 Linux For Linux, download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the Mellanox web site at: http://www.mellanox.com => Products => Software/Drivers => Infiniband & VPI Software/Drivers => Mellanox OFED => Download. Follow the installation instructions included in the download package (also available from the download page).
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 Rate: 56 Base lid: 8 LMC: 0 SM lid: 12 Capability mask: 0x02514868 Port GUID: 0x0002c903002fefe2 Link layer: InfiniBand Check the OFED version To get the version of the running Mellanox OFED/BXOFED, run the following command: # ofed_info |head -1 MLNX_OFED_LINUX-1.5.2-2.0.0 (OFED-1.5.2-20101219-1546): MLNX_OFED_LINUX-1.5.3-0.1.6 (OFED-1.5.3-0.1.
Rev 1.0 Driver Software and Firmware Figure 14: Device Manager (example) Note: If the cards are connected to a managed switch, there is no need to run openSM. Only one OpenSM should run per subnet.In InfiniBand interfaces, OpenSM is installed as a disabled Windows service. To enable it, enter at the command line: > sc start opensm 4.2 FlexBoot FlexBoot enables remote boot over Boot over InfiniBand (BoIB), or Boot over iSCSI (Bo-iSCSI).
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 number. Please contact Mellanox System Support if you cannot find the firmware binary for your mezzanine card. The following steps describe how to retrieve the PSID (firmware identification) and programmed firmware version of your mezzanine card. They also describe how to update the card with the latest firmware version available. 1. Retrieve the PSID and firmware version: a. Install the MFT package.
Rev 1.0 Driver Software and Firmware Figure 15: Support Download Assistant 3. If a newer firmware version exists for your mezzanine card, update the firmware as follows: a. Download the firmware image zip file from the Download Center (see Step 2a above) b. Unzip the firmware image c. Burn the firmware image. Enter: > flint -d /dev/mst/mt4099_pci_cr0 -i burn d. Reboot the computer e. Enter: mst start f.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 Appendix A: Specifications A.1 Board Mechanical Drawing and Dimensions The ConnectX®-3 mezzanine card mechanical drawing is depicted in Figure 16. All dimensions are in millimeters. All the mechanical tolerances are +/- 0.1mm Figure 16: Mechanical Drawing of 0J05YT/0K6V3V Mezzanine Card 23.92 64.08 72.36 78.91 57.
Rev 1.0 A.2 Specifications Table 8 - ConnectX-3 0J05YT/0K6V3V Specifications Table Physical Size: 3.11in. x3.46 in. (78.91mm X 88mm) InfiniBand: IBTA v1.2.1 QDR (10Gb/s per lane) or FDR (14.0625Gb/s per lane) port Protocol Support Data Rate: Up to 56Gb/s FDR– InfiniBand PCI Express Gen3: 3.0 SERDES @ 8.0 GT/s (1.1 and 2.0 compatible) Voltage: 12V, 3.3V, 3.3VAUX 0J05YT Typ Power: 5.75W Max Power: 7.56W Power and Environmental 0K6V3V Typ Power: 8.84W Max Power: 9.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 Appendix B: Avertissements de sécurité d’installation (French) 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 55°C (131°F).
Rev 1.0 Appendix C: Installation - Sicherheitshinweise (German) 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Außerdem sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen einwandfreien Luftstrom zu gewährleisten. 3.
ConnectX®-3 Dual Port 40Gb/s and 56Gb/s Mezzanine Card for Dell User Manual Rev 1.0 Appendix D: Advertencias de seguridad para la instalación (Warnings in Spanish) 1. Instrucciones de instalación Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de instalación. 2. Sobrecalentamiento No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 55°C(131°F).
