White paper PCIe Card Cooling with Dell EMC PowerEdge Servers Updates based on servers with iDRAC9 Abstract An overview of thermal management of PCIe adapter cards in Dell EMC PowerEdge Servers and potential airflow customization options for third-party cards December 2019 297
Revisions Revisions Date Description 11/20/2019 New white paper that consolidates all PCIe cooling related topics, including airflow customization options for third-party cards. Supported under iDRAC9 (latest available release recommended). Some features may have iDRAC license dependency. Acknowledgments This paper was produced by the following: Authors: Hasnain Shabbir, Jon Brown Support: Server Thermal Controls Team The information in this publication is provided “as is.” Dell Inc.
Table of contents Table of contents Revisions.............................................................................................................................................................................2 Acknowledgments ...............................................................................................................................................................2 Table of contents ................................................................................................
Executive summary Executive summary The adoption of PCIe adapters, especially of customer-specific appliance solutions based on PowerEdge servers, is increasing. With this increase, it is important to highlight how PowerEdge servers address thermal management of PCIe adapters, especially of third-party cards. This paper also provides details on airflow customization options available for such cases.
Introduction 1 Introduction PowerEdge servers have the cooling capacity to support a broad array of PCIe adapters. These servers use customized fans, airflow shrouding, and optimized system topologies to maximize airflow that is provided to PCIe slots. The design also ensures the temperature of the air that is delivered to the PCIe cards is at or below industry requirements.
PCIe cooling strategy in PowerEdge servers 2 PCIe cooling strategy in PowerEdge servers PowerEdge servers identify the device and provide customized cooling for each Dell EMC-designed or qualified PCIe adapter card. Many customers choose to use their own cards and those cards are also known as third-party cards. Dell EMC has designed the logic to detect these third-party PCIe adapters and provide a default cooling response that is based on an estimate of the cooling requirements for the card.
PCIe cooling strategy in PowerEdge servers iDRAC web UI snapshot for PCIe Airflow Settings The UI illustrated in the above figure enables the user to observe: • • • • The maximum LFM capability of each slot within the server when the max LFM is at full fan speeds How each slot is being managed—by airflow control, temperature control, or target LFM value The minimum LFM delivered to any third-party card slots allows users to decide whether to keep this LFM, or to customize its settings at a higher or lower
PCIe cooling strategy in PowerEdge servers Cards that are cooled based on temperature monitoring are called temperature controlled, while those adapter cards that require specific airflow only are called airflow controlled. These terms are used in iDRAC UI associated with PCIe airflow and are discussed later in this white paper. Dell EMC-qualified cards have been thoroughly validated with PowerEdge servers and require no additional customization from a cooling perspective.
PCIe cooling strategy in PowerEdge servers If the LC log does not record this message upon card installation, the server is not treating the card as a thirdparty card and no airflow customization can be done for the cards. This log is informational only, and this message does not mean that there is a problem with the PCIe card. The intent of this message is to make the user aware that a third-party card is detected in the system and that Dell EMC has provisioned cooling to the card.
PCIe cooling strategy in PowerEdge servers 2.3.2.2 Using IPMI There is no customer-facing support for configuring the third-party PCIe fan response through IPMI. 2.3.2.3 Using RACADM commands Use the following RACADM command to check the status of third-party PCI fan responses per slot. Replace the last digit which is PCIe slot number with the applicable slot number for your system. racadm get system.pcieslotlfm.1 [Key=system.Embedded.1#PCIeSlotLFM.
PCIe cooling strategy in PowerEdge servers CurrentValue = Automatic DefaultValue = Automatic Dependency = Yes DisplayOrder = 7 FQDD = System.Embedded.1 GroupDisplayName = PCIeSlotLFM GroupID = pcieslotlfm.1 InstanceID = System.Embedded.1#pcieslotlfm.
PCIe cooling strategy in PowerEdge servers To disable the third-party PCI response: winrm i SetAttribute cimv2/root/dcim/DCIM_SystemManagementService?SystemCreationClassName=DCIM_ComputerS ystem+SystemName=srv:system+CreationClassName=DCIM_SystemManagementService+Name=DCI M:SystemManagementService -u: -p: -r:https:///wsman SkipCNcheck -SkipCAcheck -encoding:utf-8 -a:basic @{Target="System.Embedded.1";AttributeName="PCIeSlotLFM.
References 3 References See one of the following references for more details on manual cooling options: • • 13 White Paper: Custom Cooling Options iDRAC9 User Guide PCIe Card Cooling with Dell EMC PowerEdge Servers
