Dell EMC DCPMM User's Guide Nov 2020 Rev.
Notes, cautions, and warnings NOTE: A NOTE indicates important information that helps you make better use of your product. CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem. WARNING: A WARNING indicates a potential for property damage, personal injury, or death. © 2017 - 2020 Dell Inc. or its subsidiaries. All rights reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries.
Contents Chapter 1: Introduction................................................................................................................. 5 System requirements..........................................................................................................................................................5 Terminology...........................................................................................................................................................................
Chapter 9: Windows.................................................................................................................... 33 PMEM in App-direct mode............................................................................................................................................. 33 PMEM Disk management..........................................................................................................................................
1 Introduction Dell EMC now offers Intel's Data Center Persistent Memory Modules (DCPMMs), a nonvolatile memory that has the same form factor as a standard DDR4 DIMM. DCPMMs come in three capacities: 128 GB, 256 GB, and 512 GB. DCPMMs are installed in the memory slots of the server and are compatible with RDIMMs and LRDIMMs.
Table 1. System Requirements (continued) Component Minimum version required DCPMM FW Build # 5375 OS Microsoft Windows 2019 VMware ESXi 6.7 with EP10 (Build #13981272) Red Hat Enterprise Linux 7.6 SUSE Linux Enterprise Server 15 Terminology Table 2. Terminology Terminology Description App-direct mode (AD) Persistent memory is accessed directly by applications as byte-addressable memory.
2 Change list Table 3.
3 Hardware Topics: • • • Server hardware configuration DIMM installation and removal DCPMM hardware configuration Server hardware configuration DCPMM is supported in R640, R740/R740XD, R840, R940, R940xa, MX740c, and MX840c PowerEdge servers with secondgeneration Intel Xeon Scalable Processors of Gold and Platinum grade. See DCPMM configurations for a list of fully supported and validated DCPMM configurations in two-socket servers.
Figure 1. Memory Layout for R740/R740XD Table 4. DCPMM configurations Number DCPMM DRAM of CPUs Populat Populat in the ion ion Server DRAM Capacit y (GB) DCPMM OS Total Capacit Memory Memory y (GB) in (GB) Memory Mode (GB) Total Memory per CPU (GB) Ratio DRAM to Optane Memory Require s an M or L CPU Support ed in App Direct Mode Support ed in Memory Mode 1 128GB x 2 16GB x 4 64 256 256 320 320 1:4 No Yes Yes 1 128GB x 1 16GB x 6 96 128 NA 224 224 1:1.
Table 4. DCPMM configurations (continued) Number DCPMM DRAM of CPUs Populat Populat in the ion ion Server DRAM Capacit y (GB) DCPMM OS Total Capacit Memory Memory y (GB) in (GB) Memory Mode (GB) Total Memory per CPU (GB) Ratio DRAM to Optane Memory Require s an M or L CPU Support ed in App Direct Mode Support ed in Memory Mode 2 128GB x 4 16GB x 8 128 512 512 640 320 1:4 No Yes Yes 2 128GB x 4 16GB x 12 192 512 NA 704 352 1:2.
Table 4. DCPMM configurations (continued) Number DCPMM DRAM of CPUs Populat Populat in the ion ion Server DRAM Capacit y (GB) DCPMM OS Total Capacit Memory Memory y (GB) in (GB) Memory Mode (GB) Total Memory per CPU (GB) Ratio DRAM to Optane Memory Require s an M or L CPU Support ed in App Direct Mode Support ed in Memory Mode 2 256GB x 64GB x 8 12 768 2,048 NA 2,816 1,408 1:2.
Table 5.
Table 6.
Table 7.
Table 7.
4 BIOS Topics: • • • BIOS configuration setting for Intel DCPMM App-direct mode configuration Memory mode configuration BIOS configuration setting for Intel DCPMM DIMM discovery All installed DCPMMs that the BIOS has discovered during system inventory is displayed in the BIOS Intel Persistent Memory tab: Memory Settings > Persistent Memory > Intel Persistent Memory > Persistent Memory DIMM Configuration. Figure 2. Persistent Memory screen NOTE: DCPMMs are shown as DIMMs.
Figure 3. Memory info NOTE: Data is always assumed to be in units of MiB/GiB/TiB even if labeled MB/GB/TB. User capacity overhead is up to 2% of capacity (GiB). Another overhead may be required for Regions, Namespace, and Filesystems. App-direct mode configuration Create goal Goal is created in BIOS. To create a goal in BIOS, go to: Memory Settings > Persistent Memory > Intel Persistent Memory > Region Configuration > Create Goal Config.
Figure 4. Goal configuration The BIOS options determine how the goal is created and the DCPMMs are configured: Persistent [%]: ● No Change - Does not apply any changes to the current goal. ● 100 - Creates a goal of 100% Persistent memory across the selected DCPMMs. ● 0 - Creates a goal of 0% Persistent memory across the selected DCPMMs. This operation configures all the DCPMM as Memory mode. Persistent memory type: ● App-direct Interleaved - Persistent mode interleave across the DCPMMs in a scoket.
Figure 5. Region configuration The number of regions that are displayed depends on the number of processors in the system and not on the DCPMMs interleaved. If the DCPMMs are configured as interleaved, one Persistent Memory Region is listed per socket in the system that has DCPMMs installed. If the DCPMMs are configured as non-interleaved, one Persistent Memory Region is listed per DCPMM installed in the system. Region information can be accessed by clicking each Persistent Memory Region link in the BIOS.
Figure 6. Region info Memory mode configuration Create goal Goal is created in the BIOS. To create a goal in BIOS, go to: Memory Settings > Persistent Memory > Intel Persistent Memory > Region Configuration > Create Goal Config. The BIOS options determine how the goal is created and the DCPMMs are configured: Operation Target: ● Platform - Applies the goal to all the DIMMs in the system (recommended). Persistent [%]: ● No Change - Does not apply any changes to the current goal.
5 DCPMM event reporting When system detects DCPMM-related event either during runtime or POST, system will log the events in Server System Event Log(SEL) and Life-Cycle Log(LCL). If an event is detected during boot time, system halts during the POST and user needs to press F1 to continue the boot process. NOTE: NVDIMM is frequently used in these messages. The term NVDIMM is generic to several different families of persistent memory including DCPMM and not meant to indicate NVDIMM-N modules.
Recommended Action: N/A ● UEFI0347 : Unable to initialize the memory because one or more errors have occurred during the NVDIMM initialization in the slot . Recommended Action : Manually remove and reinstall the NVDIMM. If the issue persists, contact your service provider. For more information about removing and reinstalling an NVDIMM, see the product Installation and Service Manual available on the support site.
Recommended Action : Retry the operation. If the issue persists, contact your service provider. ● UEFI0359 : The Overwrite DIMM operation on the Intel Persistent Memory DIMM with serial number in slot is successfully completed. Recommended Action : N/A NOTE: This is part of the DCPMM Sanitize function. ● UEFI0360 : Unable to complete the Overwrite DIMM operation on the Intel Persistent Memory DIMM with serial number in slot .
Recommended Action : Do one of the following: 1) Power off the server. 2) Disconnect the input power, wait for 30 seconds, and then reconnect to the power source. 3) Power on the server. 4) If the issue persists, contact your service provider. ● PWR2281 : Unable to perform the memory arming operation because the PSU configuration of the server is insufficient to guarantee data flush time in the event of power loss. Recommended Action : Do the following and retry the operation: ○ Turn off the server.
6 iDRAC Intel DCPMM management Topics: • iDRAC GUI iDRAC GUI DCPMM firmware version DCPMM FW version is displayed under System > Inventory > Firmware inventory tab. Figure 7. DCPMM firmware version DCPMM hardware status Select the Memory link on the Dashboard to get more information about memory health. DCPMM hardware status is displayed under System > Inventory > Hardware inventory tab.
Figure 8. DCPMM hardware status DCPMM goal configuration using iDRAC GUI 1. Log on to iDRAC GUI interface. 2. Navigate to Configuration > BIOS settings > Memory settings > Persistent Memory Settings tab. Figure 9. DCPMM goal configuration using iDRAC GUI 3. Navigate to Intel Persistent Memory > Region Configuration > Create Goal Config.
Figure 10. DCPMM goal configuration 4. Change Persistent percentage as 100% to configure Intel DCPMM into 100% App-direct Mode and 0% to configure the DIMMs into 100% memory mode. NOTE: The Persistent memory type field is for configuring a new goal. It is not for reading the status of the current goal. Use instructions under DCPMM Hardware Status to confirm current configuration. 5. Click Apply and reset the system. 6.
7 DCPMM security Topics: • • • Memory mode App-direct Cryptographic erase and DCPMM sanitize Memory mode In Memory mode DCPMMs operate as volatile system memory. User passphrase is not supported and this BIOS setting will be greyed out. App-direct Users have the option to enable Passphrase protection of DCPMM regions. The intent of the passphrase is to protect against unauthorized access to data stored on the DCPMM region.
Figure 12.
● Sanitize Both erase methods can be executed using BIOS setup options. User can choose to perform an erase on all or a subset of installed DCPMMs. Crypto erase The Crypto Erase function erases the App-direct Region Key (PM-RK) forcing the system to reboot. Cryptographic erase option can be accessed by going to: System BIOS Settings > Memory Settings > Persistent Memory > Intel Persistent Memory > Persistent Memory DIMM Configuration Figure 13.
Figure 14. Sanitize Sanitize can take up to 15 minutes with fully loaded 128 GB DIMM configuration, 30 minutes with 256 GB and 1 hour with 512 GB. NOTE: Sanitize is not supported when DCPMMs are configured in Memory mode. When the Sanitize operation is running, a prompt appears in BIOS indicating an Overwrite. Overwrite is the name for the second firmware command that is conducted.
8 DIMM configuration changes The following DCPMM migration scenarios are supported: ● Replacement of System Board due to failure All DIMMs must be re-populated in the exact same slots. DCPMMs and data content will be available for customer application access after the board has been restored to the same configuration as the original board. System Restore will automatically restore the BIOS configuration on the replacement board, including the DCPMM Passphrase, if it is set.
9 Windows Dell EMC supports Intel DCPMM with Microsoft Windows 2019 in Memory mode and App-direct mode. NOTE: Keep Windows updated with the monthly cumulative updates.
Figure 16. Memory Devices in Device Manager PMEM Disk management Windows currently supports only one namespace per interleave set (this is independent of the number of physical devices in the interleave set).The option to interleave DCPMMs can be selected during goal creation as described in App Direct and Memory mode configurations. PMEM disks have to be created with the help of "New-Pmemdisk" command by providing relevant region IDs.
○ Writes zeroes to the label storage area, writes new label index blocks, and then rebuilds the SCM stacks to reflect the changes. ○ Requires user confirmation, which can be overridden with Force. ○ This cmdlet is intended as a “big hammer” recovery mechanism. It is not recommended for normal use. List PMEM physical disks and check their health status The following image displays command usage to list all usage to list all PMEM physical devices and get their health.
Remove PMEM disks Figure 19. Remove PMEM disks Windows 2019 does not support redundant volume creation on PMEM disks by using Windows VDS (Virtual disk service). In order to create redundant volumes, use the storage spaces method. For information about storage spaces method refer to: https://docs.microsoft.com/en-us/windows-server/storage/storagespaces/deploy-standalone-storage-spaces.
Figure 21. PMEM Disk creation using Region id PMEM in memory Mode When Intel DCPMM is configured in memory mode, operating system sees it as system memory. Persistent memory size is shown as zero and volatile memory size accounts for the entire size of the DCPMM. Figure 22. PMEM in memory Mode Windows troubleshooting and event monitoring If any of the PMEM physical devices or logical devices are not functioning properly, it is suggested to check Windows event logs.
Figure 23. Windows troubleshooting and event monitoring Windows Errata The following errata affects Windows OSes and are expected to be fixed in a future OS patch. ● When a namespace is created in Windows Server 2019 (WS2019), the logged message appears as "The driver for persistent memory disk encounters internal error". The error is expected and it may occur during testing of Storage Class Memory (SCM) device. This means the namespace is created, and is considered as WAD for WS2019. Workaround: None.
10 Linux Topics: • • • Identify and configure persistent memory device Management utility Linux errata Identify and configure persistent memory device Listing DCPMM devices To list all physical devices in the system, run command ndctl list -DHi Create namespace The configuration of namespaces will decide how much memory capacity user wants to expose to the OS.
To write data into the device, run the following command: cd /mnt/pmemX echo “Hello World” >>test.txt Reset the system and the data should be persistent over power cycle. Delete namespaces Namespace can be deleted using NDCTL command: ndctl destroy-namespace is namespaceX.
Workaround: Add "x-systemd.device-timeout=0" to the mount options for the PMEM partition in fstab. Example: /dev/pmem5 /mnt/somedir ext4 defaults, x-systemd.device-timeout=0,dax 0 2 3. Dell BIOS does not support boot from DCPMM. Some Linux operating system (Red Hat Enterprise Linux 7.6, Red Hat Enterprise Linux 8.0) distros are offering early support of this feature, but Dell BIOS does not support this feature. Workaround:None.
11 VMware ESXi Dell EMC supports Intel DCPMM from vSphere 6.7 EP10 (Build #13981272) and above. The support is available for Intel DCPMM Memory mode and App-direct modes. The persistent memory inventory details are available as part of Host client. See Managing Persistent Memory in the vSphere Single Host Management - VMware Host Client which detail the persistent memory data that are presented to the user.
Figure 26. Namespaces created on ESXi out of the interleave sets exposed After creating the namespaces, ESXi automatically creates a pMem datastore and mounts it as a datastore for the users to consume it. Figure 27. PMEM Datastore PMEM in Memory mode When Intel DCPMM is configured in memory mode, ESXi sees it as system memory. NOTE: The inventory details in the vSphere HTML client will not be available if Intel DCPMM is set to Memory mode.
ESXi troubleshooting and maintenance NOTE: Go through /var/log/vmkernel.log to see if there are any errors or warnings reported related to persistent memory. The log entries related to persistent memory would be appended with "NVD" and/or "IntelNVDimm" ● Configuring Intel DCPMM in memory mode shows a reduction of system memory in system BIOS. ○ For example, a system configuration with 4 * 128 GB Intel DCPMMs, configured in memory mode provides a system memory of 504 GB instead of 512 GB.
12 System diagnostics System diagnostics in the Lifecycle Controller does not test Intel DCPMM in App-direct mode to avoid destroying customer data. NOTE: System diagnostics are not recommended to investigate Intel DCPMM failures when in Memory mode due to extremely long test time.
13 Firmware update Topics: • Dell DUP update Dell DUP update Download DCPMM DUP from www.dell.com/support/drivers. NOTE: Make sure BIOS is up to date with the latest version before applying DCPMM DUP. 1. Boot to the operating system and execute DUP package. a. On a Windows system, double click the .exe DUP file. Reboot the system and DUP will be automatically executed. b. On Linux system: i. Navigate to the folder that contains DUP bin file ii.
