Best Practices Dell EMC SC Series: Red Hat Enterprise Linux Best Practices Abstract This paper provides guidelines for configuring volume discovery, multipath, filesystem, and queue-depth management for Red Hat® Enterprise Linux® (RHEL) 6.x to 8.x with Dell™ Storage Center OS (SCOS) 7.x.x.
Revisions Revisions Date Description October 2013 Initial release December 2013 Refreshed for RHEL 6.4 May 2015 Introduce connectivity to Dell Storage SCv2x00 July 2017 Refreshed for RHEL 6.9 January 2018 Edits to address SCSI device timeouts August 2019 Content merged for RHEL 6.x, 7.x, and 8.x Acknowledgements Author: Henry Wong The information in this publication is provided “as is.” Dell Inc.
Table of contents Table of contents Revisions.............................................................................................................................................................................2 Acknowledgements .............................................................................................................................................................2 Table of contents .................................................................................................
Table of contents 3.2.4 Querying WWNs using multipath command .....................................................................................................21 3.3 Managing persistent devices with volume labels and UUIDs ...........................................................................21 3.4 Volume size ......................................................................................................................................................21 3.5 Thin-provisioned volumes ....
Table of contents 4.5 SCSI device queue settings .............................................................................................................................43 4.5.1 I/O scheduler ....................................................................................................................................................43 4.5.2 read_ahead_kb ..............................................................................................................................................
Executive summary Executive summary Red Hat® Enterprise Linux® (RHEL) is an extremely robust and scalable enterprise-class operating system. Correctly configured using the best practices presented in this paper, RHEL provides an optimized experience for use with the Dell EMC™ SC Series storage. These best practices include guidelines for configuring volume discovery, multipath, file system, and queue depth management. This paper presents RHEL versions 6.x to 8.
Overview 1 Overview SC Series storage provides Linux-compatible and SCSI-3 compliant disk volumes that remove the complexity of allocating, administering, using, and protecting mission-critical data. A properly configured SC Series array removes the need for cumbersome physical disk configuration exercises and management along with complex RAID configuration practices.
Server configuration 2 Server configuration This section discusses configuring the I/O stack on a Linux system. How the stack behaves and performs can be precisely configured to the needs of the environment where the Linux systems operate. The configuration aspects include tuning the HBA port retry count, the queue depth, the SCSI device timeout values, and many more. Depending on the types and models of the HBAs, the parameters might be set on the HBA BIOS.
Server configuration Queue depth settings for a specific Emulex HBA: # cat /sys/class/scsi_host/host12/lpfc_lun_queue_depth # cat /sys/class/scsi_host/host12/lpfc_hba_queue_depth Note: the HBA settings might not match the global settings if the HBA does not support them. To change the HBA queue depth settings, include the following line to the HBA driver module configuration file (section 2.1) and reload the driver module (section 2.1.3).
Server configuration For local-boot systems, it is recommended to unmount all SAN volumes and reload the module. The module should be unloaded from memory before it is reloaded as shown in the following. # modprobe –r qla2xxx # modprobe qla2xxx Replace qla2xxx with lpfc if working with Emulex hardware; SAN volumes can be remounted subsequently. 2.1.4 Rebuilding the initial ramdisk image For configuration persistence, rebuild the initramfs-*.
Server configuration 2.2 SCSI device command timeout On Linux, the SCSI device command timeout has a default value of 30 seconds. Verify the setting with the following command. # cat /sys/block/sdX/device/timeout Typically, this value does not need to be changed unless instructed or recommended to do so by support or application-specific directives. If necessary, the value can be changed using the following process. 1. Change the timeout in seconds for a SCSI device by writing to the device’s sysfs file.
Server configuration 2.3.1 SAS drivers Since RHEL 6.5, SAS drivers are preloaded into certain Linux kernels. The existence of the SAS drivers can be validated with the following commands. As a best practice, validate the driver versions with the Dell EMC Storage Compatibility Matrix and use the latest supported driver as indicated.
Server configuration 2.4.1 Configuration The Linux system being configured requires an Ethernet port, preferably a dedicated one that can communicate with the iSCSI ports on the SC Series storage system. The most important consideration when configuring an iSCSI volume is the network path. Due consideration is recommended to determine the confidentiality, security, and latency required for the iSCSI traffic.
Server configuration 6. In the following example, the iSCSI ports on the SC Series storage system have the IP addresses 172.16.26.180 and 10.10.140.180, respectively. a. Configure the first path: # iscsiadm -m discovery -t sendtargets -p 172.16.26.180 172.16.26.180:3260,0 iqn.2002-03.com.compellent:5000d3100000677c 172.16.26.180:3260,0 iqn.2002-03.com.compellent:5000d3100000677e 172.16.26.180:3260,0 iqn.2002-03.com.compellent:5000d31000006780 172.16.26.180:3260,0 iqn.2002-03.com.
Server configuration 2.4.4 iSCSI timeout The pre-configured iSCSI settings, defined in /etc/iscsi/iscsi.conf, work with SC Series storage system when the Linux system is also configured to use dm-multipath. It is a best practice to use dm-multipath for path redundancy and failover. It is important to note the following timeout settings because they directly affect the storage availability and failure recovery for a SAN storage system.
Volume management 3 Volume management Understanding how volumes are managed in Linux requires basic understanding of the /sys pseudofilesystem. The /sys filesystem is a structure of files that allow interaction with various elements of the kernel and modules. While the read-only files store current values, read/write files trigger events with the correct commands. Generally, the cat and echo commands are used with a redirect as STDIN instead of being opened with a traditional text editor.
Volume management The script produces the following results. The WWPN field will be used during the creation of server object in DSM (see section 3.1.4). # ./fcshow.sh Host-Port WWPN State Speed -------------------------------------------------host14: 0x21000024ff1f5820 Online 16 Gbit host15: 0x21000024ff1f5821 Online 16 Gbit 3.1.2 iSCSI HBAs SC Series storage identifies a Linux system iSCSI HBA by the iSCSI initiator name. Locate this information in /etc/iscsi/initiatorname.iscsi.
Volume management 3. To identify the correct Linux system HBAs, obtain the HBA information on the Linux system (see sections above), and cross-reference them with the list displayed. 3.2 Discovering and identifying SC Series volumes on a Linux system Once the server object for the Linux system is created, present/map the SC Series volumes to the Linux system. This section provides information on how to discover and identify these mapped volumes on a Linux system.
Volume management Rescanning the HBAs while mapping and discovering new volumes does not have any negative impact on the host. #!/bin/bash OSMajor=`uname -r |sed -rn 's/^.*(el.).*$/\1/p'` echo "INFO: OS Major rev.
Volume management SC Series Volume Serial Number and Device ID fields 3.2.3 Querying WWNs using scsi_id command The scsi_id command (located in the /lib/udev folder) can be used to report the WWID of volumes. This WWID can then be used to correlate volumes to their respective SC Series storage devices.
Volume management 3.2.4 Querying WWNs using multipath command If the system has Linux device-mapper-multipath software enabled, the multipath command displays the multipath device properties including the WWN. Section 3.6 provides more information on multipathing.
Volume management 3.5 Thin-provisioned volumes SC Series storage provides the flexibility to create large volumes upfront without consuming much space thanks to the thin provisioning feature. All SC Series volumes are thin-provisioned by default. No space is consumed until the applications start writing data to the volumes. When volumes eventually approach full capacity, administrators can easily and dynamically expand the volumes without affecting the Linux operating system and applications.
Volume management Name the Server object, define the Operating System, and select the corresponding FC HBA WWPNs associated with the Server object 3.6.2 Multipath software solutions There are a few multipath software choices, and it is up to the administrator's to decide which software solution is best for the environment. The following list provides a brief description of some of these solutions.
Volume management Start multipathd: # systemctl start multipathd # service multipathd start #For RHEL 7 and newer #For RHEL 6 and older Check the multipathd status: # systemctl status multipathd # service multipathd status 3.6.4 #For RHEL 7 and newer #For RHEL 6 and older FC/iSCSI volume multipath configuration To ease the deployment of native Linux multipath, the software has default settings for an extensive list of storage models including SC Series storage.
Volume management multipath { wwid "36000d310000067000000000000000a68" alias boot-vol uid 0 gid 0 mode 0600 } multipath { wwid "36000d3100000650000000000000017f2" alias appdata_001 uid 0 gid 0 mode 0600 } } Note: round-robin 0 is the default for RHEL 6 and older. service-time 0 is the default for RHEL 7 and newer. Both are acceptable values for SC Series FC/iSCSI volumes. Once the configuration file is in place, reload multipathd with the updated configuration. # multipath -r # multipath –ll 3.6.
Volume management A properly configured SAS volume will return the following multipath -ll output. This SC Series volume is discovered as a multipath ALUA-capable volume, where each path is capable of I/O. The path represented by prio=50 (active/optimized path) is used for all active I/O requests. The path represented by prio=1 (standby path) is a highly available, redundant path and is used when the active/optimized path becomes unavailable.
Volume management no_path_retry can be set to fail if it is preferable to fail the I/Os and let the applications to handle the errors. In which case, I/Os fail immediately and are not queued when all paths become unavailable. Within the devices section, locate the COMPELNT device definition and change the following attributes. features 0 no_path_retry fail To implement this to specific device at runtime, issue the following command for the desired multipath device.
Volume management Create and label a filesystem: # mkfs.ext4 –L testfs /dev/mapper/testvol1 3.8 Linux LVM Linux Logical Volume Manager (LVM) is a common general-purpose storage manager included in all popular Linux distributions. It provides application-level mirroring and RAID striping, but they are not necessary when the SC Series volumes are used because storage protection is already provided by the SC Series array.
Volume management 3.8.2 LVM space reclamation and SCSI UNMAP/TRIM If LVM is used with SC Series volumes, LVM can be configured to respect SCSI UNMAP/TRIM command recovered space when logical volumes are removed from the member volume group. The space recovered from this removed logical volume varies depending on whether data reduction is enabled on the SC Series volumes. This feature applies to RHEL 6 and newer. To enable the feature, edit the /etc/lvm/lvm.
Volume management Show volume labels and UUIDs on a filesystem using one of the following commands: # blkid /dev/mapper/vgapp-lvapp /dev/mapper/vgapp-lvapp: LABEL="appfs2" UUID="b060087c-fba6-4517-a38624dee0844f55" TYPE="xfs" # lsblk -o name,mountpoint,label,size,uuid /dev/mapper/vgapp-lvapp NAME MOUNTPOINT LABEL SIZE UUID vgapp-lvapp /testfs appfs2 5G b060087c-fba6-4517-a386-24dee0844f55 # xfs_admin -lu /dev/mapper/vgapp-lvapp label = "appfs2" UUID = b060087c-fba6-4517-a386-24dee0844f55 3.9.
Volume management • • For the xfs file system, the filesystem check (fsck option) should be disabled in /etc/fstab because it does not perform any check or repair automatically during boot time. The xfs journaling feature ensures the file system integrity and data is in a consistent state after abrupt shutdown. If a manual repair or check is necessary, use the xfs_repair utility to repair damaged file system. Set a value of 0 in the sixth field to disable fsck check.
Volume management volumes, and potentially cause data loss in the environment. Therefore, it is a best practice to always unmount, remove, and delete all volume information on the Linux OS after the volume is deemed no longer in use. This is non-destructive to any data stored on the actual volume itself, just the metadata about the volume stored by the OS (such as volume size or type). To remove an SC Series volume from a Linux host that is using DM-multipath, use the following steps: 1.
Volume management To use an SC Series volume as a bootable volume, the target volume needs to be presented/mapped to the target Linux system as LUN ID 0. A volume LUN ID 0 mapping with SC Series volume is performed as follows. 1. Right-click the volume and select Map Volume to Server, click Next, and click Advanced Options. 2. Select the Map volume using LUN 0 check box and click Finish to map this volume. 3.
Volume management 6. In the Installation main menu, click Installation Destination. 7. Click Add a disk.
Volume management 8. Identify the SC Series volume in the Multipath Devices tab and select the check box accordingly. 9. Click Done. The SC Series volume shown is identified and selected as the single installation target for the Linux operating system. 10. Click Done to return to the installation main menu. 11. Complete the installation and then reboot. Ensure that the multipathd service is started during boot time. See section 3.6 for enabling dm-multipath.
Volume management On RHEL 6 systems, dm-multipath might not get installed and configured during the OS installation. This prevents dm-multipath to capture and manage the Boot-From-SAN volume after the OS is installed. To correct this: 1. Install and configure device-mapper-multipath package. See section 3.6. 2. Rebuild initramfs image with multipath. # dracut –v –f –a multipath –include /etc/multipath /etc/multipath 3. Reboot the system. 4. multipath –ll now shows the boot-from-SAN volume.
Volume management • • • • 3.12.2 When view volumes are first created, they consume no storage because they share the same blocks as their parent snapshot at the beginning. As new data is written or changes are made to the existing data, new data blocks are allocated and tracked separately from the parent. Changes made to the view volumes do not affect the data on the parent volumes.
Volume management 5. Copy/export the data from the alternate system and import the data back to the parent system. 6. Validate the restored data. 3.12.3.3 Mount view volumes on the same system as the parent volumes to extract data The view volumes contain exactly the same data as the parent volumes including any filesystem labels, disk partitions, UUIDs, LVM metadata, and other application metadata (such as Oracle ASM labels).
Volume management aliases within /etc/multipath/bindings and /etc/multipath/wwids are also automatically updated to reflect any boot device changes. Note: Ensure that the original boot-from-SAN volume is no longer mapped to the server object and the snapshot view volume is mapped to the Linux system as LUN ID 0. If the OS fails to boot, the reason might be that old sdX devices or old WWIDs are still being referenced. Therefore, additional steps must be taken to allow the snapshot view volume to boot.
Performance tuning 4 Performance tuning This section provides general information and guidance pertaining to some of the more common performance tuning options and variables available to Linux. This information is not intended to be all-encompassing and the values used should not be considered final. This section provides a starting point for Linux and storage administrators to use when fine-tuning their Linux installation to achieve optimal performance.
Performance tuning The following output presents the active tuning profile (default) as well as the list of alternative tuning profiles that can be applied. It is recommended to use the default tuning profile, throughput-performance, with any SC Series storage implementation. The discussion of each tuning profile and its merits are outside the scope of this paper, and further discussion of this topic can be found in Red Hat documentation.
Performance tuning 1. Click Advanced Options. 2. Uncheck the Restrict Mapping Paths checkbox and select the controller in which to pin the volume for mapping to the server object. 4.3 Understanding HBA queue depth Queue depth refers to the number of pending I/O requests. Modifying this value can lead to an improvement in I/O performance in some workloads. Generally, increasing queue depth can increase throughput, but caution should be taken since increasing this value can also lead to higher latency.
Performance tuning unnecessarily high levels of I/O control traffic as well as raised CPU loads, causing an increase in I/O latency and potentially impacting business-critical I/O. An alternative to this implementation is using the fstrim command. The fstrim command is part of the utillinux package and allows a one-time request to discard used blocks from a mounted filesystem.
Performance tuning Show the current I/O schedule: • Show all sd devices: # egrep "*" /sys/block/sd*/queue/scheduler /sys/block/sda/queue/scheduler:noop /sys/block/sdb/queue/scheduler:noop /sys/block/sdc/queue/scheduler:noop /sys/block/sdd/queue/scheduler:noop /sys/block/sde/queue/scheduler:noop • anticipatory anticipatory anticipatory anticipatory anticipatory deadline deadline deadline deadline deadline [cfq] [cfq] [cfq] [cfq] [cfq] Show all dm-multipath devices and the associated sd devices: # multi
Performance tuning The new scheduler is in effect only for the current running instance of the OS. A script could be used to make this change persistent on all required SCSI (sd) devices (on a device-dependent basis) during boot time. Alternatively, this change can also be applied system-wide during boot time by appending the elevator= key value option to the end of the kernel string reflected in the /boot/grub/grub.conf (RHEL 6 and older) or /boot/grub2/grub.
Performance tuning 4.5.3 nr_requests This value is used by the Linux kernel to define the depth of the request queue and is often used in conjunction with changes to the HBA queue depth configuration. The default for RHEL 6 and 7 is 128 while 256 is the new default in RHEL 8. The value can be adjusted using the procedures similar to section 4.5.2. It is recommended to lower the value for latency-sensitive applications.
Performance tuning 4.6 iSCSI considerations Tuning performance for iSCSI is as much an effort in Ethernet network tuning as it is block-level tuning. Evaluate the common Ethernet kernel tunable parameters in order to determine the settings that provide the optimal performance gain with iSCSI. The use of Jumbo frames can lead to improved iSCSI performance and is a common recommendation for a large database environment.
Useful tools 5 Useful tools The following native Linux tools can be used to identify and correlate volumes to their respective SC Series devices. 5.1 The lsscsi command The lsscsi command is a tool that parses information from the /proc and /sys pseudo filesystems into human readable output. The lsscsi tool is installed using the following command. # yum –y install lsscsi The following lsscsi example filters and displays only SC Series LUNs in alphabetically order.
Useful tools Remove obsolete targets on the system: # rescan-scsi-bus.sh –r Scan for LUN size changes (after increasing it on SC Series storage system) and adjust the LUN size accordingly on the system: # rescan-scsi-bus.sh -s 5.3 The scsi_id command The scsi_id command can be applied on each volume one at a time. When there are many volumes on a Linux system, it is easier to query the volumes with a script.
Useful tools These WWID values in turn, correlate to the Serial Number or the Device ID value of the SC Series storage devices. SC Series Volume Serial Number and Device ID fields 5.4 Decoding the WWID The WWID returned by lsscsi, scsi_id, and multipath commands is a string of 33 characters. The string can be broken down into different parts that identify the vendor, make, serial number and device. The following example shows a breakdown of a SC volume WWID and their translation.
Useful tools 5.5 The dmesg command The dmesg command is useful for discovering which device names are assigned to recently discovered volumes. The following output demonstrates the discovery and assignment of a new SC Series volume to the /dev/sdh device file. # dmesg [snip] [1203830.267568] scsi 4:0:0:1: Direct-Access COMPELNT Compellent Vol 0605 PQ: 0 ANSI: 5 [1203830.267996] sd 4:0:0:1: Attached scsi generic sg8 type 0 [1203830.268089] sd 4:0:0:1: [sdh] 20971520 512-byte logical blocks: (10.7 GB/10.
Dell Storage REST API 6 Dell Storage REST API The Dell Storage REST API interface is available with the installation of Dell Storage Manager (DSM) 2015 R3 or newer. The REST API is recommended when intending to interact with SC Series storage managed by the DSM installation, either programmatically or in a command line interface. For more information, refer to the Dell Storage Manager REST API Cookbook. The use of the Compellent Command Utility (CompCU) has been deprecated.
Technical support and resources A Technical support and resources Dell.com/support is focused on meeting customer needs with proven services and support. Storage technical documents and videos provide expertise that helps to ensure customer success on Dell EMC storage platforms. A.1 53 Additional resources • Dell Storage Manager 2016 R3 Administrator’s Guide https://topics-cdn.dell.com/pdf/storage-sc2000_administrator-guide_en-us.
