Dell Fluid Cache for SAN Version 2.1.
Notes, cautions, and warnings NOTE: A NOTE indicates important information that helps you make better use of your computer. CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem. WARNING: A WARNING indicates a potential for property damage, personal injury, or death. © 2016 Dell Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws.
Contents 1 Preface.....................................................................................................................7 Audience................................................................................................................................................ 7 Related Documentation........................................................................................................................ 7 Accessing Enterprise Manager and Storage Center Documentation...............
Configuring the VSA............................................................................................................................23 Fluid Cache Dependencies................................................................................................................. 25 Fluid Cache Dependencies for Creating a Fluid Cache Cluster.................................................. 25 Fluid Cache Dependencies for Fluid Cache Configuration.........................................................
Block Device Stats......................................................................................................................... 47 A Troubleshooting Fluid Cache Installations................................................... 49 Basic Troubleshooting Steps.............................................................................................................. 49 Unable to remove a cache device from Fluid Cache Cluster using Enterprise Manager................
Storage Center is Not Available..........................................................................................................60 Fluid Cache Server is Not Available....................................................................................................60 Information Displays Differently Between Storage Centers and Fluid Cache Clusters....................61 Verify That All Parts of the Fluid Cache Cluster are Communicating with Each Other...................
1 Preface Dell Fluid Cache for SAN is a server-side caching accelerator software. Fluid Cache makes high-speed PCI Express (PCIe) SSDs a shared, distributed cache resource. Fluid Cache is deployed on clusters of Dell PowerEdge systems connected using RoCE-enabled Ethernet adapters and operates within a SAN environment employing a Dell Compellent backing store. This guide provides information on how to install, configure and manage a Fluid Cache for SAN 2.1.0 for VMware ESXi based environments.
Component Document Content Updating BIOS on Dell 12G PowerEdge Servers (iDRAC7 12G only) Describes how to upgrade the BIOS on Dell 12G PowerEdge Servers using different Dell utilities. Updating BIOS on Dell 13G PowerEdge Servers (iDRAC8 13G only) (New for 13G) Describes how to upgrade the BIOS on Dell 13G PowerEdge Servers using different Dell utilities.
2. Use the locale menu or click the link that specifies your country or region. Technical Support and Customer Service Dell’s support service is available to answer your questions about Fluid Cache for SAN 2.1.0. Contacting Dell NOTE: If you do not have an active Internet connection, you can find contact information on your purchase invoice, packing slip, bill, or Dell product catalog. Dell provides several online and telephone-based support and service options.
Dell Fluid Cache for SAN Product Overview 2 Dell Fluid Cache for SAN is a server-side caching accelerator software. Fluid Cache makes high-speed PCI Express (PCIe) SSDs a shared, distributed cache resource. Fluid Cache is deployed on clusters of Dell PowerEdge systems connected using RoCE-enabled Ethernet adapters and operates within a SAN environment employing a Dell Compellent backing store. Dell Fluid Cache for SAN Features Fluid Cache supports the following key features: • Write-back caching.
Figure 1. Fluid Cache Connectivity 1. Fluid Cache nodes communicate with each other over a private network using RDMA. 2. Enterprise Manager creates, manages, and monitors the Fluid Cache clusters. 3. Fluid Cache nodes communicate with the Management IP (VIP) of the Storage Controllers. 4. Fluid Cache communicates with vCenter to configure the VSA. 5. Enterprise Manager manages the Dell Compellent array. The interfaces illustrated in the figure are associated with specific ports.
Example Cabling Diagram Figure 2. Example Cabling Diagram 1. Enterprise Manager console 2. Management interface switch 3. Servers (4) 4. Cache network switches (2) 5. SAN switch 6. Storage Center controllers (2) 7. Storage Center expansion enclosures (2) The example above has the following features: • Fluid Cache is installed on four servers, which conforms to the minimum of three servers per each cache cluster and maximum of nine servers required.
• The servers access the Storage Center using SAN connectivity. • Enterprise Manager is used to configure and monitor Fluid Cache. NOTE: For best practices on racking and cabling each hardware component, refer to the rack placement for the component at dell.com/support/manuals. Fluid Cache for SAN Deployment Requirements The following are prerequisites for deploying Fluid Cache for SAN. Refer to the Release Notes for the most recent requirements.
Preparing the Fluid Cache Components 3 Before installing Fluid Cache, you must prepare the components of the Fluid Cache network: the servers, cache devices, network cards, and switches. The instructions that follow assume that you have an existing SAN configured and managed. All nodes in the cache cluster must be connected to the SAN and visible on the Dell Compellent array. Instructions for racking and cabling a SAN solution are beyond the scope of this document.
Checking Guest Virtual Machine Settings All guest VMs that use Fluid Cache volumes mapped to the ESXi hosts must be configured to start after the Fluid Cache VSA and stop before the Fluid Cache VSA. Make sure that any non-Fluid Cache VMs are not configured for a boot order priority of 1. Preparing the Servers Make sure that each server in the ESXi cluster has the latest supported BIOS version, Lifecycle Controller firmware, and iDRAC firmware.
Non-NVMe Cache Devices • • Firmware— The firmware version for the Dell PCIe Express Flash SSDs must be B1490908 or later. Otherwise, even though they are visible in ESXi and in Enterprise Manager, the cache devices do not appear in the list of available devices. NOTE: The installed firmware version cannot be verified in ESXi. If the cache devices do not appear in the list of available devices and you are not sure that your firmware version is B1490908 or later, update your firmware.
• The ports are in an untagged state. • The switch firmware is up to date. • Flow control (transmit and receive) is enabled and Data Center Bridging (DCB) is disabled. NOTE: Enabling flow control is a requirement for Fluid Cache. 3. Save the running configuration. 4. To implement the changes, restart the switch. Configuring a Dell Networking Switch The following procedure is for one of the supported Dell Networking switches listed in the table in Requirements For Fluid Cache.
Installing and Setting up Fluid Cache 4 NOTE: When installing and setting up Fluid Cache, you must restart the server multiple times. Enabling SSH on the ESXi Host To enable SSH on the ESXi host, perform the following steps: 1. In the vSphere Windows client, select one of the servers in the Fluid Cache cluster. 2. In the Configuration tab, go to Software → Security Profile. 3. In the Security Profile Services box, select Properties. The Services Properties window is displayed. 4.
Configuring the Network Adapters Before configuring network adapters, make sure that the network adapters are set up correctly. See Preparing the Cache Network Adapters. To configure the network adapters: 1. Download the network adapter driver for VMware. 2. In the vSphere Windows client, select one of the host servers in the Fluid Cache cluster. 3. In the Hardware area of the Configuration tab, click Storage and select a datastore. 4. Right-click the datastore and select Browse Datastore.
Configuring the iSCSI Network iSCSI networks with a single connection: You must add a virtual port group to the iSCSI vSwitch before installing the VSA. For instructions on creating a port group, refer to the topic Add a Virtual Machine Port Group available at the VMware Support website. iSCSI networks with two subnets: Follow the procedure below.
After installing the VSAs, you must edit the network adapter settings for the VSA (see Configuring the VSA). For more information on setting up and administering an iSCSI network with two subnets, refer to the following documents: • Dell Compellent Storage Center Best Practices with VMware vSphere 5.
• Management Network — Configure with a destination network that can communicate to all the VSAs that participate in the Fluid Cache cluster. • iSCSI Network 1, ISCSI Network 2 — Configure with a destination network that is connected to the iSCSI network. Using the network, the VSAs communicate to the Compellents that have the storage you want to cache.
11. When you have finished adding hardware components, click OK in the Virtual Machine Properties window. 12. Repeat this process for each Fluid Cache VM. Configuring the VSA for Two iSCSI Subnets If your iSCSI connection consists of two subnets (a single iSCSI connection is also supported), follow this procedure to change the settings on each VSA in the Fluid Cache cluster: 1. In the vSphere Windows client, right-click one of the VSAs in the Fluid Cache cluster and select Edit Settings. 2.
NOTE: If you change the default login or password on any VSA in the Fluid Cache cluster, you must change them on every VSA. The login and password must be the same on all VSAs in the Fluid Cache cluster. After pressing , the configuration home page is displayed. 5. Select Change vCenter and press to register the ESXi host with vCenter: a. Enter the hostname or IP address for your vCenter and press . b. Enter a vCenter username with the necessary privileges (See the and press .
The two iSER physical functions of the network adapter are displayed, and also the bonded interface. The bonded interface is within the VSA, while the two iSER interfaces are on the ESXi host. NOTE: The bonded and two iSER interfaces must be on the same subnet. 13. Select one of the iSER interfaces and press to associate this interface with a virtual function: a. Enter the IP address for the interface and press . b. Enter the subnet mask for the interface and press . 14.
• If the VSAs are configured to use DNS, then make sure the name resolution is available for the VSA to start. Configuring ESXi Storage Adapter Settings To ensure correct operation during certain failure scenarios, it is recommended to change the Noop Timeout and Noop Interval settings on the Mellanox iSCSI over RDMA (iSER) Adapter. 1. Log in to the vCenter Server as administrator using the vSphere Client. 2. Click the Configuration tab. 3. Click the Storage Adapters link in the Hardware section. 4.
5 Fluid Cache for SAN Cluster Creation and Management Operations After you configure and validate the Fluid Cache for SAN components, use Enterprise Manager to create and manage the Fluid Cache Environment as described in the following sections. • Creating a Fluid Cache Cluster. See Creating a Fluid Cache Cluster • Managing a Fluid Cache Cluster Environment. See Managing a Fluid Cache Cluster Environment • Maintaining a Fluid Cache Cluster Environment.
NOTE: Except for the entries in this Discover Fluid Cache Servers window, all other references to Fluid Cache servers in this document and in Enterprise Manager refer to the ESXi host and not the VSA. There is no further use of the VSA name or IP address when managing Fluid Cache. b. The Port box is autopopulated. Change only if necessary. c. In the User Name box, type the username for the VSA, which is fldc. d. In the User Password box, type the password for the VSA. The default value is calvin.
Managing a Fluid Cache Cluster Environment Configuring Fluid Cache Volumes A Fluid Cache volume extends a normal Storage Center volume to be contained across the cache devices in a Fluid Cache cluster as well as permanently stored in the Storage Center volume.
node’s cache devices, because all data is local to the client. If this option is not selected, cached data is evenly distributed among all cache devices. NOTE: Use of this option with vMotioned guest VMs may result in reduced performance for that VM. 9. Click Finish. At this stage, there is a delay while the system makes a number of configuration changes. When processing is complete, the volume is available for use in ESXi (as a datastore, RDM, and so on).
Removing Volume Mappings from a Subcluster 1. Make sure the volume is no longer in use and unmounted in vCenter. 2. In Enterprise Manager’s Storage view, expand Storage Centers if necessary and select the appropriate Storage Center. (Do not select Fluid Cache Clusters or its contents.) 3. In the Storage tab, expand Servers, and then the Fluid Cache clusters if necessary and select the subcluster whose mappings you want to remove. 4.
The system processes for some time, and then the server appears inside Fluid Cache Clusters in the Storage tab. The Add Devices to Cluster dialog box is displayed. By default, all devices compatible with Fluid Cache are selected. 6. Clear the option next to unwanted cache devices or click the Unselect All option, and then select the cache devices to be added. (Select the Select All option to use all available cache devices again.
a. b. c. d. In the Storage view, select the Fluid Cache cluster. (Do not select Storage Center or its contents.) In the Summary tab, locate the server in the Servers section. Right-click the server and select Remove Server from Cluster. When asked to confirm the action, click OK. After the system processes for some time, the server reappears outside of the cluster in the Servers area. You may need to manually refresh the display.
6. In the Resource Allocation section, in the Reservation field, change the size from 24000 MB to 34000 MB 7. Click OK and power on the VSA. 8. Repeat the steps on other Fluid Cache VSA. Removing a Cache Device from a Fluid Cache Cluster If you are removing the cache device from the server, perform a graceful removal by first ensuring that the cache device is not in active use. See Removing a Server from a Fluid Cache Cluster for the steps on gracefully removing a server from a cluster. 1.
The Assign Storage Centers window is displayed. 5. In the Assign Storage Centers window, select one or more Storage Centers to be added. 6. Click OK. Remove a Storage Center from a Fluid Cache cluster Use Enterprise Manager to remove a Storage Center from a Fluid Cache cluster while keeping the cluster. 1. Click the Storage view. 2. In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster. 3.
15. Repeat tasks 12–15 for all remaining cache devices. 16. Repeat tasks 12–15 for the network adapter virtual function (the adapters whose description contains the words “Virtual Function”). NOTE: Do not add the same device twice. 17. When you have finished adding components, click OK in the Virtual Machine Properties window. Complete the tasks in Adding Cache Devices to a Fluid Cache Cluster.
Enable Server Load Equalizing for Storage Center Volumes Server load equalizing dynamically adjusts queue depth for volumes experiencing high IOPS to minimize the performance impact on other volumes. Enable load equalizing on a Storage Center that hosts Fluid Cache volumes to prevent cache flushing operations from adversely affecting performance for other volumes. NOTE: Enable load equalizing only for environments using Fluid Cache clusters, or if directed by Dell Technical Support Services. 1.
Rolling Upgrade Requirements Before you begin the Fluid Cache VSA Rolling Upgrade process, download the following to an appropriate directory • fldc_vsa.exe — Application that enables you to import and export Fluid Cache VSA configuration during the rolling upgrade process. • Fluid-Cache-Dell-2.1.0..OVA — The OVA file for Fluid Cache for SAN, version 2.1.0.
Rolling Upgrade Process Upgrading a Fluid Cache cluster can be performed by using the rolling upgrade procedure where an existing VSA configuration is replaced with a new version 2.1.0 VSA on the ESXi host. The VSA configuration is exported from an original v2.0.10 VSA to a new v2.1.0 VSA and then the cycle is repeated for all nodes in the cluster. Fluid Cache cluster Rolling upgrade includes the following steps: 1.
cannot be validated Do you want to connect to this host anyway (y/N): 5. Shutdown the 2.0.10 VSA after the export operation is complete. 6. Power on the VSA 2.1.0, and in the console, enter the following: a. Enter the default user name and password. b. Enter the same Appliance IP address, Subnet Mask, Primary DNS IP, and the Secondary DNS IP used for 2.0.10 VSA. 7. Restart the VSA 2.1.0. 8. Import the configuration file that you have exported to the newly deployed VSA 2.1.0.
d. In the Cache Server Stats page, in the Connection column, ensure that the connection is RDMA/RC, and in the Devices column ensure that the state of the cache devices are Active. e. In the Block Device Stats page, ensure that the status of all Fluid Cache device nodes are online. Preparing ESXi host Prepare one ESXi host at a time for upgrading the ESXi host version or to patch the ESXi host. 1. If you want to use a new vCenter 6.
6. After the upgrade is complete, reboot the host. 7. Configure the Mellanox drivers. Configuring Mellanox Drivers 8. Reconnect the host to vCenter. 9. Exit maintenance mode on the host and start the Fluid Cache guest appliance. If your Fluid Cache guest for the upgraded node starts without any issues and reconnects on the Cache Overview page, the upgrade is complete on this node.
Installing the Supported Drivers Perform the following steps to install the supported drivers: 1. SSH into the ESXi host. Run the following commands to disable the ESXi 6.x inbox driver: esxcli system module set -e false -m nmlx4_en esxcli system module set -e false -m nmlx4_core esxcli system module set -e false -m nmlx4_rdma 2.
Fluid Cache Web Page Overview 6 The Fluid Cache web page collects all possible information about the Fluid Cache clusters and provides detailed status and activity report about the hosts and cache devices within the clusters. Accessing Fluid Cache Web Page In the address bar, enter the URL of one of the hosts that is part of Fluid Cache cluster on port 8082. Use the following format: http://:8082 The browser will display the Fluid Cache web page.
• Address — Displays the cache network IP addresses of the Secondary CFMs. • Port — Displays the TCP ports that are used to communicate between the CFMs. • Online — Displays the connection state of the Secondary CFMs. The possible options are Yes or No. Metadata Server Stats The Metadata Server Stats section displays the following information about the Fluid Cache cluster hosts serving as Metadata Servers. • Hostname — Displays the hostname of the three Metadata servers in the cluster.
• Flush write dirty (GB) – Displays the data (in GB), which is dirty, was written on the Compellent. • Flush write dirty rate (MB/s) – Displays the current rate at which the host is writing dirty blocks on the Compellent. This is usually limited by the Compellent. • Write Same (GB) – Not supported. • Write Same rate (MB/s) – Not supported. Cache Server Per-Path Stats The Cache Server Per-Path Stats displays the information about the activity of hosts based on each cached LUN.
volume. The rows displays the activity of individual nodes that are accessing the cache volume. A normal, healthy cluster displays a status of online for each node’s access to the Fluid Cache device. • Device — Displays the name of the Fluid Cache block device. • Hostname — Displays the hostname of the nodes that have that block device mapped to them. For VMware the device will be mapped to all nodes in the cluster.
A Troubleshooting Fluid Cache Installations If you have unresolved issues running Fluid Cache after a successful completion of the installation procedure, contact your Compellent Copilot. Troubleshooting the Compellent array and SAN architecture is beyond the scope of this document. For additional troubleshooting information, refer to the Enterprise Manager Administrator’s Guide and the documentation for other hardware and software components. See Related Documentation.
Unable to connect the vCenter Server Description If you change the vCenter server IP/network address or domain/admin password while rebooting the VSA, the connection between the VSA and Fluid Cache cluster will be lost. Solution From the VSA console, log in to the vCenter server with correct IP address and credentials, and then reboot the VSA to reconnect the Fluid Cache cluster appropriately.
The Mellanox Virtual Function is not available for Passthrough in ESXi 6.0 Possible Cause The ESXi 6.0 installation may include Mellanox drivers that do not support the virtual function. Solution Disable or uninstall the following inbox drivers and make sure to install and configure the correct versions of the Mellanox drivers. For more information about the supported Mellanox drivers, see the Dell Fluid Cache for SAN Compatibility Matrix available at http://www.dell.com/CacheSolutions.
Solution To check device function, select the device in Enterprise Manager and in the Event tab, look for a device failure message. If necessary, replace the cache device by following the instructions provided in the Dell Compellent Enterprise Manager User’s Guide. Cannot Select a Specific Cache Mode Possible Cause There is an existing cache mode configured for the volume mapping. The cache mode chosen for a volume mapping cannot be changed.
Solution Wait for the volumes to unmap. Unmapping speed should increase as more volumes are unmapped. Unable to Map Volumes to Fluid Cache using Enterprise Manager Possible Cause ESXi hosts were added to vCenter by using their IP Address and not their hostname when DNS is being used on your network. Solution Turn off the Fluid Cache VSA virtual machines on the ESXi hosts. Remove the ESXi hosts from vCenter. Add the ESXi hosts back to vCenter by using their registered DNS name.
Solution Check your network connections and verify functionality of the iSER interfaces using vCenter (Configuration → Networking). Do not use vCenter to delete or modify the iSER interfaces used by Fluid Cache, or to create new iSER interfaces. Remove any iSER interfaces that were created prior to configuration of the Fluid Cache VSA. Possible Cause The correct version of the Mellanox driver is not installed, which results in an error message when attempting to power on the VSA.
Solution Check the binding in vSphere by opening the iSCSI Initiator Properties window. If no VMkernels are shown, click Add, select the VMkernel that corresponds to your iSCSI (uncached) network, and click OK. Click Close on the Properties window and when prompted to rescan the adapter, click Yes. Cannot Connect to Cluster Nodes Possible Cause Time synch issues are preventing Enterprise Manager from connecting to the cluster nodes.
Solution Contact Dell Customer Support. Cannot Assign or Remove a Storage Center Possible Cause The Storage Center is already assigned to another Fluid Cache cluster. Solution In Enterprise Manager, see whether or not a Storage Center is listed for the Fluid Cache cluster. Possible Cause Network connectivity issues are preventing Enterprise Manager from communicating with Storage Center. Solution Make sure the network is functioning properly.
Solution Verify that all hardware iSCSI initiator WWNs configured on your ESXi hosts are unique across all ESXi nodes in your Fluid Cache Cluster. Cluster or Application Has Performance Issues Possible Cause One or more cache devices are uninstalled, have failed, or do not have the correct firmware or drivers. Solution Use Enterprise Manager to check the functionality of the cache devices. Possible Cause The Compellent storage array is overloaded.
Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the status shown on the Events or Cache tabs. Contact your Dell representative to purchase a Fluid Cache license. After activating the new license, take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and then clearing the Maintenance Mode option. Unable to add a Fluid Cache ESXi 6.
Solution 2 5. Run the following command to edit theinitiatorname.iscsi file. #vi initiatorname.iscsi 6. Change the characters with asterisk (*) with the proper ESXi Hosts vmhba## software IQN that the VSA is associated. For example: root@localhost iscsi]# cat initiatorname.iscsi InitiatorName=iqn.1998-01.com.vmware:FLDC-QAESXi55U1Srv1-48a43810 7. Run the following command to restart iscsid service.
Fluid Cache License File is Invalid Verify that the license didn’t expire or that a system change caused the license to be invalidated. • The Fluid Cache license status can be verified on either the Fluid Cache clusters’ Events tab or Cache tab. • An evaluation license is valid for only 90 days. Contact your Dell sales representative to purchase a Dell Fluid Cache for SAN license.
Information Displays Differently Between Storage Centers and Fluid Cache Clusters The Fluid Cache Clusters display shows information from the Fluid Cache server perspective while the Storage Centers display shows information from the Storage Center perspective. If there is a communication issue between the Fluid Cache servers and the Storage Center, they may each show that condition differently.
• Make sure the cluster is not in Maintenance mode. • Enterprise Manager will give the cluster a red mark if EM is having trouble communicating with the Fluid Cache servers or if the Fluid Cache servers report problems back to EM. The Cache tab for the cluster in the Fluid Cache Clusters display may have some text to indicate the issue being experienced. Also check the events for clues as to what is causing the issue.
