Dell Fluid Cache for SAN Version 2.0.
Notes, cautions, and warnings NOTE: A NOTE indicates important information that helps you make better use of your computer. CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem. WARNING: A WARNING indicates a potential for property damage, personal injury, or death. Copyright © 2016 Dell Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws.
Contents 1 Dell Fluid Cache for SAN Product Overview.................................................... 6 Key Features...........................................................................................................................................6 Connectivity...........................................................................................................................................6 2 Requirements for Fluid Cache....................................................................
7 Configuring and Managing Fluid Cache Clusters.........................................26 Creating a Fluid Cache Cluster...........................................................................................................26 Mapping Volumes in Fluid Cache....................................................................................................... 27 Mapping Volume to Servers..........................................................................................................
Fluid Cache VSA node not being added to Fluid Cache Cluster Server Object in Enterprise Manager...............................................................................................................................................43 Cluster or Application Has Performance Issues................................................................................ 43 Unable to add a Fluid Cache ESXi 6.0 host to an existing server cluster..........................................
Dell Fluid Cache for SAN Product Overview 1 Dell Fluid Cache for SAN is a server-side caching accelerator software. Fluid Cache makes high-speed PCI Express (PCIe) SSDs a shared, distributed cache resource. Fluid Cache is deployed on clusters of Dell PowerEdge systems connected using RoCE-enabled Ethernet adapters and operates within a SAN environment employing a Dell Compellent backing store. Key Features Fluid Cache supports the following features: • Write-back caching.
Figure 1. Fluid Cache Connectivity 1. Fluid Cache nodes communicate with each other over a private network using RDMA. 2. Enterprise Manager creates, manages, and monitors the Fluid Cache clusters. 3. Fluid Cache nodes communicate with the Management IP (VIP) of the Storage Controllers. 4. Fluid Cache communicates with vCenter to configure the VSA. 5. Enterprise Manager manages the Dell Compellent array. The interfaces illustrated in the figure are associated with specific ports.
Requirements for Fluid Cache 2 The following are prerequisites for deploying Fluid Cache for SAN. Refer to the Release Notes for the most recent requirements. The supported versions of firmware, drivers, and software are required to run Fluid Cache. These are listed in the Release Notes and are available at dell.com/support. For download instructions, see Downloading Firmware, Drivers, and Software.
Element Operating Systems Requirement • • • • NOTE: Dell PowerEdge R730 servers support only SR-IOV and does not support Dell Express Flash PCIe SSDs. Dell PowerEdge R730XD Dell PowerEdge R820 Dell PowerEdge R920 Dell PowerEdge T620 • • • • VMware ESXi 6.0 Update 1 (64-bit), VMware vCenter Server 6.0 Update 1 VMware ESXi 5.5 Update 2 (64-bit), VMware vCenter Server 5.5 Update 2 VMware ESXi 5.5 Update 3 (64-bit), VMware vCenter Server 5.5 Update 3 VMware ESXi 6.0 (64-bit), VMware vCenter Server 6.
Element Requirement • • • • SAN Hardware Dell Compellent Enterprise Manager 2014 R2 Dell Compellent Enterprise Manager 2015 R1 Dell Compellent Storage Center SCOS 6.5.2 Dell Compellent Storage Center SCOS 6.6.5 Dell Compellent SC8000 controller WARNING: Dell Fluid Cache for SAN requires Avahi traffic to run only on cache network interfaces running RoCE. Therefore, you must enable only the cache network running RoCE in the avahi-damon.conf file.
Fluid Cache Deployment Overview 3 Complete the following tasks to deploy Fluid Cache for SAN. The sections referenced in each task give detailed instructions. 1. Download the required software, firmware, and drivers. See Downloading Firmware, Drivers, and Software. 2. Identify and prepare the Fluid Cache servers and cache network switch. See Preparing the Fluid Cache Servers and Cache Network Switch. 3. Install the Fluid Cache software on each server in the cluster.
Downloading Firmware, Drivers, and Software 4 NOTE: It is recommended that you update servers using iDRAC. For more information, go to en.community.dell.com/techcenter/b/techcenter/archive/2013/04/17/idrac7-now-supportsupdating-server-components-using-racadm-and-web-gui.aspx. 1. Determine which firmware, drivers, and BIOS are installed for your system components by following the procedures in Checking System Configuration. 2. Go to dell.com/support. 3.
Preparing the Fluid Cache Components 5 Before installing Fluid Cache, you must prepare the components of the Fluid Cache network: the servers, cache devices, network cards, and switches. The instructions that follow assume that you have an existing SAN configured and managed. All nodes in the cache cluster must be connected to the SAN and visible on the Dell Compellent array. Instructions for racking and cabling a SAN solution are beyond the scope of this document.
Checking Guest Virtual Machine Settings All guest VMs that use Fluid Cache volumes mapped to the ESXi hosts must be configured to start after the Fluid Cache VSA and stop before the Fluid Cache VSA. Make sure that any non-Fluid Cache VMs are not configured for a boot order priority of 1. Preparing the Servers Make sure that each server in the ESXi cluster has the latest supported BIOS version, Lifecycle Controller firmware, and iDRAC firmware.
Non-NVMe Cache Devices • • Firmware— The firmware version for the Dell PCIe Express Flash SSDs must be B1490908 or later. Otherwise, even though they are visible in ESXi and in Enterprise Manager, the cache devices do not appear in the list of available devices. NOTE: The installed firmware version cannot be verified in ESXi. If the cache devices do not appear in the list of available devices and you are not sure that your firmware version is B1490908 or later, update your firmware.
• The ports are in an untagged state. • The switch firmware is up to date. • Flow control (transmit and receive) is enabled and Data Center Bridging (DCB) is disabled. NOTE: Enabling flow control is a requirement for Fluid Cache. 3. Save the running configuration. 4. To implement the changes, restart the switch.
Installing and Setting up Fluid Cache 6 NOTE: When installing and setting up Fluid Cache, you must restart the server multiple times. Enabling SSH on the ESXi Host To enable SSH on the ESXi host, perform the following steps: 1. In the vSphere Windows client, select one of the servers in the Fluid Cache cluster. 2. In the Configuration tab, go to Software → Security Profile. 3. In the Security Profile Services box, select Properties. The Services Properties window is displayed. 4.
Configuring the Network Adapters Before configuring network adapters, make sure that the network adapters are set up correctly. See Preparing the Cache Network Adapters. To configure the network adapters: 1. Download the network adapter driver for VMware. See Downloading Firmware, Drivers, and Software 2. In the vSphere Windows client, select one of the host servers in the Fluid Cache cluster. 3. In the Hardware area of the Configuration tab, click Storage and select a datastore. 4.
Configuring the iSCSI Network iSCSI networks with a single connection: You must add a virtual port group to the iSCSI vSwitch before installing the VSA. For instructions on creating a port group, refer to the topic Add a Virtual Machine Port Group available at the VMware Support website. iSCSI networks with two subnets: Follow the procedure below.
After installing the VSAs, you must edit the network adapter settings for the VSA (see Configuring the VSA). For more information on setting up and administering an iSCSI network with two subnets, refer to the following documents: • Dell Compellent Storage Center Best Practices with VMware vSphere 5.
8. In the Network Mapping window, the following source networks must be configured to an appropriate destination network. • Management Network — Configure with a destination network that can communicate to all the VSAs that participate in the Fluid Cache cluster. • iSCSI Network 1, ISCSI Network 2 — Configure with a destination network that is connected to the iSCSI network. Using the network, the VSAs communicate to the Compellents that have the storage you want to cache.
10. When you have finished adding hardware components, click OK in the Virtual Machine Properties window. 11. Repeat this process for each Fluid Cache VM. Configuring the VSA for Two iSCSI Subnets If your iSCSI connection consists of two subnets (a single iSCSI connection is also supported), follow this procedure to change the settings on each VSA in the Fluid Cache cluster: 1. In the vSphere Windows client, right-click one of the VSAs in the Fluid Cache cluster and select Edit Settings. 2.
NOTE: If you change the default login or password on any VSA in the Fluid Cache cluster, you must change them on every VSA. The login and password must be the same on all VSAs in the Fluid Cache cluster. After pressing , the configuration home page is displayed. 5. Select Change vCenter and press to register the ESXi host with vCenter: a. Enter the hostname or IP address for your vCenter and press . b. Enter a vCenter username with the necessary privileges (See the and press .
The two iSER physical functions of the network adapter are displayed, and also the bonded interface. The bonded interface is within the VSA, while the two iSER interfaces are on the ESXi host. NOTE: The bonded and two iSER interfaces must be on the same subnet. 13. Select one of the iSER interfaces and press to associate this interface with a virtual function: a. Enter the IP address for the interface and press . b. Enter the subnet mask for the interface and press . 14.
• If the VSAs are configured to use DNS, then make sure the name resolution is available for the VSA to start.
Configuring and Managing Fluid Cache Clusters 7 Make sure that your system meets the following prerequisites: • The Fluid Cache appliance must be installed and set up on all ESXi hosts to be used in the Fluid Cache cluster. • All ESXi hosts in the Fluid Cache cluster must be connected to the SAN and be visible on the Compellent array. • Enterprise Manager must be running and configured to manage the Storage Center to be used with Fluid Cache.
The Select Servers window is displayed. 6. By default, all available servers are selected. Clear the check box next to unwanted servers or select the Unselect All option, and then select three or more servers to be included. (Click Select All to use all available servers again.) 7. Click Next. The Cluster Settings page of the configuration wizard is displayed. 8. Type or select appropriate data in the Cluster Settings window. a. In the Name box, enter a name for the cluster. b.
To map a volume to a server: 1. In the Storage view, expand Storage Centers if necessary, and then select the Storage Center that contains the appropriate volume. 2. In the Storage tab, expand Volumes if necessary, and then locate the volume you want to map. 3. Right-click the volume and select Map Volume to Server. 4. In the Map Volume to Server window, select the server. 5. Click Next. 6. Select Enable Fluid Cache. 7.
Adding Servers to a Fluid Cache Cluster To add servers to a Fluid Cache cluster: 1. Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges. 2. In the Storage view, expand Fluid Cache Clusters if necessary. 3. Right-click the Fluid Cache cluster (not Fluid Cache Clusters itself) and click Tasks → Add Servers to Cluster. The Add Servers to Cluster dialog box is displayed. By default, all servers are selected that have Fluid Cache installed. 4.
Adding Hardware to a VSA Before adding cache devices or network adapters to a Fluid Cache cluster, do the following: 1. Log in to the vSphere Windows client. 2. Right-click a VSA under the ESXi host, and click Power → Power Off. 3. Confirm that you are turning off the VSA by clicking OK in the Confirm Shutdown dialog box. 4. Select the ESXi host. 5. In the Hardware area of the Configuration tab, click Advanced Settings → Edit. The Mark Devices for Passthrough window is displayed. 6.
A Maintaining Fluid Cache Installations This chapter describes tasks that must occasionally be performed after initial Fluid Cache deployment. WARNING: When performing any operation to a server platform that hosts Fluid Cache software, you must perform either one of the following methods: • Shut down a single Fluid Cache Cluster node (VSA) at a time and perform the host-based server updates. After the updates to host server are complete, reboot the host and bring the Fluid Cache Cluster node (VSA) online.
NOTE: The Fluid Cache mappings for the server are listed in the Volumes area at the bottom of the page. Removing Volume Mappings from a Subcluster 1. Make sure the volume is no longer in use and unmounted in vCenter. 2. In Enterprise Manager’s Storage view, expand Storage Centers if necessary and select the appropriate Storage Center. (Do not select Fluid Cache Clusters or its contents.) 3.
NOTE: There are important differences between powering off a VSA and deleting it entirely. Power off the VSA if it may rejoin the same cluster but you want to free up RAM (the VSA is configured to use 20 GB of RAM). Delete the VSA if you do not intend to use it in the same cluster again and you want to free up disk space (the VSA uses approximately 10GB), or if you intend to move the server to a different Fluid Cache cluster (changing clusters requires reinstallation of the VSA). 4.
Deleting a Cluster 1. Stop all I/O on cached volumes. 2. Unmount cached volumes. a. Remove the cached disks from the guest machine definition of any machine that uses a cached disk. b. Unmount the volume in the ESXi client. 3. Remove all Fluid Cache mappings from all volumes in the cluster (see Removing Volume Mappings). Note that you do not have to remove non-Fluid Cache mappings. 4. In Enterprise Manager’s Storage view, right-click the Fluid Cache cluster and click Delete. 5.
10. Contact Dell Support for instructions on the removal of claim rules. This must be done before you can redeploy Fluid Cache on the same guest machines. An alternative to removing claim rules is to completely format and reinstall the hosts (not just an upgrade install) of ESXi. 11.
B Troubleshooting Fluid Cache Installations If you have unresolved issues running Fluid Cache after a successful completion of the installation procedure, contact your Compellent Copilot. Troubleshooting the Compellent array and SAN architecture is beyond the scope of this document. For additional troubleshooting information, refer to the Enterprise Manager Administrator’s Guide and the documentation for other hardware and software components. See Related Documentation.
Server Does Not Appear in List of Servers Possible Cause A configuration issue is preventing the server from appearing in the list. Solution Make sure the procedure for configuring the server was completed successfully. See Configuring a Fluid Cache Server. Possible Cause Firewall settings are preventing network communication. Solution Check your firewall settings. See Checking Security Settings. Possible Cause The network switch is not correctly cabled or configured.
Possible Cause The cluster is in maintenance mode. Solution Take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and clearing the Maintenance Mode option. Note that certain system failures or an invalid license may prevent the cluster from being taken out of maintenance mode. Refer to the Enterprise Manager Status and Events tabs, and see Fluid Cache License Is Expired.
Solution Configure your DNS server to use static addresses for your ESXi hosts and Fluid Cache VSA VMs. Cannot Configure the Cache Network Possible Cause Firewall settings are preventing access to one or more ports required by Fluid Cache. Solution Change your firewall settings to allow access by Fluid Cache. For a list of required ports, see Checking Network Connections. Possible Cause One of the ports required by Fluid Cache is in use by another process.
Solution Make sure that all paths to the Compellent are operational. Possible Cause The TUI configuration tool could not obtain the iSCSI IQN from the ESXi host. When attempting to configure the iSCSI network, the following error message is displayed: “Cannot configure iSCSI. ESXi host is missing configured iSCSI software adapter.” Solution Manually configure the ESXi host by logging in to the ESXi host that the VSA is deployed on and configure the iSCSI software initiator.
Fluid Cache License Is Expired Possible Cause System settings such as changes to the system date cause the current Fluid Cache license to expire. You can still access data on cached volumes, but performance is degraded because the Fluid Cache cluster has been placed in maintenance mode and caching is no longer active. Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the status shown on the Events or Cache tabs.
Cannot Determine Which Cache Device Failed Possible Cause One of the cache devices has failed on a server with multiple cache devices installed, and it is not clear which of them failed. Solution In Enterprise Manager, in the Cache tab, the failed cache device is identified by a red X through it. Note the last digits of the number for this cache device. This number matches the serial number printed on the label of the failed cache device.
Possible Cause The application is not making use of the cache. Solution Make sure the application has an I/O profile that can leverage Fluid Cache, which accelerates reads, writes, and read-after writes. If the application does, check the running cache mode for that volume in Enterprise Manager. If necessary, change cache modes by deleting the volume mapping and creating a new one with a different cache mode. See Mapping Volumes.
Unable to add a Fluid Cache ESXi 6.0 host to an existing server cluster Description Using Enterprise Manager GUI, if you try to add an ESXi 6.0 Fluid Cache host to an existing server cluster within a Storage Center, the following error message is displayed: SC servers must all have the same operating system. Solution When adding a new Fluid Cache ESXi6.0 server host to an existing sub server cluster, in the Edit Server Settings dialog box, select VMware ESXi 5.
7. Solution 2 46 Run the following command to restart iscsid service. #service iscsid restart Perform the steps provided in the section “Configuring the VSA” of the deployment guide in the following sequence: 1. Perform the steps 1 through 5 as described in the deployment guide. 2. Select Configure This Appliance and press . 3. Configure your Backend Storage connection. 4. Perform the steps 6 and 7 and continue with step 10 as provided in the deployment guide.
C Example Cabling Diagram Figure 2. Example Cabling Diagram 1. Enterprise Manager console 2. Management interface switch 3. Servers (4) 4. Cache network switches (2) 5. SAN switch 6. Storage Center controllers (2) 7.
• Fluid Cache is installed on four servers, which conforms to the minimum of three servers per each cache cluster and maximum of nine servers required. • Each of the servers has a network adapter connected to a port on each of the two cache network switches. The switches are uplinked to each other. • The servers access the Storage Center using SAN connectivity. • Enterprise Manager is used to configure and monitor Fluid Cache.
D Configuring a Dell Networking Switch The following procedure is for one of the supported Dell Networking switches listed in the table in Requirements For Fluid Cache. For all other switches, see the manufacturer’s documentation. To configure a switch containing physical ports 0, 1, 2, and 3 for a cache network with four nodes connected to the switch’s 10-gigabit Ethernet ports: 1. Telnet into the switch and enter the login name and password to enter Exec mode.
E Checking the System Configuration If any of the system checks below show that the firmware or drivers are earlier than the required versions, update the components. See Requirements for Fluid Cache and Downloading Firmware, Drivers, and Software. Checking the Cache Device Firmware and Driver Non-NVMe Cache Devices • Firmware— The firmware version for the Dell PCIe Express Flash SSDs must be B1490908 or later.
4. Configure all interfaces for the management interface, the cache network, and potentially the iSCSI network. 5. Ping the management interface and cache network addresses from all nodes. To install Fluid Cache, you must be able to ping the management interface and cache network of every node in the cluster from each node in the cluster. To download updated firmware or drivers, see Downloading Firmware, Drivers, and Software.
F Related Documentation For PowerEdge server documentation, go to dell.com/support/manuals and enter your Service Tag. For cache device documentation, go to dell.com/storagecontrollermanuals and click Dell Power Edge Express Flash PCIeSSD. The following table lists documents you may want to refer to while installing Fluid Cache. Component Document Content Server PowerEdge Owner’s Manual Describes how to install, remove, configure, and troubleshoot server components.
Accessing Enterprise Manager and Storage Center Documentation Documentation for Dell Compellent products is not available at dell.com/support/manuals. To download Enterprise Manager and Storage Center documentation: 1. Go to portal.compellent.com. 2. Enter your user name and password and click Login. If you do not have a registration, send an email to customer.portal@compellent.com. 3. In the portal page, click Knowledge Center. 4.
