Optimizing Failover Time in a Serviceguard Environment, June 2007

ManualsBrandsHP ManualsSoftwareHP SAP Linux Serviceguard Cluster Extension

Optimizing failover time in a Serviceguard environment

June 2007

The HP Serviceguard failover process..................................................................................................... 2

The process when failover is caused by a node failure ......................................................................... 2

Node Failure Detection ................................................................................................................. 3

Cluster Reformation Time............................................................................................................... 3

Election of Cluster Membership ...................................................................................................... 4

Lock Acquisition ........................................................................................................................... 4

Quiescence ................................................................................................................................. 4

Cluster Component Recovery ......................................................................................................... 5

Standard Serviceguard implementation: Resource Recovery............................................................... 5

Standard Serviceguard implementation: Applications Recovery.......................................................... 5

Serviceguard Extension for RAC: Group Membership Reconfiguration ................................................ 5

Serviceguard Extension for RAC: RAC Reconfiguration ..................................................................... 5

The process when failover is caused by a package failure .................................................................... 6

Standard Serviceguard implementation: Resource Failure Detection.................................................... 7

Standard Serviceguard implementation: Package Determination ........................................................ 7

Standard Serviceguard implementation: Resource Recovery............................................................... 7

Standard Serviceguard implementation: Application Startup.............................................................. 7

Serviceguard Extension for RAC: Group Membership Reconfiguration ................................................ 7

Serviceguard Extension for RAC: RAC Reconfiguration and Database Recovery................................... 7

How you can optimize failover time....................................................................................................... 8

Some help in estimating time for failover............................................................................................. 8

Node timeout value.......................................................................................................................... 9

Testing ...................................................................................................................................... 10

Lock acquisition (cluster lock, also called tie-breaker or arbitrator)........................................................ 10

Heartbeat subnet............................................................................................................................ 11

Network failure detection................................................................................................................ 11

Number of nodes and number of packages ...................................................................................... 12

EMS resources............................................................................................................................... 12

Package control scripts ................................................................................................................... 12

System Restart Options ................................................................................................................... 12

Applications .................................................................................................................................. 13

Serviceguard Extension for Faster Failover ............................................................................................ 13

Requirements for SGeFF.................................................................................................................. 14

Environments suitable to SGeFF ....................................................................................................... 14

Cluster parameter considerations ..................................................................................................... 15

Conclusion........................................................................................................................................ 16

For more information.......................................................................................................................... 16

Summary of content (16 pages)

PAGE 1
Optimizing failover time in a Serviceguard environment June 2007 The HP Serviceguard failover process..................................................................................................... 2 The process when failover is caused by a node failure ......................................................................... 2 Node Failure Detection ................................................................................................................. 3 Cluster Reformation Time ............
PAGE 2
Executive summary One of the most important measurements for an effective high-availability/mission-critical environment is how much delay the end user notices in the event of a failure. In these environments, certain things need to take place, such as detecting a failure, finding a way to restart the work, ensuring data integrity, and restarting applications and making them available to users. Different business needs require different environments.
PAGE 3
• Quiescence—During this quiet waiting time, non-members of the newly formed cluster are rebooted. • Cluster component recovery—Serviceguard does miscellaneous tasks, such as cluster information synchronization and package determination, before the cluster resumes the work. During the application-dependent phase of the failover time, Serviceguard starts the package control scripts, which were written by the user. In standard Serviceguard implementations, there are two steps, as shown in Figure 1.
PAGE 4
Election of Cluster Membership After a node believes another node has failed, it begins the cluster re-formation process. The cluster nodes elect the nodes that will be members of a newly re-formed cluster. Each healthy node tries to take over the work of the cluster. It tries to change cluster membership to include the nodes it can communicate with and exclude the nodes it cannot reach.
PAGE 5
Cluster Component Recovery In this step, Serviceguard does miscellaneous tasks, such as cluster information synchronization and package determination. If packages are down due to a failure, Serviceguard determines on which node(s), if any, they should be restarted. (See “Standard Serviceguard implementation: Package Determination” on page 6.) The time needed for cluster component recovery depends mainly on how many packages need to be restarted.
PAGE 6
The time needed for this step is determined by RAC, and the user cannot directly change it. The process when failover is caused by a package failure These are the steps in a failover that is caused by a package failure (rather than a node failure). Figure 3.
PAGE 7
• RAC reconfiguration and database recovery—After a cluster membership change, RAC reassigns the database locks that were on failed nodes and restarts the databases. Standard Serviceguard implementation: Resource Failure Detection Serviceguard monitors the configured package services and networks. Serviceguard package configuration can include Event Monitoring Service (EMS), which monitors hardware such as storage. EMS polls an EMS resource monitor to get returns.
PAGE 8
How you can optimize failover time There are ways you can optimize the failover process for your environment to reduce the time a package is unavailable. If the failover process takes longer than necessary, you are not getting maximum availability. If the failover starts and completes too quickly, however, you may get unnecessary failovers that reduce performance of a cluster—possibly reducing availability instead of improving it. It is important to find a balance between the extremes.
PAGE 9
Cluster component recovery This depends on the number of EMS resources, packages, nodes, etc. For environments using LVM or CVM 3.5 this time is usually less than a second. For environments using VERITAS CVM 4.1 or VERITAS CFS, VERITAS components, depending on the failure type, this time can range from 5 seconds to an additional cluster reformation time. Resource recovery This depends on the number of volume groups, IP addresses, services, etc.
PAGE 10
Testing To fine-tune the parameters, it is important to test the cluster in an environment that imitates the actual production environment. Test the cluster, running all of its packages, with the heaviest expected loads on networks, CPU, and I/O. To time failover, force each package to fail over to another node. One way to force a failover is to power off the node where the package is running. Read the logs of the failover, noting the time stamps.
PAGE 11
Quorum server considerations If you choose a quorum server, be sure the network between it and the nodes is highly available and reliable. If you encounter delays reaching the quorum server, you can configure a QS_TIMEOUT_EXTENSION, but the extension time adds directly to the lock acquisition time. Serviceguard calculates the time for the actual lock acquisition. You cannot change it directly, but (in Serviceguard releases earlier than A.11.
PAGE 12
Number of nodes and number of packages Serviceguard’s Package Manager starts the package control scripts using the exec process. This time is noticeable only when you have a large number of packages configured on a small number of nodes. On most systems, for example, if 50 packages failed over to one node, it would take a few seconds to start the scripts.
PAGE 13
cluster); cmhaltnode (to halt just this node); or the shutdown(1M) command (which will perform a cmhaltnode before rebooting the node). Applications There is no single solution to optimizing the efficiency of the many different applications. However, here are some general tips. Failed-over applications may spend a long time recovering data. See if you can reduce this time. Consider contacting the application vendor and the systems integrator for specific tuning tips.
PAGE 14
Requirements for SGeFF Every SGeFF cluster must meet these configuration requirements: • Serviceguard A.11.16 and SGeFF must be installed on every node in the cluster. • The cluster must have no more than two nodes.* • The cluster must have a quorum server cluster lock installed outside the cluster. • More than one heartbeat network must be configured. • An RS-232 link is not allowed for a heartbeat network. • VERITAS CVM and VERITAS CFS cannot be used in an SGeFF cluster.
PAGE 15
Cluster parameter considerations There are risks associated with Serviceguard Extension for Faster Failover’s shorter failover time. If there is a temporary interruption, the timeout should be long enough for the interruption to recover. Each administrator must decide how much time the cluster needs to confirm the likelihood (if not certainty) of a failure. On a busy system where the networks, I/O, or CPU have frequent or large spikes in activity, transient problems are likely to cause delayed heartbeats.
PAGE 16
Conclusion The total failover time in a Serviceguard cluster depends on many different factors and the interactions between them. Serviceguard users can optimize failover time to help maximize the time applications are available. Optimizing failover time means finding a balance between waiting longer than needed to act and acting too hastily. Less time spent in the failover process means less time packages are unavailable.