53-1001769-01 30 March 2010 Fabric OS Troubleshooting and Diagnostics Guide Supporting Fabric OS v6.4.
Copyright © 2008-2010 Brocade Communications Systems, Inc. All Rights Reserved. Brocade, the B-wing symbol, BigIron, DCX, Fabric OS, FastIron, IronPoint, IronShield, IronView, IronWare, JetCore, NetIron, SecureIron, ServerIron, StorageX, and TurboIron are registered trademarks, and DCFM, Extraordinary Networks, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.
Contents About This Document In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Supported hardware and software . . . . . . . . . . . . . . . . . . . . . . . . . . . x What’s new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x Document conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Time issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Switch message logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Switch boot issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Rolling Reboot Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 FC-FC routing connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 4 Configuration Issues In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Configupload and download issues . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . 50 Brocade configuration form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Chapter 5 FirmwareDownload Errors In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 8 ISL Trunking Issues In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Link issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Buffer credit issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Chapter 9 Zone Issues In this chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Overview of corrective action . . . . .
Equipment status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Checking the temperature, fan, and power supply . . . . . . . . . . 97 Checking the status of the fans . . . . . . . . . . . . . . . . . . . . . . . . . 97 Checking the status of a power supply. . . . . . . . . . . . . . . . . . . . 98 Checking temperature status . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 System message log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
viii Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01
About This Document In this chapter • How this document is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix • Supported hardware and software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x • What’s new in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x • Document conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi • Additional information . . . . . . . . . . .
Supported hardware and software In those instances in which procedures or parts of procedures documented here apply to some switches but not to others, this guide identifies exactly which switches are supported and which are not. Although many different software and hardware configurations are tested and supported by Brocade Communications Systems, Inc. for 6.4.0, documenting all possible configurations and scenarios is beyond the scope of this document.
• Information that was changed: - All commands have been updated. • Information that was deleted: - All obsolete information. This information was obsoleted because it was no longer supported in the current version of firmware. - The iSCSI chapter has been removed and put in the iSCSI Administrator’s Guide. For further information about documentation updates for this release, refer to the release notes.
... Repeat the previous element, for example “member[;member...]” value Fixed values following arguments are printed in plain font. For example, --show WWN | Boolean. Elements are exclusive. Example: --show -mode egress | ingress Command examples This book describes how to perform configuration tasks using the Fabric OS command line interface, but does not describe the commands in detail.
Additional information This section lists additional Brocade and industry-specific documentation that you might find helpful. Brocade resources To get up-to-the-minute information, go to http://my.brocade.com and register at no cost for a user ID and password. For practical discussions about SAN design, implementation, and maintenance, you can obtain Building SANs with Brocade Fabric Switches through: http://www.amazon.
• Description of any troubleshooting steps already performed and the results • Serial console and Telnet session logs • syslog message logs 2. Switch Serial Number The switch serial number and corresponding bar code are provided on the serial number label, as illustrated below.
Chapter 1 Introduction to Troubleshooting In this chapter • Troubleshooting overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Most common problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Questions for common symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Gathering information for your switch support provider . . . . . . . . . . . . . . . . .
1 Most common problem areas Most common problem areas Table 1 identifies the most common problem areas that arise within SANs and identifies tools to use to resolve them.
Questions for common symptoms TABLE 2 1 Common symptoms Symptom Areas to check Chapter or Document Blade is faulty Firmware or application download Hardware connections Chapter 2, “General Issues” Chapter 5, “FirmwareDownload Errors” Chapter 7, “Virtual Fabrics” Blade is stuck in the “LOADING” state Firmware or application download Chapter 5, “FirmwareDownload Errors” Configupload or download fails FTP or SCP server or USB availability Chapter 4, “Configuration Issues” E_Port failed to come
1 Questions for common symptoms TABLE 2 Common symptoms (Continued) Symptom Areas to check Chapter or Document LEDs are steady Links Chapter 3, “Connectivity Issues” License issues Licensing Chapter 2, “General Issues” LSAN is slow or times-out LSAN tagging Chapter 2, “General Issues” Marginal link Links Chapter 3, “Connectivity Issues” No connectivity between host and storage Cables SCSI timeout errors SCSI retry errors Zoning Chapter 3, “Connectivity Issues” Chapter 8, “ISL Trunking I
Gathering information for your switch support provider TABLE 2 1 Common symptoms (Continued) Symptom Areas to check Chapter or Document User is unable to change switch settings RBAC settings Account settings Chapter 6, “Security Issues” Virtual Fabric does not form FIDs Chapter 7, “Virtual Fabrics” Zone configuration mismatch Effective configuration Chapter 9, “Zone Issues” Zone content mismatch Effective configuration Chapter 9, “Zone Issues” Zone type mismatch Effective configuration
1 Gathering information for your switch support provider 2. Type the appropriate supportSave command based on your needs: • If you are saving to an FTP or SCP server, use the following syntax: supportSave When invoked without operands, this command goes into interactive mode. The following operands are optional: -n Does not prompt for confirmation. This operand is optional; if omitted, you are prompted for confirmation. -c Uses the FTP parameters saved by the supportFtp command.
Building a case for your switch support provider 1 3. Set the utility to capture output from the screen. Some utilities require this step to be performed prior to opening up a session. Check with your utility vendor for instructions. 4. Type the command or start the process to capture the required data on the console. Capturing command output 1. Connect to the switch through a Telnet or SSH utility. 2. Log in using an account assigned to the admin role. 3.
1 Building a case for your switch support provider • Do you have IPsec installed on the switch’s Ethernet interface? Yes or no. Use the ipsecConfig --show command to determine the answer. • Do you have Inband Management installed on the switches GigE ports? Yes or no. User the portShow iproute geX command to determine the answer. • Are you using NPIV? Yes or no. Use the switchShow command to determine the answer.
Building a case for your switch support provider 1 • Host information: OS version and patch level HBA type HBA firmware version HBA driver version Configuration settings Storage information: • Disk/tape type Disk/tape firmware level Controller type Controller firmware level Configuration settings Storage software (such as EMC Control Center, Veritas SPC, etc.) 8. If this is a Brocade 48000, Brocade DCX or DCX-4S enterprise-class platform, are the CPs in-sync? Yes or no.
1 10 Building a case for your switch support provider Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01
Chapter 2 General Issues In this chapter • Licensing issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Time issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Switch message logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Switch boot issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 Switch message logs Switch message logs Switch message logs (RAS logs) contain information on events that happen on the switch or in the fabric. This is an effective tool in understanding what is going on in your fabric or on your switch. Weekly review of the RAS logs is necessary to prevent minor problems from becoming larger issues, or in catching problems at an early stage. Below are some common problems that can occur with or in your system message log.
Switch boot issues Symptom 2 I2C bus errors are appearing in the switch log. Probable cause and recommended action I2C bus errors generally indicate defective hardware or poorly seated devices or blades; the specific item is listed in the error message. See the Fabric OS Message Reference for information specific to the error that was received. Some Chip-Port (CPT) and Environmental Monitor (EM) messages contain I2C-related information.
2 Switch boot issues • • • • reboot haFailover fastBoot firmwareDownload The RRD feature is activated and halts rebooting when an unexpected reboot reason is shown continuously in the reboot history within a certain period of time. The period of time is switch dependent. The following are considered unexpected reboots: • Reset A reset reboot may be caused by one of the following: - Power-cycle of the switch or CP. Linux reboot command. Hardware watchdog timeout. Heartbeat loss related reboot.
FC-FC routing connectivity 2 Restrictions The following restrictions are applicable on the RRD feature: • • • • The RRD works only on CFOS based systems and is not available on AP blades. If FIPS mode is enabled, then the RRD feature works in record-only mode. Rolling Reboot Detection relies on the bootprom and Linux kernel working properly.
2 FC-FC routing connectivity The fcPing command sends a Fibre Channel ELS ECHO request to a pair of ports. It performs a zone check between the source and destination. In addition, two Fibre Channel Extended Link Service (ELS) requests will be generated. The first ELS request is from the domain controller to the source port identifier. The second ELS request is from the domain controller to the destination port identifiers.
FC-FC routing connectivity 2 Example of one device that accepts the request and another device that rejects the request: switch:admin> fcping 10:00:00:00:c9:29:0e:c4 21:00:00:20:37:25:ad:05 Source: 10:00:00:00:c9:29:0e:c4 Destination: 21:00:00:20:37:25:ad:05 Zone Check: Not Zoned Pinging 10:00:00:00:c9:29:0e:c4 [0x20800] with 12 bytes of data: received reply from 10:00:00:00:c9:29:0e:c4: 12 bytes time:1162 usec received reply from 10:00:00:00:c9:29:0e:c4: 12 bytes time:1013 usec received reply from 10:00:
2 FC-FC routing connectivity • Determines the specific ISLs and internal links with failures. • Exercises all ISL links in the base fabric for a logical fabric configuration. The number of actual paths covered when using the superping tool depends on two other parameters that you can optionally specify. When you issue the fcPing --allpaths command without any other options, superping covers all ISLs in the routes between source to destination.
FC-FC routing connectivity 2 NOTE Superping provides an indication if all ISLs are covered. If all the ISLs are not covered, you can increase the coverage count and maximum retries to transmit, so that complete coverage of all ISLs is achieved. Consider the example below in which a few errors are recorded on ISLs 3/205-->2/25, 3/204-->2/27, 2/42-->101/3, and 2/1-->101/8. But a maximum of 100 percentage errors are recorded on internal port 0/284 on domain 2, which is the potential faulty link.
2 FC-FC routing connectivity • In frame redirection configurations, where there is a physical host, physical target, virtual initiator and virtual target; superping only identifies the path from the physical host to the physical target regardless if the data path consists of the path from physical target to virtual target through the virtual initiator.
FC-FC routing connectivity 2 Timeout: (1..
2 Symptom Host application times out. Probable cause and recommended action The FCR tends to take a long time, more than 5 seconds, to present and setup paths for the proxy devices. Certain hosts are able to do discovery much faster as a result they end up timing out. Use the speed tag feature to always present target proxy to the host and import them faster. This helps sensitive hosts to do a quick discovery without timing out or cause an application failure.
Chapter 3 Connectivity Issues In this chapter • Port initialization and FCP auto discovery process . . . . . . . . . . . . . . . . . . . . • Link issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Connection problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Link failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Marginal links . . . . .
3 Port initialization and FCP auto discovery process • VEX_Port—A virtual EX_Port. It connects a Fibre Channel router to an edge fabric. From the point of view of a switch in an edge fabric, a VEX_Port appears as a normal VE_Port. It follows the same Fibre Channel protocol as other VE_Ports. However, the router terminates VEX_Ports rather than allowing different fabrics to merge as would happen on a switch with regular VE_Ports. Figure 2 shows the process behind port initialization.
Link issues 3 Link issues Symptom Port LEDs are flashing. Probable cause and recommended action Depending on the rate of the flash and the color of the port LED this could mean several things. To determine what is happening on either your port status LED or power status LED, refer to that switch’s model hardware reference manual. There is a table that describes the LEDs purpose and explains the current behavior as well as provides suggested resolutions. Symptom Port LEDs are steady.
3 Connection problems Checking the logical connection 1. Enter the switchShow command. 2. Review the output from the command and determine if the device successfully logged into the switch. • A device that is logically connected to the switch is registered as an F_, L_, E_, EX_, VE_, VEX_, or N_Port. • A device that is not logically connected to the switch will be registered as a G_ or U_Port, if NPIV is not on the switch. 3.
Connection problems Type Pid *N NL COS PortName NodeName 3 TTL(sec) 021a00; 2,3;20:00:00:e0:69:f0:07:c6;10:00:00:e0:69:f0:07:c6; 895 Fabric Port Name: 20:0a:00:60:69:10:8d:fd 051edc; 3;21:00:00:20:37:d9:77:96;20:00:00:20:37:d9:77:96; na FC4s: FCP [SEAGATE ST318304FC 0005] NL Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee0; 3;21:00:00:20:37:d9:73:0f;20:00:00:20:37:d9:73:0f; na FC4s: FCP [SEAGATE ST318304FC 0005] NL Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee1; 3;21:00:00:20:37:d9:76:b3;20:00:
3 Link failures Link failures A link failure occurs when a server, storage, or switch device is connected to a switch, but the link between the devices does not come up. This prevents the devices from communicating to or through the switch. If the switchShow command or LEDs indicate that the link has not come up properly, use one or more of the following procedures. The port negotiates the link speed with the opposite side.
Link failures 3 Checking for a loop initialization failure 1. Verify the port is an L_Port. a. Enter the switchShow command. b. Check the last field of the output to verify that the switch port indicates an L_Port. If a loop device is connected to the switch, the switch port must be initialized as an L_Port. c. Check to ensure that the state is online; otherwise, check for link failures.
3 Marginal links After becoming an active port, the port becomes an F_Port or an E_Port depending on the device on the opposite side. If the opposite device is a host or target device, the port becomes an F_Port. If the opposite device is another switch, the port becomes an E_Port. If there is a problem with the host or target device, enter the portCfgGPort to force the port to try to come up as point-to-point only. Correcting a port that has come up in the wrong mode 1. Enter the switchShow command. 2.
Marginal links 3 Only one frame is transmitted and received at any given time. An external cable is not required to run this test. The port LEDs flicker green rapidly while the test is running. Table 5 shows the different loopback modes you can use when using the portLoopbackTest to test a marginal link.
3 Marginal links switch:admin> porterrshow frames tx rx enc in crc crc too too bad enc disc link loss loss frjt fbsy err g_eof shrt long eof out c3 fail sync sig ============================================================================ 0: 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 665k 7.0k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 78 60 12 4 0 0 0 0 0 0 0 0 0 0 0 0 665k 7.
Device login issues 3 Device login issues A correct login is when the port type matches the device type that is plugged in. In the example below, it shows that the device connected to Port 1 is a fabric point-to-point device and it is correctly logged in an F_Port. switch:admin> switchshow switchName:brcd5300 switchType:64.
3 Device login issues In some cases, you may find that the port has been locked as an L_Port and the device attached is a fabric point-to-point device such as a host or switch. This would be an incorrect configuration for the device and therefore the device cannot log into the switch. To correct this type of problem, remove the Lock L_Port configuration using the portCfgDefault command.
Media-related issues 14 Online In_Sync LOGIN NOELP LED ACCEPT 15 Online In_Sync LED 3 PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE PRESENT ACTIVE E_PORT G_PORT U_PORT SEGMENTED LOGIN 6. Enter the portLogDumpPort portid command where the port ID is the port number; then, view the device-to-switch communication. switch:admin> portlogdumpport 8 | more time task event port cmd args ------------------------------------------------Thu Nov 6 16:52:39 2008 16:52:39.
3 Media-related issues TABLE 6 Component test descriptions Test name Operands Checks portTest [-ports itemlist] [-iteration count] [-userdelay time] [-timeout time] [-pattern pattern] [-patsize size] [-seed seed] [-listtype porttype] Used to isolate problems to a single replaceable element and isolate problems to near-end terminal equipment, far-end terminal equipment, or transmission line. Diagnostics can be executed every day or on demand.
Segmented fabrics TABLE 7 3 Switch component tests Test Function portBeacon Sets port beaconing mode. portLoopbackTest Performs a functional test of port N to N path. Verifies the functional components of the switch. turboRamTest Verifies that the on chip SRAM located in the 4 and 8 Gbps ASIC is using the Turbo-Ram BIST circuitry. This allows the BIST controller to perform the SRAM write and read operations at a much faster rate.
3 Segmented fabrics Reconciling fabric parameters individually 1. Log in to one of the segmented switches as admin. 2. Enter the configShow -pattern “fabric.ops” command. 3. Log in to another switch in the same fabric as admin. 4. Enter the configShow -pattern “fabric.ops” command. 5. Compare the two switch configurations line by line and look for differences. Do this by comparing the two Telnet windows or by printing the configShow -pattern “fabric.ops” output.
Segmented fabrics 3 4. Choose the fabric on which to change the duplicate domain ID; connect to the conflicting switch in that fabric. 5. Enter the switchDisable command. 6. Enter the configure command. 7. When the Fabric Parameters prompt displays, type y. 8. When the Domain prompt displays, type in the new number. 9. Press enter on all prompts to accept their default settings. 10. Enter the switchEnable command.
3 Port mirroring Reconciling incompatible software features Earlier releases of software may not be supported in new versions of Fabric OS code. This may be due to a software feature changing or new services being supported. If you suspect that you are trying to introduce a switch into a fabric that has an older version of code, check the release notes to verify that any features on that switch are supported in the fabric with the newer code.
Port mirroring 3 Port mirroring considerations Before creating port mirror connections, consider the following limitations: • A mirror port can be any port on the same switch as the source identifier port. • Only one domain can be mirrored. After a domain is defined, only mirror ports on the defined domain can be used. The first connection defines the restriction on the domain, which can be either the local domain or a remote domain.
3 Port mirroring Supported platforms Port Mirroring is supported only in FC ports. In general, a platform or blade supporting port mirroring supports both the mirror ports and the mirror connections. Exemptions are listed in Table 8. TABLE 8 Port Mirroring platform supportability Brocade Model Fabric OS v6.2.0 Fabric OS v6.1.1_enc Fabric OS v6.34.0 Fabric OS v6.4.0 FCOE10-24 Port Mirroring is not supported on Europa. Port mirroring is supported on Europa.
Port mirroring TABLE 9 3 Maximum number of mirror connections Model Maximum Number of Mirror Connections (chassis-wide) Brocade 5000 3 Brocade 5100 3 Brocade 5300 1 Brocade 5410 1 Brocade 5450 1 Brocade 5460 1 Brocade 5470 1 Brocade 5480 1 Brocade 7500 3 Brocade 7500E 3 Brocade 7600 3 Brocade 7800 1 Brocade DCX 3 Brocade DCX-4S 3 Brocade Encryption Switch 3 Brocade M5424 1 Brocade VA-40FC 3 FA4-18 3 FC10-6 3 FC4-16IP 3 FC4-48 3 FC8-16/32/48/64 3 FR4-18i 3
3 Port mirroring Adding a port mirror connection 1. Connect to the switch and log in using an account assigned to the admin role. 2. Enter the portMirror --add slotnumber/portnumber SourceID DestID command. The configuration database keeps information about the number of port mirror connections configured on a switch, the number of chunks of port mirroring data that are stored, and the chunk number. When removing a mirror connection, always use this method to ensure that the data is cleared.
Port mirroring 3 Displaying port mirror connections 1. Connect to the switch and log in using an account assigned to the admin role. 2. Enter the portMirror --show command.
3 46 Port mirroring Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01
Chapter Configuration Issues 4 In this chapter • Configupload and download issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 • Brocade configuration form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Configupload and download issues It is important to maintain consistent configuration settings on all switches in the same fabric because inconsistent parameters (such as inconsistent PID formats) can cause fabric segmentation.
4 Configupload and download issues • The FTP or SCP server’s IP address cannot be contacted. Verify that you can connect to the FTP server. Use your local PC to connect to the FTP server or ping the FTP server. Example of a successful ping C:\> ping 192.163.163.50 Pinging 192.163.163.50 with 32 bytes of data: Reply from 192.163.163.50: bytes=32 time=5ms TTL=61 Ping statistics for 192.163.163.
Configupload and download issues 4 • On a Virtual Fabric-enabled switch, you do not have the chassis role permission set on your user account. Implement one change at a time, then issue the command again. By implementing one change at a time, you will be able to determine what works and what does not work. Knowing which change corrected the problems will help you to avoid this problem in future endeavors. Symptom The configuration download fails.
4 Configupload and download issues Probable cause and recommended action Verify that the switch was rebooted by checking the system log. If you are doing this on a enterprise-class platform, verify that both CPs rebooted by checking the system log. If any error occurs during the download, such as an error about a particular key, it is important to issue the configDefault command and attempt to repeat the configDownload command.
Brocade configuration form 4 Brocade configuration form Use this form as a hard copy reference for your configuration information. In the hardware reference manuals for the Brocade 48000, DCX, and DCX-4S modular switches there is a guide for FC port setting tables. The tables can be used to record configuration information for the various blades.
4 52 Brocade configuration form Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01
Chapter 5 FirmwareDownload Errors In this chapter • Blade troubleshooting tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Firmware download issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Troubleshooting firmwareDownload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • USB error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Firmware download issues Symptom The blade is faulty (issue slotShow to confirm). Probable cause and recommended action If the port or application blade is faulty, enter the slotPowerOff and slotPowerOn commands for the port or application blade. If the port or application blade still appears to be faulty, remove it and re-insert it into the chassis. Symptom The AP blade is stuck in the “LOADING” state (issue slotShow to confirm).
Firmware download issues Symptom 5 Firmwaredownload times out. Probable cause and recommended action This can be caused by an excessively slow network. If it takes more than 30 minutes to download firmware on a switch, or on each CP in a director, the firmware download process times out. If a timeout occurs on a switch, the firmware download process synchronizes the two partitions on the switch by starting a firmware commit operation.
5 Firmware download issues Password: Server IP: 192.126.168.115, Protocol IPv4 Checking system settings for firmwaredownload... Firmware access timeout. The server is inaccessible or firmware path is invalid. Please make sure the server name or IP address, the user/password and the firmware path are valid. Symptom Cannot download the requested firmware. Probable cause and recommended action The firmware you are trying to download on the switch is incompatible.
Troubleshooting firmwareDownload 5 Troubleshooting firmwareDownload A network diagnostic script and preinstallation check is a part of the firmwareDownload procedure. The script and preinstallation check performs troubleshooting and automatically checks for any blocking conditions. If the firmware download fails, see the Fabric OS Message Reference for details about error messages. Also see, “Considerations for downgrading firmware” on page 58.
5 Considerations for downgrading firmware TABLE 11 USB error handling Scenario under which download fails Error handling Action An access error occurs during firmwaredownload because the removal of the USB device, or USB device hardware failure, etc. Firmwaredownload will timeout and commit will be started to repair the partitions of the CPUs that are affected. See previous table for details. None. USB device is not enabled.
Considerations for downgrading firmware 5 File Name: /home/userfoo/v6.3.0 Network Protocol (1-auto-select, 2-FTP, 3-SCP) [1]: Password: Checking System Settings... Version compatibility check passed. Checking version compatibility... Version compatibility check passed.. The following items need to be addressed before downloading the specified firmware: Downgrade is not allowed because there are more than 2 FX8-24 blades plugged in to the chassis. The limit after downgrade is 2.
5 Considerations for downgrading firmware IPSec on FCIP tunnels requires version v6.3.1 or higher for 7800, and v6.4.0 or higher for FX8-24. Please use "portcfg fciptunnel" to disable IPSec for all FCIP tunnels and try again. Downgrade is not allowed because Persistent PID is enabled. Please use "ag --persistentpidenable 0" to disable the feature before downgrading. Downgrade is not allowed because one or more ports are locked as E-port.Please use "portcfgeport" CLI to remove the lock.
Considerations for downgrading firmware Message 5 The FS8-18 (type 43) blade is not supported by the target firmware. Please use slotshow to find out which slot it is in and remove it first. Probable cause and recommended action The firmware download operation was attempting to downgrade a system to Fabric OS v6.1.1_enc or earlier with one or more Brocade FC8-18 AP blades (blade ID 43) in the system. Brocade Encryption Blade FC8-18 AP blades are not supported on firmware v6.1.
5 Considerations for downgrading firmware Message Downgrade is not allowed because VF is enabled. Please run "lscfg --config" and "lscfg --delete" commands to remove the non-default LS first, then run "fosconfig --disable vf" to disable VF before proceeding. Probable cause and recommended action You cannot downgrade because Virtual Fabrics are enabled. Delete the logical switches, delete the base switch, and disable Virtual Fabrics prior to downgrading the firmware.
Chapter 6 Security Issues In this chapter • Password issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Device authentication issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Protocol and certificate management issues . . . . . . . . . . . . . . . . . . . . . . . . • SNMP issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • FIPS issues. . . . . . . . . . . . . . .
6 Device authentication issues Symptom Unable to log into the boot PROM. Probable cause and recommended action To recover a lost boot PROM password, contact your switch service provider. You must have previously set a recovery string to recover the boot PROM password. This does not work on lost or forgotten passwords in the account database. Password recovery options The following table describes the options available when one or more types of passwords are lost.
Protocol and certificate management issues Symptom 6 Switch is unable to form an F_Port. Probable cause and recommended action Regardless of the device authentication policy mode on the switch, the F_Port is disabled if the DH-CHAP protocol fails to authenticate. If the HBA sets the FC-SP bit during FLOGI and the switch sends a FLOGI accept with FC-SP bit set, then the switch expects the HBA to start the AUTH_NEGOTIATE.
6 SNMP issues • If you think this is a policy-related issue, FCS switch or other security server-related issue then use supportSave -n to collect data from the Primary FCS switch and all affected switches.
FIPS issues Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 6 67
6 68 FIPS issues Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01
Chapter 7 Virtual Fabrics In this chapter • General Virtual Fabric troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Fabric identification issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Logical Fabric issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Base switch issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Logical switch issues. . . . . . .
7 Fabric identification issues • VE_Ports cannot exist in a logical switch that has XISL use turned on. Although VE_Ports are allowed in a base switch, Fabric OS v6.2.0 and later do not support the use of VE_Ports to carry traffic for logical fabrics using XISLs. They can be used to carry FCR traffic through EX_ and VEX_Ports. You should make sure your configuration does not result in the use of VE_Ports in a base switch for logical fabric traffic.
Base switch issues Symptom 7 The switch with domain with firmware version has joined the FID fabric and may not be compatible with XISL use. Probable cause and recommended action This message indicates the specified switch in the logical fabric using XISLs is running an incompatible firmware version and must be upgraded to Fabric OS v6.2.0. Base switch issues All logical switches in a fabric should have the same base switch attribute.
7 Logical switch issues Logical switch issues CAUTION When a logical switch is created, all configuration for the logical switch is set to factory defaults. When a logical switch is deleted, all configuration for the logical switch is deleted permanently and is not recoverable. Symptom The indicated slot is empty. Probable cause and recommended action You used the lsCfg command and an empty slot was specified. Reissue the command with the appropriate slot number.
Switch configuration blade compatibility Symptom 7 Unable to remove ports from their current switch. Probable cause and recommended action When moving ports to a switch, they are first removed from the switch in which they reside. This error message is displayed if this step fails. Symptom A non-GE blade is within the slot range. Probable cause and recommended action You are attempting to configure a GE port on a slot that does not contain GE ports.
7 Switch configuration blade compatibility Gathering additional information For Virtual Fabric-related issues, use the following guidelines to gather additional data for your switch support provider: • Perform the supportSave command. • If not sure about the problem area, perform the supportSave command on all chassis and logical switches in the fabric.
Chapter 8 ISL Trunking Issues In this chapter • Link issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 • Buffer credit issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Link issues This section describes trunking link issues that can come up and recommended actions to take to correct the problems. Symptom A link that is part of an ISL trunk failed.
8 Buffer credit issues • • • • Port trunking is disabled. The port is not an E_Port. The port is not 2 Gbps, 4 Gbps, or 8 Gbps. The port connects to a switch other than the one you want it to. To correct this issue, connect additional ISLs to the switch with which you want to communicate. • The ports are not the same speed or they are not set to an invalid speed. Manually set port speeds to a speed supported on both sides of the trunk. • The ports are not set to the same long distance mode.
Chapter 9 Zone Issues In this chapter • Overview of corrective action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Segmented fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Zone conflicts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Gathering additional information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9 Segmented fabrics If you enter the cfgShow command to display information about all zones, the TI zones appear in the defined zone configuration only and do not appear in the effective zone configuration. 1. Connect to the switch and log in as admin. 2. Enter the zone --show command. zone --show [name] where: name The name of the zone to be displayed. If the name is omitted, the command displays information about all TI zones in the defined configuration.
Zone conflicts Symptom 9 Zone conflict appears in logs and fabric is segmented. Probable cause and recommended action This issue is usually caused by incompatible zoning configurations. Verify the following are true: • The effective cfg (zone set) on each end of the segmented ISL is identical. • Any zone object with the same name has the same entries in the same sequence. Symptom Fabric segmentation is caused by a “configuration mismatch”.
9 Zone conflicts TABLE 14 Commands for debugging zoning (Continued) Command Function cfgEnable Use to enable and activate (make effective) the specified configuration. cfgSave Use to save changes to the zone configuration database. cfgTransAbort Use to abort the current zoning transaction without committing it. cfgTransShow Use to display the ID of the current zoning transaction.
Zone conflicts 9 Changing the default zone access A switch is not allowed to merge with another switch that has an active effective configuration if the default zone is set to “no access”. Before the switch can join, the default zone setting has to be set to "all access". When the default zone no access option is enabled and the active configuration is disabled by using the cfgDisable command, a special hidden configuration with no members is activated.
9 Zone conflicts Checking for Fibre Channel connectivity problems Enter the fcPing command (refer to the Fabric OS Command Reference for more information on this command), which checks the zoning configuration for the two ports specified by: • Generates an ELS (Extended Link Service frame) ECHO request to the source port specified and validates the response. • Generates an ELS ECHO request to the destination port specified and validates the response.
Zone conflicts 9 received reply from 0x020800: 12 bytes time:1006 usec received reply from 0x020800: 12 bytes time:1008 usec received reply from 0x020800: 12 bytes time:1038 usec received reply from 0x020800: 12 bytes time:1010 usec 5 frames sent, 5 frames received, 0 frames rejected, 0 frames timeout Round-trip min/avg/max = 1006/1044/1159 usec Pinging 22:00:00:04:cf:75:63:85 [0x217d9] with 12 bytes of data: Request timed out Request timed out Request timed out Request timed out Request timed out 5 frame
9 Gathering additional information Gathering additional information Collect the data from a supportSave -n command. Then collect the data from the cfgTransShow command. For the port having the problem, collect the data from the filterPortShow command.
Chapter Working With Diagnostic Features 10 In this chapter • About Fabric OS diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 • Diagnostic information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 • Power-on self test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 • Switch status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10 Power-on self test To save a set of files that customer support technicians can use to further diagnose the switch condition, enter the supportSave command. The command prompts for an FTP server, packages the following files, and sends them to the specified server: • The output of the supportShow command. • The contents of any trace dump files on the switch. • System message (RAS) logs. See also “Automatic trace dump transfers” on page 103.
Power-on self test 10 eth0: Link status change: Link Up. 100 Mbps Full duplex Auto (autonegotiation complete). INITCP: CPLD Vers: 0x95 Image ID: 0x19 uptime: 2008; sysc_qid: 0 Fabric OS (Paulsa45) Paulsa45 console login: 2005/03/31-20:12:42, [TRCE-5000], 0,, INFO, ?, trace:, trace_buffer.c, line: 1170 2005/03/31-20:12:42, [LOG-5000], 0,, INFO, SW4100_P45, Previous message repeat 1 time(s), trace_ulib.
10 Switch status Disabling POST A reboot is not required for this command to take effect. 1. Connect to the switch and log in with a user account that has admin privileges with the chassis-role permission. 2. Enter the diagDisablePost command. This disables POST1 and POST2. Enabling POST A reboot is not required for this command to take effect. 1. Connect to the switch and log in with a user account that has admin privileges with the chassis-role permission. 2.
Switch status Faulty ports monitor Missing SFPs monitor 10 HEALTHY HEALTHY All ports are healthy For more information on how the overall switch status is determined, see the switchStatusPolicySet command in the Fabric OS Command Reference. Displaying switch information 1. Connect to the switch and log in as admin. 2.
10 SpinFab and portTest The switchShow command also displays the following information for ports on the specified switch: Index Index follows Area up to 255. Then it continues to the maximum port of the platform. Index identifies the port number relative to the switch. Index column is only displayed on enterprise-class platforms. Slot Slot number; 1-4 and 7-10. Port Port number; 0-15, 0-31 or 0-47. Address The 24-bit Address Identifier.
SpinFab and portTest 10 CAUTION During the spinFab testing, the switch remains in normal operation. However, some performance degradation will occur due to the ISL links being saturated with the test frames. This test should be run with caution on a live fabric. Table 15 lists the supported ports for the specified version of Fabric OS when using the spinFab command. TABLE 15 Port type support Port type Supported in v6.2.0 Supported in v6.3.0 Supported in v6.4.
10 SpinFab and portTest Pt0/3(33) Ch0/33 Enc_out Error Counter is 187725412 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1, Pt0/3(33) Ch0/33 TruncFrm Error Counter is 32 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1, Pt0/3(33) Ch0/33 FrmTooLong Error Counter is 32 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1, Pt0/3(33) Ch0/33 BadOrdSet Error Counter is 32 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1, Pt0/3(33) Ch0/33 BadEOF Error Counter is 1 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1, Pt0/3(33) Ch0/33 DiscC3 Error Count
SpinFab and portTest 10 The following are debugging procedures: • Check whether the same port is reporting Link Errors as discussed in “Link Errors” on page 91. If yes, follow the same set of debugging procedures as discussed in “Link Errors” on page 91. • Check whether the local/remote port is beyond port 255. If yes, try connecting to the lower number of ports. This behavior is found in Fabric OS v6.2.0 and earlier versions only. • Check whether the local/remote port is part of shared-area region.
10 Port information Port information Use the following instructions to view information about ports and to help diagnose if your switch is experiencing port problems. Viewing the status of a port 1. Connect to the switch and log in as admin. 2. Enter the portShow [slot/] port command, specifying the number that corresponds to the port you are troubleshooting.
Port information 10 Displaying the port statistics 1. Connect to the switch and log in as admin. 2. At the command line, enter the portStatsShow command. Port statistics include information such as the number of frames received, number of frames sent, number of encoding errors received, and number of class 2 and class 3 frames received. See the Fabric OS Command Reference for additional portStatsShow command information, such as the syntax for slot or port numbering.
10 Port information Displaying a summary of port errors for a switch 1. Connect to the switch and log in as admin. 2. Enter the portErrShow command. See the Fabric OS Command Reference for additional portErrShow command information. switch:admin> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy tx rx in err g_eof shrt long eof out c3 fail sync sig ============================================================================ 0: 665k 7.
Equipment status TABLE 16 10 Error summary description (Continued) Error type Description too shrt Frames shorter than minimum too long Frames longer than maximum bad eof Frames with bad end-of-frame delimiters enc out Encoding error outside of frames disc c3 Class 3 frames discarded link fail Link failures (LF1 or LF2 states) loss sync Loss of synchronization loss sig Loss of signal frjt Frames rejected with F_RJT fbsy Frames busied with F_BSY Equipment status You can display stat
10 Equipment status Fan 1 is Absent Fan 2 is Ok, speed is 6553 RPM Fan 3 is Ok, speed is 6367 RPM The possible status values are: • • • • • • OK—Fan is functioning correctly. Absent—Fan is not present. Below minimum—Fan is present but rotating too slowly or stopped. Above minimum—Fan is rotating too quickly. Unknown—Unknown fan unit installed. FAULTY—Fan has exceeded hardware tolerance. The output from this command varies depending on switch type and number of fans present.
System message log 3 Ok 18 10 64 Information displays for each temperature sensor in the switch. The possible temperature status values are: • OK—Temperature is within acceptable range. • FAIL—Temperature is outside of acceptable range. Damage might occur. Refer to the hardware reference manual for your switch to determine the normal temperature range. System message log The system message log, or RASLog, feature enables messages to be saved across power cycles and reboots.
10 Port log Displaying the system message log one message at a time 1. Connect to the switch and log in as admin. 2. Enter the errShow command. Clearing the system message log 1. Connect to the switch and log in as admin. 2. Enter the errClear command. 3. Repeat step 2 on the standby CP for a complete erasure of the message log. All switch and chassis events are removed from both CPs. Port log The Fabric OS maintains an internal log of all port activity.
Port log 10
10 Syslogd configuration Syslogd configuration The system logging daemon (syslogd) is an IP-based service for logging system messages made available by default on Unix and Linux operating systems. It is available as a third-party application for Windows operating systems. Fabric OS can be configured to use a UNIX-style syslogd process to forward system events and error messages to log files on a remote host system.
Automatic trace dump transfers 10 3. Verify that the IP address was entered correctly, using the syslogdIpShow command. The syslogdIpadd command accepts IPv4 and IPv6 addresses. You can specify up to six host IP addresses for storing syslog messages, as shown in this example: switch:admin> switch:admin> switch:admin> switch:admin> switch:admin> switch:admin> syslogdipadd syslogdipadd syslogdipadd syslogdipadd syslogdipadd syslogdipadd 1080::8:800:200C:417A 1081::8:800:200C:417A 1082::8:800:200C:417A 10.
10 Automatic trace dump transfers • Setting up a periodic checking of the remote server so that you are alerted if the server becomes unavailable and you can correct the problem. After the setup is complete, you can run the supportSave -c command to save RASLog, TRACE, supportShow, core file, FFDC data and other diagnostic support information to the server without specifying server details. The following procedures describe the tasks for setting up automatic transfer.
Appendix A Switch Type and Blade ID The switchType is a displayed field listed when you run the switchShow command. When you are gathering information to give to your switch support provider, you may be asked the switch model. If you do not know the model, you can use this chart to convert the switchType to a B-Series model number. switch:admin> switchshow switchName:Sprint5100 switchType:66.
A Switch Type and Blade ID TABLE 19 switchType switchType to B-Series model converter B-Series switch model ASIC Base switch speed 58 5000 Condor 4 Gb 32-port switch 61 4424 GoldenEye 2 Gb 24-port embedded switch 62 Brocade DCX Condor2 8 Gb 798-port core fabric backbone 64 5300 GoldenEye2 8 Gb 64-port switch 66 5100 Condor2 8 Gb 32-port switch 67 Brocade Encryption Switch Condor2 8 Gb 16-port encryption switch 69 5410 GoldenEye2 8 Gb 12-port embedded switch 71 300 GoldenE
Switch Type and Blade ID TABLE 20 Blade ID A B-series blade model descriptions (Continued) B-series blade model ASIC Description 33 FA4-18 Condor 4 Gb 16-FC ports with two 10/100/1000 BaseT Ethernet copper interfaces Fibre Channel Application Services blade 36 FC4-48 n/a 4 Gb 48-FC ports blade 39 FC10-6 Condor 10 Gb 6-FC ports blade 51 FC8-48 Condor2 8 Gb 48-FC ports blade 55 FC8-32 n/a 8 Gb 32-FC ports blade 68 FS8-18 Condor2 8 Gb 16-port encryption blade 74 FCOE10-24 Condor
A 108 Switch Type and Blade ID Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01
Appendix B Hexadecimal Hexadecimal, or simply hex, is a numeral system with a base of 16, usually written using unique symbols 0–9 and A–F, or a–f. Its primary purpose is to represent the binary code that computers interpret and represent in a format easier for humans to read. It acts as a form of shorthand, in which one hexadecimal digit stands in place of four binary bits. For example, the decimal numeral 79, whose binary representation is 01001111, is 4F (or 4f) in hexadecimal (4 = 0100, F = 1111).
B Hexadecimal TABLE 21 Decimal 01 02 03 04 05 06 07 08 09 10 Hex 01 02 03 04 05 06 07 08 09 0a Decimal 11 12 13 14 15 16 17 18 19 20 Hex 0b 0c 0d 0e 0f 10 11 12 13 14 Decimal 21 22 23 24 25 26 27 28 29 30 Hex 15 16 17 18 19 1a 1b 1c 1d 1e Decimal 31 32 33 34 35 36 37 38 39 40 Hex 1f 20 21 22 23 24 25 26 27 28 Decimal 41 42 43 44 45 46 47 48 49 50 Hex 29 2a 2b 2c 2d 2e 2f 30 31 32 Decimal 51 52
Hexadecimal TABLE 21 B Decimal to Hex conversion table (Continued) Decimal 181 182 183 184 185 186 187 188 189 190 Hex b5 b6 b7 b8 b9 ba bb bc bd be Decimal 191 192 193 194 195 196 197 198 199 200 bf c0 c1 c2 c3 c4 c5 c6 c7 c8 201 202 203 204 205 206 207 208 209 210 c9 ca cb cc cd ce cf d0 d1 d2 Decimal 211 212 213 214 215 216 217 218 219 220 Hex d3 d4 d5 d6 d7 d8 d9 da db dc 221 222 223 224 225 226 227 228 229
B 112 Hexadecimal Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01
Index A account management lost password recovery options, 64 recovering forgotten passwords, 63 unable to modify switch settings, 64 user forgot password, 4 user unable to change switch settings, 5 B blade errors AP blade type 24 is inserted, 73 faulty, 3, 54 stuck in the ’LOADING’ state, 3, 54 browser troubleshooting certificates, 65 C certificates corrupt, 65 invalid, 65 not installed, 65 troubleshooting, 65 command output, 7 common problem areas, 2 symptoms, 2 configdownload fails, 3 configupload fai
F F_Port, 23, 30 fabric issues, 2 merge fails, 3 parameters, 37 parameters, reconcile, 38 segments, 3 fabric merge problem, 77 fabric segmentation configuration mismatch, 79 content mismatch, 79 default zone access, 81 fcPing, 82 Fibre Channel connectivity, 82 type mismatch, 79 zone configuration members, 81 zone conflict, 79 zoning problems, 83 fans, status of, 97 FCIP tunnel bounces, 3 tunnel does not come online, 3 tunnel does not form, 3 tunnel is sluggish, 3 FCR is slowing down, 3 feature is not workin
LSAN is slow or times-out, 4 M M_Port, 23 marginal links, 2, 4 media-related issues functional tests, 35 structural tests, 35 message logs, 1 missing devices, 2 N Name Server, (See also NS), 26 network time protocol, (See also NTP), 1 no connectivity between host and storage, 4 no connectivity between switches, 4 no light on LEDs, 4 NS, 26 NTP, 1 port mirroring, 23 adding port connections, 44 considerations, 41 deleting port connections, 44 port type E_Port, 23, 30 EX_Port, 23 F_Port, 23, 30 FL_Port, 23
storage devices, 2 management applications, 2 structural tests, 35 superping tool, 17 supportSave, 5 supportSave timeout value, 6 switch configuration, 2 constantly reboots, 4 disabling port, 93 panic, 4 reboots during configup/download, 4 system status, 88 unable to join fabric, 4 switch boot issues, 13 switch message logs, 12 switch panic, 66 switch reboots during the configuration download, 49 switch status, viewing, 88 switchType, 105 symptoms, 2 synchronize switches, 1 syslog messages, 4 T targets, 2
