Update 1905 for Cloud Platform System (CPS) Standard Dell Hybrid Cloud System for Microsoft Dell Engineering June 2019
Revisions Date July 2016 August 2016 August 2016 October 2016 November 2016 December 2016 January 2017 February 2017 March 2017 May 2017 May 2017 June 2017 August 2017 September 2017 October 2017 November 2017 January 2018 January 2018 March 2018 April 2018 May 2018 June 2018 July 2018 August 2018 September 2018 October 2018 November 2018 December 2018 January 2019 March 2019 April 2019 April 2019 June 2019 Description Initial release 1605 Release 1606 Release 1607 Release 1608 Release 1609 Release 1610 Re
Table of contents Revisions ............................................................................................................................................................................. 2 1 Overview of the Patch and Update framework ............................................................................................................. 5 2 Update 1905—Summary ..............................................................................................................................
WARNING: You cannot run the 1905 Patch & Update framework—1.5.13—directly without first upgrading your environment to 1803 Patch & Update framework—1.5. You can directly upgrade to 1905 only after the DHCS stamp is at the 1.5 version, P&U 1803. Also be advised that the addition of any non-DHCS hardware to your system will cause the Patch & Update process to fail.
1 Overview of the Patch and Update framework The Dell Hybrid Cloud System for Microsoft includes the Patch and Update (P&U) framework. This framework enables you to easily update the infrastructure components of the Dell Hybrid Cloud System for Microsoft stamp with minimal or no disruption to tenant workloads. The framework automates the installation of software, driver, and firmware updates on the physical hosts and the infrastructure VMs. Note: The P&U framework does not update tenant VMs.
2 Update 1905—Summary Update 1905 for CPS Standard includes updates for Windows Server and other software components. This update includes the following components: 1905 update. This is the main package. It can contain Windows Server, System Center, and SQL Server updates. (See payload details in Chapter 5.) IMPORTANT: Update 1803 is a prerequisite for installing update 1905. IMPORTANT: Update 1812 contains a security related WAP update (See CVE-2018-8652).
2.2 How to check which update package is installed To check the version of the update package that is currently installed on the stamp, do the following: 1. On the Console VM, open the DeploymentManifest.xml file at the path: C:\Program Files\Microsoft Cloud Solutions\DeployDriver\Manifests. 2. At the top of the file, look for the following entries: • • “Version=”: This is the version of the Dell-provided update package.
3 1905 Patch and Update Prerequisites You must do the following in order to run the P&U successfully. 3.1 Prepare the patching environment You must first prepare the environment.
3.5 Step 4: Ensure that LaJollaDeploymentService is not running in the background on the Console VM You can ensure that the service LaJollaDeploymentService is stopped by doing the following: 1. On the Console VM, open up the services MMC console that is located under Control Panel>System and Security->Administrative Tools->Services. 2. Look for LaJollaDeploymentService. 3. Ensure that Status is Stopped. 3.
3.7 Step 6: (Optional): Exclude external SOFS storage clusters from P&U IMPORTANT: This procedure applies only if you attached external Scale-Out File Server (SOFS) storage clusters to the CPS Standard stamp. If you attached external Scale-Out-File-Server (SOFS) storage clusters to the CPS Standard stamp (for additional workload capacity), you must exclude them from P&U. If you do not, P&U will fail. To exclude external storage clusters, do the following: 1. Open the VMM console. 2.
4 1905 Patch and Update Process IMPORTANT: Be sure to follow the prerequisites listed in the previous section before you run the 1905 Patch and Update process. You must first prepare the environment. This section covers the preparation steps. In the "Update the computers" section of the CPS Standard Administrators Guide, complete "Step 1: Restart the Console VM" and "Step 2: Run a health check and fix any discovered issues." This includes functionality to check for and disable any running backup jobs.
\\CON01\PUShare\PU_DellEMC1905\PU\Framework\PatchingUpgrade\InvokePURun .ps1 -PUCredential (Get-Credential) -ScomAlertAction "Continue" 4. When prompted, enter the account credentials of the account that you used to log into the ConsoleVM. 5. The Invoke-PURun script performs a one-time environment setup and may prompt you to restart Windows PowerShell on its for invocation, for example: PowerShell environment settings have changed. Please restart the PowerShell console before proceeding.
update for a while. You can use the following steps to view the progress of cluster updates in Failover Cluster Manager. i. Open Failover Cluster Manager. ii. Connect to the cluster. a. In the navigation pane, right-click Failover Cluster Manager, and then click Connect to Cluster. b. In the Select Cluster dialog box, click Browse. c. Click the desired cluster, and then click OK two times. iii.
4.2 Step 2: Upgrade the Intel NIC Driver IMPORTANT: You must manually upgrade the NIC driver on all your physical servers. Failure to do so could result in “Blue Screen of Death” errors. 1. Extract the NIC driver from the zip package a. Unzip the NIC drivers from the file Network_Driver_M1P35_WN64_18.8.0_A00_01.zip which is located in the folder C:\PUShare\DellEMC1905\Payload\PatchingUpdates\DellUpdate\CAUHotfix_All\Binaries to the folder C:\PUShare\NIC 2.
d. Verify that the driver was successfully updated i. Go back to the PowerShell window and run the following commands: ii. Enter-PSSession C iii. Get-WmiObject Win32_PnPSignedDriver | Select-Object -Property devicename, driverversion | Where devicename -like 'Intel(R) Ethernet*' Note: The driver version should now be 3.14.78.0 e. Repeat steps a through d on every Compute cluster node in the stamp. 4. Upgrade the Intel NIC Driver on the File Servers a.
4.3 Step 3: Run the 1905 Microsoft P&U package (DHCS_Update_1905_Run_Second) IMPORTANT: You must run the DHCS_Update_1905_Run_First package before you run the 1905 Microsoft P&U package (DHCS_Update_1905_Run_Second). Because of the size of this package, estimates for deployment duration are 12 to 18 hours. Run the 1905 Microsoft P&U update package by doing the following: 1.
6. DPM agents on the DPM servers are in an enabled state. If this is the case, the health check output indicates that you must run the Set-DPMBackupMode script to cancel the jobs and disable the agents. The PowerShell output looks similar to the following screenshot: 7. To cancel the jobs and disable the agents, do the following: a. From an elevated Windows PowerShell session, run the following commands. Press Enter after each command: cd \\CON01\PUShare\\PU\Framework\PatchingUpg
Note: After Cluster-Aware Updating (CAU) completes, you can click Generate a report on past Updating Runs to view details about what was installed through CAU. • If you have the VMM console open, and it reconnects, patching of the VMM server may be in progress. This is expected behavior. 2. To monitor the progress, you can use the following methods: • View the verbose output on the screen. • View the P&U events in Event Viewer.
4. If you disabled DPM agents on the DPM servers earlier, do the following to restart any canceled jobs and enable the DPM agents: a. On the Console VM, make sure that you are logged on as the account that is a member of -Setup-Admins. b. Open an elevated Windows PowerShell session, and run the following commands. Press Enter after each command. cd "\\CON01\PUShare\\PU\Framework\PatchingUpgrade" Import-Module .\PatchingUpgrade\DPM.
Post-update manual steps KB#3000483 is a Windows Group Policy related security update (CVE-2015-0008). It requires both the binary files (delivered in P&U 1611 and 1703), and a Group Policy update. See the KB article for details on the Group Policy changes, including a section titled “Minimum recommended configuration for domainjoined computers”. (Review the KB article and decide how to implement for your organization and CPS domain.
Post-update clean up After you have verified that patching has completed successfully, do the following to clean up the environment. 1. If you disabled DPM agents on the DPM servers earlier, do the following to restart any canceled jobs and enable the DPM agents: • On the Console VM, make sure that you are logged on as the account that you created for patching, such as CPS-Update-Admin.
5 Microsoft payload for Update 1905 Payload for Update 1905 Update Details KB Number Title CVE / ADV 890830 Windows Malicious Software Removal Tool x64 - May 2019 N/A 3151800 .NET Framework 4.6.
KB Number 4499408 23 Title 2019-05 Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows 8.
5.1 Troubleshooting the P&U process Issue 1 Symptoms: The P&U install process fails with an SMA MAX Timeout Error: Exception calling "InvokeRunbook" with "2" argument(s): "Max Timeout reached for SMA runbook 'Import-OmManagementPack'. P&U fails after a two-hour timeout waiting for the Runbook to complete. Description: SMA Service is hanging when processing runbooks for P&U, specifically the “Import-OmManagementPack” Runbook.
Issue 2 Symptoms: The P&U process updates the console, including reboots, but does not finish final P&U processing. Description: This can include examining the Deployment Manifest and running compliance checks. Detection: Run the following script (updating the $prefix variable before running with the prefix of your stamp) $prefix = "" (Get-SmaVariable -WebServiceEndpoint ("https://{0}APA01" -f $prefix) -Name PUSubsystemVersions).Value If any of the values for “MicrosoftVersion” are not "1.0.1905.
Detection: 1. Open Failover Cluster Manager. 2. Right-click on the CCL cluster and choose More Actions, and then choose Cluster-Aware Updating. 3. Once the Cluster-Aware Updating dialog opens, select Analyze cluster updating readiness.
4. The analyzer runs for a minute or two, and then shows you the results, as illustrated by the following graphic: Under the Title “A firewall rule that allows remote shutdown should be enabled on each node in the failover cluster” you should see a green ‘Passed’ result. If there are any compute nodes that are members of this CCL cluster listed as having failed this test, you have run into this issue. Resolution: Reboot the affected nodes.
Resolution: Browse to the location where the Dell EMC Patch and Update package has been extracted. Under C:\PUShare\PU_DellEMC1905\ Subsystems\PU, you can find the Test-PUHealth.ps1 script, and in the following snippet, add the highlighted workaround: Write-HealthLog -TelemetryInfo $TelemetryInfo -EventType "Progress" -Message "Checking PU custom property for '$($server.ComputerName)'.
Issue 5 Symptoms: From the Console VM, the CPS Administrator cannot access the OEM OOB (Out-of-Band Management) webpage through Internet Explorer. The error will be similar to the following: This page can’t be displayed Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to https://URL again. If this error persists, it is possible that this site uses an unsupported protocol or cipher site such as RC4 (link for details), which is not considered secure.
Cause: Microsoft Update KB# 4093492, that impacts CredSSP authentication protocol and RDP functions. All servers in a CPS environment are now forcing the registry key “HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\CredSSP\Parameter s\AllowEncryptionOracle” to a value of “1”. Workaround #1: Open a Microsoft Management Console and add the Group Policy Editor Snap for the WSUS server.
6 Dell EMC Payload for Update 1905 Dell EMC Update 1905 for CPS Standard includes the following driver and firmware updates. WARNING: Once the Intel NIC firmware is updated to 18.8.x, do not downgrade to previous A-Rev versions below 18.8.x. • Dell Server PowerEdge BIOS R630/R730/R730XD Version 2.9.1 Fixes & Enhancements o Fixes - Fixed an issue pertaining to the PCR2 measurement. When TPM is enabled, the PCR2 values were sometimes inconsistent when the CSIOR was enabled.
• Dell PERC H330 Mini/Adapter RAID Controllers firmware version 25.5.5.0005 Fixes & Enhancements o Fixes -Fixed an issue where congested "I2C 4 PID 10 TIMEOUT but stop sent!" PERC debug TTY log events are reported in 14G server systems.
• Intel NIC Family Version 18.8.0 Firmware for I350, I354, X520, X540, and X550 adapters Fixes & Enhancements Note: Firmware downgrade from 18.8.x to 18.5.x or older versions is not supported. o o • Intel NIC Family Version 18.8.
• Dell 12Gbps HBA firmware version 16.17.00.03 Fixes & Enhancements o o 34 Fixes: Resolved an issue which would cause I/O latency during a cable failure in a multipath topology. Resolved several issues which could cause delays in recovery from communication errors on backplanes or external cabling. Fixed an issue where iDRAC / RACADM could lose communication with HBA until the systems was rebooted Improved device error recovery logic.
